A Quick Guide to Kali

Richard A. Kelsey
NEC Research Institute
kelsey@research.nj.nec.com

Address spaces and servers

Address-spaces are an abstraction of Unix processes. An address space is identified by the machine on which it runs and the socket on which it listens for connections from other address spaces. New address spaces can be added as a Kali program runs.

All of the procedures described in this section are in structure kali.

This starts a server for the current process, making it an address space. The socket that the server is listening on is printed out. Normally the server is started as a separate thread, using spawn from structure threads. 
> ,open kali threads
> (spawn start-server 'kali-server)
Waiting for connection on port 1228
>
Socket-id->address-space returns the address-space corresponding to the process whose server is listening at socket on machine-name. Socket should be the socket number printed out by the call to start-server that created the address space.

Address-space? is the predicate for address spaces.

Remote-run! and remote-apply transport procedure and arguments to address-space and do the application there. Remote-run! returns immediately, while remote-apply blocks until procedure returns, and then returns whatever values procedure returned. Procedure, arguments, and values are all transmitted by copying, with the exception of proxies and symbols. Objects are shared within a particular message, including the message that send procedure and arguments and the message returning values. Objects are not shared between messages. 

  (let ((x (list 1 2)))
    (remote-apply a1 eq? x x))
    -> #t

  (let ((x (list 1 2)))
    (eq? x (remote-apply a1 (lambda () x))))
    -> #f

  (let ((x (list 1)))
    (remote-apply a1 (lambda () (set-car! x 2)))
    (car x))
    -> 1
If address-space is the local address space, no messages are sent and no copying occurs. For a remote-run! where address-space is the local address space, a separate thread is spawned to do the application of procedure to arguments.

There is currently no mechanism for GCing address spaces. Kali makes socket connections between address spaces only as needed, but once made they stay forever.

Proxies

Proxies are globally-unique, distributed cells. Every proxy potentially has a distinct value in every address space.

These procedures are in structure kali.

Make-proxy makes a new proxy, whose value in the current address space is value. Initially the new proxy has no value on other address spaces. Proxy-owner returns the address space on which the proxy was created.

Proxy-local-ref and proxy-local-set! access and set the value of the proxy in the current address space. Proxy-remote-ref and proxy-remote-set! do the same for the value on the address space on which the proxy was created. They could be defined as follows: 

  (define proxy-remote-ref
    (lambda (proxy)
      (remote-apply (proxy-owner proxy)
                    proxy-local-ref
                    proxy)))

  (define proxy-remote-set!
    (lambda (proxy value)
      (remote-run! (proxy-owner proxy)
                   proxy-local-set!
                   proxy
                   value)))
Any-proxy-value returns either the local value of proxy, if there is one, or the value on the proxy's owner.

Note that the remote values may be transmitted between nodes and thus may be a copy of the original value. Each proxy is itself a unique global object and is never copied. 

  (let ((x (make-proxy #f)))
    (eq? x (remote-apply a1 (lambda () x))))
   -> #t

Typically, a proxy only has a value on the owning address space. Local values, via Proxy-local-ref and proxy-local-set!, are only used when a per-address-space cell is needed. An example might be a per-address-space queue of tasks.

A proxy is required whenever a remote-run! or remote-apply may refer to an object that should not be copied. This includes lexically bound variables that are set!. 

  (let* ((call-count 0)
         (f (lambda ()
              (set! call-count (+ 1 call-count)))))

    (remote-apply a1 (lambda () (f) (f) (f)))

    call-count)
   -> 0 if a1 is not the local address space, 3 if it is.

  (let* ((call-count (make-proxy 0))
         (f (lambda ()
              (proxy-remote-set!
                call-count
                (+ 1 (proxy-remote-ref call-count))))))

    (remote-apply a1 (lambda () (f) (f) (f)))

    (proxy-remote-ref call-count))
   -> 3

Many system-supplied data structures, including locks, tables, queues, placeholders and so forth should be put in proxies if they are used remotely.

The current proxy GC algorithm does not collect proxies that are given values on remote nodes. All other proxies are eventually GC'ed when no longer referenced.

Debugging Kali programs

Kali programs run in a distributed, multithreaded environment, making debugging a non-trivial task. As described in doc/threads.txt, when any thread raises an error, Scheme 48 stops running all of the threads at that command level. Kali does not extend this between address spaces, so other address spaces will keep running as if nothing had happened. Messages to the stopped address space are buffered until the user restarts the stopped command level.

Another difficulty in debugging Kali programs is that redefinitions are not propagated between address spaces. A redefinition is handled as a set! to the local cell for the variable. Other address space have their own copies of the cell, which are not updated automatically. The following example shows this effect. 

> (define (f) 10)
> (define (g) (f))
> (g)
10
> (remote-apply a1 g)
10
> (define (f) 20)
> (g)
20
> (remote-apply a1 g)
10
The remote application of g gets the original value of f, not the new one. The remote f can be updated by hand. 
> (remote-run! a1 (lambda (x) (set! f x)) f)
> (remote-apply a1 g)
20
Note that the argument to remote-run! is evaluated in the local address space, and so gets the new value of f. Doing 
(remote-run! a1 (lambda () (set! f f)))
would have had no effect. Both occurrences of f would refer to the binding on the remote address space. When in doubt it is best to restart the program from scratch.

The following procedure is useful in debugging multi-threaded programs.

Debug-message prints the elements to `stderr', followed by a newline. The only types of values that debug-message prints in full are small integers (fixnums), strings, characters, symbols, boolean, and the empty list. Values of other types are abbreviated as follows.
pair (...)
vector #(...)
procedure #{procedure}
record #{<name of record type>}
all others ???

The great thing about debug-message is that it bypasses Scheme 48's I/O and thread handling. The message appears immediately, with no delays or errors.

Debug-message is exported by the structure debug-messages.

Code sharing between address spaces

In Kali, Scheme code in one address space is treated as distinct from code in all other address spaces. If the same file is loaded into two different address spaces, each will have its own copy. If these two address spaces both run that code in a third address space, that third space will get two copies of the code. To avoid duplication, it is a good idea to load a file into only one address space.

The same lack of sharing occurs if, after a file is loaded, an image is dumped and then used to start two address spaces. The two spaces are each considered to have loaded the file. To circumvent this, each Kali image contains a table of loaded code that can be shared between address spaces, assuming that all spaces are using the same table. Address spaces started with different tables are assumed to have nothing in common, and all code needed for remote evaluation must be copied to the remote address space. The following procedure, from the structure address-spaces, can be used to rebuild the shared-code table after loading additional code.

This creates a table containing all existing code and and proxies. This table is shared between all address spaces that are started from the current image. Code loaded before the call to initialize-shared-address-space! will not be copied between address spaces.
kelsey@research.nj.nec.com