Beckman Institute            University of Illinois at Urbana-Champaign             

Documentation Table of Contents

• Overview
• Required Ports: A Note on Firewalls
• Reserved Words and Characters
• Creating a Cluster
  • Creating a Cluster: The Network Configuration File
    • Creating the Network Configuration File
    • Network Configuration File Command Reference
  • Creating a Cluster: Running a Syzygy Server
    • Computer Names According to the Syzygy Server
  • Creating a Cluster: The Syzygy Login Files
    • Creating the Login File
  • Creating a Cluster: The Remote Execution Daemon
  • Creating a Cluster: Specifying VR-Related Parameters
• Using Virtual Computers
• Component Contexts
  • What is the Component Context?
  • Manipulating Component Contexts
    • Setting SZGCONTEXT
    • Command-line Context Manipulation
• Connection Brokering and Service Names
  • Base Service Names
  • Examining Running Services
  • How it All Fits Together
• Locks
• Command Line Tools Reference
  • Parameter Database Management
  • Interprocess Communications
  • System Monitoring
• Troubleshooting
  • Connection-Brokering Test

Overview

A Syzygy cluster is comprised of a set of computers connected by a network. Each cluster is controlled by a Syzygy server. There is only one instance of the Syzygy server program, szgserver, per cluster. The group of computers comprising a Syzygy cluster is controlled by an instance of the Syzygy server program, szgserver, which performs several functions:

• Manages multiple users.
• Stores a configuration parameter database for each user.
• Maintains a list of the software components registered with the cluster, with each component being identified by the computer on which it is running, a symbolic string (equivalent to executable name), and a unique ID (equivalent to a Unix process ID).
• Contains a connection brokering service that allows components that need to connect to automatically find each other.
• Maintains locks to prevent more than one component at a time from accessing certain cluster resources.
• Routes messages from one registered software component to another. These messages can be used like Unix signals. For example, messages can cause a component to terminate or to reload its parameters.

Membership in a cluster for a given user is determined by two files:

• szg.conf, (normally one/computer) which specifies the computer's Syzygy name, describes its network interfaces, and assigns each to a named Syzygy network.
• szg_<user>.conf (where <user> refers to the operating system user name), which specifies the Syzygy server that OS user <user> is currently logged into.

A Syzygy cluster can be controlled in two ways:

1. By issuing Syzygy commands from a command shell or from a scripting language such as Python.

2. Programatically using methods of an arSZGClient object, either in C++ or Python. There is almost a one-to-one mapping between the methods of this object and the shell commands

It's not entirely necessary, but it's a lot easier to run cluster applications if each computer in a cluster runs a copy of the remote execution daemon szgd. This allows all of an application's components to be launched by a single command issued from any computer logged into the Syzygy server.

Required Ports: A Note on Firewalls

IMPORTANT: Many operating systems today will install a firewall. The default configuration of this software might very well keep Syzygy from functioning. As a networked system, Syzygy expects to be able to communicate on particular ports. The easiest option is to simply not have a firewall at all. However, this may not be acceptable.

These ports must be allowed through a firewall:

• For Syzygy to operate in a distributed fashion, you will need to create a cluster. This involves running an instance of the Syzygy server szgserver, which in turn requires specifying a port for incoming TCP connections. The computer on which szgserver runs must allow connections to this port.

• Each computer that is part of the cluster will require a block of ports for Syzygy components to connect to one another. This block is defaults to port number 4700-48999, but it can be set manually using the dports command. TCP connections to this block of ports must be let through your firewall.

• Besides this block of ports, UDP port 4620 is used for automated discovery of Syzygy servers. It's not a major problem if this port is unavailable, it just means that you have to use the more specific form of the dlogin command (see Commandline Programs for Logging Into and Out of a Server) and the dhunt command won't find your server.

Reserved Words and Characters

"NULL" or "*" cannot be user-defined user, computer, group, or parameter names. "NULL" internally functions as the indication of an undefined value, while "*" is used in the Syzygy server automated discovery process. The characters

    :  ;  |  /

should also not be used in names.

Creating a Cluster

Creating a Cluster: The Network Configuration File

Each computer in a cluster must contain a network configuration file named "szg.conf". in which network interface data are stored. The default location of this file varies by platform:

• On Unix-like operating systems (Linux, Mac OS X, Irix), the default location is in /etc/. Note that non-root users cannot write to /etc, so initial network configuration must be done as root if the default location is used.
• On Windows, the default location is in c:\szg\.

If desired you can set the environment variable SZG_CONF to the path to a directory where this file should be created. Note that all Syzygy users MUST be able to change into the directory containing the szg.conf file.

Every Syzygy program will try to read the szg.conf file, but they generally do not try to write to it. It must exist for programs to work in Cluster Mode but it is not necessary for programs to work in Standalone Mode.

szg.conf is an XML file contains the following information:

# The computer's Syzygy name (typically the first part of the DNS name, but it doesn't have to be). Defaults to "NULL". # For each network interface card in the computer there must be a record specifying: # The Syzygy network that this interface is part of. Interfaces connected to the internet should specify "internet"; multiple private networks are allowed and should be assigned distinct names that are meaningful to you. # The interface's IP address and netmask. The netmask defaults to 255.255.255.0. # A "type" value that must currently be set to "IP". Note that there's no default interface record, if you don't add any you simply won't be able to perform any Syzygy networking functions. # A range of ports to be used in connection brokering. This is specified by two numbers, the starting port number and number of ports in the block. The default values are 4700 and 200.

Here is an example of an szg.conf file:

  <computer>
    <name>my_computer</name>
  </computer>
  <interface>
    <type>IP</type>
    <name>internet</name>
    <address>999.999.999.999</address>
  </interface>
  <interface>
    <type>IP</type>
    <name>private_net</name>
    <address>192.168.0.1</address>
  </interface>
  <ports>
    <first>
      4700
    </first>
    <size>
      200
    </size>
  </ports>

By default, the szg.conf file is located at c:\szg\szg.conf on Win32 systems and at /etc/szg.conf on Unix systems. An alternate location can be specified by setting the environment variable SZG_CONF to the path of the desired directory (without trailing path delimiter, e.g. /tmp/szg or G:\szg. Remember that to un-set a Syzygy environment variable you can set it to the string "NULL", which Syzygy interprets as "no such variable". Note that this means it is possible to have distinct szg.conf files for different operating system users, should that be desirable.

Creating the Network Configuration File

You don't need to edit szg.conf by hand; command line programs are provided. Each of these commands parse the whole szg.conf file and re-write it from scratch upon completion, minimizing the possibility of error.

To create szg.conf, perform the following steps (to perform these steps you must have write access to the directory where szg.conf will be created):

1. Login to the computer to be configured.
2. Assign the computer a Syzygy name. This name must be unique across all the computers in your system. A good choice is the short version of your computer's DNS name (e.g. foobar.isl.uiuc.edu becomes foobar) but other names are possible. We assume that you can write to the file's location. For example, type:

       dname my_computer
   

3. szg.conf must contain information about the network interfaces in the computer, specifically the IP address and netmask. In addition, each interface must be assigned to a Syzygy network. To add an interface to the config file:

       daddinterface <network_name> <address> [<netmask>]
   

   <network_name> assigns the interface to a particular Syzygy network. Internet addresses should use the name "internet". Private networks can use an arbitrary name, but this should be consistent across the private network and different from that assigned to other private networks in the distributed system. For example, the distributed system might contain 2 clusters, each connected internally by a distinct 192.168.0.XXX private network. In this case, the <network_name> associated with each should be different. This lets Syzygy operate properly with respect to connection brokering.
   The primary or default network for each computer should be added first. This network should be one to which all computers in the distributed system are connected. Components will expect to connect to the szgserver on this network. Furthermore, if components have a choice about how to connect to one another (i.e. they are connected by several networks), they will default to using the first network.
   You can optionally specify a <netmask> for the interface. By default, a netmask of 255.255.255.0 is used.
   You can remove networks from the config file using the following command:

       ddelinterface <network_name> <address>
   

   Syzygy operates using connection brokering. This means that you will not explicitly assign the ports on which servers will listen for connections. Instead, Syzygy will assign the servers' ports based on a pool it maintains on a per-computer basis. By default, the block of ports numbers 4700-4799 is used, which is likely to be OK on both Unix and Windows machines. However, you can change this block using the following command:

       dports <first_port> <num_ports>
   

   This command allocates a block of ports beginning at <first_port> and containing <num_ports> ports. IMPORTANT: you'll want every port in this block to be free and user services to be able to bind to them. The block should be of reasonable size in order to accomodate all the services that might run on the computer. It is best to be generous when assigning the size. NOTE: some Windows versions do not like user services to bind to ports 5000 and above.

4. After issuing these commands, you can check the stored config file by typing:

       dconfig
   

   The output might look something like this for a computer with two network interface cards:

       computer = foobar
       network = internet, address = 999.999.999.999, netmask=255.255.255.0
       network = private_net, address = 192.168.0.1, netmask=255.255.255.0
       ports = 4700 - 4899
   

Network Configuration File Command Reference

    dname <name>

Sets the computer's name in the szg.conf file. If no ports information is present, it goes ahead and adds a default ports record, reserving the range 4700-4899.

    dconfig

Parses the szg.conf file and prints the information in compact form.

    daddinterface <network_name> <address> [<netmask>]

Adds an interface to szg.conf file with the given name, IP address, and (optionally) netmask. The name should be descriptive (like "internet") and is meant to uniquely identify a network. The netmask can also be given (the default is 255.255.255.0).

    ddelinterface <network_name> <address>

Removes the interface with the given name/address pair from the szg.conf file.

    dports <first_port> <num_ports>

Changes the "ports" record in the szg.conf file, with "first_port" giving the number of the first port to be used and "num_ports" giving the number of ports in the block.

Creating a Cluster: Running a Syzygy Server

To run a Syzygy server, type:

    szgserver <server_name> <server_port>

e.g.

    szgserver my_server 8888

The <server_name> is arbitrary and is the name that users will use for logging in with the dlogin command (see Commandline Programs for Logging Into and Out of a Server). The <server_port> is the TCP port that the server will listen on for connections. All Syzygy components in the cluster will attempt to connect to the server via this port and will maintain that connection throughout their operation, so of course if you're running a firewall this port must be open. Note that this port must not be within the block of ports printed by dconfig; that block must be available for connection brokering.

You can restrict the IP addresses from which connections will be accepted by appending an optional sequence of whitelist entries, which can either consist of single IP addresses or blocks of IPs in address/netmask format. The latter means that addresses that match the specified address after being AND'ed with the netmask will be allowed to connect. For example, If you typed:

    szgserver my_server 8888 192.168.0.11 130.126.127.0/255.255.255.0

The first whitelist entry, 192.168.0.11, specifies a single address from which connections will be accepted. The second entry specifies that connections from all IP addresses beginning with 130.126.127 will also be accepted.

NOTE: szgserver is meant to run in the background for a long time (like a linux daemon or windows service). Here are some common misunderstandings:

• Do not run a new copy of szgserver for each application.
• Do not run a copy of szgserver on every computer in your cluster.

Computer Names According to the Syzygy Server

The Syzygy server does not do name resolution (i.e. does not use DNS). Clients read the network configuration file szg.conf before communicating with the server and transmits the name contained therein to the server.

Creating a Cluster: The Syzygy Login Files

The network configuration file determines the identity of an individual computer in the cluster. This information is insufficient to join a cluster: In order to do that you also need to specify a Syzygy server and user name. This information is maintained in a set of login files, potentially one for each operating system user on the computer. The default locations for these files are:

• Unix: /tmp/
• Windows: c:\szg\

An alternate directory can be specified by setting the environment variable SZG_LOGIN to the path to the directory (without a trailing path delimiter). The login file names are of the form szg_<user>.conf, where <user> refers to the operating system user name.

Each such file contains the following records:

1. The user's Syzygy user name.
2. A complete specification of the Syzygy server that the user is currently logged into. This consists of the server's name, IP address and port number.

Here is an example login file:

  <login>
    <user>me</user>
    <server_name>my_server</server_name>
    <server_IP>999.999.999.999</server_IP>
    <server_port>
      8888 
    </server_port>
  </login>
  

If the user is not logged into any Syzygy server (e.g. after issuing the dlogout command, necessary for running programs in Standalone Mode), the login file will look like this:

  <login>
    <user>NULL</user>
    <server_name>NULL</server_name>
    <server_IP>NULL</server_IP>
    <server_port>
      0 
    </server_port>
  </login>

Of course, the fact that this information is stored in a file means that it persists across operating system login sessions; if you log off of the computer and then log in again, you will still be logged into the same Syzygy server.

Creating the Login File

You use the following commands to create and modify the Syzygy login files:

    dlogin <server_name> <syzygy_user_name>
    dlogin <server_IP> <server_port> <user_name>

Two forms of the dlogin command. In the first form you specify the server name and you Syzygy user name and the dlogin program attempts to discover the server by broadcasting a discovery packet on the local network. If that doesn't work (because the current computer can't broadcast to the one running the server), you can use the second form, in which you explicitly specify the server's IP address and port. For example:

    dlogin my_server me

or

    dlogin 127.0.0.1 8888 me

Upon successful dlogin, the information in the login file will be printed, looking something like:

    OS user     = johndoe
    syzygy user = me
    szgserver   = my_server, 999.999.999.999:8888

This command associates a Syzygy user name with the current operating system (Windows, Unix) user name. Any subsequent commands issued on that computer while logged in as that operating system user will be interpreted by Syzygy as being issued by the specified Syzygy user. To change Syzygy users for a given OS user you can either (a) dlogin as a different Syzygy user, or (b) dlogout, which disconnects you from any szgserver. As stated above, if you log out of your computer's operating system and then log back in you will still be dlogged in as the same Syzygy user.

Note that the second form of the dlogin command is the one you would use when setting up a mini-cluster on a single computer that is not connected to a network. In this case you should use the localhost or loopback address for the current computer, 127.0.0.1. This address does not support broadcast traffic.

Not also that there is currently no authentication of Syzygy users. Other users that can dlogin to the server can change your Syzygy database parameters or execute your Syzygy applications without restriction.

If both forms of the dlogin command fail: check the following:

• The networks in the Syzygy config file on the machine on which dlogin was issued may be incorrect. Check them again using dconfig.
• The Syzygy server tells dlogin to connect using the first address in the szg.conf file on the computer running the server. The machine from which you are trying to dlogin must be able to reach this IP address. If not, use ddelinterface and daddinterface to reorder the addresses in the szg.conf file on the server machine.

To log yourself out of the Syzygy server and reset the login file, type:

    dlogout

This command resets the fields in the current user's login file to "NULL" or 0 as appropriate. Issue this command before running programs in Standalone Mode.

    dwho

Prints the login file of the current operating system user in compact form.

    dhunt

Searches for Syzygy servers. This command reads the szg.conf file and sends a broadcast discovery packet out through each network interface listed in that file. Any servers receiving the packet respond with their names, IP addresses, and port numbers.

Creating a Cluster: The Remote Execution Daemon

The remote execution daemon szgd is a special Syzygy component that executes programs in response to messages (normally sent by users using the dex command). It launched native (i.e. C++) and Python programs, supports multiple users, and manipulates environment variables controlling dynamic linker and python module paths to ensure that loadable objects behave as expected. szgd must connect to an szgserver as it starts, so you must have dlogged in beforehand.

The exec message sent by dex contains the name of current user on the computer on which it was run. For C++ programs, each szgd queries the Syzygy database (see System Configuration) for the value of the variable SZG_EXEC/path defined for the current Syzygy user and computer. The value of this variable is used as a search path for the specified executable. For Python programs, the variable SZG_PYTHON/executable should point to the python executable and SZG_PYTHON/path is the search path for the user program. See Syzgy Resource Path Configuration for details.

By convention, dynamic libraries (dlls) are assumed to live in the same directory as the executables that load them. Consequently, when executing a program, szgd prepends the directory in which the program lives to the front of environment variable giving the dynamic linker search path (LD_LIBRARY_PATH on Linux).

The form of the szgd command is as follows:

    szgd <base_paths> [-r]

The required <base_paths> argument should be a semicolon-delimited list of paths to directories or executables; paths to executables should not include the '.exe' suffix on Windows. szgd is in charge of launching programs on each machine in your cluster in response to a remote command. It will only launch programs whose full path begins with one of the paths in the base_paths list (For example, on Windows you probably do not want it to include c:\Windows\). Note that for many command shells the semicolon is the end-of-command delimiter, so you may need to enclose the base_paths argument in quotes if it contains a semicolon.

szgd immediately attempts to connect to the szgserver that you are currently dlogged into. By default, it exits on failure to connect. If the optional -r argument is passed it will instead repeatedly attempt to reconnect on failure.

For example, you might type:

  szgd C:\myapps -r

After starting szgd on each computer of your cluster, type:

    dps

You'll see something like:

    computer1/szgd/0
    computer2/szgd/1
    computer3/szgd/2
    computer4/szgd/3
    computer5/szgd/4
    computer6/szgd/5

i.e. "computer name"/"process name"/"process ID". There should be a line for each computer on which you have run szgd.

Note that only one instance of szgd can run on each computer; this is ensured by having it attempt to acquire the Syzygy lock <computer>/szgd, where <computer> is the current computer name. If another szgd already holds this lock the current one quits with an error message.

Creating a Cluster: Specifying VR-Related Parameters

At this point you have the basic Syzygy infrastructure in place. You need to read about setting up the information in the Syzygy database to specify what components should run on which computers, the configuration of your graphics displays, and so on. See System Configuration.

The rest of this chapter contains reference material about the distributed operating system.

Using Virtual Computers

For information on defining virtual computers, see Virtual Computer Configuration.

To launch an application on a virtual computer, you will need a copy of szgd running on every "mapped" machine, i.e. any real computer whose name appears as the value of a "<something>/map" parameter value in the virtual computer definition. Once this is done, to launch atlantis (a sample application included with the szg source) on virtual computer vc, use the command:

    dex vc atlantis

You can also launch a single component as part of a virtual computer. This allows you to either launch all the components of a virtual computer by hand one at a time or to re-start a component that has exited for some reason. Type:

    dex <computer> <component_name> [<args>] -szg virtual=<virtual_computer_name>

For example, suppose we've had to quit the copy of atlantis running on computer "computer3", which is part of virtual computer "vc". Then we could type:

    dex computer3 atlantis -szg virtual=vc

to restart it. This works for services (e.g. DeviceServer, SoundRender) as well. See Component Contexts for more information.

There are two ways to kill an application running on the virtual computer. You can directly kill the trigger instance. For that you need to know the location parameter of the virtual computer. For the example virtual computer defined in Virtual Computer Configuration, that would be "my_room", so you would type:

    dmsg -c my_room quit

Or you can type:

    dkillall vc

Here are several useful commands for managing a virtual computer:

    dkillall <virtual_computer>

Kill any application currently running on the given virtual computer, along with any associated services such as szgrender, SoundRender, or DeviceServer.

    dmsg -c <virtual_computer_location> quit

Kills the application, if any, currently running in the given virtual computer location, but leaves services alone.

    restarttracker <virtual_computer>

Restart services associated with virtual_computer (e.g. input devices and sound).

    setdemomode <virtual_computer> [true, false]

If second argument is "true", this sets each render machine to use a fixed head position rather than that reported by the tracking device. The head position is read from the active screen definition (see Graphics Configuration) on each render computer. Each rendering machine will assume that the direction of gaze is perpendicular to its display, and the up direction of the head will be the screen's up direction rotated by the angle (in degrees) given by the screen's fixedheadupangle element. To set things back to normal mode, use "false" as the second argument.

    setstereo <virtual_computer> [true, false]

Turns active stereo rendering on and off for each rendering machine of the cluster.

    screensaver <virtual_computer>

Start szgrender programs on each render machine in the cluster. Since szgrender is black when no application is connected, this is effectively a screensaver.

    calibrationdemo <virtual_computer>

Display a calibration screen useful for alignment and color matching, looking for the calibration picture cubecal.ppm in the SZG_DATA/path of each render machine.

Component Contexts

What is the Component Context?

Every component in Syzygy is executed in a "context". The context determines the overall behavior of the component, whether the component is executing as part of a virtual computer, and what networks it will use for various sorts of communication. It consists of a list of variable/value pairs, as listed below:

    virtual=virtual_computer_name

The name of the virtual computer on which this component is executing. Set to the value "NULL" if it is not executing on a virtual computer (the default).

    mode/default=[trigger, master, component]

The overall behavior of the component. The "trigger" value causes a syzygy application to run as a trigger instance, launching other necessary components on a virtual computer. The "master" value causes a master/slave component to take the master role. The "component" value is a default and has no effect.

    mode/graphics=SZG_DISPLAY(n)

One of SZG_DISPLAY0, SZG_DISPLAY1, etc. If the component opens a graphics window, this determines the screen configuration it will use from among those defined for the computer it is running on. If you specify a display that isn't defined, a default display configuration will be used.

    networks/default=network_list

The value network_list is a slash-delimited list of network names, like internet/my_private_net. It indicates, in descending order of preference, the network across which service connections should occur, assuming that the service type (input, graphics, sound) has not already specified this.

    networks/input=network_list

Defines the network across which input service connections should occur.

    networks/graphics=network_list

Defines the network across which graphics service (szgrender) connections should occur.

    networks/sound=network_list

Defines the network path upon which sound service connections should occur.

    parameter_file=parameter_file_name

If the application is running in Standalone Mode it reads its parameter database from a file in its current working directory. By default it looks for szg_parameters.xml and then szg_parameters.txt, but this can be changed by passing this special parameter.

    user=syzygy_user_name

The Syzygy user under whose auspices the application will run.

    server=IPaddress/port

Identifies the location of the szgserver to which the component should attempt to connect.

Note those last two items in particular: By manipulating them you could run a component as a different user and connecting to a different Syzygy server from the one you are currently dlogged into.

Manipulating Component Contexts

Normally when you launch an application on a virtual computer the application launching process sets up the context appropriately for each launched component. However, sometimes you will want or need to set the context manually; for example, you may need to restart an individual component of an application running on a virtual computer. This can be done in two ways: (1) By setting the SZGCONTEXT environment variable, or (2) by passing special arguments on the command line.

Setting SZGCONTEXT

The value of SZGCONTEXT should consist of a sequence of the above pairs seperated by ';'. You can set this environment variable manually to control the behavior of Syzygy components on a particular computer. A sample one might look something like:

    virtual=vc;mode/graphics=SZG_DISPLAY0;networks/graphics=my_private_net

Command-line Context Manipulation

To manipulate a context from the command line, pass "variable=value" pairs preceded by "-szg". For example:

    syzygy_executable -szg virtual=vc -szg networks/graphics=internet

Obviously, this method is more flexible than setting an environment variable, but it has a weakness: It depends on all Syzygy components (including any applications that you write) processing command-line arguments after the Syzygy client object, arSZGClient, has been initialized (arSZGClient::init()). In framework-based applications (see Programming this happens in the framework object's init() method. When this method is called the special arguments are parsed and removed from the global argv variable, so the application or component won't be faced with them.

Yet another advantage to this method of manipulating the context: It can be used remotely. When you launch an application using the dex command, most context variables specified on the command line are passed on to the component being launched. For example:

    dex render1 szgrender -szg virtual=vc

will launch szgrender on computer "render1" and tell it to behave as a component of virtual computer "vc".

dex will interpret the "user" and "server" arguments, allowing you to launch applications as a different user or even on a different cluster from those specified in the login file. For example:

    dex vc_far_away my_app -szg server=faraway_IP_address/faraway_port -szg user=somebody_else

Connection Brokering and Service Names

Connections to Syzygy services such as input event streams, sound output, and scene-graph rendering are brokered by the Syzygy server szgserver using a service name. Components that provide a service with a particular name will fail to launch if another provider of that same service name is already running, so it is important to understand how services get their names and how to find out what services are offered.

Service names are parially determined by the nature of the service (the "basic service name") and partially by the circumstances under which a component is launched. If a user launches a component, but not in the context of a virtual computer, the component's service name will be:

    basic_service_name/syzygy_user_name

This prevents the actions of one syzygy user from interfering with those of another.

On the other hand, if a user launches a component in the context of a virtual computer, the component's service name will be:

    virtual_computer_location/basic_service_name

(see Virtual Computer Configuration). This allows different users to share application components running in a particular virtual computer location, allowing them to be used as a communal resource.

Base Service Names

Base service names for the various syzygy service are listed below.

    SZG_INPUT#

An input-event service: One of SZG_INPUT0, SZG_INPUT1, etc. The number refers to the driver slot.??? inputsimulator and DeviceServer offer this service. DeviceClient and Syzygy applications in general must connect to it in order to receive input from trackers, joysticks, gamepads, and so on.

    SZG_SOUND

A source of sound information. The sound-playing component SoundRender needs to receive or connect to this service. Syzygy VR framework applications (see Programming) offer it.

    SZG_SOUND_BARRIER

Synchronization for the SZG_SOUND service. SoundRender connects to this service. Syzygy VR framework applications offer it.

    SZG_GEOMETRY

A source of scene graph information, generally a distributed scene graph framework application (see Programming). The scene-graph rendering component szgrender needs to connect to this service.

    SZG_GEOMETRY_BARRIER

Synchronization for the SZG_GEOMETRY service. szgrender needs to connect to this service. Distributed scene graph applications offer it.

    SZG_MASTER_<application_name>

The master instance of a master/slave application (see Programming) offers data to the slaves via this service. Slave instances try to connect to this service.

    SZG_MASTER_(application_name)_BARRIER

Synchronization for the above service.

Examining Running Services

Examining services can be helful in debugging problems with components not connecting properly.

For a list of currently-offered services, type:

    dservices

For a list of pending service requests, i.e. requests that have not yet been able to connect to a provider of the requested service, type:

    dpending

How it All Fits Together

Consider a brief example.

You launch a distributed scene graph application. It signals the connection broker in the server that it requires an SZG_INPUT service (for user input) and provides SZG_GEOMETRY and SZG_SOUND services (i.e it is a source of virtual-world-object and sound information).

The connection broker notes that there is a running DeviceServer providing SZG_INPUT0, several szgrenders on various computers requesting SZG_GEOMETRY, and a SoundRender requesting SZG_SOUND. In each case, it sends the IP address and port of the service provider to the components requesting the service, allowing them to connect and satisfy their cravings.

Locks

Locks are Syzygy's mechanism for synchronizing access by multiple, networked processes to resources that can only be accessed by one process at a time. They are exposed to the network by the Syzygy server, but they work analogously to mutexes within a multi-threaded program. When a process requires access to a resource, it requests the associated lock; if no other process holds the lock, it immediately acquires it and proceeds to use the resource. If another process is holding the lock, however, the first process is blocked and cannot continue until the second one releases it.

For example, as noted in Virtual Computer Configuration, overlapping virtual computers can be assigned a common "location" parameter. Virtual computers with the same location parameter are assumed to share machines. If one user launches an application on one such virtual computer, you don't want anyone else to be able to start another program on an overlapping virtual computer at the same time.

To prevent this, the first application to launch requests a lock with the name "<virtual_computer_location>/SZG_DEMO/lock". It holds the lock until all of its components have launched, then releases it. Other applications attempting to launch on another virtual computer with the same location parameter will request the same lock and will be blocked until they can acquire it.

Locks are also used to reserve resources. Some Syzygy components are not allowed to run if another instance of the same component is already running on the same computer. When such a component starts, it requests a lock and does not release it until it exits; other instances attempting to start in the meantime will be unable to aquire the lock and will exit. Here are a few components and their named locks:

• szgd: <computer_name>/szgd.
• SoundRender: <computer_name>/SoundRender.
• DeviceServer: <computer_name>/DeviceServer.
• szgrender: <computer_name>/SZG_DISPLAY# (# is determined by the display it is using, see Graphics Configuration). Multiple szgrenders can run on the same computer if they use different displays.

Finally, locks facilitate automatic shutdown of previously running applications. If a new application requires a particular resource (for example, it needs to run a rendering component on render1/SZG_DISPLAY0), it requests the associated lock. If the request fails, the server sends the Syzygy process ID of the component holding the lock to the application. The new application then sends the offending component a "quit" message and waits for the lock to be released.

To see the existing locks, type:

    dlocks

Command Line Tools Reference

Syzygy also has commands for managing the paramter database (used for holding system configuration data), for sending messages to running components, and for monitoring the status of the system (e.g. finding out what components are running).

Parameter Database Management

The commands for manipulating the Syzygy database are described in the System Configuration chapter.

Interprocess Communications

Syzygy components communicate by sending messages to one another, with the Syzygy serving providing routing information.

Messages have a type and optionally a body. The type is generally a single word, such as "exec" (to launch a component) or "quit" (telling the receiving component to exit). Some message types are only meaningful to certain components.

Of particular interest to applications programmers is the "user" message type. This gets passed on to the application code by means of the application frameworks' user-message callback or method (see Programming). This allows you to control your application by means of messages sent using the dmsg command.

The following commands can be used to send messages to running components:

     dex [-v] <execution_location> <executable_name> [<arguments>]

Launch a Syzygy component or application remotely. <execution_location> should be either

1. A virtual computer name; or
2. The Syzygy name of a real computer.
   If a virtual computer, an "exec" message is sent to szgd on the virtual computer's "trigger" computer; it in turn sends messages to start other components of the applicatoin. On the other hand, if <execution_location> is a real computer the the message will be sent to szgd on that machine and a single instance of the executable will be launched there.
   Most components will return messages to dex regarding their success or failure in launching. By default, only the contents of the final message are printed, but the -v option prints all such messages.
   The executable can be either a native (C++) or a Python program. For native programs on Windows hosts, ".EXE" is automatically appended to <executable_name> (so don't add it yourself). Python programs must end in ".py".
   Any trailing <arguments> are passed on to the component as command-line arguments.

       dex [-v] <executable_name>
   

   A special case, this launches executable_name on the local computer. No additional arguments may be passed to the program.

  dkill [-9] [<computer_name>] <executable_label>

Send a "quit" message to the first entry in the Syzygy process table with the specified label (i.e. process name as returned by dps) and (optionally) running on the specified computer. Like unix "kill -9", the -9 option forcibly closes sgzserver's connection to the component. This is what you use if a client has crashed without the Syzygy server realizing that its connection to the client is gone.

The remainder are variants on the dmsg command, which is the most generic message-sending command.

  dmsg <ID> <message_type> [<message_body>]

Send a message to the process with the specified <ID> as printed by dps. The message type and optional body are both strings. for example, you might launch an executable by typing (assuming 99 is an szgd ID):

    dmsg 99 exec /home/randomuser/bin/linux

  dmsg [-r] -p <computer_name> <component_name> <message_type> [<message_body>]

Find the Syzygy component with name <component_name> that is running on computer <computer_name>. Send it a message with type <message_type> and optional body <message_body>. If the -r flag is specified, dmsg requests a reply to its message and waits for that reply, printing it upon receipt.

  dmsg [-r] -m <virtual_computer> <message_type> [<message_body>]

Find the component running on the master screen of the specified virtual computer. Send it a message with type <message_type> and optional body <message_body>. If -r, wait for and print reply.

  dmsg [-r] -g <virtual_computer> <display_number> <message_type> [<message_body>]

Find the component running on the specified virtual computer's display indexed by <display_number>. Send it a message with type <message_type> and optional body <message_body>. If -r, wait for and print reply.

  dmsg [-r] -c <virtual_computer_location> <message_type> [<message_body>]

Find the trigger component for the currently running application in the given virtual computer location (locations are discussed in Virtual Computer Configuration). Send it a message with type <message_type> and optional body <message_body>. If -r, wait for and print reply.

  dmsg [-r] -s <service_name> <message_type> [<message_body>]

Find the component providing the service <service_name>. Send it a message with type <message_type> and optional body <message_body>. If -r, wait for and print reply.

  dmsg [-r] -l <lock_name> <message_type> [<message_body>]

Find the component holding the lock <lock_name>. Send it a message with type <message_type> and optional body <message_body>. If -r, wait for and print reply.

System Monitoring

  dps [<search_tag>]

List all Syzygy processes in the format:

    computer_name/process_name/ID

Adding the <search_tag> parameter lists only those lines containing it.

  dtop [d <milliseconds>] | q | t

Unix Only. Repeated dps, like Unix "top" with pretty color coding. Specify the repeat interval as:

1. "d" followed by a value in milliseconds;
2. "q" (no delay, cpu hog).
3. "t" (for stress testing the server, no delay and no display). Hit the "q" key to quit dtop, or kill it using dkill.

     dlocks
   

   Prints a list of the locks currently held by Syzygy components, along with their component IDs.

     dservices
   

   Prints a list of services currently offered by Syzygy components.

     dpending
   

   Prints a list of unfilled service requests. Useful in understanding why something is failing to connect to or otherwise get information from another component.

Troubleshooting

When things go wrong, e.g. an application or application component does not launch or two components refuse to connect, the user needs to understand connection brokering, locks, and the tools to examine them. Misconfiguration or misuse of these features is a major cause of problems when using Syzygy.

• szgrender, SoundRender, szgd refuse to launch: Each of these components holds a Syzygy lock corresponding to a real world resource during its operation. For szgrender, it is a given chunk of screen real estate. For SoundRender, it is the sound card. For szgd, it is the computer itself. In the case of szgrender, executing a second copy to display on the same virtual screen (SZG_DISPLAY0, SZG_DISPLAY1, etc.) as an already running szgrender will cause the second copy to quit with the following error message (the component ID will vary):

      szgrender error: failed to get screen resource held by component <ID##>.
  

  This is the desired behavior when a resource cannot be shared.
• A Syzygy VR framework application (see Programming) that is run on a real computer but not as part of a virtual computer fails to launch with the following error message:

      arSyncDataServer error: failed to register service.
  

  Again, this is an expected behavior. The VR framework application wants to offer an SZG_SOUND service. If it is not run as part of a virtual computer, its service name will default to "SZG_SOUND/<syzygy_user_name>". If the same user runs a second framework application on a different computer, it will attempt to provide exactly the same named service. This is not allowed, otherwise SoundRender would not know which of the two to connect to.

• Component A does not connect to Component B even though it should: First, you should run a Connection-Brokering Test. If this tests succeeds, one of the following is almost certainly true:
  • Neither component is being run on a virtual computer, but they are being run with different user names. You might have dlogin'ed with different Syzygy user names on the different computers. In this case, you must be dlogin'ed with the same Syzygy name on both computers.
  • One component is being run on a virtual computer but another is not. Again, this will fail. Both must be running in the same virtual computer location. Recall that every virtual computer has a location; if not specified, it defaults to the virtual computer name.
  • One component is being run in virtual computer location L1 while the other is being run in virtual computer location L2. Again, this is going to fail. The virtual computer locations must be the same for the components to connect.
    You can diagnose these problems by running "dservices" to see the names of services which are being offered and by running "dpending" to see the names of unfulfilled service requests. You should see the full name of the offered service, the full name of the request, and see how they fail to match.

• A computer running some Syzygy components crashed, had its power cord yanked, etc. Now these components are stuck in the Syzyg process table, i.e. they still show up int the list of processes printed by dps. Furthermore, new instances of those components cannot be launched. Assume for the sake of example that the relevant component is szgd. The problem occurs because the <computer_name>/szgd lock was never released because szgd did not exit normally. The solution is to type:

      dkill -9 szgd
  

  (or whatever the actual component name is). This forcibly removes the component name from the Syzygy process table and releases the relevant lock.

Connection-Brokering Test

To verify that basic connection brokering works between computers in your cluster:

1. Start BarrierServer on one of your cluster machines.

2. Start BarrierClient on another machine. If all is well, the BarrierClient will begin printing the average time necessary to perform each synchronization through the BarrierServer.
   Troubleshooting:
   • If BarrierClient fails to start, make sure dlogin succeeded on the machine on which it is running..
   • If BarrierClient starts but does not start printing, you have entered the IP address incorrectly in the szg.conf file on the BarrierServer machine. Use dconfig to examine it and ddelinterface/daddinterface to fix it.

3. Start BarrierClient on other machines.

4. Kill BarrierServer and restart on another machine. If the various running BarrierClients do not reconnect, see the troubleshooting section above.

[Schedule] [Labs] [Beckman Meeting Rooms] [Equipment] [Projects] [CUBE Projects] [Syzygy] [VSS] [People] [Events] [Publications]