The Viz Architecture Tech Report March 1999 Laurent Nguyen, Kurt Schwehr, Deanne Tucker, Eric Zbinden, Alex Derbes, Paul Henning, Nick Vallidis. Keywords: client/server, XDR, visualization, graphics library, OpenInventor, Robotics, multi-thread, rendering, Python TOC 1. Introduction 1.1 IMG Applications 1.2 Current tools limitations 2. Objectives 2.1 State-of-the-art 2.2 Goal of this development 2.3 Additional considerations 3. Description of viz 4. Server Architecture 4.1 Intro 4.2 Communication 4.2.1 Executing an incoming command 4.2.2 Message packing using External Data Representation (XDR) 4.2.3 Process stage (prep) 4.2.4 Message queue 4.2.6 Output 4.3 Renderer thread 4.3.1 Why OpenInventor ? 4.3.2 Timer sensor 4.3.3 Key/Mouse handling/callbacks 4.4 Add command and scene graph layout 4.5 Other capabilities 4.6 Implementation 5. Clients 5.1 General client design philosophy 5.2 The python support code (PyCom) and C API 7. Applications. 7.1 Data representation/visualization 7.2 Telepresence Remotely Operated Vehicle and the Arctic Mission 7.3 Walker robot 8. Discussion. 8.1 Limitations 8.2 Future work: viz as a library 8.4 Conclusion 9. References A. Messages description B. Related visualization tools B.1 GRASS (http://www.baylor.edu/~grass) B.2 IMEdit (www.innovmetric.com) B.3 Visualization Toolkit - VTK (http://www.kitware.com/vtk.html) B.4 Virtual Environment for Vehicle Interface (VEVI version 3 and 5) B.5 DView (http://dview.jpl.nasa.gov) B.6 Marsmap (http://img.arc.nasa.gov/Pathfinder/marsmap2/www_index.html) B.7 dVISE (http://www.division.com) B.8 Telegrip & RobCAD (www.deneb.com) B.9 Libptk (http://imtsg7.epfl.ch/projects/virgy/public/main/goal.html#Goal) B.10 WITS (http://robotics.jpl.nasa.gov/tasks/wits/) B.11 MUSE Technologies, Inc. (http://www.musetech.com) B.12 Alice (www.alice.org) C. Related graphics libraries/rendering engines C.1 WTK (http://www.sense8.com) C.2 OpenInventor (http://www.tgs.com) C.3 Performer (http://www.sgi.com/software/performer/overview.html) C.4 Java 3D (http://www.sun.com/desktop/java3d) C.5 OpenGL (www.opengl.org) C.6 Direct 3D (www.microsoft.com/directx/default.asp) D. Viz user's guide D.1 Keyboard D.2 Command line options D.3 Messages Abstract The Intelligent Mechanisms Group (IMG) at NASA Ames Research Center has built a series of Virtual Environments for Vehicle Interfaces (VEVI) for telerobotic control. Virtual environments provide a high-bandwidth feedback to the operator and allow an intuitive interaction with the remote robotic mechanism. In order to handle a tighter integration with a growing number of tools, while still maintaining the graphics performance necessary for a realistic telepresence, the IMG has built a visualization package called 'viz'. Viz is a 3D graphics rendering server, one of the modules of IMG's core architecture that allows distributed processing across machines on the same network. Clients such as the user interface, or telemetry display modules can connect to Viz via a socket connection and send messages, in order to modify the scene graph. Some examples of messages include 'add object', 'change color', 'move object', 'remove object'. Clients can be written in any language assuming the language supports socket communication and External Data Representation (XDR) encoding/decoding. Viz supports multi-connections of one or more clients. The server also provides sub-threads to process incoming messages, thus allowing the user to interact with the scene graph with minimal interruptions from communications. The 3D capabilities of Viz are based on the graphics library OpenInventor. 1. Introduction 1.1 IMG Applications The Intelligent Mechanisms Group (IMG) at NASA Ames Research Center is performing research in telerobotic applications and has built a series of Virtual Environments for Vehicle Interfaces (VEVI) for planetary surface and orbital missions. They are utilized to allow the operator to visualize the state of the remote robotic mechanism from sensor data. Typically, stereo images taken by the robot are processed in order to produce photorealistic 3D terrain models that can be visualized in the virtual environment along with the CAD model of the robot. Furthermore, planning, simulation and science analysis tools are integrated into the VEVI, so that control the robot becomes an intuitive task for the operator. 1.2 Current tools limitations The virtual environment is an important component of the user interface. Not only the operator interacts directly with it, but a suite of tools helping the operator's in his/her task are expected. Earlier versions of IMG's interfaces focused on teleoperation. In this context, high frequency telemetry feedback, such as robot pose, are important. The interface has to provide an acceptable rendering framerate, while still handling a large volume of data inputs. Decoupling the hardware from the interface when necessary is crucial to accomodate any hardware or communication failure and provide a continuous 3D rendering of the scene. In addition, low latency in communication with the robot will maintain a short control loop. Beside data inputs, no modules could be connected to the program, and the interaction with the virtual scene was poor. Following versions were built for planetary missions where communication delays are non-negligible. In this situation, the interface must provide more high-level tools and interaction modalities (such as planning, simulation, science analysis) and realistic feedback (high-resolution, and accurate photorealistic 3D models). Current tools provide a set of tools, but rendering performance and processing becomes an issue. 2. Objectives 2.1 State-of-the-art Appendix B provides a non-exhaustive list of visualization packages and related tools. While GIS tools offer a large sweep of terrain visualization and editing capabilities, other tools such as Deneb's products allow to control robotic mechanisms in a structured environment, such as a factory. Furthermore, some of these tools can't handle an environment of unlimited or unknown size. Also, because of their proprietary nature, some tools architecture don't allow the integration of external modules. 2.2 Goal of this development While previous visualization packages from IMG include more features than just rendering in the same program, the design of viz is focused on providing a solid rendering engine. The main goal is to provide a more efficient basic rendering server, to allow the designer to isolate the rendering functions from the rest of the application that (s)he is designing. We focused on a modular open architectural design and rapid proto-typing. This approach allows quick configuration of a system where modules of different nature such as data processing programs or user interfaces can be distributed across the network on different machines. Furthermore, we did not try to replicate the entire functionality of the graphics library outside of the server, but rather identify the most useful functions of a 3D renderer, and implement them into a higher level API. We used messages to try to provide a clear and simple API for several reasons: 1. To make it easy to add support for additional scripting or compiled languages. 2. To make it easier to maintain the server code. 3. Easier to learn than the full OpenInventor API -- rapid prototyping. If you want more elaborate interaction with the 3D environment, we recommend that you write a custom C++ Inventor application. Viz is not intended to support any arbitrary change to the scene graph or user interaction. This allows us to build a scene graph that we can count on to have a number of necessary properties. 2.3 Additional considerations Some constraints influenced the design of viz. First, the computing environment: the IMG is equipped with high-performance 3D graphics UNIX workstation, such as Silicon Graphics Octane SSI, MXI and Infinite Reality running IRIX 6.2, 6.4 and 6.5. Linux machines are also present in the lab and sometimes used to interface with the robot. In addition, rapid-prototyping is a useful necessary feature that allows the developer to accomodate very late changes during a mission. Rendering performance is important, but usually dependent on the type of machine and operating system or the graphics library involved. However, it is possible to improve rendering performance by distributing the processing load on other machines, if the program supports it. Finally, cost can be an issue. We chose to use a graphics library that is provided with the OS for free. This will give us the freedom of distributing the software to other organization without requiring them to buy expensive support software. 3. Description of viz In order to illustrate the general philosophy of viz, we will describe a typical application. Viz is one binary program that you run. It is a multi-threaded application written mostly in C++ with a little bit of C. Once viz is running, a separate client with file/directory browsing capability can help the user load a 3D model. A measuring client that can read outputs of viz (coordinates of points picked by the user) and calculate distances, area, or volume information can be added. Telemetry replay can be performed by connecting a client that will feed viz with updates. 4. Server Architecture 4.1 Intro The server process creates and runs two main threads: the communication thread and the rendering thread. The communication thread listens to the service port for any TCP/IP socket connection. Basically, its only role is to detect if a new client has connected by calling the listen() function in an endless loop. Every time a client connects to Viz, the main communication thread spawns a new thread that we will refer to as working thread. The accept() function assigns that client a new port number. Therefore, one socket descriptor is associated to each client-working thread combination. In this scheme, each working thread is then responsible for taking care of its client's request and process incoming messages. The working thread consists of a call to the select() function in an infinite loop that can be broken by the quitting or breaking the connection. Once a second, the select times out to let the loop check to see if it should terminate from either the quit flag being set or the socket being closed. We will describe how the thread processes any incoming message in the next sections. The rendering thread is responsible for the virtual environment and the interaction with the user. It is running the OpenInventor mainloop(). The working threads are linked to this thread with a queue by which they can pass commands that they have processed. All commands that modify the scene graph are executed in the rendering thread context. 4.2 Communication 4.2.1 Executing an incoming command The client encodes each message to the server into an XDR message defined by that message's XDR definition file. Once a message arrives, the working thread decodes the byte count and message name. Then it reads the entire message. Once the message has been completely received, it calls the XDR deserialize function call for that message type and processes the message. The message name is used to retrieve the right processing function from a lookup table (prep()). This is the main work function for each call. We try to do as much of the processing as possible here, inside the working thread. Depending on the message, the preprocessing function will search the scene graph for the referred object, create new sub-scene graphs, and/or read data from disk. After the prep() function executes, the thread enters the necessary information into a queue of commands to be executed by the rendering thread to modify the scene graph. 4.2.2 Message packing using External Data Representation (XDR) As mentioned before, each message is encoded using XDR. XDR provides an easy way to serialize the message: Each XDR string sent to the server consists in the byte count of the message followed by the message name and the message. The byte count, message name and message are XDR encoded to provide a uniform format for the data stream. There are a couple of reasons why XDR was chosen. XDR is designed to be a platform independent serialization protocol for exchanging data. It is widely used in software packages, including RPC, NFS, NDDS, and NetCDF. As such, XDR libraries are available on just about every interesting platform. It provides a nice, concrete definition language that is easy and fast to parse. The structure is similar to C structure definitions. The overhead of using XDR is light and the necessary library is small. 4.2.3 Process stage (prep) The lookup table of the process functions is implemented using an STL map. Each map entry has a message name associated to a pointer to a function. Upon initialization, the STL map is created and filled with these entries and remains constant for the rest of run time. The function pointer is a pointer to the process function that knows how to create the class for that type of command and run the prep() function. Every time a new message arrives, the map is used to get the pointer to the process function. That process function creates a command object instance and calls the prep() method of that command object, passing the XDR string as an argument. The prep() command XDR decodes the message and then preforms the necessary operations on the scene graph to do as much work as possible without modifying the scene graph. The process command creates this new object and once prep() is done, it adds the command to the queue. 4.2.4 Message queue The queue is known in OOP technologies. It is necessary to use a queue, because only one thread can access the scene graph at a time, since Open-Inventor is not designed to be thread safe. Therefore, only the rendering thread is allowed to modify the scene graph. All other threads that want to modify the scene graph must queue their changes for the render thread to process. Hence, every working threads and the rendering thread have access to the queue, which is owned by the rendering thread. In other words, it is instantiated in the render class. The queue is a queue of commands. Each command is a completely self-contained object. It includes a function to modify the scene graph (modSceneGraph()) and the necessary data and pointers to accomplish that task. We use a superclass called SceneOps as a virtual base class. Each command type is a subclass that defines the data needed and how to actually change the scene graph. The working threads will create a SceneOps object from each message and will add it to the queue. In fact, in order behave like a First In/First Out (FIFO), the queue uses the RendererCmd class, a wrapper class around SceneOps that contains two pointers, previous and next, to implement a double-linked list. At a given frequency, the rendering thread asks the queue to execute a command. The queue extracts the first commands, executes its SceneOps::modSceneGraph() function and then removes it from the queue. This is executed within the rendering thread context. It is worth mentioning that the queue's internal data structures are protected by semaphores, to make it thread safe. Therefore, any working thread can add an object to the queue at any time. As an example, we will describe the add command. The working thread receives an add message. It then XDR-decodes it, and calls the prep() function. The prep() function searches the scene graph for the parent of the new object to add, and creates a separate subscene graph containing the new object geometry, as well as its position and orientation nodes. Pointers to the parent and the subscene graph are then stored into an add command object (derived from the SceneOps class) that is added to the queue. The rendering thread then calls the queue to execute a new command. The queue extracts that new add command, and calls its modSceneGrah() function which executes Separator::addChild(), the OpenInventor function that will add the subscene graph to the main scene graph at the parent node. Thus the render loop never has to wait for the creation of the new object, since an object is added only when it's completely ready. What resides in the subscene graph for each object will be explained later. Notice that the queue contains an additional internal map, an instance of the addTable class, in order to avoid race conditions. Imagine that an 'add object A' is added to the queue, but not yet executed. A new command affecting object A is queued, while 'add object A' is still in the queue, which means object A does not appear in the scene graph, yet. The prep() function of that new command must not only search the scene graph for object A, but also the queue, otherwise it will fail. The addTable class uses an STL map to store pointers to all nodes that are to be added to the scene graph with respect to their corresponding object name. 4.2.6 Output In addition to reading messages coming in, viz also outputs data to the connected client. Currently, the measuring tool mainly uses the the output to returns the coordinates of the points designated by the user. The client reads the points pair by pair and sends back to viz messages to display the calculated distance. 4.3 Renderer thread The role of the renderer thread is to provide the user with responsive interaction with the scene graph and to execute queued commands from the communications threads. At a given frequency and during a given period of time, it will check the queue and execute any pending commands. In order to give a fast response to the user and avoid annoying delays, this thread must minimize processing work and insure a given rendering framerate, as much as possible. This is where previous visualization usually have usually failed, because loading models or managing communication would often take too long, preventing the main rendering loop from handling the user's request. Because of its multi-threaded nature, viz can devote an entire high-priority thread is devoted to the user. Inputs from the user through the mouse and keyboard are processed in callbacks. These callbacks are designed to take minimum time, in order to return as quickly as possible. 4.3.1 Why OpenInventor ? Choosing which graphics rendering package is always a tough decision. We chose OpenInventor for a number of reasons. On the SGI, there is no license required for runtime and IRIX ships with the runtime support libraries installed by default. We like the C++ OOP (Object Oriented Programming) design of the Inventor class hierarchy. There is a default GUI provided by OpenInventor that has a rich set of interaction modalities. Together with the manipulators and draggers, this provides a powerful way to interact with 3D worlds. The Inventor file format (IV) provides a very powerful scene graph structure that is a superset of VRML 1. Complex applications can be written as an IV file and loaded into any Inventor application, including the stripped down program "ivview." Inventor is becoming a cross platform development environment. TGS has ported Inventor to Windows, Linux, and Solaris. TGS has also added almost all of the functionality found in other graphics packages, such as collision detection, mesh optimization and VRML 2 support. 4.3.2 Timer sensor We can divide what happens during the rendering loop into two types of Inventor events. First, the keyboard and mouse events, which will call the interaction callbacks, explained in the next section. Secondly, the timer event which takes care of executing queued commands, explained in this section. Basically, the only relevant call inside the renderer thread function is Inventor::mainLoop(). Everything else is implemented inside the approriate callbacks. To begin with, two parameters controls the execution of the the timer callback. First, the frequency at which the timer goes off. Second the amount of time we allow for executing queued sceneOps. Inside this timer callback, the queue is checked for new commands. A possible next sceneOp is pulled out, its modScene method is executed, and then it is removed from the queue. This operation is repeated until the callback runs out of time allowed to execute queued commands or the queue is empty. Notice, that a low frequency at which the timer goes off would yield a slow response from viz, but would allow more responsiveness to user interaction. Compared to simpler visualization programs that would execute any commands as soon as its received, viz has a higher latency between the time when a command is sent and when it actually effects the scene graph. The other parameter is the length of time that is allowed to execute commands. Every time the timer event goes off, the callback records the current clock time. It then executes a command off of the queue and check to see if it has run out of time. If there is time left it will try to execute another sceneOp. This method allows the renderer thread a better chance to maintain the update rate. We did not want to have the rendering engine pause for long periods of time if viz gets a burst of commands from clients. One limitation is that if scene graph operation takes a long time, the rendering thread is blocked until that operation has been executed. However, most sceneOps are designed to take the minimum amount of time possible, typically a few milliseconds. Some exceptions are the printToFile and saveSceneGraph functions which must white large amounts of data without the scene graph being modified while the run. Another interesting effect is observed at a low timer frequency and long execution time: commands that affect the same object may be overridden by following commands before ever seen by the user. Two examples: First if an object is commanded to be red and then commanded to be blue and both commands are executed in one timer callback, the user will only see a blue object. Second, an Add/Remove combination. An object can be added and removed next. The user will not see any object appearing. This has both good and bad consequences. On one hand, it is bad, because the user may not see the effect of all the commands sent to viz. Interesting events may be hidden. On the other hand, the good news is that if the latency is maintained low enough, what is displayed in viz is realtime. Therefore, what the user sees is the latest update. In a simpler model which displays all high-frequency updates as they come in, the changes will start appearing later and later, not reflecting the current state of the robot. For instance, a user can only see changes faster than 60 HZ by slowing down the rate of display to something slower. In conclusion, these parameters must be tuned appropriately, depending on the application. 4.3.3 Key/Mouse handling/callbacks In OpenInventor, mouse and keyboard events are handled by adding event callback nodes to the scene graph. Everytime a key is pressed, the appropriate callback is called. In the current implementation, mouse events are used in two situations. First, the user has the ability to designate 3D points in the virtual environment. By moving the cursor to the desired position, the user can designate any point lying on any visible surface of an object in the virtual environment. When the button is pressed, the 2D location of the cursor in the window viewport is used to cast a ray in the 3D world from the viewpoint parallel to the viewing position. The resulting 3D point is the intersection with the closest object from the viewpoint. Viz then outputs these 3D coordinates along with the object name, and surface normals to any client that have requested this output type. This mechanism is typically used for measuring, where clients can calculate linear distances, or even surface areas or volumes, based on the coordinates of two points. Secondly, an additional mode has been added to the viz internals: the following cursor. In this mode, viz outputs the coordinates of the 3D points everytime the mouse moves, not just when the button is pressed. This mode is useful to display in real-time what the position of the 3D cursor is. 4.4 Add command and scene graph layout The most important command to viz is the add command. It allows the user to either load a model into viz from a file or use a modified URL to create the basic geometric primitives. While other commands merely change attributes of exisiting node, this command actually dictates how the scene graph will look like. This is where new nodes are added, and, in viz, a set of rules for inserting new objects to the scene graph have been defined, so that assumptions can be made about the scene graph layout when searching the graph, for faster lookup. This section describes the sub-scene graph structure, which is the core unit of the main scene graph. The scene graph is organized based on objects, "object centric." It is a hierarchical collection of objects. Each object has an associated sub-scene graph. The basic form of the sub scene graph consists of a separator root node which contains the name the object, as defined in its message name field. This root is the parent of a number of components that viz counts to be present. The two mandatory components are, in order, a translation node, a group node containing x, y, and z, rotation nodes, (Euler x, y, z rotations). Then, a possible geometry node can be present. It is often a separator node containing the scene graph of the model geometry below. Furthermore, there are option nodes that are not required to be present to save on memory. They usually appear much more rarely, so we decided to only add them when necessary, on the fly. The optional nodes are material node, scale, and drawing style. These nodes are contained between the geometry and rotation group node in the scene graph. All child objects are added at the end of the roots children, after the geometry, on the right side. Note that objects without any geometry node are used to group objects together, like a terrain model containing numerous children for each wedge, or the robot object containing all the robot parts as children. Consequently, Viz is using a fixed organized scene graph that cannot be modified from outside. The user does not have artibrary access to the scene graph. This excludes any fancy scheme like downloading Python scripts into viz, for example. Hence, doing things like running SWIG on the entire Inventor API and allowing inventor scripting clients to be executed by the renderer thread won't work. If this were to be allowed, the rest of viz the viz commands will not function anymore. Scripting should be done at the client level. It is possible for the user to create Inventor files that are Viz compatible -- they contain scene graph sections that follow the Viz specification for objects. This allows objects to be created with hooks for clients to modify their behavior. For example, it is possible to create an entire robot in one Inventor file. Each joint can be named so that clients can move all of the robots degrees of freedom. Also note that adding a camera or a light differs from adding a regular object. In this case, the position and orientation are embedded inside the nodes. However, this is handled by both the changePos and changeOrient messages in the current implementation. Finally, special nodes/objects/entities have special reserved names starting with "_". 'world' is also a reserved name to designate the top level of a scene graph (see Appendix for a complete list). Other nodes such as sensors do not conform to the scene graph rules, but are in other portions of viz. These generally are not acted upon by the standard viz commands. 4.5 Other capabilities Other capabilities have been implemented in viz. The Disk File logger allows to log viz internal activity to a log file (defined at the command line). The terrain following rover is a first test in using the OpenInventor library to determine the contact points of the rover wheels on the terrain by vertically ray picking. Although this feature was implemented inside Viz, a very more precise simulation should be performed outside viz, where the client could calculate contact points as well as other dynamic parameters. Viz also supports built-in billboards, a concept first introduced in VEVI. Billboards are images textured on a 'billboard' in the 3D environment. The billboards are desgined to always face the viewpoint, wherever it is, so that the user always sees the billboarded image. Viz uses the billboard engine, which consists of a sensor node attached to the viewpoint. Every time the viewpoint moves or changes its orientation, the sensor callbacks adjusts all billboards orientation. 4.6 Implementation Viz was written in C++, with a small portion of communication code in C. It uses the C++ OpenInventor graphics library, which was easy to intergrate. However, the design of Viz does not fully utilize the power of the object-oriented approach. Only the sceneOps class is used for inheritance. The other main classes, such as ivQueue, ivThread or serverClass, were only instantiated once. A major design flow lead to too much cross talk between classes concerning the queue. It turns out that the working threads needed to know about some of the rendering threads attributes. Since the queue was the only object to be global to the application, numerous access functions were added to it, until we realized that making the rendering thread global made more sense. The Standard Template Library (STL) was used at various locations. It worked well for us, once we figured out how to avoid temporary object code compiling that would break our build mechanism. In addition, we used two STL maps that were instantiated along with the classes definition, which unfortunately had the side effect of generating code in the header files. The build mechanism consists in a hierarchical set of Makefiles and source directories. Each Makefile compiles the source code in the current directory and calls any sub-directory Makefile, to produce a static library. The final binary is produced by compiling and linking these libraries. 5. Clients 5.1 General client design philosophy Each client is a separate heavy weight process (program). Clients can run on the same machine or be distributed across a network. They can be part of a separate application, a small GUI, or a script. In fact, anything that can send XDR encoded data through a socket can be a client. 5.2 The python support code (PyCom) and C API So far, we have implemented client support for two languages: Python and C. Adding languages is fairly easy and could be done by in a couple days at the longest. Each language requires two components to support writing clients. The first is socket support. The second is a code generator to produce code for each message that serializes and deserializes the data into XDR format. We have written a generic XDR parser in Flex and Bison (GNU versions of Lex and YACC). This parser produces a parse tree data structure. It is then a simple process to write a small bit of C code that walks the parse tree to produce the XDR code for whatever language. The Python scripting language has turned out to be very easy to work with generating clients. Each message is implemented as a class. After a socket is open, a message can be created and can send itself to the socket. Based on that API, a Python-based shell for Viz (vsh) has been put together. It allows the user to interactively modify the scene graph from the command line, using the full set of the Python interpreter features. 7. Applications. This section describes the various applications that used Viz. 7.1 Data representation/visualization Viz was the visualization module of a Masters thesis on sensor data visualization. Using a generic multi-dimensional sensor data model, multiple rules was defined to display the data, based on the dimensions and application. A separate client was implemented by the student to generate or receive sensor data and translate them into the generic data model. This C program, which used Xforms for its 2D GUI, then would display the data according to the desired modality specified by the user. Viz was mainly used for visualization. The viz program did not have to be modified. Also, no real interaction with the user was necesary, except for moving the viewpoint. Hence, no data needed to be returned to the client from Viz. The performance of the rendering engine was sufficient. Only small datasets were used. The application was able to display realtime data. The integration with Viz took only two weeks including writing the basic client. 7.2 Telepresence Remotely Operated Vehicle and the Arctic Mission Viz used to represent 3D models of TROV in a fake terrain for the Oceans 97 conference in Monterey. This was again a one way client. A 3 axis compass was connected via serial port to a Silicon Graphics Inc. workstation. A Python interface, derived from the Nomad Virtual Dashboard, displayed a 2D compass. This interface then converted the compass data into commands for the TROV model in Viz. This was the first major usage within Viz of the animation and manipulation powers provided by OpenInventor. An Inventor engine was connected to the propellers to turn them at a rate proportional to speed. A dragger was attached to the vehicle allowing the user to move TROV about the 3D world. Manipulators were attached to the forward camera's pan/tilt unit. In order to have a Viz camera smoothly flying around the model, an additional client was interpolating intermediate positions between defined viewpoint poses and sending viewpoints position updates. 7.3 Walker robot The IMG has designed and developed a prototype of a 6-legged 'Spider' robot. Unlike wheeled-robots, the spider was designed to take minimal space undeployed. In order to study the behavior of various walking algorithms before the hardware is built, the team implemented a simulator of the leg planner which was interfaced to Viz. This is a good example where the simulator computes robot-terrain interaction outside of Viz. 8. Discussion. 8.1 Limitations This section will describe the main limitations of this architecture as well as mention the problems we encountered and unsolved issues. First, the main limitation is expandability. As feature are added, new messages to support that feature must be implemented. Then, the terrain following application raises the question whether it should be implemented inside or outside Viz. The advantage of doing it inside is that the 3D graphics library functions are directly available to deal with the virtual objects. There is no protocol or overhead involved. For instance, ray picking can be performed within a single function. However, one might choose to implement a better ray picking algorithm outside Viz, hence leaving Viz generic. The easy quick solution is usually to implement everything inside the renderer. The question of what Viz should really output to the client wasn't really solved. How do we deal with queries ? How do we deal with only some clients wanting subsets of messages ? Do we want to add an object database or some way of using the scene graph to allow clients to know what objects exist in the world ? OpenInventor provides the ability to attach sensors to any node in the scene graph. Whenever the node changes, the sensor callback is executed. This is very useful for monitoring an object position, for example. However, do we offer that capability to the Viz client in a generic way ? Another important question: Is is possible to make Viz independent of the rendering engine/graphics library? In theory, yes, definitely ! Most 3D graphics library use a scene graph that they traverse when rendering a scene. Therefore, the set of RenderCore message should apply to any graphics library, eventhough the API for searching nodes in the scene graph or modifiying the scene graph, for instance, differs. Viz was designed to allow that flexibility. However, there a tradeoff between flexibility and capabilities. The more flexible the visualization is designed to handle any 3D library, the more minimal is the supported common features between these libraries, the less use of the library's specific powerful concepts or tools you allow. For instance, OpenInventor's draggers and manipulators don't exist in World Toolkit. Should they be supported ? if so, what happens if we switch the library to World Toolkit ? Lastly, there are still questions about what is really needed for thread stack sizes. The default values were definitely too small. 8.2 Future work: viz as a library Since the first implementation of Viz resulted in a program that runs, the core of the future work is to build clients to exploit the range of features. In addition, the concept of Viz as a library offers a way to allow the user to customize Viz's internal behavior through a clean safe API. The idea is that the user would call two or three functions (init(), mainloop(), finish(), for example) from the library to run a standard Viz. Additionally, the user would also be able to redefine these functions, in order to change the behavior of the terrain following rover, for instance, or even add custom callbacks. However, this concept defeats the purpose of having a program that is easy to run and to connect to. 8.4 Conclusion This is the first implementation of Viz. It has proved to be a descent renderer that provides acceptable performance while offering a very open architecture. The package has been evaluated with various kinds of applications. A truly object-oriented analysis and design should be applied, whenever a rewrite turns out to be necessary. 9. References Dante 1 & 2, www.frc.ri.cmu.edu Nathalie A. Cabrol, G. Chong Diaz, et al (1998). Atacama I: Science Results of the 1997 Nomad Rover Field Test in the Atacama Desert, Chile (pdf,32K) In Lunar and Planetary Science Conference, Houston, TX, 1998. Nathalie A. Cabrol, G. Chong Diaz, George Dunfield, et al (1998). Atacama II: Nomad Rover Sample 1-250697 and Implications for Fossil Characterization During Mars Exploration. (pdf, 87K) In Lunar and Planetary Science Conference, Houston, TX, 1998. N. Cabrol, P. Lee, G. Chong Diaz, et al (1998). Atacama III: Meteorite Search During the Nomad Field Tests: Perspectives on Automated Field Operations by Teleoperated Vehicles in Extreme Environments (pdf, 44K) In Lunar and Planetary Science Conference, Houston, TX, 1998. C. Stoker, T. Blackmon, J. Hagen, B. Kanefsky, D. Rasmussen, K. Schwehr, M. Sims, E. Zbinden Marsmap: An Interactive Virtual Reality Model of the Pathfinder Landing Site (pdf, 67K) In Lunar and Planetary Science Conference, Houston, TX, 1998. David Wettergreen, Maria Bualat, Daniel Christian, Kurt Schwehr,Hans Thomas, Deanne Tucker, Eric Zbinden (1997). Operating Nomad during the Atacama Desert Trek (pdf, 719K) In Proc. Field and Service Robotics Conference, Canberra, Australia, December 1997. David Wettergreen, Hans Thomas, Maria Bualat (1997). Initial Results from Vision-based Control of the Ames Marsokhod Rover (pdf, 351K) In Proc. IEEE/RSJ International Conference on Intelligent Robots and Systems, Control of Wheeled Robots, pp. 1377-1382, Grenoble, France, September 7-12, 1997. Daniel Christian, David Wettergreen, Maria Bualat, Kurt Schwehr, Deanne Tucker, Eric Zbinden (1997). Field Experiments with the Ames Marsokhod Rover (pdf, 670K) ps version (2.7MB) In Proceedings of the 1997 Field and Service Robotics Conference, Canberra, Australia, December 1997. John E. Bares, David S. Wettergreen (1997). Lessons from the Development and Deployment of Dante II (pdf, 590 K) In Proceedings of the 1997 Field and Service Robotics Conference, Canberra, Australia, December 1997. T. Fong (1993). A Computational Architecture for Semi-Autonomous Robotic Vehicles (postscript; 1,514 KB). pdf version; (195 KB) In AIAA 93-4508, Computing in Aerospace 9 Conference, San Diego, CA, October 19-21,1993, L. Piguet, B. Hine, P. Hontalas, T. Fong and E. Nygren (1996). The Virtual Environment Vehicle Interface: A Dynamic, Distributed And Flexible Virtual Environment (763 KB, pdf document). C. Stoker and B. Hine (1996). Telepresence Control Of Mobile Robots: Kilauea Marsokhod Experiment (24 KB). pdf version; (44 KB) S. Schneider, V. Chen, J. Steele, and G. Pardo-Castellote (1995). The ControlShell Component-Based Real-Time Programming System, and its Application to the Marsokhod Martian Rover (778 KB). pdf version; (260 KB) B. Hine, P. Hontalas, T. Fong, L. Piguet, E. Nygren, and A. Kline (1995). VEVI: A Virtual Environment Teleoperations Interface for Planetary Exploration (1,433 KB). pdf version; (514 KB) In SAE 25th International Conference on Environmental Systems, San Diego, CA July 1995 T. Fong, H. Pangels, D. Wettergreen, E. Nygren, B. Hine, P. Hontalas, C. Fedor (1995). Operator Interfaces and Network-Based Participation for Dante II (4,573 KB). pdf version; (502 KB) In SAE 25th International Conference on Environmental Systems, San Diego, CA July 1995 L. Piguet, T. Fong, B. Hine, P. Hontalas, and E. Nygren. (1995). VEVI: A Virtual Reality Tool For Robotic Planetary Exploration (962 KB). pdf version; (266 KB) In Proceedings of Virtual Reality World, Stuttgart, Germany, February 1995. B. P. Hine III, C. Stoker, M. Sims, D. Rasmussen, P. Hontalas, T. Fong, J. Steele, D. Barch, D. Andersen, E. Miles, and E. Nygren. (1994). The Application of Telepresence and Virtual Reality to Subsea Exploration (1,651 KB). pdf version; (375 KB) In The 2nd Workshop on Mobile Robots for Subsea Environments, Proc. ROV'94, May 1994, Monterey CA. Appendices: A. Messages description Add: Assuming that it is a file on disk, we call the Inventor SoDB::readAll call with the filename. All relative paths end up being relative to where Viz was started. This is often confusing. People often think that a file will be found relative to the client. File lookup happens in the normal manner encounter in the Unix file system with calls such as fopen. The scene graph created by SoDB::readAll is attached to our special sub-scene graph. We store the root pointer for this subscene graph and the parent pointer into the Add class and queue the command. B. Related visualization tools B.1 GRASS (http://www.baylor.edu/~grass) GRASS is a free Geographic Information System (GIS) package developed by the Army Corps of Engineers. GRASS is the first place that one of the authors saw the rendering system strongly separated from much of the processing of 2D and 3D data. B.2 IMEdit (www.innovmetric.com) InnovMetric designs high-end software solutions for processing 3D digitizer data, and for optimizing 3D models. The IMEdit/IMCompress package provides a series of tools to manipulate 3D meshes. Among interesting features are measuring, reducing models, translation to VRML 1, VRML 2 and other standard formats. B.3 Visualization Toolkit - VTK (http://www.kitware.com/vtk.html) VTK is a free software package that provides a large set of tools to manipulate and visualize 3D models. It is supported and is available on standard platforms/OS. Although wrapped on OpenGL, it provides a slightly different way of interacting with the scene than the regular scene graph. B.4 Virtual Environment for Vehicle Interface (VEVI version 3 and 5) VEVI 3 was developed by the Intelligent Mechanisms Group. VEVI 3 is based on World Toolkit 2.1 for rendering and NDDS 1.9 for communications. The program was successfully used for several robotics missions including CMU's Dante 1 and 2, TROV, and Marsokhod field experiment. VEVI 3 does not use a scene graph, as World Toolkit did not support that concept at the time VEVI 3 was developed. VEVI 3 provided the inspiration for much is what Viz is. We tried to improve the system performance and scriptability. VEVI 3 introduced two concepts, Vshell and Vscript. For Viz, these became our python shell for Viz. We decided that we did not want to write our own scripting language and were frustrated with programming in Tcl, so we went with Python. Execution of VEVI is controlled by files known as Vevi Configuration Files (VCF). VEVI had 2D image billboards and was integrated with Netscape to provide links to user data. VEVI 5 (http://www.4thplanet.com) The developers of VEVI 3 decided to spin off a company from NASA Ames Research Center that is known as Fourth Planet. Since leaving NASA, Fourth Planet has been continually developing VEVI for use in a number of projects, such as the Ford factory automation, or for visualizing large computer networks in real time. VEVI has been ported to Solaris, HP-UX, and Windows NT. The NDDS networking package was replaced by FPcom, the company's in-house communications package. B.5 DView (http://dview.jpl.nasa.gov) DView has been under development at NASA's JPL for a number of years. It has been created and is maintained by Patti Koenig. DView is based on OpenGL and Motif. It integrates a number of large packages from JPL to support planetary missions, including SPICE. It provides an external string-based scripting interface with the Tcl language. It is not multithreaded. It runs on IRIX, Solaris, and Windows. It is currently being ported to Java 3D. B.6 Marsmap (http://img.arc.nasa.gov/Pathfinder/marsmap2/www_index.html) MarsMap is a World Toolkit based application developed by Blackmon, from his previous work on the Mars Virtual Explorer at Berkeley. MarsMap was used by the Intelligent Mechanisms Group (IMG) for the Mars Pathfinder mission. It provides a good example of the some of the common techniques of measuring distances and angles of a 3D model by picking with the mouse into the scene. This and other techniques can be found in CAD, GIS, and mesh editing tools. MarsMap has also been used quite extensively for the Pathfinder Mission as a visual cataloging tool. Data markers such as 3D text symbols are superimposed on the terrain model to indicate the sequence of APXS measurements taken by the Sojourner rover, soil mechanics experiments and other science related events. B.7 dVISE (http://www.division.com) DIVISION's dVISE family of products provides cost-effective access and viewing of product information, and can dramatically improve communication between all members of the enterprise. dVISE also provides Virtual Mockup and Functional Simulation tools which help manufacturers to optimize assemblability, maintainability, and operability, dramatically reducing the need for physical prototypes. Although dVise provides the capability for external products to interact with the renderer, it is more focused toward industrial applications. B.8 Telegrip & RobCAD (www.deneb.com) Deneb offers an integrated suite of tools that begin at the conceptual phase of product and process design enabling design for assembly, design for manufacturability and design for maintainability. Deneb's suite includes products for: discrete event simulation for factory layout and throughput analysis; workcell tooling design and validation; assembly planning and analysis; human motion and task analysis; robotic off-line programming; CNC program validation; CMM program verification; and simulation-based training. All products can be viewed in a Virtual Reality environment, shared around the world using Deneb's Virtual Collaborative Engineering environments, and run from a PC on the shop floor or in the field for operator instructions. While quite expensive, this package is more suited to factory/industrial applications, structured environment where prior knowledge of the world objects is necessary. igrid does not deal well with very large unstructured terrains. B.9 Libptk (http://imtsg7.epfl.ch/projects/virgy/public/main/goal.html#Goal) Libptk is a 3D graphic engine designed to run various real time simulations. Using a description file, the program loads a scene and allows the operator to interact with it. Interactions are performed through a Gizmo based interface, so the user can set on the fly various parameters of the simulation. A scene is made of a population of objects. Some of them are rendered objects (visualized) such as a cube, a robot, an organ or a tool. Most of them are description objects which are not visualized but control the simulation, they are for instance deformation units, texture contexts, graphic environment or sensors (This enumeration is not exhaustive). Libptk stands for Library Performer Toolkit. It has been developed by the VRAI lab at the Swiss Federal Institute of Technology, Lausanne. Libptk is mainly targeted at medical applications. It is Object-oriented, with a very clean C++ API. Based on the graphics library Performer, it only runs on Silicon Graphics Inc. (SGI) workstations, but can fully use the multi-processor and graphics resources of these high-end machines. B.10 WITS (http://robotics.jpl.nasa.gov/tasks/wits/) WITS is an Internet-based tool that enables scientists to collaboratively participate in planetary lander and rover missions from their home institutions. WITS enables the viewing of downlinked images and results in various ways, terrain feature measurement and annotation, and planning of daily mission activities. WITS is written in the Java language and is accessible by mission scientists and the general public via a commercial web browser. The public can use WITS to plan and simulate their own lander and rover missions. WITS was used in the 1997 Mars Pathfinder mission for public outreach and limited science operations and will be a nominal science operations tool in the 1998 Mars Volatiles and Climate Surveyor (MVACS) lander and 2001 and 2003 rover missions to Mars. B.11 MUSE Technologies, Inc. (http://www.musetech.com) This company develops and markets software products that enhance a computer user's ability to visualize, analyze and understand large and diverse forms of data using UNIX and WindowsNT(TM)-based systems. The Company's products provide solutions to complex data integration and data management problems encountered by scientists, engineers, developers and other computing professionals. The Company also develops advanced collaborative software that enables groups of users to work together over local and global networks. B.12 Alice (www.alice.org) Alice is a 3D interactive plugin to a web browser. It is available only on Windows 95/98/NT since it uses Microsoft's Direct-3D library and MS Visual Basic. It uses Python as its scripting language. It is being developed at Carnegie Mellon University. C. Related graphics libraries/rendering engines C.1 WTK (http://www.sense8.com) World Toolkit provides virtual environments capabilities for PC and Unix through a C library. It also supports a variety of peripherals. It is quite an expensive package. C.2 OpenInventor (http://www.tgs.com) OpenInventor comes with IRIX for free on Silicon Graphics Inc (SGI) workstations. Template Graphic Software Inc has bought the development license and is currently maintaining and providing support for subsequent releases. It provides a hierarchical C++ library. C.3 Performer (http://www.sgi.com/software/performer/overview.html) IRIS Performer is for developers who need to extract maximum performance from SGI machines for visual simulation, virtual reality, game development, and high-end CAD systems. Often these applications need multi-processor Onyx systems with multiple RealityEngine pipelines with a high degree of parallelism and running at fixed frame rates. C.4 Java 3D (http://www.sun.com/desktop/java3d) Java 3D is a network-centric, scene-graph based API for developing 3D applets and applications. It is said to provide reasonable performance on PC. C.5 OpenGL (www.opengl.org) Originally developed by SGI, OpenGL (for "Open Graphics Library") is a software interface to graphics hardware. It is the low-level layer on which graphics library such as Inventor, or Java 3D are based on. OpenGL is an open specification, and implementations for most of the major platforms and OS are available. Some are even free. SGI developed OpenGL optmizer (http://www.sgi.com/software/optimizer/index.html) a software package, to optimize the 3D models to render. C.6 Direct 3D (www.microsoft.com/directx/default.asp) Direct3D -- Direct3D is a part of Microsofts DirectX. D. Viz user's guide Viz needs the following developing environment: - SGI IRIX 6.3,6.4 or 6.5 - C++ compiler (for development) (not gcc) - Standard Template Library (STL) - OpenInventor (SGI 2.1.5) - GNU make D.1 Keyboard h - help l - add directional light editor t - terrain following rover -/+ on the keypad -- timing level. The '+' and '-' keys are mapped to an internal variable known as the debug level. This allows the user to increase with + and decrease with - the amount of debugging information sent to stdout. At higher levels, the amount of information sent to stdout will great effect the behavior of the system, so this information is generally not produced unless needed. The 'p' key. The 'p' key prints the entire scene graph to stdout. This is very useful debugging small scenes. It can take a VERY long time is there are many objects, especially polygon meshes. 't' starts the terrain following rover (experimental). 'q' is the end of the world as we know it. 'q' quits out of viz without asking! Is this a feature or a bug? Some say feature! 'm' is the mail key ala www.JWZ.org (experimental). It tells netscape to bring up its mail reader. 'n' tells netscape to start reading news. 'm' They illustrate how to interface with Netscape from separate programs. D.2 Command line options --debug-level=num Set the debug level as described above. --input-port-number=num Set the service port to which clients can connect. --log-file=name Set the log filename. --iv-timer=int Set the time interval in ms between each check of the queue by the rendering thread. --iv-execute=int Set the allowed time in ms for the rendering thread to execute queued commands. --follow-cursor=bool Set Viz in a mode where each mouse movement is output to interested clients. --version Outputs Viz build version. --help Display the help page. D.3 Messages The following messages are currently implemented. The XDR definition file describes the necessary arguments for each message. Because of the way the renderer traverses the scene graph, a node will affect everything in the scene below it and to its right. Consequently, in viz, a command affecting an object will also affect its children. Note that: 'world' is a reserved name for the scene graph root. Use it only to refer as a parent. In the current implementation, it is the user responsibility to make sure objects have unique names. ADD: Add an object called 'name' to 'parent', an object in the scene. 'URL' defines where the object is created from: - 'shape:cube' (cone, cylinder or sphere) creates a primitive. - 'http://path/filename' or 'ftp://path/filename' load an object from a specified URL (if python is supported). - 'text2d:text_to_add' adds a 2d text. - 'text3d:text_to_add' adds a 3d text. - 'billboard:path/image_filename' loads an image. - 'filename' is the default way to load VRML-1 or IV model into viz. struct AddStruct { string name<>; /* Name of the object */ string parent<>; /* Parent name */ string URL<>; /* URL path of a model to load */ float pos[3]; /* Position x,y,z (meters) */ float dir[3]; /* Roll, pitch, yaw (radians) */ int visible; /* 0=hide on creation, 1=show immediately */ }; CHANGE ALPHA: change the transparency of an object. struct ChangeAlphaStruct { string name<>; /* Name of the object */ float alpha; /* change the alpha blending. */ }; CHANGE COLOR: change the color of an object. Note that '_background_0' as a name (respect. '_background_x') refers to the background color of Viewer 0 (respect. x). struct ChangeColorStruct { string name<>; /* Name of the object */ float rgb[3]; /* change the color. Each value is [0..1] */ }; CHANGE FONT: change the font of a text object. Supported fonts are currently: "Palatino-Italic", "Palatino", "Palatino-Bold", "Palatino-BoldItalic", "Helvetica", "Helvetica-Bold", "Helvetica-Oblique", "Helvetica-BoldOblique", "Courier", "Courier-Bold", "Courier-Oblique", "Courier-BoldOblique", "Times-Roman", "Times-Bold", "Times-Italic", "Times-BoldItalic", "DefaultFont". struct ChangeFontStruct { string name<>; /* Name of the object */ string font_string<>; /* change the font */ }; CHANGE ORIENTATION: change the orientation of an object. struct ChangeOrientStruct { string name<>; /* Name of the object */ float dir[3]; /* radians */ }; CHANGE PARENT: change the parent of an object. struct ChangeParentStruct { string name<>; /* Name of the object */ string parent<>; /* New parent name */ }; CHANGE POSITION: change the 3D position of an object in the parent's coordinate frame. struct ChangePosStruct { string name<>; /* Name of the object */ float pos[3]; /* Position x,y,z (meters) */ }; CHANGE SCALE: change the scale of an object in the three directions. struct ChangeScaleStruct { string name<>; /* Name of the object */ float scale[3]; /* Scale the object by 'scale' */ }; CHANGE TEXT: change the string of a 2D or 3D text. struct ChangeTextStruct { string name<>; /* Name of the object */ string text_string<>; /* change the text */ }; HIDE: hide an object (but do not remove it). struct HideStruct { string name<>; /* Name of the object */ }; REMOVE: Remove an object from the scene. struct RemoveStruct { string name<>; /* Name of the object */ }; SHOW: Show an object (if hidden) struct ShowStruct { string name<>; /* Name of the object */ }; FOLLOW CURSOR: turn on the following cursor feature. If on, viz output the 3D location of the cursor by ray casting from the camera viewpoint and intersecting with an object. struct FollowCursorStruct { int on; /* 1 == Turn on messages; 0 == don't send any msgs */ }; OUTPUT CONTROL: if on, viz outputs certain types of events to the client (what viz outputs needs to be refined). struct OutputControlStruct { int on; /* 1 == Turn on messages; 0 == don't send any msgs */ }; PRINT TO FILE: Take a snapshot of the scene and write the image to a postscript or rgb file. struct PrintToFileStruct { string type<>; /* "ps" or "rgb" */ int imgWidth; /* -1 for the maximum */ int imgHeight; /* -1 for the maximum */ string fileName<>; int background[3]; /* RGB background color */ /* Set to 0 if rgb file. */ int dpiX; /* Printer dots per inch. */ int dpiY; /* Printer dots per inch. */ }; SAVE SCENE GRAPH: Save the scene graph onto a file (currently IV file). Can be considered as a 3D snapshot. struct SaveSceneGraphStruct { string fileName<>; /* Name of the file to save the SceneGrahp to */ string rootName<>; /* root node to save from*/ string asciiObin<>; /* "ascii" or "bin" */ };