4/2001 NOTE: Email addresses have been removed and machines are no longer on the net. PLEASE send me any updates you would like added to this document. It is in no way complete and has not been edited by anyone for accuracy, grammar, spelling, etc. This doc is pretty confusing and jumpy. Good luck reading it :) The goals of this document are: * Allow outside people to see how the system was setup. * Allow future teams to learn from my mistakes/experience. * I'm sure that I will forget alot of this stuff when the next mission comes up. * Maybe allow for some reuse of code and practices. ---------------------------------------------------------------------- $Id: james.txt,v 1.6 2000/01/05 01:28:18 schwehr Exp $ MVACS Image Pipeline / JAMES Pipeline Author: Kurt Schwehr copyrite 2000 NOTE/WARNING: Acronym section at the end may help decrypt this. **************************************** * TABLE OF CONTENTS/DOOM **************************************** - Quick reference to data access. - Quick start proceedure. - Monitoring. - Machine Descriptions. - A walk through the pipeline. - fwDMS_raw - Calibration - fwDMS_ical - MCP - Master Control Program - stereopair2wits - pioToWeb - General Environment Variables - Description of scripts and programs - Runtime control for scripts - MCP.bash TODO LIST --------- - Kerberos section and/or add it to the open source software section. - Frame of reference for programming and style used. - Explain the Ames Stereopipeline, Viz setup and pushing of images. - Fix up section on monitoring - discuss different use of left or right eye as reference. **************************************** * Quick reference to data access. **************************************** So you're a science type and you just want to find the data and start looking at it. Assuming that your on a Unix box (A Sun, SGI, or Linux), you can do the following. If your on a Mac or PC, you'll need someone else to help you. To look at images: cd /proj/ops/ssi/current/data/vap/images/jpg xv & Ctrl-V in the xv window to open the "Visual Schnauzer" Double click on the "sol" directory you are interested in. To look at mosaics: cd /proj/ops/ssi/current/data/vap/mosaics To run Viz or Wits, find some other document to help you... :) **************************************** * QUICK START PROCEDURE **************************************** Warning: Don't just apply this literally. Best to read through all of the documentation first. MISSION PHASE: landed (or if link set, can use "current") ___ mail ( email removed to spare us from spam ) schwehr zbinden breid falevsky jdweinb lemmon chriss Justin.N.Maki sfhviid Subject: Starting phase landed Starting up mission phase landed /proj/ops/ssi/current -> landed -Your name here Window: ___ ssh gluttony.mvacs.ucla.edu ___ top Window: ___ ssh gluttony.mvacs.ucla.edu ___ cd /proj/ops/ssi/ ___ rm current ___ ln -s landed current Window: ___ Log into any GDS or Viz computer (sgi or sun): e.g. sloth, viz, k9, etc. ___ Run /proj/ops/ssi/bin/fwStatus.bash landed Window: ___ ssh gluttony.mvacs.ucla.edu ___ setenv SSI_SOL_UNKNOWN sol000_unknown ^ for sol 0 only... ___ /proj/ops/ssi/bin/fwMaster.bash landed Window: ___ ssh rage.mvacs.ucla.edu ___ source /proj/mvacs/ssi/fei/feiinit.sh ___ kinit kds -l minutes: 1275 passwd: ___ klist ___ cd /proj/ops/ssi/landed/queues/raw_in ___ feisubscribe m98_mist_ssi_edr using /proj/ops/ssi/landed/bin/config/fei_ssi.options ___ feisubscribe m98_mist_rac_edr using /proj/ops/ssi/landed/bin/config/fei_rac.options Window: (Must be on k9.mvacs.ucla.edu) ___ cd /projects/viz/src ___ source vizsource ___ cd /projects/viz/src/Gui ___ set path = (/apps/sup/packages/python-1.5.1/IRIX-6.4/bin $path) ___ ./goViz Window: (Must be on k9.mvacs.ucla.edu) ___ /proj/ops/ssi/bin/sgiSay.bash landed Window: (Must be on k9.mvacs.ucla.edu) ___ cd /proj/ops/ssi/landed/data/vap/mosaic ___ xv -poll -nod jnm-mosaic.tif & ___ xv -poll -nod jnm-mosaic.tif & ___ cd /proj/ops/ssi/landed/data/images/jpg ___ xv -poll -nod -maxp ames-latest.jpg & NOTE: * See Justin Maki for information on his mosaics. * See Eric Zbinden for how to have wedges/terrain patches automatically pushed into Viz as they are created. * See Eric Zbinden, Mike Wagner (wagnermd), or Larry Edwards for how to make Ames mosaics. ====================================================================== TO SHUTDOWN: ___ mail schwehr, zbinden, breid, falevsky, jdweinb, lemmon, chriss, Justin.N.Maki, sfhviid Subject: SHUTDOWN of phase landed SHUTDOWN of mission phase landed -Eric ___ cd /proj/ops/ssi/landed/queues/raw_in ___ touch stop.rtc AND/OR ___ /proj/ops/ssi/bin/fwStop.bash landed ___ Watch the fwStatus.bash display for things to quit. TO PAUSE: ___ cd /proj/ops/ssi/landed/queues/raw_in ___ touch pause.rtc ___ Watch the fwStatus.bash display for things to pause. ___ touch resume.rtc ___ Watch the fwStatus.bash display for things to start back up. **************************************** * MONITORING **************************************** What you can do if your not the person directly controlling the pipeline. How can you see what is going on. fwStatus.bash fwStatus.bash landed Example output:
############################################"
## Pipeline status for <$phase>"
##
Fri Dec  1 10:10:33 PST 1999

FIX: cat of files...

FIX: ls -g of files.

FIX: how to interpret what you see in the display. **************************************** * MACHINE DESCRIPTIONS **************************************** This is a quick overview of the computers involved, their type, configuration, and purpose. The goal is to give enough background for figuring out how we got things to work out and how much compute power we were using. gluttony.mvacs.ucla.edu - This was a Sun Ultra-5 running SunOS 5.7 with 128 MB of ram. Not the speediest machine at MVACS, but it did not have a monitor, so people would not be logged into it unless they were running the James pipeline. This was the machine used to run one or two instances of the James pipeline, one for the real lander (???) or test bed (mist???), and one for the test unit at Lockheed Martin (stl). > uname -a SunOS gluttony.mvacs.ucla.edu 5.7 Generic_106541-07 sun4u sparc heresy.mvacs.ucla.edu - iMac G3 running Common Lisp (What flavor). This was Bob Kanefsky's Mac that ran the sequence tracking code. It ran a web server interface to the Lisp code. See Kanefsky for further descriptions of the tools that he developed. rage.mvacs.ucla.edu - Sun Ultra-5 running SunOS 5.5. This machine was one of the few that was setup with kerberos configured to access the JPL MIPL databases with fei. The other machine setup for MIPL kerberos was shaggy.mvacs.ucla.edu. This was also the internal time server. rage.schwehr> uname -a SunOS rage 5.5.1 Generic_103640-29 sun4u sparc SUNW,Ultra-5_10 k9.mvacs.ucla.edu - SGI Octane MXI running IRIX 6.5 with 256 MB of RAM. This machine was the Ames machine in the SSI cubical. It was hooked up to one of the science work area project screens (the one on the right) and was used to show viz during meetings, DMD, packet_watch, mosaics and live images. IRIX64 k9 6.5 07151433 IP30 mips k9.schwehr> hinv 2 195 MHZ IP30 Processors CPU: MIPS R10000 Processor Chip Revision: 2.7 Main memory size: 384 Mbytes Graphics board: EMXI shaggy - Sun Ultra 60 with RAID array. Server. Allowed ftp. scooby - Sun Ultra 60 with RAID array. Server. 256 MB ram SunOS scooby.mvacs.ucla.edu 5.7 Generic_106541-07 sun4u sparc Filesystem 1k-blocks Used Available Use% Mounted on /dev/dsk/c0t0d0s0 97306 53249 34327 61% / /dev/dsk/c0t0d0s6 817976 593014 167704 78% /usr /dev/dsk/c0t0d0s3 336931 20876 282362 7% /var /dev/dsk/c0t0d0s4 7131707 9 7060381 0% /u/1 /dev/dsk/c1t1d0s0 122851802 30041298 91581986 25% /r/1 /dev/dsk/c1t2d0s0 122851802 49614650 72008634 41% /r/2 /dev/dsk/c1t3d0s0 122851802 77044864 44578420 63% /r/3 / 97306 53249 34327 61% /net/scooby cruelty.mvacs.ucla.edu -- Sun Ultra 80 used primarily for Wits by the Robotic Arm team. > uname -a SunOS cruelty 5.7 Generic_106541-08 sun4u sparc 1024MB of ram 4 cpu fury.mvacs.ucla.edu - Sun Ultra 10. This machine was the main Robotic Arm team computer until cruelty arrived. viz - SGI's loaner Onyx2 IR with 4 R12K (???) and 1024MB of memory. This was the primary computer used for presentations in the main press area. t2 - NASA Ames SGI Onyx2 IR with 2 R10K CPUs and 512MB of memory. The server for the 50 GB disk for the Ames team. erratic - Hooked up to the lander testbed? Also served an NFS filesystem. **************************************** * A WALK THROUGH THE PIPELINE. **************************************** This first couple paragraphs are very terse, since the pipeline did not deal with those processes. Uplink command generation. Seqgen, Apgen, Fairwood SASF Editor (FSE) Stubbe Hviid's program. Produces sequences and from multiple sequences, a pef is generated. A matching ekernel is generated with pef2ek. The ekernel is submitted to MIPL via fei. The commands are uplinked to the spacecraft in the form of a ????. The commands run on the spacecraft. Data is sent back to Earth in the form of CCSDS packets or SFDU packets. Which is it? Data is sent back to Earth in the form of CCSDS packets (embedded in a CCSDS Transfer Frame protocol). Once on Earth, they are encapsulated into Standard Formatted Data Units (SFDU). For example, information like Earth Received Time, DSN station number, etc. are in the SFDU header. The SFDU data contains a CCSDS packet, whose headers give packet generation time, APID, packet number, grouping if multi-packet, and checksum. These packets are processed by MIPL. The telemetry is processed and values from the ekernel are merged into the data to fill in missing data using m98telemproc. The image files are put into the data base by MIPL and are pulled over to MVACS via fei. Fei dumps the files into a directory that is watched by FwDMS, written by Stubbe Shviid. FwDMS is a sorting program written in C++. It has a configuration file that tells it where to look for files and for each outgoing queue, what kind of images to send out. * fwDMS_raw The first instance of FwDMS reads data from the raw_in queue or from the fei directory in Ian's directory structure (/proj/ops/mist??). These files are "raw" vicar images straight from MIPL. They have not been processed by any programs yet. They are considered "raw" because they have not been calibrated yet. Each image is physically stored under the data/store directory tree. Then the file is parsed and a whole bunch of symbolic likes are created under the data/raw tree so that images are available by sol, filter number, and module number. This is known as the raw archive. There are a whole bunch of outgoing queues from this fwDMS_raw. These queues are called "pipes" by Stubbe's configuration file. toNtDB goes to Stubbe's database program running on Windows NT using Interbase. This database is query-able via cgi? The rawToWorld script is used to feed scp scripts to Germany. The most important pipe is toCal. These images are fed into Bob Reid's calibration routines. There is an "npipe" which just sends new images and does not repeat an image if it is send through again. This is used for the PIO queue. This data is for the "realtime" update of the web pages for outside people. This queue is processed by pioToWeb.bash. * Calibration - StartCal.pro The calibration code is written by Bob Reid at the Lunar and Planetary Lab at the UofA. For the exact details of how calibration is done, see Bob's documentation. The software is all written in IDL and is based on ImpSoft and ???. It removes dark current, handles bad pixels, and a bunch of other stuff. Calibration has always been done using the FM (Flight Model) even though the data has been taken on the EM (engineering model). This was done to make sure we did not have the wrong model at landing. Instead, Bob has disabled a number of calibrations that make things really bad. Once calibration is done, the routines dump the corrected images into the ical_in queue for the next FwDMS to pick up. * fwDMS_ical ical stands for instrument calibration. fwDMS_ical has the job of directing the initial flow of the images once they have been calibrated by Bob Reid's calibration code. fwDMS first puts the actual data files in data/store/ical. Then it populates the links in data/ical based on module, sol, and filter. Then it passes the images off to two queues. icalToWorld is for images to go to Germany via a script of Stubbe's responsibility. The second queue is toJames. This is the hand off to the Master Control Program, which is Kurt's responsibility. * MCP - Master Control Program This script is named after TRON's Master Control Program since it holds a lot of power, assimilates lots of other programs, but it really pretty dumb. Its job is to watch for complete stereo pairs in queues/toJames. When there is a stereo pair, it runs stereopair2wits with the two images. * stereopair2wits stereopair2wits has the job of converting stereo pairs into numerous products. The program takes in two vicar files that are a matched stereo pair and the main products are wedges to be used by the RA team with WITS. In order to do this, we have to traverse a long and arduous process that is much less painful than when we started with this script way back when. The process starts with the Ames "stereo" program (see Eric Zbinden) which produces horizontal and vertical disparity files (.hdisp and .vdisp) along with a mask (eMask) file that records how the disparity for each pixel was generated. The hdisp and vdisp files are then passed to marsxyz program (see Justin Maki). marsxyz takes the camera model and combines it with the disparity for each pixel to generate an xyz point in 3D for each pixel. This is often refered to as a dotcloud. These points are stored in a 2D array vicar file, where each XY pixel becomes a 3D point instead. The xyz file from marsxyz is passed to vicarXYZtoVRML (see Eric Zbinden) which produces a VRML 1.0 file for use in Viz or other VRML browsers and plugins. There is an alternate path for VRML files that comes from Eric Zbinden which exclusively uses the Ames stereo program. The second path for xyz files is to produce data for the wits wedge. A wedge is a database of information for a stereo pair of images that contains both 2D and 3D information. There are two java programs maintained by Kam Tso, but both are stored in the same Java archive file (wits.jar). The first is the Index Triangle Strip Array (itsa) file generator. The second is the data file generator. These files describe general characteristics of the stereopair. This produces left.data and right.data. A quick note on building of models. When building 3D models, they are usually mapped with respect to either the left or right eye of the camera system. This allows the texture from that eye to be mapped directly onto the 3D model without having to use disparity to map from an image to the space that is the reference point of the camera. Wits uses the left eye as the reference. This works pretty well for most things, but has one major problem. Only the right eye has filters in the filter wheel that cover red, green, and blue. Since color makes a big difference for the quality/feel of the terrain, the Ames Stereo Pipeline was configured to used the right eye. However, the Ames Stereo Pipeline can easily be switch to either the left or right eye in the stereo.default file. The next step, buildImages, is to generate a whole bunch of image files for various sources. Also here is m98cam (see Justin Maki). This program extracts the cahvor camera model information to produce left.cahvor and right.cahvor. m98cam is in buildImages only for historical reasons. Two special products are generated for Wits. One is the left-masked-hi.jpg. This file is a colored representation of the quality of the range data. The left black and white image is merge with an eMask file. Good correlation data comes through as normal grey. No range data areas are marked with cyan and areas of interpolated range data are colored with yellow. The regular texture is visible through these colors. The color is produced by suppressing the other color channels. The second product is a rough cut red-blue anaglyph. Both of these products are produced with vicar2jpg (see Kurt Schwehr). Then the standard Wits image products are produced. These are standard left and right black and white jpgs from vicar2jpg. Wedges that fail are left as tmp.s*_*.timestamp. Wedges usually fail because of bad correlation in the stereo phase. This frequently happens because of poor quality images. Once the wedge is successfully created, it is renamed to not have the tmp. at the beginning of the directory name. Once stereopair2wits returns for whatever reason, MCP removes the images from the toJames queue. * pioToWeb pioToWeb is in charge of creating the jpg images for live updates on the web and for the tiff & jpg images in data/vap/images. Bob Marcialis is also producing images in data/vap/images, but Bob Marcialis's should be better quality since they are produced from Bob Reid's calibrated images. These other images with be generated by hand in IDL. For the live web updates, two products are created: a jpeg (.jpg) and a text (.txt) file. The text file is a dump of some of the vicar label entries. These entries are used by Louise for image annotations and by Dave Page for his mosaic program. Once the images and text files are created, the "hose" program notifies the web server scripts (see Louise Falevsky and Anil Venkata) via a socket connect that is explicitly allowed through the firewall. FIX: See what document to explain how the web stuff was setup? The live web pages are done with perl and apache. Describe what is on the web page for each image, how they are layed out left and right with the cool spectral band bar taken from Stubbe's web page. FIX: See what for Dave Paige's mosaic program? This program is written with Fortran and Image Magick. **************************************** * GENERAL ENVIRONMENT VARIABLES **************************************** FIX: Write about environment variables across the programs. SSI_ SSI_VERBOSITY - This environment variable is used to control the debugging output of each of the subprograms. Usually this data is redirected to log files if fwMaster.bash is used. The default is 4 which is equivalent to the TRACE level. To set the programs to be very talkative while running, do something like this: setenv SSI_VERBOSITY 20 SSI_TOP_DIR **************************************** * DESCRIPTION OF SCRIPTS AND PROGRAMS **************************************** FIX: where do I talk about access permissions. Damn... it would be so much better to have real access control lists (like with afs) so that I could give myself only insert permission on some directories and other people only delete and etc etc. fw == Fairwood == Stubbe Shviid * Runtime control for scripts Since this script needs to be controlled by a team of people working across multiple shifts, most of the long running programs listed to run time control (.rtc) messages that come in the form of 0 length files in the incoming queue directories. An example for the JAMES stereo pipeline ( MCP.bash and stereopair2wits.bash ): cd /proj/ops/ssi/landed/queues/toJames touch pause.rtc # Should suspend and wait for resume.rtc touch resume.rtc # Back to normal mode touch stop.rtc # MCP should exit. * MCP.bash The Master Control Program is a play off of the movie TRON. This program tries to control the stereopair2wits program, but is pretty dumb. Its main goal is to watch a queue directory for stereo pairs and call stereopair2wits. It works in a simple loop. It checks a the queue directory for images. If there are images, it processes all left-right image pairs that it finds. This way it will ignore just a left or just a right image. It then checks to see the elapse time since it started. If it was less time elapsed compared to the update time, it will sleep for the rest of the left over update time. If the update time has expired, it will loop immediately to check the input queue directory. There are two ways to run MCP. The normal way is to run it in continuous mode. You can optionally specify the the minimum time between updates. If the update time is not specified, it currently defaults to 30 seconds. There are shorter versions of each command. Examples: MCP.bash uc MCP.bash update-continuous MCP.bash uc 15 MCP.bash update-continuous 15 The other way to run the program can be run is to just do one update. It will check the directory, process any stereo pairs that it finds and then exit. Examples: MCP.bash u MCP.bash update To control the execution, there a bunch of environment variable that change the parameters. The main one is SSI_TOP_DIR. Generally, this is the only variable used. It sets up top level structure of the sub tree used for this system. If this is the only variable set, it is used as the root for all the other variables. During testing and development, MCP was used standalone without fwDMS and company to just process Wits wedges and VRML models. So general usage for of MCP as a stand alone program is like this: setenv SSI_TOP_DIR /proj/ops/ssi/landed MCP.bash uc 15 To go through the rest of the variables, they are less important, but still useful. For example, the script files can be found elsewhere in the filesystem if the script directory is set. SCRIPT_DIR is the location of all of the bash scripts. The main thing used from SCRIPT_DIR is stereopair2wits.bash. BIN_DIR is the location of the compiled, machine dependent programs. The main program used here is the GNU version of date. Normal ATT & BSD date programs will not return a number of seconds, but gnu date will. SSI_DATA_TOP_DIR is the location of the data tree. This is where stereopair2wits puts its output data products. SSI_STATUS_DIR is the location of where fwStatus.bash looks for the current status of each program. MCP writes the current status using a simple SetStatus function. SSI_STEREO_QUEUE_DIR is the directory that is checked for incoming stereo pair images. There are a few general notes about MCP that are important. The first is that it looks in the queue directory for images of a certain pattern. It will ignore other image files. Images must have the form of "s*_*[lr]" with the extension of .img or .cal. Also, it will ignore images until it finds a left/right pair. Like the other main scripts, this one also listens to run time control (.rtc) messages placed in the incoming queue directory. There are three messages: pause.rtc, resume.rtc, stop.rtc. To put a hold on the script, use pause.rtc. Its pair, resume.rtc, will get the program running again. stop.rtc will kill the process as soon as possible. * fwMaster.bash fwMaster.bash was created by Stubbe to start up the whole pipeline. It creates the directory structure for a mission phase, fills in a number of configuration files, and starts up each of the programs that make up the image pipeline. There are a number of programs that it does not start. The most important is fei. It also does not start any of the visualization programs, including xv, Viz, and WITS. fwMaster takes between 1 and 3 arguments. The normal operation is to just specify the mission phase, e.g. fwMaster.bash landed This phase is created under the base directory, which defaults to /proj/ops/ssi. It is possible to move this base directory to anywhere else by specifying two arguments when fwMaster is started, e.g. fwMaster.bash kurtTest03 /u1/schwehr/testArea The last argument is the data source for incoming files. This is partially obsolete. We changed things late in the game to just run fei in the raw_in directory. But originally, the last argument was specified as mist, stl, or mspl. mspl is what the spacecraft landed on Mars is supposed to be called. Now, I believe the phase is used as the location to write Ian's tree of files with date stamp tracking of each file. Ian wanted an audit trail at UCLA for each image. So if MIPL re-released the image with some header change, then there would be a new copy of this image with a date stamp appended to the filename. I don't remember the status of this. FIX: look at the latest code and figure out what this actually was. *** Walk through of fwMaster *** Setting the umask to 002, so that all files and directories are group writable. This script was supposed to be run only on one machine so that: 1. everyone would know where it was running to monitor the processes 2. Not all the GDS computers were correctly configured to run the pipeline. 3. The pipeline can load the machine down if it gets a large number of images all at the same time. The createVerifyDir function exists so have directory creation have error checking and to make sure directories are group sys.ssi. Argument checking needs lots of improvement. The first major task is to create a whole batch of directories. The main categories of directories are programs (bin), queues and data. It also puts in a couple pre- written readme files into the directory structure. If the bin directory needs to be created, the program does a recursive copy of the binary tree from /proj/ops/ssi/bin. The goal is to have a snapshot from each test of working binaries. We tended to stay away from putting binaries in CVS. It can be done, but it takes a lot of space and can sometimes bite back. The script then clears out all of the run time control (rtc) files from the queue directories. This puts all programs into a known state. Otherwise a program might startup and find an old control file. NOTE: RTC is a frequently used acronym. Other uses are: Real Time Controller Real Time Computer Real Time Communications Oh well... If the user does not set SSI_SOL_UNKNOWN to a value, the script picks one. We started with just "sol_unknown", but during the tests, we never received a valid planetary day number. This meant that all images got dumped into one sol_unknown area. This really sucked. So now, the unknown directory has the date appended, so that the sol_unknowns are at least separated by the day that the script started. We do not have on the fly changing of sol_unknown. The script must be restarted with a different SSI_SOL_UNKNOWN. The Java binary was a special case. We never had a proper install of Java 1.2, so we used a link to Kam Tso's copy of Java 1.2 in his home directory. This was not optimal, but did work. The link was important because Java will follow the link and figure out where all of its associated files are. If it is a copy of the script, it will be unable to find itself. Next, the script creates two configuration files that are very similar. They control the running of fwDMS, the Database Management System. These entries are where all of the main queue are specified for raw and calibrated images. After the environment is setup, each of the programs is run. They can really be started in any order, but we tried to lay them out in the general order that data passes through them. ctmIFPD is really obsolete. It used to be used to pull data from the fei incoming directories in /proj/ops/{mist,stl,mspl} and import the image files into the raw_in queue. This functionality was taken over by fwDMS_raw. Two instances of fwDMS are started. The first is fwDMS_raw which handles raw images. The second if fwDMS_cal which handles images that have passed through the calibration software. StartCal is called to begin running the IDL based configuration software. fwADSC_MSOC is also an obsolete program. Its job is to export data to the web mirror that lives in Europe, run by Stubbe. Lastly, to similar scripts are started. pioToWeb handles the images that go to the live official web page, www.marspolarlander.com. MCP watches for stereo pairs and starts the JAMES pipeline when images arrive. * fwStartCal.bash fwStartCal is a pretty simple script to start up Bob Reid's calibration code. The script creates an IDL routine that sets the path, changes to the toCal directory and runs the IDL function cal_queue. * fwStatus.bash fwStatus is a simple loop that watches the contents of the bin/status directory. It runs cat on each of the files that end in .status. Each program is responsible for writing a one line description of its status to a file in this directory. To go along with this, it also does a directory listing of the status files. This is important to keep track of the last update time of the status file. At this time none of these programs catches things like Ctrl-C and many other signals. When programs get killed some way other than a stop.rtc message, the status file will just sit there being updated. When a program dies strangely, you will see that its status file has an older time that others. Programs so periodically touch their status files, even when idle. * fwStop.bash fwStop is the intelligent way to stop all of the programs as fast as possible. It is not as fast as a killall, but the programs get to cleanly shutdown. The program writes a stop.rtc file into every queue directory that it can find. * pioToWeb.bash pioToWeb is a cheap knock off of MCP.bash. MCP.bash was copied and modified to handle the web images. It has then grown from there to include a number of similar features, such as creating jpg and tiff images for the local team. As such, it has a maintenance skew problem with MCP.bash. Some fixes to one have not been applied to the other. NOTE: see MCP.bash for more information about the general characteristic of this script. pioToWeb takes raw, uncalibrated images as its input, but can handle calibrated images. The primary product of pioToWeb is, for each vicar image, a jpeg image and a matching text file. The text file contains labels from the vicar header. These are placed into the toWeb queue directory. To go along with toWeb, the products are also written to the webToWorld queue for one of Stubbe's scripts to use for Stubbe's web pages. The second product of these web pages is to fill in the vap/images/{jpg,tif} directories with jpeg and tiff images respectively. This came out of Bob Marcialis generating directories full of different image formats. All images created by this script are names ames-*.{jpg,tif} so that they can be differentiated from Bob M.'s images. It was assumed that the "ames" images were a rapid first cut image and that Bob M would use the calibrated image and work on getting the best possible images and have additional image formats. Also, to simulate the functionality of Jedi, the script has a link in data/vap/images/jpg/ames-latest.jpg that is always set to point to the latest image. It is up to people to view this latest image however they wish. An easy way to watch this file is with xv like this: xv -poll -nod -maxp /proj/ops/ssi/landed/data/vap/images/jpg/ames-latest.jpg There are a couple different settings that affect the image quality. For most conversions with vicar2jpg and vicar2tiff, the -i option is used to ignore a top number of pixels when doing an auto scale of pixel values. This is to allow for uncorrected bad pixels that come out at the maximum value. Also, for jpeg images, the non-web images have the "--quality 99" option to make the images as lossless as possible. Web images currently default to a jpg quality of 75%. Originally the text file to go with each image was only meant to have a couple entries. This got really out of hand as people asked for more and more entries. As a result, the command line is pretty large and needs to be called more than once to get them all in. FIX: anaglyphs. * sgiSay.bash This program is basically a modified MCP used to run "say" on an SGI. The program was a 15 minute hack one night for fun. None of the programs actually output stuff to the say queue, but it was quite amusing. To have text spoken, put a file containing the text into the toSay queue. This does have the problem that large files will tie up the system for a long time. Also, where is the official origin of this say program? * stereopair2wits.bash -- JPL/Ames (JAMES) Pipeline. stereopair2wits.bash is the script that is the JAMES pipeline. It is the most complicated of all the scripts in that it calls a large number of programs to do processing. At this point, the script could use some work to clean up the source code and the temporary files that it creates. It is derived from a large number of scripts and hand run sequences of commands. Many programs in the original have been phased out as the system is simplified. There is still quite a few things that can be done to further simplify and speed up the process. Command line arguments stereopair2wits normally takes two arguments: the left and right images of a stereo pair. It assumes this is a valid path to the image files. It optionally takes a third argument that is used as the output sub-directory for the data products that are created. If the third argument is not specified, the program tries to use the PLANET_DAY_NUMBER to set the sub directory to be sol###. If the value of PLANET_DAY_NUMBER is set to the pds null value (-9.99e+31), then it sets the output dir to be whatever SSI_SOL_UNKNOWN is set to. FIX: describe the wedge naming convention with the date stamp for wits but not for data/vap/{disparity,range,vrml}. A walk through the park - as the script flows: This script has one additional main environment variable that none of the other scripts have, JAVA_DIR. This is the location of the wits.jar file. It is a Java archive file provided by the WITS team that contains all of their routines. One of the programs in the script, marsxyz, is based on the JPL vicar library so it requires Fortran shared libraries to be in the library path (LD_LIBRARY_PATH). On the GDS suns, this is done by sourcing a file. The detection of this condition could be done much better! The solution is to either not link against Fortran or to have smarter detection of the Fortran libraries. Also, many SGIs don't have the Fortran libraries at all. The exit handler function should really remove the temporary directory and set a .status file and/or log file to record success/failure and file names. PLANET_DAY_NUMBER handling for the outputDir is described above. After checking the version of the operating system and BASH, the script checks to the filter number. It only does stereo on the red filter, #5. This is the only filter that we have calibration data for marsxyz. buildRangeMap() NOTE: All operations in buildRangeMap are in the MVACS coordinate frame and referenced to the left camera. This will make a difficult time with color images, since the RGB filter set is only on the right eye. This is why the Ames Stereo Pipeline can use either eye, but is normally referenced to the right eye. Next the script calls buildRangeMap. The function does the majority of the work processing a stereo pair and produces a high CPU load for a few seconds. First it runs the NASA Ames stereo program to produce hdisp, vdisp, and eMask files. All three are in VICAR format. hdisp and vdisp are the horizontal and vertical disparity files respectively. See http://img.arc.nasa.gov/PhotoVR for more info. The eMask file is a mask file of the quality of the correlation for each pixel disparity value. The most common values are for normal correlation, no correlation, and horizontal/vertical interpolation. stereo is controlled by the stereo default file. Next the hdisp and vdisp files are passed to marsxyz, with is maintained by Justin Maki. This file uses calibration data to convert vdisp and hdisp into a VICAR xyz dotcloud file. NOTE: The symbolic link from left-range-1.xyz to left-range-1.ran is a HACK for the WITS team. .ran files have been obsoleted for this project, but the WITS code makes it difficult to change the extension. .ran files are a dangerously simple binary format. The only header information is width and height of the data. That is a very bad idea. Next vicarXYZtoVRML is used to produce VRML files (.wrl) for use with Viz and other software that can read VRML files. These files follow the VRML 1.0 standard. Note that additional code is needed to convert the altitude texture to rgb format. The next stage of buildRangeMap is to run two routines from WITS. The first builds an Index Triangle Strip Array (itsa) file from the .xyz file. After that, the another Java routine reads the .xyz file to produce data files for the left and right images that wits needs to use to lay out the mosaic. These are done just once to save recalculating these parameters each time WITS loads a wedge. buildImage() The next function of stereopair2wits is buildImages. This section first extracts the CAHVOR camera model from the left and right vicar images. Then the images are converted to jpg format. The best way to convert to jpg has not been figured out. Currently, the images are auto scaled, but it was decided that the process would be tweaked for better results once images were acquired from Mars. In addition to left and right jpgs, several other jpgs are created that support the use of WITS. The first are simple red-blue anaglyphs for stereo with red/blue glasses. These anaglyphs are not as good as those that are tweaked, but it is a good starting point. Also, the eMask is merged with the left image to produce a grey scale image with color information overlayed to depict correlation quality. cleanup... While the script is working, it puts the in-progress wedge into a directory that has a "tmp." in front of the wedge name. This prevents people from loading partially built wedges into wits. When completed, the directory is renamed to its final form. Also, if the process fails, the "tmp." is left on the front of the directory name. This is done to document that stereopair2wits was attempted on a stereo pair but did not succeed. ------------------------------ Binaries from PhotoVR / stereo ------------------------------ * disp2pgm * doLOD * mosaic * stereo * vicGetLabelEntries * vicar2jpg * vicar2rgb * vicar2tiff * vicarFileInfo vicarFileInfo is an older program that dumps all of the labels, one per line. Has not been maintained much. You can also use "vicGetLabelEntries --all". * vicarHist vicarHist is a rather dumb program which produces an image of a histogram of dn/pixel values in an image. * vicarMerge See Bob Kanefsky (kanef) about this. Probably should not be used again. Takes the label from one vicar file and dumps them onto another file. Note from Kanef: Yes, it was a kludge Eric asked me to write when he didn't have time to integrate label copying into his code. There's almost nothing to the vicarMerge program itself, it was the label-output routines that took a couple of days. * vicarXYZtoVRML See Eric Zbinden. * warp Eh? ---------------------------------------- xv xv 3.10a has a bug with 16 bit (HALF) images. It frees a lookup table and then proceeds to use it. xvtech@devo.dccs.upenn.edu Here is the patch I tried to submit to the xv bug list: xv-3.10a> diff xvpds.c~ xvpds.c 861c861 < unsigned short *pShort; --- > unsigned short *pShort, *toBeFreed; 962c962,963 < free(pShort = (unsigned short *)pinfo->pic); --- > /*free(pShort = (unsigned short *)pinfo->pic);*/ > toBeFreed = pShort = (unsigned short *)pinfo->pic; 966a968 > free (toBeFreed); ---------------------------------------- WITS Where to get info on WITS? http://robotics.jpl.nasa.gov/tasks/wits/homepage.html FIX: there was another off site WITS page... what was it? ---------------------------------------- VIZ See Laurent Nguyen, NASA Ames http://img.arc.nasa.gov/~nguyen/viz/doc/html/ **************************************** * General notes on coding styles, tricks, and hacks. **************************************** **************************************** * SOFTWARE SOURCES **************************************** Where the heck did we find this random software from to do all this crazy stuff? flex bison jpg tiff **************************************** * SUGGESTIONS FOR FUTURE WORK, IMPROVEMENTS, and REWRITES. **************************************** **************************************** * ACRONYMS & DEFINITIONS **************************************** NOTE: Best to read the Mars Polar Lander EDR-SIS document along with this. The EDR-SIS is the software specification document that MIPL codes to for their software. Many of the terms in this document are defined there (http://rushmore.jpl.nasa.gov/mars98) CAHVOR stands for "CENTER,AXIS,HORIZONTAL,VERTICAL,OPTICAL,RADIAL" It's probably not worth describing the details of the model here, but a good reference is a paper by Yakimovsky and Cunningham, (Computer Graphics and Image Processing 7, pages 195-210, 1978), A less-technical overview can be found on the web at: http://rushmore.jpl.nasa.gov/~jnm/talks/camera_models/ APID - Application Packet Identifier CCSDS - www.ccsds.org - format for packets sent from deep space vehicles. DMD - Data Monitoring and Display. A tool provided by JPL's Multi-Mission Ground Data System (MGDS) for displaying channelized telemetry as numbers or named states (ON, OFF), usually in columns that change a few times per minute, in a large number of different windows. If you see someone running it, you can recognize it by its white text on a blue background, with changed items in yellow and a few items highlighted in red. DPT - Downlink Priority Table - This sets the priority of downlink data. The APID's are assigned priority. So, does SSI image data come down before or after data from the RAC camera. Each APID is assigned a priority. This makes it possible to control whether currently queued engineering data is downlinked before or after science data, whether SSI images come down before or after RAC images, and so forth. For example, on Pathfinder, the Insurance Pan was originally given a lower priority than the Super Pan, but after the Insurance Pan was taken and it was clear the Super Pan would block it for longer than the battery might last, the science team decided to change it by temporarily assigning its APID to have a higher priority. ekernel (ek) - An ekernel is generated from a pef file. If contains a subset of information and is in a binary format. It is used with the ccsds packet EDR - Experiment/Engineering Data Record - VICAR files. FIX: is this correct? Comes from the telemetry processor, m98telemproc. The definition of the EDR can be found at http://rushmore.jpl.nasa.gov/mars98/ Software Interface Specification (SIS) Documents EDRSIS EDR arealmost-raw data from an instrument, as opposed to a data product like a mosaic or graph. For a camera, an EDR is an individual monochrome image, and MIPL tends to use Vicar format for them. But data products that happen to be in Vicar format are not EDRs. eMask - Why is it an "e" mask? A simple pgm style??? image from stereo that contains the type of disparity quality for each pixel. There about 5 types, but only three are shown from vicar2jpg. FEI - file exchange interface - A program by MIPL in the Vicar package. It is a secure way to fetch files from the mipl data base. It uses kerberos for security. As files are added to the data base in a channel that is subscribed to, the data automatically transfered to the local directory. Contact Pam Wonzik GDS - Ground Data System. These are the computers at MVACS maintained by Ian McEwan, Drav, Steve, and Hyun. ical - Instrument calibration - files that have been calibrated for an instrument. IDL - www.rsi.com - Image processing package that has been around for 20+ years. Very obvious that it has Fortran origins. IRIX - www.sgi.com - Unix operating system by Silicon Graphics. JEDI - Java EDR Dislay Interface LPL - Lunar and Planetary Lab and the University of Arizona. http://lpl.arizona.edu http://imp.lpl.arizona.edu MCP - Master Control Program - Simple/dumb program that watches for stereo pairs and calls stereopair2wits. MIPL - Multimission Image Processing Lab at JPL - http://rushmore.jpl.nasa.gov MSOP - Mars Surveyor Operations Program MSPL - Lockheed Martin's name for the flight lander that goes to Mars. ??? MVACS - Mars Volatile And Climate Surveyor - This was the instrument package on top of the lander. PDS - Planetary Data System An consortium that archives spacecraft data and data products. They have their own labeled format that differs from Vicar format. http://pds.jpl.nasa.gov pef - Predicted Event File - Generated from a number of sasf file. I think this generated by apgen. The APGEN documentation says in a couple of places that "a PEF file is generated", but its glossary defines PEF as "Predicted Events File (output from SEQ_GEN)". It also has an entry defining "TOL" as "Time-Ordered Listing (output from APGEN)". A PEF predicts what will happen when, by expanding any macros and variables in SASF files into a single timeline, and adding non-spacecraft events like DSN windows. PIO - Public Information and Outreach - or whatever stuff for normal folks is supposed to be called. RAC - Robotic Arm Camera - Camera attached near the end of the robotic arm that can view the scoop. SASF -- Spacecraft Activity Sequence File. A text file containing commands that will be sent to the spacecraft or testbed. This is used as input to SEQ_GEN, and may also be read by sequence analysis or visualization programs. It may be generated with a text editor or a sequence generation program. SCLK -- Spacecraft Clock time. Number of seconds elapsed since the clock was initialized sometime before launch, as measured on the spacecraft. SCET -- SCLK expressed as a date and time, usually UT. OWLT -- One-Way Light Time -- time for a signal to get from the spacecraft to Earth. ERT -- Earth Received Time -- time a packet "hits the ground" at the DSN, usually expressed as a UT. This is equal to SCET + OWLT (where SCET is the time the packet is transmitted) , although if the clocks disagree this probably also accounts for that. In some contexts, where "SCET" means "the time an event happened, e.g. an image being acquired" and ERT means "the time the data was received", the delay could be much longer than in the strict meaning of ERT. UT -- Universal Time (timezone 0) also known as Greenwich Mean Time. Pacific Standard Time is 8 hours earlier, and Pacific Daylight Time is 7 hours earlier. UTC -- Strictly speaking, this stands for Universal Coordinated Time, adjusted for leap seconds. In practice, it usually just means UT, because people expect a timezone to have three letters. SOE -- Sequence of Events file. Similar to a PEF, and used interchangably on MVACS. sol - One Martian day. Aprox 24 hrs and 35 minutes long. Sol 0 is defined as the day we land on Mars. the length of a Martian (sidereal) day is: 24 hours, 37 minutes and 22.663 seconds. Solaris - aka SunOS. This is the operating system from Sun Microsystems. uname returns "SunOS" on these machines. SSI - Surface Stereo Imager. University of Arizona LPL stereo camera. See the EDR-SIS documents from Mars Pathfinder IMP camera and from MVACS for details of the camera. stereo.default - Configuration file for stereo and vicarXYZtoVRML programs. See Eric Zbinden. Contains a large number of parameters. SunOS TLA - Three Letter Acronymn UofA - University of Arizona, Tucson, AZ. Vicar - Video Image Communication and Retrieval http://www-mipl.jpl.nasa.gov/vicar/vic_file_fmt.html Viz VRML - Virtual Reality Markup Language - www.vrml.org. Derived from SGI's OpenInventor. VICAR - Video Image Communication and Retrieval a subroutine library, and a package of tools which has its own file format also known as VICAR that JPL uses for images during missions. Viz - Visualization architecture from the Autonomous and Robotics Area (ARA) / Intelligent Mechanisms Group (IMG) at NASA Ames Research Center. Point of contact is Laurent Nguyen. http://img.arc.nasa.gov/~nguyen/viz/doc/html wedge WITS - Web Interface for TeleScience - Paul Backes and Kam Tso **************************************** * LESSONS LEARNED **************************************** * During ATLO data collection with cameras, especially with stereo, it would be much better to have texture on as many surfaces as possible. For example, putting camouflage or posters the walls will let stereo algorithms correlate, where white walls will not. This will allow stereo algorithms and camera calibration to be better tested with the flight hardware before we land on another planet. * Would be best if programs that do not require Fortran are not linked against it. SunOS does not install Fortran libraries by default. When they are installed, they are normally in /usr/local/SUNspro/lib directory??? Perhaps an -rpath option is in order for the building of such programs. The main program linked against libF77 is marsxyz. * Would be nice to take calibration data for more than just one filter for stereo processing. * Need to have a valid PLANET_DAY_NUMBER set for ORTs and MORTs before it comes through fei, even if the spacecraft clock is before landing. I am not sure what the SCLK should have been set to, but how ever it is done, when we are on sol1 of a ORT, then the planet_day_number for those images should be 1. Should the SCLK match the spacecraft or should it match what the spacecraft will have after landing on the respective Sol? I am not shure which is better, but my hunch is that it should be the same as the current spacecraft SCLK with programs having a different definition of time of landing per ORT. I do not know how it would impact the other software on the project. * Needed better configuration control on the whole GDS system, including computers, networking, and software. The GDS was far from being done when we got to landing. * Tracking images by downlink does not really make sense. There is no way to detect a downlink. Can packets for one image come across different downlinks? * Need to have stable and fault tolerant networking. It would seem to be a good idea to have a backup router and net connection working and tested early on. Need to have more than just the TDS with a backup available. Many teams need continual access to their home institutions. Maybe the web team needs more than one subnet and/or their own primary access to the outside world that is separate. * Doc, doc, doc... the GDS faq was ridiculously pathetic. A real FAQ would have been a big help to many. * The MIPL web page for mars98 was a good idea. Yeah!!! * Mailing list archives for all the mailing lists would be good. Not just the 4 that were archived and on the secure web site. * Backups of the network drives is really important. Did not happen for several weeks close to landing. * Kanef might want to do a "Gap Report" listing in text what was not received at the packet level. * Person to person communication needs to be better. Between the ends of the building, the wireless phones we not great. * BASH is way better than csh for shell scripts. Perl or Python would probably be a lot better than bash. How would pdksh do? Scripts need to be written in a maintainable way with documentation. Hacked together scripts are the ingredients for a big mess. * For the ORT/MORT test bed setup, would be better to put out a real geology scene instead of plastic things. Jim's trench was excellent. It was a good simulation. How about doing a geologic setting per test. E.G. metamorphic assemblage for one. Next would be a volcanic environment cross cutting sedimentary. Things from intro to geology... Let people do "real" analysis to recover a realistic story. How does the story match what the designer thought? Test the science teams analysis. Should coordinate between the sand box and the TEGA test bed. Do a similar sample. * Would have been really good to have meeting notes for each of the large project meetings that are available so that people can catch up. * Should we be going to PDS format right away when possible, since that is our end goal? Or will this cause more trouble than it is worth? * Not having RA data downlinked from the spacecraft was crazy. **************************************** * Performance review **************************************** How fast is this thing? **************************************** * FILE EXTENSIONS USED **************************************** .ccsds -- .bmp -- MS Windows bitmap images. .cal -- .gif -- Compuserve Graphics Interchange Format .jpg -- http://www.jpeg.org/public/jpeghomepage.htm .img -- http://www-mipl.jpl.nasa.gov/vicar/vic_file_fmt.html .pict .ps .rgb .tif -- Tiff format. http://www.libtiff.org/ .vic -- http://www-mipl.jpl.nasa.gov/vicar/vic_file_fmt.html **************************************** * OPEN SOURCE SOFTWARE - USED FOR WHAT AND BY WHO **************************************** NOTE: There were lots more things that are open source type software that we used, but that was required but not directly worked with. There is no way I could list all those out. autoconf - http://www.gnu.org automake - http://www.gnu.org Used these automake/autoconf to create a new build system for netpipes so that it was easier to install. Okay, so it took me an hour to learn these two programs and would have saved me time in the short run, but I wanted to learn these two handy programs. bash - http://www.gnu.org All of the major scripts for the imaging pipeline on MVACS were written in bash. This was a compromise between perl and csh. csh would have killed me and perl was a little scary to my management :) Many recommended pkdsh instead of bash, but I have no experience with it. bison - http://www.gnu.org flex - http://www.gnu.org Parse me babe! I use these two to parse XDR message definitions in Viz and just flex to tokenize vicar format labels. Seems to scare off many of the other engineers for some reason. May be switching to Antlr if I can figure it out since I get c++ and java at the same time. But, of couse Antlr seg faulted java version "1.2.2" Classic VM (build Linux_JDK_RC3, native threads, sunwjit) java 1.2.2: emacs - http://www.gnu.org Okay... I couldn't code or write as easily as I do without emacs. gcc 2.95.? - http://www.gnu.org The compiler that worked for SunOS 5.7 gimp 1.0.4 sgilib.c -- http://www.gimp.org Revision 1.6.2.1 19998/060/06 by Yosh. Grabbed the SGI iris rgb format reader/writer and used it in libvicar for linux. Seems to have some strange problmes at the end of images on IRIX/Solaris, but worked well on x86 Linux redhat 6.1. lesstif - What the heck did I need to keep grabbing lesstif for? I can't remember. But, it kept coming up that we needed motif for stuff. libjpeg - ftp://ftp.uu.net/graphics/jpeg/jpegsrc.v6b.tar.gz Linux - www.linux.org - www.kernel.org Used Redhat 6.1 for my laptop. Offline development. Attempting to straighten out byte order problems in libvicar, but did not have time. We also have Viz running on Redhat 5.2. kerberos - somewhere at MIT. make - http://www.gnu.org netpipes - contains the hose and faucet programs libtiff - perl - http://www.perl.org python - http://www.python.org This is the main scripting/prototyping language that we use for the Viz clients. Great for getting things together quickly. We use tkinter right now for GUIs. sgiImage - by Paul Haeberli http://reality.sgi.com/grafica/sgiimage.html This one is really not a happening item. It has byte order problems and was really a mess. It limps along on Sparc Solaris and SGI IRIX, but died with byte order problems on x86 Linux. Did not have time to straighten it out. xmms - www.xmms.org Kept my ears happy while I spent way to many hours getting this stuff to work. **************************************** * People Listing for those involved with the pipeline. **************************************** These are the people that I had to coordinate with for the whole tamale. They are probably referenced in the above text. NOTE: Emails, phone #s and addresses removed for privacy... DEFINITELY INCOMPLETE breid, falevsky jdweinb, lemmon, chriss Louise Marie Falevsky Stubbe (Fairwood) Hviid Justin Maki Bob Marcialis Bob Reid Kurt Schwehr Kam Tso Anil Madhavapeddy Venkata Jonathon Weinberg Pam Wonzik Eric Zbinden