DRAFT DRAFT DRAFT Kurt Schwehr Copyright (C) 2003 kdschwehr _at_ ucsd dot edu Viz Lecture Series Lecture 6 Building La Jolla $Revision: 831 $ CONTENTS DEMOS OF THE DAY INTRODUCTION CHIRP SEISMIC DATA: READING TAPES VERIFYING DATA INTEGRITY CONVERTING FROM XSTAR (GNU Make) CONVERTING FROM XSTAR (BASH) PREPARING CHIRP LINES FOR 3D
* DEMOS OF THE DAY
* INTRODUCTION For this lecture, I will attempt to talk you through the entire process of making a demo for a study area. We will use the area between La Jolla on the southern end and New Port, CA on the northern end. Since much of the data has yet to be published, we will only release samples of each data file for now. But there will be enough raw data that you can try each of the steps yourself and then apply it to some other field area.
* CHIRP SEISMICS For this field area, we have a large collection of chirp seismic data that were collected by Driscoll et al. using Driscoll's EdgeTech chirp. This chirp is modified from the standard design in that it uses an ADSL digital link that runs over the 400V tether power lines. Many other chirp systems generate the outgoing waveform topside and send it in analog form down the tether. This particular design is not on the web pages below. For more info: www.edgetech.com www.edgetech.com/subbotto.htm
* READING TAPES The data also sits at /net/archive/mcs/2000/Eel_River/ After a cruise, the first step to handling chirp seismic data is to get it off of the 2GB tapes. Each tape contains a number of files. Each file contains one line of data. We use the unix "dd" command to pull the files off of the tape. It goes something like this: dd if=/dev/rmt/tps4d3nrnsv ibs=36384 of=xstar_lajolla224.sgy The data on the tape in written in segy format which is described here: http://seg.org/publications/tech-stand/ dd if=/dev/rmt/tps4d3nrnsv ibs=36384 of=outputfile.sgy 8 at a time mt -t /dev/rmt/tps4d3nrnsv rewind For more info: man dd
* VERIFYING DATA INTEGRITY Occasionally, you may have to transfer large amounts of data from one computer to another. With the huge data volumes associated with chirp, is often possible to have a problem in copying that doesn't get noticed. The most common is truncation, but it is also possible to have some other type of data write over the original file. To catch this kind of problem, we can use program that calculate a checksum hash. A good one to use is MD5. It is very hard to corrupt a file and have the file still have the same md5 hash. The libmd package is used to create a md5sum or md5 binary. So a quick command to calculate sums: md5sum *.xstar > md5sum.txt Then to check the files that are listed in md5sum.txt just run this: md5sum -c md5sum.txt For more info: http://www.penguin.cz/~mhi/libmd/
* CONVERTING FROM XSTAR (GNU Make) The data format written by the X-Star Edge is slightly different than the format we would like to have it in. We can use sioseis to read in the xstar format and write it back out as standard segy format. We will use sioseis for the conversion. You can do the processing in sioseis, su, or some other program from this point, but I will go with su after the initial conversion. The sioseis commands are wrapped in a bash script (convxstar.bash): procs diskin xstar header diskoa end diskin format edgetech ipath ${infile} end end xstar end end filter ftype 1 pass 500 dbdrop 1000 end end header fno 0 lno 999999 ftr 0 ltr 99999 l2 = l1 end end diskoa ofmt 1 opath ${outfile} end end The script starts by specifying the input raw data with the diskin command with the format as "edgetech". Depending on the hardware architecture, you may need to add the "swapped" flag to handle endian conversion for differing byte-orders. The xstar command changes the EdgeTech chirp format into a more standard SEG-Y format. For filtering, this specifies a type "1" filter which is a lowpass with a cutoff off 500 Hz. (FIX: is this right?) Errr... FIX. Don't really know this stuff. Now that we have this little script that can handle the conversion of one edgetech xstar file, we can build a script to convert them all. In previous lectures, I have used simple bash scripts with for loops that do all of the work. This time, I will use a GNU style makefile to automate the process (Makefile): 1 XSTARDIR := xstar SEGYDIR := segy 2 XSTAR := $(wildcard xstar/*.xstar) XSTAR_SHORT := $(notdir $(XSTAR)) SEGY_SHORT := $(XSTAR_SHORT:.xstar=.sgy) SEGY := $(SEGY_SHORT:%.sgy=$(SEGYDIR)/%.sgy) 3 VPATH := ${XSTARDIR} $(SEGYDIR)/%.sgy: %.xstar ./convxstar.bash $< $@ 4 targets: ${SEGY} For this makefile, we first define two directories (1). The original raw data will be in "xstar" and the results of the conversion go in "segy". We then want to define XSTAR and SEGY variables that contain lists of files in each format (2). Here I use three GNU Make tricks. The first is the wildcard operator. This puts all files that match "xstar/*.xstar" in the XSTAR variable. We then create the same list, but without the directory "xstar" in front of each filename using the notdir command. From the XSTAR_SHORT variable, we switch the .xstar suffix to a .sgy suffix. Finally we set SEGY to the files name plus the SEGYDIR prefix. Now that we have the file list, we need to tell make how to convert a xstar to a sgy. We do this with a rule in the third group (3). The percents (%) are matching wildcards. This rule runs convxstar.bash with $< being the input file on the right of the depency and $@ being the output which is on the left of the dependency. I have used VPATH here to allow the xstar files to be in a subdirectory and automatically found. WARNING: You must have an actual tab character before the ./convxstar.bash. If you end up with spaces, make will complain about a "missing separator" on that line. To wrap it up, we have a "targets" rule which tells make what our end goal is (4). This tell make that we want to create SEGY files. If no target is specified on the command line, make tries to build the first one that it finds in the makefile. Watch out that you do not stick some other target in front, unless you want it to be the new default. Now give it a try: make segy/lj101.sgy Which will run: ./convxstar.bash xstar/lj101.xstar segy/lj101.sgy You could also just type make and it will run the convxstar.bash for each file in the xstar directory. This was a very advanced example of GNU Make and it takes quite a while to learn make. Don't be worried if you are very confused by it! To help you see a bit more about what is going on, there is an another target that prints out some of the variables. For more info: http://sioseis.ucsd.edu/ http://www.gnu.org/manual/bash-2.05a/bashref.html - BASH manual http://www.gnu.org/manual/make-3.80/make.html - GNU Make manual in emacs: M-x info C-s make
* CONVERTING FROM XSTAR (BASH) The above makefile is confusing for those who are not heavily into make. So here is a small bash script that does the same basic job. It loops through the list of files in the xstar subdirectory and runs the convxstar.bash script on each file.
* CONVERT TO SU FORMAT We now want to convert to an Seismic Unix (SU) format and extract some key data from within the su format file. For more info: http://www.cwp.mines.edu/cwpcodes/
*PREPARING CHIRP LINES FOR 3D To import the seismic lines into our 3D models, we have two options. The first is to just bring the image in as is with lots of white space above the sea floor. This works, but the extra white space can really get in the way of being able to see the relationship between the sea floor and the chirp or even chirp line to chirp line. The second way, what we normally want, is to bring in the dataset with the white space above the sea floor made transparent. This can be done with Adobe Photoshop or with GIMP. FIX: does anyone want to contribute a little text on how to do this with the GIMP? Photoshop: I will describe the process of doing this on MacOSX with Photoshop 7.0. I will try to add anything that I know will be different for MS-Windows, but I have not tested this on MS-Windows. First, it is a good idea to have a Photoshop alias on your desktop. This will let you drag and drop images onto the icon to load them in Photoshop. This is often a lot easier than doing File->Open. To do this, double click on your hard disk. Go to /Applications/Adobe Photoshop 7.0 and select Adobe Photoshop 7. Then select File->Make Alias from the top bar (or just press Apple-L). Then rename Adobe Photoshop 7 alias to just Photoshop. Finally drag it to a nice spot on your desktop. For MS-Windows, once you have photoshop running, you can drag images to the Photoshop entry on the toolbar (usually across the bottom). Hold it there until Photoshop pops up. Then release the image on Photoshop and images should load. You must make sure that your advanced tiff options are enabled to be able to export transparent tiffs. This is not obvious, so here is how to do it on the Mac version: Photoshop -> Preferences -> Saving Files Now look under File Compatibility and make sure that Enable adaved TIFF save options is checked. Now we are ready to begin editing a file. Go to the folder with your jpg (or what ever format you use) images from sioseis or su. Drag one line onto the Photoshop icon. Here I will use a composite of several lines called "lj150_162rev.jpg". Now save the image as a Photoshop psd as jpg does not support transparency. You can do this either by the menu File->Save As or the shortcut Apple-Shift-S. Under format, select Photoshop First we are going to get rid of everything outside the bounding box of the plot. These parts will mess up the location and depth of the plot. Use the Selection tool (top left) aka Rectangular Marquee Tool. You will want to be zoomed in to at least 50 or 100% of the image size. You can zoom with Apple = (the same key as the plus... you don't need to hit the shift key). Go to one corner of the box and start the selection. Then drag to the opposite corner so that you have selected the part of the image that you want to keep. Now crop the image: Image->Crop and save. Now select the Magnetic Lasso tool. It is 2nd down from the top on the left. It is in the same box as the Lasso Tool (a small loop with a line coming off). Use the tool to select a region of the image that you want transparent. Drag the cursor around the region. Click on the starting point to finish. Click the right mouse button inside the selected region and choose Select Inverse. Copy the selection (Apple C) and then create a new layer (Apple-Shift-N). Make sure that Opacity is 100%. Then paste the selection in the new layer (Apple-V). You can see the transparency by turning off the Background layer in the Layers window by clicking on the Eye. You should now see a portion of the image with the checkered white and gray showing through in the region you want to be transparent. Now, in the layers window, drag the background to the trash and save your image. You are ready to delete the rest of the water column white. Use the Magnetic Lasso Tool to highlight more regions and use Edit->Cut (Apple-X) to make the rest of it transparent. A handy trick when editing in close is to hold down the space bar. You get a hand icon that lets you scroll the image with the mouse. Also, if there is a small section of sea floor that is uniformly out of place, it is easy to push it back into place. Select the region using the Rectangular Marquee Tool, then select the Move Tool. You can drag it or use the arrow keys to move it down into place. IVS provides instructions for how to make transparent subbottom images: http://www.ivs.unb.ca/docs/vertical_images.pdf I have been unable to save transparent TIFs from my version of Photoshop 7, so I use ImageReady. From there ImageReady, open the PSD and then do File->Export Original. Select the tif format. For compression, choose LZW. You can open the tif back up in Photoshop to make sure that it actually is transparent. FIX: does LZW work with fledermaus To create a transparent gif, open up your PSD or TIFF and do File -> Save For Web FIX: write a converter to create an SFImage with transparency for inventor. For More Info: http://www.sgi.com/software/inventor/vrml/chap6.html http://www.sgi.com/software/inventor/tech_info.html FIX: move this somewhere http://vsvr.medien.fh-duesseldorf.de/~herder/tech/OpenInventor/Workshop/Texture/SCCP/image.html http://www.ivs.unb.ca/products/fledermaus We now need to get the navigation for these lines: 096-108 150-162 165-177 here is a small sample from lj100.txt: lj100 1 -117.26121167 32.88557000 15617 -11769: 0:2816:10752 lj100 2 -117.26121333 32.88558500 15617 -11769: 0:3072:10752 lj100 3 -117.26121333 32.88558500 15617 -11769: 0:3072:10752 lj100 4 -117.26121333 32.88558500 15617 -11769: 0:3072:10752 lj100 5 -117.26121333 32.88558500 15617 -11769: 0:3072:10752 Let's make some Inventor lines for these nav files. Yet again, I have put together a small python script called nav2iv.py: nav2iv.py lj096-108-nav.iv lj09[6-9].txt lj10[0-8].txt nav2iv.py lj150-162-nav.iv lj15?.txt lj16[0-2].txt nav2iv.py lj165-177-nav.iv lj16[5-9].txt lj17[0-7].txt # Use the coast files from Lecture 7... ivview lj*-nav.iv coast*.iv sandiego-topo8.2.iv
* Configuring fledermaus <<<<<<< VizLecture6.html cat lj6[5-9].* lj7?.* lj80.txt > ../lj065-080.txt cat lj8[1-9].* lj9[0-4].* > ../lj081-094.txt cat lj9[6-9].* lj10[0-8].* > ../lj096-108.txt cat lj15[0-9].* lj16[0-2].* > ../lj150-162.txt cat lj16[5-9].* lj17[0-7].* > ../lj165-177.txt cat lj179.* lj18[0-8].* > ../lj179-188.txt cat lj20[7-9].* lj210.* > ../lj207-210.txt cat lj22[89].* lj23[01].* > ../lj228-231.txt cat lj90[6-9].* > ../lj906-909.txt #!/bin/bash /bin/rm -f linecounts.txt touch linecounts.txt for file in lj*.txt; do base=${file%%.txt} awk '{print $3, $4}' $file | uniq > ${base}.xy wc -l ${base}.xy wc -l ${base}.xy >> linecounts.txt done cat linecounts.txt 5245 lj065-080.xy 4890 lj081-094.xy 4525 lj096-108.xy 3840 lj111-121.xy 4796 lj150-162.xy 3605 lj179-188.xy 955 lj207-210.xy 1255 lj228-231.xy Then schrinking the images down to this image width since the fledermaus guys insist that the xy file is the same length as the image width. Load lj065-080.xy, then Image->Image Size. Type in 5245 and click ok. Make sure that the "save transparency" and "save pyramid" options are checked mkvcurtain -in lj-tifs/lj096-108rev3small.tif -out lj096-108.sd -xy nav/lj096-108.xy -zrange -180 0 mkvcurtain -in lj-tifs/lj065-080small.tif -out lj065-080.sd -xy nav/lj065-080.xy -zrange -180 0 mkvcurtain -in lj-tifs/lj081-094small.tif -out lj081-094.sd -xy nav/lj081-094.xy -zrange -180 0 mkvcurtain -in lj-tifs/lj096-108small.tif -out lj096-108.sd -xy nav/lj096-108.xy -zrange -180 0 mkvcurtain -in lj-tifs/lj111-121small.tif -out lj111-121.sd -xy nav/lj111-121.xy -zrange -180 0 mkvcurtain -in lj-tifs/lj150-162small.tif -out lj150-162.sd -xy nav/lj150-162.xy -zrange -180 0 mkvcurtain -in lj-tifs/lj179-188small.tif -out lj179-188.sd -xy nav/lj179-188.xy -zrange -180 0 mkvcurtain -in lj-tifs/lj207-210small.tif -out lj207-210.sd -xy nav/lj207-210.xy -zrange -180 0 mkvcurtain -in lj-tifs/lj906_909small.tif -out lj906-909.sd -xy nav/lj906-909.xy -zrange -180 0 178-188 had some bad geo-ref! one point is missing a minus sign 117.27007000 32.91873000 converted to -117.27329890 32.91873000 ---------------------------------------------------------------------- If you get a blue fledermaus window and black on black fonts in the open and save dialogs you need to set up your environment for fledermaus xresources. One way to do this is with your .bashrc: export XKEYSYMDB=/usr/lib/X11/XKeysymDB export XUSERFILESEARCHPATH=/usr/IVS/resources/%N export SVHFONTPATH=usr/IVS/hershey here is what that would be with tcsh/csh setenv XKEYSYMDB /usr/lib/X11/XKeysymDB setenv XUSERFILESEARCHPATH /usr/IVS/resources/%N setenv SVHFONTPATH=/usr/IVS/hershey I think the svhfontpath is not required.
*MAKING A MOVIE OF THE EARTH Earth and Moon in Inventor Earth in Fledermaus http://www.ivs.unb.ca/docs/S09-Movie_Tools.pdf 7.5 m is aprox 10ms at 1500m/s(?) FIX: CHECK the image is 0.25 sec == 250 ms -> 25*7.5 m = 187.5 meters mkvcurtain -in lj96_108rev2.tif -out lj096_108.sd -xy lj096_108.xy -zrange 0 187.5 mkvcurtain -in lj96_108rev2-small2.tif -out lj096_108.sd -xy lj096_108-works.xy -zrange 0 187.5 line91_92.tif: TIFF file, little-endian cd /stoweraid/Jenna/cracksbathy/seisimag && imginfo line91_92.tif Warning: line91_92.tif: associated alpha treated as unassociated Image file: line91_92.tif File format: TIFF image Dimensions (w,h,z,c): 4626, 819, 1, 2 Page Size (w,h,z,c): 4626, 1, 1, 2 Data type: unsigned char Dimension order: interleaved Color model: luminance-alpha Coordinate space: upper-left Statistical Min/Max: 0 - 255 Display Min/Max: 0 - 255 Data Compression: Lempel-Ziv & Welch Resolution unit: Inch X resolution: 72 Y resolution: 72