Class 5: File types and Emacs (DRAFT)

You can do this either from inside your ubuntu virtual machine inside the ccom network or by using ssh/putty to connect to researchtools.ccom.nh. If you are using the class virtual machine, first install the irssi IRC client:

sudo apt-get install irssi

Now start irssi and connect:

irssi
/connect researchtools.ccom.nh
/join #unhresearchtools

You will need to read the irssi documentation about how to use the client, but the most important command you probably need to start with is how to quit:

/quit

From the virtual machine, run Pidgin. From the menu, it is Applications -> Internet -> Pidgin Internet Messenger. Then from the Buddy List, Accounts -> Manage Accounts. The press the "Add…" button. Under the Add Account "Basic" tab, select the protocol as: IRC. Pick a username (should match your CCOM username), and select the server as researchtools.ccom.nh. There is no password. Leave password blank.

Make sure that the account is "Enabled" (put a check mark in the account).

Then under buddies, select "Join a Chat". You can put in "#unhresearchtools" or you can push the "Room List" button.

There is a command line program that can attempt to open files based on its best guess for how a file should be opened: xdg-open (or just open on the Mac) This works well for some image types.

xdg-open Field_Procedures_Manual_May_2011.pdf

You might be accustomed to using the "double click" in graphical interfaces, but knowing how to open files in the default application is very helpful for working from a scripting environment.

A quick aside: Stop and take a quick look at the NOAA Field Procedures Manual. There are a number of very helpful documents available for material related to this course. NOAA has put together this document to talk about how they do Hydrographic Surveying. This might be different for you if you do hydrographic surveying for some other organization or surveying for goals other than hydrography. However, this and many documents like it provide excellent reference and background material.

If you are at CCOM, you can see some of the documents we have links to in the wiki:

http://wiki.ccom.unh.edu/index.php/Survey_manuals

We will talk more about helpful references and how to manage these documents throughout the rest of the semester.

Now we can try asking the computer more about these files. There is a The file command tries to look at a little bit of the beginning of each file to see if it can figure out what type of data is in that file.

WARNING: Always be aware that file names are just a hint to a file type. Renaming a file to some random characters does not change the contents of the file. Some programs count on the "extensions" on the end of the file name (e.g. ".tar"), but you will find that those are not always consistent with the content of the file.

file *
0479_20080620_175447_RVCS.all.bz2:    bzip2 compressed data, block size = 900k
13003_1.KAP:                          data
13003.BSB:                            ASCII English text, with CRLF line terminators
20110912-1801.jpeg:                   JPEG image data, JFIF standard 1.01
a-folder:                             directory
bags.sqlite:                          SQLite 3.x database
delicious.htm:                        exported SGML document text
dos-text.txt:                         ASCII text, with CRLF line terminators
empty-file:                           empty
Field_Procedures_Manual_May_2011.pdf: PDF document, version 1.4
foo.csv:                              ASCII text
...

There are a large number of files in this directory, but just look at the first file that comes up:

0479_20080620_175447_RVCS.all.bz2:    bzip2 compressed data, block size = 900k

The file command does not get past the fact that the file is compressed. We need to uncompress all of the files that have been shrunk with bzip2 or another program called gzip. gzip files tend to end in ".gz".

ls -l *.bz2 *.gz
-rw-r--r-- 1 schwehr schwehr  3715206 2011-09-13 08:11 0479_20080620_175447_RVCS.all.bz2
-rw-r--r-- 1 schwehr schwehr 42990137 2011-09-13 08:08 reson7111-201005.s7k.bz2
-rw-r--r-- 1 schwehr schwehr     8868 2010-10-16 12:16 terrain.grd.gz
-rwxr-xr-x 1 schwehr schwehr 15541594 2011-09-13 08:11 y1104-02.segy.bz2

bunzip2 *.bz2
gunzip *.gz

ls -l *.bz2 *.gz

Now when we run file on 0479_{20080620175447RVCS}.all, we get:

file 0479_20080620_175447_RVCS.all 
0479_20080620_175447_RVCS.all: data

So, it turns out that file does not know anything about our ".all" file. It's binary data, but we can try some other means to see if we can identify the file… we can look into the file. If we try the pager command less, it will ask us if we are sure we want to look at the binary data. Yes, we would like to take a look.

less 0479_20080620_175447_RVCS.all 
"0479_20080620_175447_RVCS.all" may be a binary file.  See it anyway?

Answer yes, and you will see lots of weird stuff with some characters in there. Type a "q" to quit out of less and we will try another helpful command to try to hide the "noise" of the binary data and see if there is any useful text in the file. The command strings will go through a file and return only readable characters from a file. We can see how many strings there are first by "piping" the output from strings to a command called word count (wc).

strings 0479_20080620_175447_RVCS.all | wc
  21912   26248  143940

The vertical bar is the "pipe" command. wc tells us the number of lines followed by the number of words in the middle and the number of characters on the right. 21 thousand matches is too many to look at, so we should just look at the first few strings returned using the head command (there is also an equivalent tail command for the other end of the stream).

strings 0479_20080620_175447_RVCS.all | wc
  21912   26248  143940
schwehr@ubuntu:~/examples-20110913$ strings 0479_20080620_175447_RVCS.all | head
WLZ=0.57,SMH=481, ... DSV=3.0.7 040104,SID=Hydro2008_Day2,COM=Day1 of Summer Hydro 2008. 
Area1
06/14/1501,
G":s
" xj
K!)k
u!Wk
H"5m
Z!?!
.!h!

I have replaced a whole lot of text with "…" above, but this first line returned tells us quite a bit about this file. From this, I can know that the file has something to do about the CCOM Summer Hydro class in 2008. We don't have anything that tells us much more, but now we have something we can use for a web search or to ask around about the Summer Hydro class.

We can use a program called "grep" to search for text patterns in the results of the file command. The pictures in the file return a line that contains the word "image". For example, I found this JPEG image:

20110912-1801.jpeg:                   JPEG image data, JFIF standard 1.01

We can pass the output through a pipe to the grep command and ask it to search for the word "image".

file * | grep image
20110912-1801.jpeg:                   JPEG image data, JFIF standard 1.01
H11296_5m-hillshade.tif:              TIFF image data, little-endian

We have a JPEG image and a tiff image. That doesn't tell us much about those images, but we can find out more with other tools. The first tool that we will use is ImageMagick. It comes with a command called identify.

identify  *.jpeg *.tif 
20110912-1801.jpeg JPEG 1280x960 1280x960+0+0 8-bit DirectClass 78.4KB 0.000u 0:00.000
H11296_5m-hillshade.tif[1] TIFF 3205x1278 3205x1278+0+0 8-bit Grayscale DirectClass 4.107MB 0.000u 0:00.000
identify: H11296_5m-hillshade.tif: unknown field with tag 33550 (0x830e) encountered. `TIFFReadDirectory' @ warning/tiff.c/TIFFWarnings/704.
identify: H11296_5m-hillshade.tif: unknown field with tag 33922 (0x8482) encountered. `TIFFReadDirectory' @ warning/tiff.c/TIFFWarnings/704.
identify: H11296_5m-hillshade.tif: unknown field with tag 34735 (0x87af) encountered. `TIFFReadDirectory' @ warning/tiff.c/TIFFWarnings/704.
...

Ignore the warnings with "unknown field" in them. This is a geotiff with information that ImageMagick does not know about. identify has told us the size of the images a little about the type of content. Better yet to take a look at the images.

display  *.jpeg *.tif 

The jpeg turns out to be an image from the camera above the bridge of the USCG Ice Breaker Healy. The tiff looks weird, but is actually a gray scale image of some Lidar data from the area near the UNH campus.

The Geospatial Data Abstraction Library ( GDAL ) library, has tools for identifying both raster (e.g. images) and vector (e.g. line) data that has spatial data attached to it. First try gdalinfo on the JPEG image:

gdalinfo 20110912-1801.jpeg 
Driver: JPEG/JPEG JFIF
Files: 20110912-1801.jpeg
Size is 1280, 960
Coordinate System is `'
Metadata:
  EXIF_XResolution=(72)
  EXIF_YResolution=(72)
  EXIF_ResolutionUnit=2
  EXIF_YCbCrPositioning=1
  EXIF_GPSVersionID=0x2 0x2 00 00
  EXIF_GPSLatitudeRef=N
  EXIF_GPSLatitude=(81) (14) (24.5814)
  EXIF_GPSLongitudeRef=W
  EXIF_GPSLongitude=(126) (47) (30.5148)
Image Structure Metadata:
  SOURCE_COLOR_SPACE=YCbCr
  INTERLEAVE=PIXEL
  COMPRESSION=JPEG
Corner Coordinates:
Upper Left  (    0.0,    0.0)
Lower Left  (    0.0,  960.0)
Upper Right ( 1280.0,    0.0)
Lower Right ( 1280.0,  960.0)
Center      (  640.0,  480.0)
Band 1 Block=1280x1 Type=Byte, ColorInterp=Red
  Image Structure Metadata:
    COMPRESSION=JPEG
Band 2 Block=1280x1 Type=Byte, ColorInterp=Green
  Image Structure Metadata:
    COMPRESSION=JPEG
Band 3 Block=1280x1 Type=Byte, ColorInterp=Blue
  Image Structure Metadata:
    COMPRESSION=JPEG

JPEG images have special data called "EXIF" tags that can record more than just the image. In this case it has save the GPS location of the ship from when the picture was taken. The ship was at roughly 81 North and 126 West when the picture was taken. That is way up in the Arctic!

Now take a look at the results for the "GeoTiff":

gdalinfo H11296_5m-hillshade.tif 
Driver: GTiff/GeoTIFF
Files: H11296_5m-hillshade.tif
Size is 3205, 1278
Coordinate System is:
GEOGCS["WGS 84",
    DATUM["WGS_1984",
        SPHEROID["WGS 84",6378137,298.2572235629972,
            AUTHORITY["EPSG","7030"]],
        AUTHORITY["EPSG","6326"]],
    PRIMEM["Greenwich",0],
    UNIT["degree",0.0174532925199433],
    AUTHORITY["EPSG","4326"]]
Origin = (-70.778190923162583,43.023240535377035)
Pixel Size = (0.000058406003444,-0.000058406003444)
Metadata:
  AREA_OR_POINT=Area
Image Structure Metadata:
  INTERLEAVE=BAND
Corner Coordinates:
Upper Left  ( -70.7781909,  43.0232405) ( 70d46'41.49"W, 43d 1'23.67"N)
Lower Left  ( -70.7781909,  42.9485977) ( 70d46'41.49"W, 42d56'54.95"N)
Upper Right ( -70.5909997,  43.0232405) ( 70d35'27.60"W, 43d 1'23.67"N)
Lower Right ( -70.5909997,  42.9485977) ( 70d35'27.60"W, 42d56'54.95"N)
Center      ( -70.6845953,  42.9859191) ( 70d41'4.54"W, 42d59'9.31"N)
Band 1 Block=3205x2 Type=Byte, ColorInterp=Gray
  NoData Value=0

The results for the JPEG picture were different than for this GeoTiff. The picture is located just at one point on the Earth. In the tif, we see that there is a Coordinate System definition (WGS84) and that there 4 corners of the rectangle defined to say where this area is (Newcastle, NH to the Isle of Shoals).

First, we should add GNU Emacs to the quick start short cut location on the toolbar at the top of the screen.

Right click on the gray bar, and select "Add to Panel…". Double click "Application Launcher… Copy a launcher from he applications menu". Press the small arrow to the left of "Accessories" to expand the items inside of accessories. Double click on "GNU Emacs 23". Click "Close" to stop adding programs to the "Panel".

Click the "E" icon to start Emacs. You are now faced with a very powerful tool that can handle very complicated editing jobs. Emacs is very complicated, but it is worth the time to learn.

On your own time, I strongly suggest that you go through the "Emacs Tutorial" and "Emacs Guided Tour". Emacs has key strokes that you can use for every command, but starting off, you can use the menus at the top.

To start with, open a new file. "File" -> "Visit New File". Click the "Browse for other folders" arrow to see directories. Double click "examples-20110913". Now type in "my-script.sh" into the name.

You are now looking at a blank space where you can create your script. Start off by creating a very basic script to print on line of text just to get started:

echo "hello from my script"

Save this file "File" -> "Save".

Now switch back to your terminal and take a look at the file your created. It will be the newest file, so we can do a list files, but add sort by time (-t) and reverse the sort order (-r) to put the newest files at the bottom. I will pipe the output through tail to give us only the last 10 lines.

ls -ltr | tail
-rw-r--r-- 1 schwehr schwehr    46393 2011-09-13 08:29 sample-audio.m4a
-rw-r--r-- 1 schwehr schwehr    70310 2011-09-13 08:30 sample-audio.mp3
-rw-r--r-- 1 schwehr schwehr   416460 2011-09-13 08:31 sample-audio.wav
-rw-r--r-- 1 schwehr schwehr   208512 2011-09-13 08:31 sample-audio.ac3
-rw-r--r-- 1 schwehr schwehr       78 2011-09-13 08:33 foo.csv
-rw-r--r-- 1 schwehr schwehr      117 2011-09-13 08:34 dos-text.txt
-rw-r--r-- 1 schwehr schwehr      294 2011-09-13 08:36 sample.org
-rw-r--r-- 1 schwehr schwehr      980 2011-09-13 08:36 sample.tex
-rw-r--r-- 1 schwehr schwehr    64942 2011-09-13 08:37 sample.pdf
-rw-r--r-- 1 schwehr schwehr       28 2011-09-13 10:38 my-script.sh

You should see my-script.sh. Take a look the file. We can use cat to take a look.