Research Tools log file for FIX: your name

Up to this point, we have been working hard as a class to get comfortable with a powerful text editor and the terminal. We will do a little more setup today and in lecture 10 on 2011-09-29 Thu, but we will be starting into learning Python in lecture 10. Hopefully, you will appreciate the 4 weeks of setup when working with Python in emacs.

Today we will setup org-babel so that we can execute code from inside of emacs org-mode files and write a shell script file.

http://orgmode.org/worg/org-contrib/babel/

The first 4 videos are available: 2011 Research Tools YouTube playlist. These supplementary videos cover using emacs. They are modelled after the videos on http://showmedo.com.

If you try to execute a "sh" (bash shell) source code block in an org-mode file (using C-c C-c), you will probably see an error message like this in the mini buffer at the bottom:

No org-babel-execute function for sh!

We need to setup emacs to have org-mode fully installed. Open the file ~/.emacs

; Must have org-mode loaded before we can configure org-babel
(require 'org-install)

; Some initial langauges we want org-babel to support
(org-babel-do-load-languages
 'org-babel-load-languages
 '(
   (sh . t)
   (python . t)
   (R . t)
   (ruby . t)
   (ditaa . t)
   (dot . t)
   (octave . t)
   (sqlite . t)
   (perl . t)
   ))

; Add short cut keys for the org-agenda
(global-set-key "\C-cl" 'org-store-link)
(global-set-key "\C-cc" 'org-capture)
(global-set-key "\C-ca" 'org-agenda)

Now restart emacs.

echo $SHELL

#+results:
: /bin/bash

For the rest of the semester, you need to keep a log file for this class with C-c C-c inside the src block.

mkdir ~/Dropbox/logs

Start a log file. Replace YOURUSERNAME below with your short username at CCOM.

log=~/Dropbox/logs/researchtools-YOURUSERNAME.org
echo "#+STARTUP: showall" > $log 

echo "Log contains:"
cat $log

You can now open up that log file in another emacs window. Add AUTHOR and EMAIL lines just line in homework 2.

#+TITLE:     Research Tools log file for FIX: your name
#+AUTHOR:    FIX: your name
#+EMAIL:     FIX: your email

And you can make entries that look something like this:

* Log entry for a day <2011-09-27 Tue>                            :day:

I start them by typing line this:

* Log entry for a day 

Then I use C-c . to add the date. I follow it up with at least the day tag by doing C-c C-c on the header line and typing "day". I also use "travel" as a tag for days that I'm traveling.

You can now make entries for that day with two "*" entries for a 2nd level heading:

** Research Tools Class 09                              :researchtools:class:

# FIX: Start writing about the class here.

Now we are going to create our first KML file. We are going to cheat a bit and not try to understand the file format, but this will at least show you how easy it can be.

First, make sure we have a working directory for this class:

mkdir -p ~/class/09
ls -ld ~/class/09

Now, get the header and footer text for the KML line format:

cd ~/class/09

curl -O http://vislab-ccom.unh.edu/~schwehr/Classes/2011/esci895-researchtools/google-earth-line-start.kml
curl -O http://vislab-ccom.unh.edu/~schwehr/Classes/2011/esci895-researchtools/google-earth-line-end.kml

These two pieces give you the front and back of the KML and all we need to do is provide the coordinates for the

Get the coordinates file from the Boston Construction file used during the homework:

cd ~/class/09

curl -O http://vislab-ccom.unh.edu/~schwehr/Classes/2011/esci895-researchtools/examples/2007-boston-construction.csv.bz2

bunzip2 2007-boston-construction.csv.bz2

Take a look at the file:

cd ~/class/09

head 2007-boston-construction.csv 

We see that it has longitude, latitude, and a Unix UTC timestamp. We will ignore the timestamp for now.

-70.5014566667,42.1006833333,1179617934
-70.5016466667,42.101755,1179617991
-70.501845,42.1028766667,1179618051
-70.5020833333,42.1039,1179618111
-70.5022083333,42.1049116667,1179618176
-70.5022883333,42.1059316667,1179618233
-70.502515,42.1069266667,1179618296
-70.5027566667,42.10796,1179618356
-70.5028616667,42.1090066667,1179618416
-70.5029816667,42.1102133333,1179618486

We can reuse the cut command to get just the X and Y coordinates:

cd ~/class/09

cut -d, -f1,2 2007-boston-construction.csv | head

Which should look like this:

-70.5014566667,42.1006833333
-70.5016466667,42.101755
-70.501845,42.1028766667
-70.5020833333,42.1039
-70.5022083333,42.1049116667
-70.5022883333,42.1059316667
-70.502515,42.1069266667
-70.5027566667,42.10796
-70.5028616667,42.1090066667
-70.5029816667,42.1102133333

We are lucky! KML expects coordinates to come as x,y,z or x,y. If you want to read more, you can look at the KML Reference section on coordinates, Placemark and LineString. We will talk more about KML in future lectures quite a bit.

<Placemark>
  <LineString>
    <coordinates>
      -125.810021667,48.4840316667
      -125.810295,48.483705
    </coordinates>
  </LineString>
</Placemark>

Let's create the x,y pairs in a file:

cd ~/class/09

cut -d, -f1,2 2007-boston-construction.csv > 2007-boston-construction.xy

We can now put the header, points and tail together to create a KML file. Google Earth has trouble with lines with too many points in them, so we will use head to only output some of the points.

The ">" redirects output to a file just as we have done in the past. However, if we use it a 2nd time to the same file, it will overwrite the first file and destroy anything that we did before.

The ">>" string does redirection, but appends the data to any existing file.

cd ~/class/09

cat        google-earth-line-start.kml >  2007-boston-construction.kml
head -1000 2007-boston-construction.xy >> 2007-boston-construction.kml
cat        google-earth-line-end.kml   >> 2007-boston-construction.kml

We can now open the file in Google Earth. It appears that Google Earth on a Ubuntu 11.04 virtual machine on the Mac is very crash prone. You can put the kml file into your dropbox folder and then download it through http://dropbox.com.

# This does NOT work
google-earth ~/class/09/2007-boston-construction.kml

Or you can try running Google Earth and then doing a File->Open to select your KML. That appears to be more stable.

You can also open the KML file in QGIS. In the Ubuntu menu's, go to Applications->Science->Quantum GIS.

If you have the time, head over to James hall for the Earth Science Tea!

DEADLINE: 2011-10-04 Tue by 5PM EDT

Note: These check boxes are just to help you.

• [X] Watch youtube videos: 1 2 3 4
  • [X] Put any questions that you have into your org-mode log file
• [X] Make sure you have log entries for both 2011-09-27 Tue and 2011-09-29 Thu for the Research Tools course
• [X] Turn in your log file on the researchtools server as this file ~/hw/03/log-$USER-$(date +%Y%m%d).org
• [X] Email mailto:kurt@ccom.unh.edu the md5 sum of the log file. Do not make this an attachment.