Class 3: Weather Demo, Basic Command line, VM Ware Linux (DRAFT)

Table of Contents


Internet Relay Chat (IRC)

Start Firefox

Under "Tools," select "ChatZilla" at the bottom.

ChatZilla will use your login name (often referred to as you "shortname") by default. You can pick anything you like, but if you don't use something related to your name, people may have trouble which user in a chat is you. For this class, make sure to use the same short name as your CCOM account - please do not change your alias.

We need to now sign into an "IRC Server" that hosts chat rooms. There are many thousands of servers in the world and we will even setup on at CCOM later in the semester. For now, I have created an unofficial channel on http://freenode.net. FreeNode supports free and opensource software and this class is primarily about just those topics. Type this command right into the bottom window of ChatZilla:

/attach irc.freenode.net
/join #unhresearchtools

Say "hi" in the IRC channel

Remember that you must write at least one message in the IRC chat each class. It can be as simple as "hi".

TODO "Turn in" your homework by 5PM EDT today

Demonstration - Processing weather data

For this section, you do not need to try to follow along. Just watch what I am doing. The goal is to give you a feeling for what we will be covering. I do not you expect to understand all the steps just yet. It will take us a number of classes to cover all of this material.

This material has also been written up here on my blog:


First, log into a linux computer and setup a project directory.

ssh researchtools.ccom.nh # or use putty
mkdir wx
cd wx # wx is short for "weather"

Check out the weather data that is available over the CCOM internal network. socat is a network data helper. It can dump what it sees from a network connection, pass it to other programs, or forward the data.

socat TCP4:datalogger1.ccom.nh:36000 - | head

You should see some NMEA weather data.

2011/09/06 06:07:40 socat[29595] E write(1, 0x9d6b410, 53): Broken pipe

What we really want is some "exciting" weather. Hurricane Irene coming through New Hampshire just over a week ago will do. I logged the weather station and put the files online on this web page: examples. The unix "wget" command will pull the data down for us into the "wx" project.

pwd # make sure you are in the "wx" directory
wget http://vislab-ccom.unh.edu/~schwehr/Classes/2011/esci895-researchtools/examples/ccom-airmar-2011-08-28.bz2
wget http://vislab-ccom.unh.edu/~schwehr/Classes/2011/esci895-researchtools/examples/ccom-airmar-2011-08-29.bz2
wget http://vislab-ccom.unh.edu/~schwehr/Classes/2011/esci895-researchtools/examples/ccom-airmar-2011-08-30.bz2

# If you are unfamiliar with .bz2 files, ask the computer if it knows 
file ccom*
ccom-airmar-2011-08-28.bz2: bzip2 compressed data, block size = 900k
ccom-airmar-2011-08-29.bz2: bzip2 compressed data, block size = 900k
ccom-airmar-2011-08-30.bz2: bzip2 compressed data, block size = 900k

# Uncompress the data.  "*" matches any text
bunzip2 ccom*.bz2

file ccom-airmar-2011-08-*
ccom-airmar-2011-08-28: ASCII English text
ccom-airmar-2011-08-29: ASCII English text
ccom-airmar-2011-08-30: ASCII English text

head ccom-airmar-2011-08-29
# START LOGGING UTC seconds since the epoch: 1314576000.23
# SPEED:       4800
# PORT:        /dev/ttyS0
# TIMEOUT:     300.0
# STATIONID:   ccom-airmar

We now have weather data logs. We need to get the software to parse the logs.

# Check out the NMEA weather parsing code using mercurial
hg clone http://schwehr@bitbucket.org/schwehr/nmeadec

# Add it to the python path
cd nmeadec
export PYTHONPATH=`pwd`
cd -

Find some sample weather messages. They have the NMEA code "MDA" for "Composite Weather". The grep command searches for text.

grep MDA ccom-airmar-2011-08-29 | head

Now start "ipython" and try out the parser!

ipython -pylab  # ask ipython to load plotting and science modules

import nmeadec # pull in Kurt's NMEA parsing library

line = '$WIMDA,29.1166,I,0.9860,B,19.6,C,,,,,,,170.4,T,185.8,M,2.1,N,1.1,M*2D,rccom-airmar,1314576000.72'

# Your results should look something like this:
{'air_temp': 19.6,
 'pressure_bars': 0.986,
 'pressure_hg': 29.1166,
 'sentence': 'MDA',
 'talker': 'WI',
 'wind_dir_mag': 185.8,
 'wind_dir_true': 170.4,
 'wind_speed_knots': 2.1,
 'wind_speed_ms': 1.1}

You can then use the "getwx" function from processwx described in the blog post to pull in a large record of data.

One warning about the blog post: "pylab" pulls in math functions and one of them, "sum", causes trouble. There is a python "sum" builtin function. If you need the normal python sum, here is how to use it:

import __builtin__
# Result: 6

Research tools is not setup for graphics, so please follow along in the blog post.

The resulting image made by matplotlib using the weather data from CCOM's Airmar weather station during hurricane Irene:


We will go slowly through all the skills that are required to build plots like this of data over the next couple of classes.

Basic command line

Why learn about the command line?

Today people are often uncomfortable working on the command line to get things done with computers or perhaps have never even used the command line. Before windowing systems and mice were common, this was really the only way that people were able to tell a computer what to do. The advent of the Graphical User Interface (GUI) made some tasks easier, but it also made many tasks harder. If you need to rename hundreds of files, using a mouse is going to take you a long time or you are going to have to find and learn a small utility program. With the command line, using a "shell", you can write a quick command to rename large numbers of files easily. In the process, you have gained something over the GUI method: an inherently easy way to document or repeat the task - the text command. You can paste that command into a text file for documentation. You can even make the file executable and run it as a "script" in the future. The shell will remember commands that you have run before and let you rerun them the same way you did before or help you edit the commands to run slightly altered versions.

Why choose bash as your shell environment?

There are many flavors of shells with the most common being tcsh, sh bash, zsh, Windows/DOS command, and Windows PowerShell. The Microsoft Windows shells are too limited and are not portable to other operating systems. Unix systems started with C-shell (csh) and the Bourne shell (sh) in the 1970's and 1980's. Both of these shells were pretty limited in features. tcsh, bash, and zsh are improved versions of the old csh and sh shells. If you gain experience with csh and sh, you will find the syntax of sh to be more flexible and consistent than csh. sh provides basic functions that you can call that make writing scripts a bit easier. Additionally, sh is used on Unix type systems to start up the system and manage server type processes ("daemons", not demons) that work in the background to make the computer more functional. You will likely want to create or modify a daemon as you get comfortable with the Linux environment to do tasks such as logging data from serial ports. If you learn csh/tcsh, you will likely later have to learn at least some sh/bash. You are better off to just learn sh/bash and avoid having to waste time learning two slightly different shells.

The Bourne Again Shell (bash) has become the defacto standard rewrite of sh that provides a more usable experience than the limited sh. It gives us command completion (hit tab to finish a word if it can), histories and scrollback of previous command, the ability to control processes, etc.

Debugging stategies

If you are typing in the commands you find here, you might occasionally make mistakes that prevent the command from giving you exactly what you see here for output. Here are a couple things that can help you figure out what is going on.

First, read aloud the command that you have typed and what is in the document.

Check for common characters to confuse. It is easy to replace a "1" (number one) with an "l" (Lima) or vice versa if the fonts you have in your terminal and web browser make those two characters. Make sure you are using the right quote character (e.g. ", ', or ` are all different). Another pair of characters that is sometime trouble are the 0 (zero) and O (Oscar).

Note that the pipe character is a vertical bar: "|". This character is sometimes two vertical dashes. On US keyboards it is located between the delete and return/enter keys and is the shift of "\".

What to do if you get stuck?

Before we get into the commands, we need to talk about what to do if things get stuck. If you mistype a command and it just sits there doing nothing, you should first try holding down the "control" key and hitting the "C" key. This sends a "break" or "kill" message to the program. This is often written as "Ctrl-C" or "C-c". Here is a command that hangs. I then use Ctrl-C to get out of it. The bash shell responds with a "C" and gives a prompt again.

egrep some-string

If the command really gets stuck and does not respond to the Ctrl-C, you can close the terminal window and open a new window. Later on, you will learn fancier techniques for controlling programs (also known as processes), but this will work for now.

Managing files - Where am I and what is here? (pwd and ls)

To start, we need to become comfortable with managing files in a Linux command line environment with the bash shell and the commands that come with it.

The first command that you need to know tells you the working directory: pwd (print working directory). This command writes where you are to the terminal.

# And the response will have your user name, not "schwehr"
# /home/CCOMNH/schwehr

You type pwd, press enter/return and it will tell you where you are.

The path that you see will be different than I show above, but hopefully, you get the idea.

If you are accustomed to DOS or Microsoft windows, you have seen that directories (called "Folders" on Windows) are separated by the "\" character. With bash, directories are separated byt the "/" character. It is definitely annoying that Microsoft decided to change the character, but we are now stuck with this difference.

We can create a new directory with the mkdir (make directory) command.

mkdir example

Let's now move into that directory with the cd (change directory) command.

cd example

We should take a look at what is in that directory with the ls (list directory contents) command.


This will print out nothing. There are no files in the directory. Now is a good time to learn about options to command line programs. You can ask the ls command to behave differently. First let's try asking for all files with the "-a" option. This means it will show any hidden files that have a name starting with a ".". These are refered to as "dot" files.

ls -a
# .  ..

You can pass multiple options to a command. With the ls command, we might also want to see the "long" output. This will give us a lot more information than we want right now, but it will show you the date and time that the files were last changed and who "ownes" each file.

ls -a -l # That is "l" as in Lima
# total 8
# drwxr-xr-x  2 schwehr schwehr 4096 2010-10-15 08:13 .
# drwxr-xr-x 42 schwehr schwehr 4096 2010-10-15 08:13 ..

You can often combine these options into one short option. The previous command can be written like this.

ls -la

When working with bash, each directory has two special dot files. One "." refers to the current working directory. This is only occasionally useful. More interesting is the file with two dots. The ".." entry refers to the directory above this one. Let's try moving to the parent directory.

# /home/CCOMNH/schwehr/example

cd ..  # change directory to the parent - also known as "up"

# /home/CCOMNH/schwehr/example

Finally, if you are somewhere on the disk and want to get back to your home directory, the "~" points back to your home directory. We can use the echo command to see what the "~" means and then give it a try. echo prints what it is given to the terminal.

echo ~
# /home/kurt

cd ~

# /home/kurt

cd ~/example

# /home/kurt/example

bash keeps track of all the commands that you run. This is helpful when you want to run a command that you typed before or want to save what you have done to a notes file.


The results:

1  cd example
2  ls
3  ls -a
4  ls -a -l
5  ls -la
6  pwd
7  cd ..
8  pwd
9  cd -
10 pwd
11 echo ~
12 pwd
13 cd ~/example
14 pwd
15 history

You can scroll back to previous commands, edit them if necessary, and rerun them. Press the up and down arrows to scroll back through previous commands and left/right to edit a command. We will get into more advanced editing of commands later.

We can also ask the shell to tell us which disks are "mounted" (aka "attached" or "installed") on the computer with the df (disk free) command. Here is an example from a Linux system. Windows with cygwin will look pretty different. You can also ask it to write out the space on the device in a more "human-readable* format with the "-h" option. Note, you will see "non-disk" things on a linux computer, that I have hidden from you here. Please ignore these extraneous entries.

df   # This is going to give us lots of stuff we do not care about
Filesystem           1K-blocks      Used Available Use% Mounted on
                      19206868   2140996  16090208  12% /
none                    505412       188    505224   1% /dev
none                    512448         0    512448   0% /dev/shm
none                    512448       564    511884   1% /var/run
none                    512448         0    512448   0% /var/lock
/dev/sda6              3844152     73352   3575524   3% /tmp
/dev/sda1               188403     28307    150368  16% /boot
/dev/sda5              3844152    683612   2965264  19% /var
                      22585212    340952  21096972   2% /home

# Better:  ask for just the disk associated with the your home directory
# Pass it a period, meaning the "current directory"
df .
df .
Filesystem           1K-blocks      Used Available Use% Mounted on
                      22585212    340952  21096972   2% /home

# Ask for the size to be a bit easier to read with the "-h" option
df -h .
Filesystem            Size  Used Avail Use% Mounted on
                       22G  333M   21G   2% /home

Find help and documentation for commands

Linux and cygwin have what are called "manual pages" or "man pages" that describe most commands. Give it a try.

man df

The results:

DF(1)                            User Commands                           DF(1)

       df - report file system disk space usage

       df [OPTION]... [FILE]...

       This  manual  page  documents  the  GNU version of df.  df displays the
       amount of disk space available on the file system containing each  file
       name  argument.   If  no file name is given, the space available on all
       currently mounted file systems is shown.  Disk space  is  shown  in  1K
       blocks  by  default, unless the environment variable POSIXLY_CORRECT is
       set, in which case 512-byte blocks are used.

       If an argument is the absolute file name of a disk device node containā€
       ing  a  mounted  file system, df shows the space available on that file
       system rather than on the file system containing the device node (which
       is  always  the  root file system).  This version of df cannot show the
       space available on unmounted file systems, because  on  most  kinds  of
       systems  doing  so requires very nonportable intimate knowledge of file
       system structures.
 Manual page df(1) line 1

NOTE: 'q' gets you out of the above screen!

When you are in the man page, you are interacting with a "pager" program (it's actually a program called less). You have use the up and down arrow keys, the space bar, the b key, <, and > to move up and down the manual. A very important key to know is q to quit out of the manual.

You can also search for commands that might help you get a job done. This is known as "apropos". For example "apropos editor" You can also ask for it with the "-k" option to man.

man -k sort
alphasort (3)        - scan a directory for matching entries
apt-sortpkgs (1)     - Utility to sort package index files
bsearch (3)          - binary search of a sorted array
bunzip2 (1)          - a block-sorting file compressor, v1.0.4
bzip2 (1)            - a block-sorting file compressor, v1.0.4
comm (1)             - compare two sorted files line by line
qsort (3)            - sorts an array
sort (1)             - sort lines of text files
sort-dctrl (1)       - sort Debian control files
texindex (1)         - sort Texinfo index files
tsort (1)            - perform topological sort
versionsort (3)      - scan a directory for matching entries
winop (3blt)         - Perform assorted window operations#+END_EXAMPLE

On the right, after the dash ("-"), is a description of the command. On the left is the name of the command. Entries with a "(1)" after the name are things you can access from the bash command line. Entries with a "(2)" or "(3)" are things that are accessible from a full programming language such as C, perl, python, etc.

Specifying groups of files (pattern matching)

It is time to jump into the example directory and start working with directory listings.

cd ~/example

Now we can use a command called touch to create some files. touch is designed to update the last modified time, but if the file does not exist, it will create an empty file. Here we will create three files. Many commands can work on many files at the same time.

touch 1 2 3

ls -l
total 0
-rw-r--r-- 1 kurt kurt 0 2010-10-15 09:39 1
-rw-r--r-- 1 kurt kurt 0 2010-10-15 09:39 2
-rw-r--r-- 1 kurt kurt 0 2010-10-15 09:39 3

We can now try removing the files with the "rm" (remove) command.

rm 1 2 3

Now, let's create a bunch of files to give ourselves something to work with.

touch 1 2 3 4 5 6 7 8 9 10 11 12 13 100

We can now start trying out some of the shells abilities to select groups of files. This is know in shell terminology as pattern matching or "glob". The complete bash manual on matching files is here.


This is a bit of a big topic, but just jump in and over time you will pick up these tricks. I will use them throughout the rest of the book and with repetition, you will start to get the hang of them.

First, the "*" matches anything. By itself, it will match all the files. When combined with text, it will match anything with that text. Here are some examples to give you the idea. In bash, the "#" character starts a comment on a line. I will use comments to explain each entry.

# all files in a directory (effectively the same a just a plain "ls")
ls *
# 1  10  100  11  12  13  2  3  4  5  6  7  8  9

# anything starting with "1"
ls 1*
# 1  10  100  11  12  13

# anything ending with a "0" - This is the number zero
ls *0
# 10  100

# anything starting with 1 and ending with a 0
ls 1*0
# 10  100

The "?" is more specific than the "*". The "?" matches any single character. Give it a try.

# Match anything that has just 1 character
ls ?
# 1  2  3  4  5  6  7  8  9

# anything with exactly two letters
ls ??
# 10  11  12  13

# the letter "1" followed by any single character
ls 1?
# 10  11  12  13

You can get fancier by using square brackets for "[]" specifying sets of characters or ranges by putting a dash between two characters. It's best to just see some examples.

# List files that are one character of the number 2 through 5
ls [2-5]
# 2  3  4  5

# List files that start with 1 and have a 1 or 3 following.
ls 1[13]
# 11  13

# Combine the * and [] to ask for any file ending in 1 or 3
ls *[13]
# 1  11  13  3

# Here we are using a special system directory for an example using a
# range of alphabetical characters (x, y, & z).
# Please do not worry about what these files are
ls /sbin/*[x-z]
# /sbin/fsck.minix  /sbin/getty  /sbin/iwspy  /sbin/mkfs.minix  /sbin/pam_tally

Using a Virtual Machine (VM) version of Ubuntu Linux

Before we go any more into the shell, it is worth taking the time to show you how to be able to do this type of thing on your own computer and not always have to log into researchtools.ccom.nh.

Author: Kurt Schwehr

Date: <2011-09-06 Tue>

HTML generated by org-mode 7.4 in emacs 23