8 – Data and the table class

Tabular Data

This tutorial is taken from here.
It focuses on the Table class – see the reference and functions of this class here.

You may have noticed that the graphing example above loads a text file with numbers separated by commas. Processing includes a Table class which will load a comma-separated (CSV) or tab-separated (TSV) file and automatically parse the data into columns and rows for you. This is a great deal more convenient than struggling to manually parse large data files with split(). It works as follows. Let’s say you have a data file that looks like:
data_03_table

Instead of saying:

String[] stuff = loadStrings("data.csv");

We can now say:

Table table = loadTable("data.csv");

Now we’ve missed an important detail. Take a look at the data.csv text file above. Notice how the first line of text is not the data itself, but rather a “header row.” This row includes labels that describe the data included in each subsequent row. The good news is that Processing can automatically interpret and store the headers for you, if you pass in the option “header” when loading the table. (In addition to “header” there are other options you can specify. For example if your file is called data.txt but is comma separated data you can pass in the option “csv”. If it also has a header row, then you can specifiy both options like so: “header,csv”). A full list of options can be found on the loadTable() documentation page.

Table table = loadTable("data.csv","header");

Now that the table is loaded, we can look at how we grab individual pieces of data or iterate over the entire table. Let’s look at the data visualized as a grid.

data_05_headers copy

In the above grid we can see that the data is organized in terms of columns and rows. One way to access the data would be to therefore request a value by its numeric column and row location (with zero being the first row or first column). This is similar to accessing a pixel color at a given (x,y) location. Here we are getting a piece of data at a given (row, column) location.

int val1 = table.getInt(2,1);      // val now has the value 235
float val2 = table.getFloat(3,2);  // val2 now has the value 44.758068
String s = table.getString(0,3);   // s now has the value "Happy"

While the numeric index is sometimes useful, it’s generally going to be more convenient to access each piece of data by the column name. For example, we could pull out a specific row from the Table.

TableRow row = table.getRow(2);  // Gets the third row (index 2)

Note in the above line of code that a Table object refers to the entire table of data while a TableRow object handles an individual row of data within the Table. Once we have the TableRow object, we can ask for data from some or all of the columns.

int x = row.getInt("x");             // x has the value 273
int y = row.getInt("y");             // y has the value 235
float d = row.getFloat("diameter");  // d has the value 61.14072
String s = row.getString("name");    // s has the value "Joyous"

The method getRow() returns a single row from the table. If you want to grab all the rows and iterate over them you can do so with the method rows().

for (TableRow row : table.rows()) {
  float x = row.getFloat("x");
  float y = row.getFloat("y");
  float d = row.getFloat("diameter");
  String n = row.getString("name");
  // Do something with the data of each row
}

So I know this for loop is weird and different to what we have seen so far, but it is a short hand way available in the table class to iterate over all the rows in the table. So the bit “for (TableRow row : table.rows())” means “a for loop that iterates over every table row”. So each cycle of the loop will give you these values for each row, moving down your table. Then the row.getXX(“something”); will return the values from the row it is on, from the columns labelled with what is in the brackets.

(Side note: See how row is an object of the TableRow class that is locally declared just in this loop).

If you want to search for a select number of rows within the table, you can do so with findRows() and matchRows().
In addition to being read, Table objects can be altered or created on the fly while a sketch is running. Cell values can be adjusted, rows can be removed, and new rows can be added. For example, to set new values in a cell there are functions setInt(), setFloat(), and setString().

// Update the value of column "x" to mouseX in a given TableRow.
row.setInt("x",mouseX);

To add a new row to a Table, simply call the method addRow() and set the values of each column.

// Create a new row
TableRow row = table.addRow();
// Set the values of that row
row.setFloat("x", mouseX);
row.setFloat("y", mouseY);
row.setFloat("diameter", random(40, 80));
row.setString("name", "new label");

To delete a row to a Table, simply call the method removeRow() and pass in the numeric index of the row you would like removed. The following code for example, removes the first row whenever the size of the table is greater than ten rows.

// If the table has more than 10 rows
if (table.getRowCount() > 10) {
  // Delete the first row
  table.removeRow(0);
}

For an example that creates objects from and saves objects to a Table, take a look at LoadSaveTable under Topics –> Advanced Data.

Loading and Visualizing your OpenPaths Data

This example takes a geolocation data file from openPaths and loads it into processing.

Before you do anything, get the file in csv format and open it up in a spread sheet program to take a look at your data. You will want to pars, filter and mine your data.

*Parse: Provide some structure for the data’s meaning, and order it into categories. Once you have obtained your data, before you load it into Processing it is important to parse the data. Parsing means checking the file’s format. Is the dataset tagged correctly? Check that each line of your data is broken up consistently across columns. This can be done in a number of ways such as, printing your file out in the terminal or opening your file in a text editor or spreadsheet program and checking for inconsistencies or gaps.

*Filter: Remove all but the data of interest. Your dataset is likely to contain extra information not relevant to your visualisation. For example in the csv file, the file has columns that might not be relevant. Remove irrelevant data so so that they do not interfere with the your visualisation process.

*Mine: Apply methods from statistics or data mining to discern patterns in your data and place the data in mathematical context. As Fry (2008) outlines, the mining stage of visualising data involves applying statistical methods and math to your dataset to analyse patterns and trends within it. This might be as simple as identifying the minimum and maximum values so that you know the range of variation in its values. Depending on your data, you may wish to calculate an average or a median value.

Load in your data

This requires you to get your openpaths data file and put it into your data folder. Download it from here. If you can’t get openPaths on your phone, here is some sample data

//import your data
Table table;
 
//data variables
float latm, latp;
float lonm, lonp;
 
//arrays for x coordinates and y coordinates
float[] locationsX;
float[] locationsY;
 
 
void setup() {
  size(800, 600);
 
  //load your data file. Check if you have a header.
  table = loadTable("openpaths.csv", "header");
 
  println(table.getRowCount() + " total rows in table");
 
  //make arrays as long as our table length. table.getRowCount() evaluates to the length of your table
  locationsX = new float[table.getRowCount()];
  locationsY = new float[table.getRowCount()];
}
 
void draw(){
 //change the values in this line to see how this will print one of your data points to the consol
 println("table="+table.getFloat(0, 1));
}

We then want to map the coordinate data to your processing sketch. This means we need to consider the longitude and latitude values and their range. Take a look at their range by inspecting your csv file.

We want to remap these values to the coordinates of the screen so that these points show up on our screen but remain relatively positioned. We use this using a very useful function called map.

The Map function

Map takes a variable and maps it from one range to another. So in this case we want to look at the range of our latitude values and remap them so that we can get x values that fall into the range of the width of the screen (0 – width). We also want to look at the range of our longitude values and remap them to the range of the screen height from (0 – height).

map(value, start1, stop1, start2, stop2);
value: the incoming value to be converted
start1: lower bound of the value's current range
stop1: upper bound of the value's current range
start2: lower bound of the value's target range
stop2: upper bound of the value's target range

Consider how your coordinate locations would be positioned on a map using longitude (runs longwise) and latitude (runs horizontal).
nyCoords

And then we want to draw them in the same position but using the coordinates of our screen.
nymappedCoords

So we use the map function to remap the values of longitude and latitude to the screen coordinate values. You will have to determine your own range, but for me in New York, these values work.

    //remap coordinates to the dimensions of our screen
    lat=map(lat, 40.5, 41.5, 0, height);
    lon=map(lon, -73.5, -74.5, 0, width);

We can then display these longitude and latitudes by remapping them to the coordinates of the screen and by using a for loop to draw an ellipse at each point.

void draw(){
//use a for loop to y which we use to call each row number
  for (int y=0; y<table.getRowCount (); y++) {
    //to animate this data, change the getRowCount() to a counter
 
    //get latitude using getFloat(row, column) - from the first column
    float lat = table.getFloat(y, 0);
 
    //get longtitude using getFloat(row, column) - from the second column
    float lon = table.getFloat(y, 1);
 
    //remap coordinates to the dimensions of our screen
    lat=map(lat, 41.5, 40.5, 0, height);
    lon=map(lon, -74.5,-73.5,  0, width);
 
    fill(0, 0, 255);
 
    ellipse(lon,lat, 20, 20);
 
 
  }
}

Exercise:

  • Connect the points with lines instead of ellipses.
  • Can you find a way to animate this path. (you need to use a counter).