Data Mining: Text Mining, Visualization and Social Media

Updated d8taplex Design

As I mentioned in a previous post, I've been working on a redesign of d8taplex as a means to learn more about the grid based approach to content design.

d8taplex has now been updated with a more thoughtful design and some bug fixes and experience improvements.

I've learned a lot from the process. It is amazing how simple principles can help clarify the structure of a site.

Home page:

Directory:

Results:

Data set page:

Comparing Surface Temperature Between Japan and Scotland

It is interesting to compare data sets from different sources that describe the same variable. Below we have a data set from Japan describing the surface temperature anomaly for the world, the northern hemisphere, the sourther hemisphere and Japan (with the baseline average taken from 1971 to 2000). Compare this with the second graph which shows the anomaly from the Scottish Met (with the average taken between 1961 and 1990).

[permalink]

[permalink]

The Joy of Time Series Data

I'm not ashamed to admit that part of the fun of playing with data sets is simply the pleasure in looking at rich examples where the ebb and flow of the real world is captured in the data without necessarily being concerned with the value of the data to my own personal area of interest. Below are some time series selected by artistic merit from the updated d8taplex index. Note that you will need to read the version of this post on my blog as your feed reader won't always render the embedded graphs.

A data set entitled 'Monthly confidence consumer survey' from the French data. Many of the tables on official French sites appear to have text in both French and English [permalink].

Another wonderful French graph detailing prices of various food stuffs [permalink].

Industry indices from the Minister of Finance in Japan [permalink].

Some Japanese demographic model projections [permalink].

Some stats on drug types in Canada [permalink].

International Time Series Data Available on d8taplex

I've made a moderate update to d8taplex, adding time series data sets from Canada, France and Japan in addition to those from the UK. The site now includes the ability to limit the search for time series data to data sets sources from specific countries.

With this new index, the site now contains 367, 189 time series from 20, 524 data sets found on 72 sites. Take a look at the directory linked off the home page for full details on the data available per site.

In a recent blog post covering a presentation at Web 2.0 Expo, Hjalmar Gislason indicates that only 14 of the 193 countries of the world have established any sort of open data. This list of 14 does not include France or Japan (according to the shaded map used to make this point). The data sets surfaced by d8taplex are discovered by crawling the web and establishes that official 'open data portals' are not required to bring data to the masses. In many cases, these open data sites are aggregators and normalizers of data - they are not themselves data generating agencies. Thus, an alternative to the current flavour of open data - requiring a government portal which aggregates data - is a stack of technologies capable of discovering and interpreting data in any form that it appears in be it HTML tables, spreadsheets, text files or even images.

Note that the new d8taplex index includes data from Canada's National Statistics Agency site, not the much maligned data.gc.ca site.

Data Engine Roundup #8

I came across EidosSearch recently - a company founded by Steven Zhang and David Kedmey. Their product is a visual search engine for time series. You can select an exemplar data set (part of a time series) and it will return close matches to that time series from an indexed data set. While I don't think this is a novel idea, it is perhaps the first time I've seen it implemented with any degree of elegance.

Their homepage (linked above) has a good video introduction explaining what they do.

 

Thermal Growing Season Time Series

I recently implemented a simple directory page on d8taplex which lists the sites from which data has been extracted. Looking at data from The Commission for Rural Communities I found this time series data set on the thermal growing season.

The thermal growing season is defined in the data set as follows:

The thermal growing season length is defined as beginning when the temperature on five consecutive days exceeds 5˚C and ending when the temperature on five consecutive days is below that threshold.

[See wikipedia for another, similar definition.] A simple - and reasonable - interpretation of this data, then, is that things are getting warmer.

WeatherSpark - The Perfect Storm of Weather Data Visualization

WeatherSpark is:

a new type of weather website, with interactive weather graphs that allow you to pan and zoom through the entire history of any weather station on earth.

Get multiple forecasts for the current location, overlaid on records and averages to put it all in context.

but you really need to see it for these words to do justice.

Among other things, the site has a great interactive time series control that does all the right things at different scales.

Via Statistical Modeling, Causal Inference and Social Science