Data Mining: Text Mining, Visualization and Social Media

International Time Series Data Available on d8taplex

I've made a moderate update to d8taplex, adding time series data sets from Canada, France and Japan in addition to those from the UK. The site now includes the ability to limit the search for time series data to data sets sources from specific countries.

With this new index, the site now contains 367, 189 time series from 20, 524 data sets found on 72 sites. Take a look at the directory linked off the home page for full details on the data available per site.

In a recent blog post covering a presentation at Web 2.0 Expo, Hjalmar Gislason indicates that only 14 of the 193 countries of the world have established any sort of open data. This list of 14 does not include France or Japan (according to the shaded map used to make this point). The data sets surfaced by d8taplex are discovered by crawling the web and establishes that official 'open data portals' are not required to bring data to the masses. In many cases, these open data sites are aggregators and normalizers of data - they are not themselves data generating agencies. Thus, an alternative to the current flavour of open data - requiring a government portal which aggregates data - is a stack of technologies capable of discovering and interpreting data in any form that it appears in be it HTML tables, spreadsheets, text files or even images.

Note that the new d8taplex index includes data from Canada's National Statistics Agency site, not the much maligned data.gc.ca site.

Data.gov, Others on the Verge of Extinction?

It seems like a twisted April Fools joke, but there are reports that data.gov and other US government data and transparency portals are to be defunded. If this failure to invest is localised then open data will live on in other forms within the US. However, I only hope that other nations don't attempt to follow the lead.

In my experience, one of the functions of these sorts of sites is to act as governmental data aggregators. In the UK at least, the data is often distributed on other sites. The aggregation function removes the barrier to discovery. Thus, removal doesn't necessarily mean that the data itself evaporates. I'm not sure if the US situation is the same.

Transparency is not Open Data

I recently wrote a short post outlining my thoughts on the distinction between open data (the term generally associated with the behaviour of governments and other public bodies providing declarative and programmatic access to various data sets) and transparency (the behaviour of governments publishing honest accounts of their behaviour without restriction).

In reviewing a few of the open data sets that BuzzData linked to on their recent post about these open data portals, I noticed that some of the data is provided in formats such as PDF. While it may be transparent to publish data in this format (PDF is a document format with reasonably high market penetration) the data that they contain is obscured by the purposes of the document encoding. PDF is designed to make documents look good, not to separate out the content abstractly from the format. Consequently, the data - while being distributed for human consumption in an accessible manner - is not open in the sense of being in a declarative or programmatically accessible representation.

Open Data is not Transparency

There are many encouraging signs of late in the general area of open data. However, one thing that has to be kept in mind with this movement is that open data is only part of transparency - it is necessary but not sufficient. If the data is not understandable by the intended audience (and open data suggests a very broad audience) then there is no transparency. The information and knowledge locked in the data will be hiding in plain sight.

This thought suggests that any open data movement has to be combined with a 'plain English' (or 'plain <insert your language here>') programme and an investment in data literacy. In addition, to take the whole movement to its obvious conclusion, there should be some well defined success criteria. What is the answer to the question: what happens to whom when citizens experience open data?

You may also be interested in posts on data engines.