Data Mining: Text Mining, Visualization and Social Media

Data Engine Roundup #7

• The BuzzData blog has a nice post about how the open data movement is measuring up. I like how this team thinks - looking at the value proposition of the Open Knowledge Foundation and figuring out how to make it measurable.
• The same post links to a map created by the CTIC Foundation in Asturias, Spain. This map geolocates 86 open data catalogues. This is similar to the map that BuzzData created earlier in the week which links to a similar map (some dead links in there, BTW). To get a slightly different view of what data is available, take a look at the directory for d8taplex which lists hosts on the UK government domain which contain time series data.
• On March 28th, DataMarket announced its first Premium Data Sets. These are provided in partnership with Xignite and are in the financial domain.

Read other data engine roundups and commentary here.

Feed Readers, Data Embedding and the Challenges of HTML 5

Blogs are an important channel of information on the web. However, because feeds (RSS, Atom in various forms) represent cross site risks to feed readers built on the web, not to mention the potential for javascript to mess up the HTML based reader's DOM, many feed readers strip out embed, iframe and script elements.

There is only moderate consistency in when this is done. For example, Google Reader subscribers won't see the embedded graph below from d8taplex, but Bloglines readers will.

Video is one of the most important media types online, and web-based readers - through a white list mechanism - permit embedding and iframe inclusion in feeds if they source their content from certain blessed sites. However, again, this behaviour (the content of the white list) varies between online feed reader services.

One of the promises of HTML 5 is the ability for more and more application functionality to be delivered via standard browser interpretable mechanisms including javascript, SVG, the canvas element and so on. However, if online feed readers are cautious with this type of dynamic embedded content, then the blogging channel is going to be stunted when it comes to innovations in discourses that involve non-textual elements.

As Google Reader is the dominant player, they would, ideally, set up some mechanism to establish the validity and reliability of sources of such dynamic content, thus allowing all legitimate contributors the means to publish via feeds. Otherwise, dynamic data providers are forced to use Flash or similar runtime based inclusions and not the HTML 5 stack.

How Big is That Really?

The BBC site Dimensions (with the url howbigreally.com) takes well known objects and extents (e.g. the moon, floods in Pakistan) and projects their geometry onto a map. This provides an interesting way to help understand our world - both natural and manmade.

Here's an example : the moon versus Australia.

Transparency is not Open Data

I recently wrote a short post outlining my thoughts on the distinction between open data (the term generally associated with the behaviour of governments and other public bodies providing declarative and programmatic access to various data sets) and transparency (the behaviour of governments publishing honest accounts of their behaviour without restriction).

In reviewing a few of the open data sets that BuzzData linked to on their recent post about these open data portals, I noticed that some of the data is provided in formats such as PDF. While it may be transparent to publish data in this format (PDF is a document format with reasonably high market penetration) the data that they contain is obscured by the purposes of the document encoding. PDF is designed to make documents look good, not to separate out the content abstractly from the format. Consequently, the data - while being distributed for human consumption in an accessible manner - is not open in the sense of being in a declarative or programmatically accessible representation.

Data Engine Roundup #6

• BuzzData has a blog - and while they haven't yet posted much about their product, they have a couple of posts on open government data and in particular the reality of the new Canadian portal.
• DataMarket and others talk about Data 2.0, held on April 4th in San Francisco. The speakers list reads like a who's who of people speaking at Data 2.0.

Read more data engine posts here!

Building Data Discourses - Second Attempt

A bug in the embedding script prevented the embedding of d8taplex data from functioning correctly for a while, so if you were bemused by my earlier post on data discourse, please give it another try: Building Data Discourses.

Comparing US State Economies to other Nations

The Economist has an interesting map which labels each US state with the name of a nation state with comparable GDP.

 

The visualization can be flipped to present population comparisons in the same way.

 

Target Maps : Social Spatial Data

Information Aesthetics points to Target Maps, an online tool for uploading and sharing spatial data.

 

By looking at the embedding and visitor statistics, the site can surface maps that are getting bursts of attention (like the Japan radiation levels map).

 

Building Data Discourses

[Please note that while the feeds for this post correctly display the embedding, both Google's feed reader and Bloglines appear to strip out the embedding code - I'll investigate when I can. Meanwhile, if you are reading this on a feed reader, please click through to the permalink.]

I've written before about embedding - and some initial experiments on d8taplex with code for embedding svg graphs, with data drawn from the site, directly into web pages. In further playing with the data, and slowly getting to the point where I can actually start writing about insights in the data itself rather than the site per se, I realised that there was the potential for more subtlety than I had originally thought.

Certainly, I can embed a graph like any other object, thus:

However, just as d8taplex allows you to explore and modify the data (go on, hit the expand button on the graph and find out), by implementing embedding in a certain way, I can programatically get the same effect. The graph below replicates user interactions with the data simply by calling the operations on the graph from the embed code. Here I have selected some of the lines, then removed the unselected lines via the 'keep' command.

In addition, I'd like to be able to embed data, such as the rise in sea level in Liverpool, as a sparkline . This can all be done over the same data set (the data is cached by the embedding mechanism and subsequent uses of the same data set are pulled from the cache).

I can even embed the fully interactive version of the data (each version is described by a simple tag) inline. Hit the 'smooth' button, then the 'hide' button to get to an altered version of the data with smoothed instead of original values.

What I'd like to explore with this is ways in which the data can be more closely woven with the discourse and still interactive so that readers can poke around themselves.

Data Engine Roundup #5

Recent news for the data engine shed:

• Kasabi completes it hackday and starts preparing for its private beta with developers. You can register for the beta at kasabi.com. See Leigh Dodds ruby client for the Kasabi API on github.
• Socrata (here in Seattle) announces hiring Charles Blanchet as VP of sales.
• Olga Pavlovsky at Timetric writes a post on "reinventing business research".
• As noted by many bloggers, Canada rolls out its open data portal.

Read other posts on data engines.