Data Mining: Text Mining, Visualization and Social Media

Consumers of News, and Vegetables

One of my motivations for playing with interfaces to news data is to better understand the relationship between the upstream news creation, selection and propagation process and the downstream news consumption behaviour of the audience (or product).

While the middle classes (in the US at least) are becoming more attentive to where there food comes from and what it is made of (witness how Wholefoods promotion of locally sourced organic produce has forced many other supermarkets to follow suite) the same attention is not given to news.

There are many interesting news aggregators out there (e.g. Google News), but they mostly play the same role (and give the same results) as editors of the front page. It is unlikely that the Google News front page doesn't show pretty much the same major stories as the BBC front page, for example. In addition, they are tied to the decision process encoded in the data that they mine to determine the selection of articles for promotion.

The d8taplex news page sticks with the timeline of the news being generated, clusters similar news content (with a more longitudinal view of the story than most aggregators) and ties in the bit.ly statistics to indicate what users are clicking on (via sharing the bit.ly links via twitter, etc.).

With this simple interface, I can spot news stories of interest to readers but which aren't being broadly passed around by the news outlets being crawled (e.g. the recent earthquake in Japan had almost no coverage but got a lot of clicks). In addition, I can see news articles which are getting a lot of attention from the agencies but with less engagement from readers (e.g. the Pope's visit to Africa).

Eventually, I'm hoping to arrive at an interface that will allow users to both select news of interest, but also be aware of the tyranny of preferential attachment.

Updates to d8taplex News Stream

I've made some updates to the news stream experiment on d8taplex. The page now displays a stream of news articles with clusters of related articles being shown with emboldened titles. Each article also shows the number of clicks tracked by bit.ly. The bit.ly data is coloured according to how hot it is (red represents the most clicked articles, orange represents 'warm' stories and gray represents stories that haven't yet received any significant attention).

With two view of news (the clusters represent what the media companies are interested in and the bit.ly data represents what the weberati are interested in) the stream provides a slightly different overview of the news cycle. One can find clusters for which there is no significant bit.ly data, yet also find stories that are getting a lot of clicks but which don't belong to any clusters. In addition, as each story is annotated with the source (CNN, BBC, etc.) you can also find clusters generated by only one source.

Here's a cluster showing a number of articles as well as a few with significant bit.ly attention:

Note that to expose the additional information for a story - including the clustered stories - you simply need to hover the pointer on the appropriate annotation (Q = quotes, C = countries, G = group / cluster). In addition, if you click on the G it will pin the cluster open for further interaction.

Dynamic filtering is still available (just type in the filter box), and keywords from the selected articles will be populated in the keyword area.

Take it for a spin!

A Simple News Exploration Interface

I've just pushed out the next version of the hapax page. I've changed the interface to allow for dynamic filtering of the news stories presented. You can now type in filter terms (such as 'bbc' or 'greece') and the page will only display those stories that are related to those terms.

Here we see a filter for the term 'greek':

The terms that are used for filtering are extracted from the news articles and include the title, country names, keywords, quoted statements and hapax.

To see these fields, you simply hover over the appropriate indicator below the news article title.

Here, for example, we see the quotes exposed for a story:

In addition, to give an indication of how popular a story is, I've pulled the bit.ly click information for each URL.

Because the number of articles loaded is relatively high and the filtering is instantaneous, this approach allows the user to rapidly develop topics interactively without relying on either automated clustering or having to click through multiple screens to find out what the cluster is.

There are some hidden features to the filtering code which I will write up when I get a chance.

Re-Educating Search Users

Briefly, there are a couple of products out there that are intent on getting search users to change there 2.4 words per query habit. The most visible might be Siri (though recently it was operating at a zero word per query rate due to an outage). But perhaps a more subtle one is 'A Google A Day'. At the site quiz questions are presented which essentially require long form queries (and often more than one) to solve. By creating such a site, Google is indicating that it wants to train users to leverage what it believes are its strengths and what will be viewed as the emerging area of natural language search.