Data Mining: Text Mining, Visualization and Social Media

Future of News at Princeton

I'm back in Seattle after attending the excellent Future of News workshop hosted by David Robinson and Ed Felten. There were many contributing factors to the success of the workshop, not least of which was the cross disciplinary nature of speakers, panelists and attendees. In addition to participants in my line of work, or with similar areas of interest, such as David Blei (former colleague at WhizBang!Labs) and Kevin Anderson (whose career spans both the BBC, with some involvement with Backstage, and the Guardian, an organization that is not caught with with fretting about the past; and who blogs on Corante, for his sins) I met representatives (and survivors of) traditional newsprint organizations, individuals centrally involved in transforming news in the Web 2.0 world, academics with impressive access to the entire trajectory of media evolution, hackers and so on.

Given the rich range of voices, while we may not have solved any specific problems (despite Ed's closing remarks) I certainly feel as if we aired a reasonably good sample of them. On reflection, I stand by yesterday's summary: there is plenty of pessimism around old media structures and plenty of optimism around the opportunities that new sources and new forms of information, combined with new ways to filter, analyse and aggregate this data presents. Bridging the two positions, there is concern around issues of quality and value with respect to the nature of the content (that is to say, a contributor's ability to provide transparent and supportable content) - will the new information ecology support reasonable ideals for news? Note, to me, those ideals centre on making the reader better informed and more efficient at selecting content.

For other coverage of the event, a great starting place would be Kevin's posts (starting here with an account of Paul Starr's opening talk). Also Steve Boriss, Tim Lee, and Jack Kemp. Note that the presentations and discussions will soon be available online.

Spectra Visual News Reader

Another interesting find from Information Aesthetics. News classes, selected via the top menu, populate a rotating column of articles that are then read at the bottom of the display. Fun - not sold on the utility.

News, Opinion and Efficiency

Jeff Jarvis writes up some thoughts spring boarding from Nick Denton's post regarding the news/opinion divide. At the highest level, this is about the value of humanizing information. There are two related points that I think are missing from this discussion. The first is the value a source of information provides to the user by enabling them to be efficient consumers of that information. The second is a little more complex, and is to do with network effects and homophily.

Efficiency: news sources, or rather, news aggregators, must make decisions about which pieces of news to present to the consumer. In addition, they must figure out how to present this news. Objectivity and the editorial role play in to this by removing distractions and providing a relevance function to the possible set of news items. Opinion - that is to say - removing either or both of these filters - may well lead to a lack of efficiency on the side of the consumer.

Homophily: consumers, being human, are subject to homophily. Thus, the more human/emotional an information source is, the more it will strengthen reading behaviours that are driven by this seeking of like minded writers. With ideal information distribution goals in mind (allowing information consumers to be more efficient and better informed) this will do a disservice to readers.

I think the bigger picture here is to do with trust. If we could trust our news sources, then objectivity and editorial control would be fine. However, the forces that determine what a news source reports work directly against trust as they are financial. Bringing in the emotional element - the personality of the writer - into the picture provides a powerful connection with the reader, thus replacing trust with a personal relationship.

ABC - How to Lie with Pictures

This is off topic, but given that it is about news media and a little about science, I thought I'd squeeze it in. ABC News has an article today, authored by Ashley Phillips, about pollution in American cities. The article is fronted with an image of Pittsburgh (on top of the list of polluted places) in all its smoggy glory. However, the image includes the Three Rivers Stadium. This stadium was demolished early in 2001. So why post an out of date image with this story? Are the images of the other cities somehow incorrect as well? Perhaps the image is doctored to boot.

I'm guessing that 'news' sources of this type have a constant stream of such inaccuracies - more strength to the fifth estate and algorithmic news.

Africa: Nothing To See Here

Infosthetics posts about Yahoo!'s news visualization (another one of those push pin news apps - this time in 3d). Interacting with this display is another reminder of the sad lack of coverage that the commercial western (US) media has for Africa. Read Ethan Zuckerman for some perspective.

The Simpsons

News Explorer

News Explorer is a pretty comprehensive news aggregation site created by the European Commission. It shows a number of aggregate views of recent news, including the obligatory geographic view. In addition, it mines the news data for relational information. For example, it can display the names of individuals in the news, relationships to other people and entities, etc. In this regard, it has many similarities with TextMap, though the later also provides subjectivity anlaysis.

Madeleine McCann and the Media

On Thursday the 3rd of May, Madeleine McCann - 3 years old at the time - was snatched from her unattended holiday house in Portugal while her parents were out of the house having a meal. One is naturally sympathetic to the distress of the parents, but the ensuing media frenzy and outpouring of celebrity concern makes this an interesting case for trend watching.

Google news shows 6, 873 articles as the result of a search in the date range 4-5 May (note, however, that Google shows the exact same number of results for 5-6 May, so it doesn't look like that control is too precise; between 4-20 Math, Google news returns 5, 776 results - WTF?).

Yahoo! news search appears to have better control over its dates. Selectively searching for daily ranges produces the chart below.

Blog search engines can be used to track the popular attention that the story draws. BlogPulse, which shows a normalized measure of articles, produces the chart below.

Technorati's chart gives an idea of the absolute count.

As interest in the story grew, a number of contributors gave money to help with the search. Included in this movement were a number of celebrities including David Beckham, J. K. Rowling (who share another connection). In addition, a site has been put up to help with publicizing the search.

From a trend mining point of view, what is interesting about these stories is their initial slow burn (in the blogosphere). This can be contrasted sudden onset stories as illustrated in the following graph.

In developing technologies that mine signals in the blogosphere, one would like to discover early these slower burn signals and, at the same time, be able to predict how they will grow.   

Financial Times Debuts Trends

Briefly, Guy Valerio (Information Architect, Financial Times), just dropped me a note describing the FT's newest addition to their search interface. Using an idiom similar to Sphere's blog search and Topix's news search, the interface indicates the volume of FT articles (or those from other licensed sources) and allows for manipulation of the graph to narrow the search by date. Below is an example search for 'iraq'.

As can be seen from this example, the service provides a full 5 years of results. It is fascinating to look at examples like 'flu', 'snow', etc. - where there is seasonality, but also some unpredictable spikes.

Latest Backstage Project: BBC Touch

Chris Riley has posted an interesting site to the backstage mailing list: BBC Touch. What the site does is compare the news stories on the front page (which are described as the stories the beeb 'wants' us to read) and compares these with the most popular (most read) stories based on the most read stories RSS feed (this is described as the stories 'we' want to read). On top of this, some key term extraction is done via the Yahoo term extraction API.

The site is definitely fun, however, one needs to have a bit more perspective as to the function of the BBC and its relationship to other news media services. Research by Ethan Zuckerman demonstrated that the global coverage of the BBC news (that is to say, the distribution of countries in articles published by the BBC) is quite different from that for other (US based) media. Note that currently, Ethan is not supporting his Global Attention Profile research page, so you may have to dig a little to get the details. Secondly, one has to be careful in interpreting popularity. The BBC has a clear obligation to  provide informative and broad coverage to the nation (and the world). Guiding news via popularity is in part an admission of news-as-entertainment which is a very slippery slope. Remember the BBC is not a commercial enterprise.

One could almost look at this site as indicating how in touch readers are :-)