Data Mining: Text Mining, Visualization and Social Media

Perceptive Pixel: Update

I'm very late to this. When I first heard about Perceptive Pixel, I was very excited. Several videos of the very slick, very fun wall-mounted multi-touch based interface zipped around the blogosphere last year. But then there didn't seem to be much going on. Meanwhile, a number of companies started popularizing some of the elements of that interface, Apple with the iPhone (multi-touch), and Microsoft with the surface computer to name two obvious examples.

It turns out that Perceptive Pixel were in fact deploying their product. I don't watch broadcast tv so I had missed the fact that it was being used as a differentiator in discussing the primary elections in the US.

Here's an example.

Here's another one:

Worldwide Telescope

Microsoft Research's Worldwide Telescope is available for download. I'd encourage you to go and take a look. I would have been writing up some details of it for this blog, but it is so easy to spend time exploring the sky and the planets that I'm left with no time right now! Note that it does provide planet models (including earth) as well as astral data.

The Trending Trend

Dan Taylor at fabric of folly writes up a summary of a number of trending tools (particularly for social data). I've not yet blogged about Facebook's Lexicon, so check it out in Dan's post. Dan's blog looks pretty interesting in general - subscribed!

Freebase and Data Visualization

This is a nice post [via Cool Inforgraphics] which uses Freebase as a data source to create an animation charting the growth of Wal-Mart over time. Toby says:

Freebase has a topic for every zip code, along with it’s longitude and latitude. Here’s one example. One query pulls out all the ZIP codes along with their longitudes and latitudes. You can turn longitudes and latitudes into graphical coordinates with some simple transformations (which will vary based on the region you’re plotting and how big your image is) — here are the ones I used:

x=(longitude+127)*16
y=(50-latitude)*20

If you plot all the ZIP codes using a library like PIL, you get a nice map with dots that roughly match population density, which has the advantage of looking a little bit like a night-time satellite photo of the United States.

Freebase also contains a list of Wal-mart locations, along with their addresses and the year that they opened. Here’s an example. One query pulls all of these out of Freebase.

Map of Iranian Blogosphere

Nathan points to this interesting paper on the structure and topics of the Iranian blogosphere. I've not yet had time to read it, but wanted to pull out this visualization for starters.

New York Times Movies Graphic

Many people have pointed to this interesting visualization in the New York Times which shows data describing movie hits (and misses) over a long time period. Neoformix also has a nice composition which stitches the entire line of data together to form a complete picture. But there are some question marks against this type of view. Take a look at the sample below.

The problem - to me - is that absolute height has weight here. So it looks like National Treasure: Book of Secrets is really, really important. National Treasure's first two weekends brought in $45MM and $65MM ($110MM) where as I Am Legend brought in $77MM and $33MM ($110MM). This problem, however, is something of a have-cake-and-eat-it issue. The graphic is primarily intended to show the aggregate view of the entire space. It then uses in situ mechanism to allow the user to browse down to the details (thus making comparison hard, but achieving the overall goal).

GapMinder: The Opportunity

There were a couple of sessions at CIFOO which related two areas that I believe are very important. The first was a session on GapMinder/Trendalyzer - the software that Hans Rosling uses to give illustrate and motivate his wonderful presentations. The second was a session on data visualization.

GapMinder is the birthplace of Trendalyzer, a piece of software that Google acquired recently. GapMinder has three great assets: a data collection (publicly available statistics describing economic, environmental and other national and cultural measures), some software (which is basically a scatter graph with a temporal aspect - elegant, but quite simple) and a personality (Hans Rosling).

In the session on data visualization, I brought up a number of issues which I believe get at a central question: how do we go beyond the list as the only interface with the huge wealth of information that can be found on line. These included:

• The user is lazy - users will discover the simplest path of least commitment.
• Value of the web cannot be expressed as a list.
• A user's investment in an interaction is related to the expected quality and value of results (thus, if you have low expectations, you will only suffer close to immediate response times, but if you have high expectations you will be prepared to wait longer for a response).

So what relates these two sessions? I believe that the Trendalyzer software is a great example of the type of step we need to make to better educate people about the power of visualization and the power of data types other than lists of documents. It is a single idea, a single way to look at things and such focus is a powerful way to keep the attention of the user and remove possible distractions and complexity.

Ola Rosling, who presented the project, had something of a tough time answering questions about the direction and plans for the system, however. Will corporations be able to upload their data and visualize it? Is the system intended purely for (high school) educational purposes? I assume that there is actually a well thought out plan for the system, but the Googlification of the project may make these objectives less than transparent at this time.

One clear answer that Ola gave which, I'm afraid, indicates a real lost opportunity, is that Trendalyzer will not involve any community aspect. The system does allow for the sharing of links that reconstitute a specific state of the interface, but there won't be any Many Eyes style community. There are a number of reasons why I believe this to be a mistake:

• Community will help to build attention and thus lead to increased visibility and educational opportunities.
• Community will help refine existing data (those pesky 'official' stats may be less than true) and identify new sources of data.
• Community will help develop the tool.

GapMinder and Trendalyzer have always been very cool things - I hope that somewhere there is a strong vision guiding them forward to even more ambitious heights leveraging their new home.

(BTW, I am aware that the GapMinder organization is separate from Google - it is just the software that was acquired).

Game Visualization

I posted recently about an article in Wired which presented player state at different time slices in Halo maps.

I'm currently playing Half Life 2: Episode 2 and recalled that Information Aesthetics had linked to similar maps from that game. SteamPowered has a nice set of stats for Valve games. For example, it shows the distribution of completion times for HL2: EP1 and HL2: EP2 (the modal time seems to be 4 hours for EP1 and 5 for EP2). The maps, of course, are the most interesting - showing the distribution of deaths by location. This is the map I'm currently playing:

Debate Transcript Analyzer

Neoformix has been posting a number of interesting applications recently. Check out Jeff's analyzer for debate transcripts.

Refer to the image below. The top section shows the distribution of some selected words within the text across a 'timeline' which goes from left to right. Each speech segment is the same width and the height of the small white bars show the number of occurences of that word for that segment. You can add new words with the text box in the top right corner or you can remove existing words by clicking on them.

Right below the word distribution graphs is a similar coloured set showing a spectral decomposition of the text based on who spoke and how much was said. In this case the bar heights give the amount of text for each segment. Click and drag the mouse left to right to move along the timeline and show the actual text for 3 consecutive segments. Mousing into this lower region will cause the blocks to expand and show more text.

JC Herz and Completely Useless Visualization

I've just returned from attending - and enjoying - the first Defrag. I'll probably write a number of posts, but the first one I want to write is on the topic of Visualizing Social Media - the subject of my talk. I'm not actually going to write too much about the talk itself, but rather on the presentation that preceded mine, given by JC Herz.

JC Herz opened with some comments - mirrored in my own presentation and generally understood by the visualization community - about the distinction between graphics that look attractive and graphics which deliver value. Fine - I completely agree with this. However, she then went on to use one of my visualizations - specifically, the map of the blogosphere which I produced a year or so ago - as an example of this refering to it as (approx) "completely useless". This was a cheap shot. Firstly, I suspect that she had an idea of what she wanted to say (how network diagrams are far too common and often not well understood or applied) and then did a simple image search online for an example. However, she didn't take the time to read anything about the work behind the visualization, or the comments of the author. Secondly - and this is where her comments highlight a more worrying weakness in her approach to this field - she presented one of her own pieces of work which, taken out of context as mine was, confronted the viewer with far more ambiguity and lack of meaning. This was demonstrated at question time when her quasi 3d image of a social network of soldiers spread over time with their communications rendered in musical notation (you wonder why it was hard to understand?) was puzzled over by the audience.

JC Herz finished her presentation indicating that she was working on something aimed at the upcoming US Presidential election that would have an impact in Washington DC. Hopefully she will have learned to add context to ensure that that impact is well understood. Meow!