Data Mining: Text Mining, Visualization and Social Media

Data, Statistics and Unbelievers

I was pretty impressed with Obama's inauguration speech. He almost slipped by without using the word 'terrorist' or any of its relations (he used 'terror' once) and he gave respect for science and rationalism a shot in the arm by appealing to data and statistics:

These are the indicators of crisis, subject to data and statistics. Less measurable, but no less profound, is a sapping of confidence across our land; a nagging fear that America's decline is inevitable, that the next generation must lower its sights.

He almost did right by the rest of us by referring to those of us for whom faith is not a concept, although he used a term which centres on faith. One can be sure that every word in the speech had a great deal of thought behind it, and that every word was balanced with its synonyms. We can infer, therefore, that Obama's team considers atheist as being in some way wrong. It is arguably the most accurate term, and so it must be wrong for political reasons. Perhaps 'unbelievers' is a way of being inclusive while still indicating his theological position.

Naturally, Neoformix has the whole thing in perspective.

The Edited Palin: Can Journalism Be Too Objective?

You may prefer ‘objective’ reporting, or you may prefer character driven news-o-ticians. Either way, one thing I’ve noticed more and more is the use of verbatim transcripts including misspeaking and repairs. Traditionally, Bush has been an easy target for this, but Palin is another example. Take this recent CNN article “Palin says she doesn’t regret Couric interview” which includes:

• "Certainly I should have done the interview. And to attribute I think that interview to any kind of negativity in the campaign or a downfall in the campaign, I think it's ridiculous."
• "In retrospect, in hindsight, I wish I would had more opportunities or that we would have seized more opportunities to speak more,"
• "However, the campaign is over. That chapter is closed. Now is the time to move on and make sure all of us are doing all that we can to progress this nation."

Now, does Palin speak with perfect ineloquence? or does the media report too literally what is said? Couldn’t this have been reported:

• "Certainly I should have done the interview. And to attribute I think that interview to any kind of negativity in the campaign or a downfall in the campaign, I think it's [is] ridiculous."
• "In retrospect, in hindsight, I wish I would had more opportunities or that we would have seized more opportunities to speak more,"
• "However, the campaign is over. That chapter is closed. Now is the time to move on and make sure all of us are doing all that we can to progress this nation."

All Eyes on Obama

Obama has the attention – will he get the vote? Of course he will!

Visualizing Tax Policy

William’s been doing some nice work on visualizing the tax policies of the two US presidential candidates: here, here, and here. For readers not in the US, note that the term ‘socialism’ is considered a scary word over here, and – at least in the mainstream media – the frequency with which a pundit uses it is in indirect proportion to the speakers understanding of its meaning. Note also that there is an idea over here that taxes are a form of punishment (at least, this is how Joe “Wurzelbacher” the Plumber expresses his relationship with this basic instrument of government).

Political Streams Update; ACORN Boating

We experienced a hiccup early this morning, but the site is back up – hopefully this wasn’t too disruptive.

While we don’t provide any supported API’s right now, you can persuade a trend chart out of the site. Here’s one showing the attention around ACORN.

Political Streams: Update

We’ve just rolled out some fixes to Political Streams. Among them

• Improved extraction of news article titles: we do pretty well at this using a generic system, but were failing in some specific cases.
• Fixes to trend graphs: due to some issues with our content store, our trend graphs were a little suspect. The trend graph are now providing an accurate signature for the terms they show.
• Improved classification: a lot of what drives our system is the ability to classify content either by type (news articles, weblogs posts, …) and by topic (politics, etc.). The system is set up to allow incremental improvements moving forward.
• UX improvements – paging on the home page now works with the real time pushed updates.

I’m going to be keeping a sharp eye on the trend graphs now, especially tracking how Obama and McCain attract attention.

Another thing to note: because we use the most excellent Freebase for our semantic content, we can take advantage of updates made there. For example, here is the page for Joe “the plumber” Wurzelbacher.

Animated Weblog Diffusion

As part of my presentation at the Personal Democracy Forum 2008, I showed an animation of a simulation of diffusion through the blogosphere. Anthony and Guilhem (also presenting at PDF) have just released a version of this on real data over on their Presidential Watch 08 site. They talk about it here, and the two demonstrations can be found here and here.

The visualization is an augmentation of the existing graph based view with a new display/control showing the attention over time to the video that is the subject of diffusion.

Political Streams Released

Take a look.

Palin’s Linguistics

Kitty Burns Florey has a nice little piece, making the assumption that the sequences of words coming out of Sarah Palin’s mouth are, in fact, sentences in the English language.

Baby Palin Brings Home the Bacon

Having failed to better Obama’s level of attention, the McCain campaign finally goes over the top and bursts through by revealing that Palin’s 17 year old daughter is 5 months pregnant.

Palin is, for the moment, getting more attention than either Obama or McCain!