Data Mining: Text Mining, Visualization and Social Media

Dealing With Real Time Media

I attended a panel yesterday (at the Personal Democracy Forum 2008) on the live web (Robert Scoble, Bhaskar Roy, Max Haot, Keith McSpurren). A couple of observations about streaming live video, tweets, etc.

1. The production of these streams leaves little room - or time - for self editing.
2. The consumption of this data leaves little room for filtering - why would we consume real time data yesterday?

One problem that I see with the tools we currently have for handling this data is that they will follow the path of email. Email clients rapidly evolved to the 3-pane approach to consumption and never went anywhere else. There are no real analytical tools (except, perhaps, Xobni) for helping us deal with email.

I see an opportunity in this space for user facing tools that leverage the advances in social media analysis (including text mining, network analysis, etc.) to help us summarize and select the data, making the data more relevant, and the consumption more efficient.

Politics, meet Social Media Analysis

In preparing for speaking at Personal Democracy Forum 2008 I spoke with a number of colleagues in the social media analysis space asking if they were engaged with any of the presidential or primary campaigns. I didn't hear of any who were.

There was a panel in the morning session today in which a number of staffers from the various campaigns present and past talked about the tools and strategies of campaigning. It was apparent from their answers that they didn't use any real social media analytics systems. They did talk about dealing with lots of email, and building internal systems to help with that. Again, it didn't sound like they were using any text mining tools to help with this.

So, it seems like there is a big opportunity for companies in the social media analysis space, but there is also something of a disappointment in hearing that the politicians are not exploring what can be done with these technologies.

Personal Democracy Forum 2008

I'll be in New York next week speaking at Personal Democracy Forum 2008. The title of my talk is 'When Worlds Collide: Social Media, Mainstream Media and Politics. In it I will discuss the value of large aggregates of social media data as a lens for understanding the political process and landscape. The presentation is very visual and I hope to have some version of it online soon. Readers of this blog will much of the content, but there is some new stuff in there as well!