Data Mining: Text Mining, Visualization and Social Media

Unrest in the Middle East, Social Media, Text Mining and Trending Topics

In the early 2000s, after the collapse of WhizBang!Labs, I joined Intelliseek as part of the group of researchers who formed the ARC. In a fit of hacking and inspiration, we created a simple blog tracking site called BlogPulse. We actually launched the site without informing our bosses, which led to some interesting dialgoue.

Back in those days, blogs were thin on the ground (I remember first hearing of them from an early advocate - Fernando Pereira). Instead of crawlling RSS feeds, we simply crawled the front page of the blog and did a diff on the content to extract the new material. We then took that content and ran it through some simple but (generally) elegant keyword and phrase extraction technology to determine, on a daily cadence, the hot topics that people were spending their attention on.

The internet archive's first capture of the site is dated Apirl 8th, 2003. Keyphrase from that day were as follows:

On that date, we  crawled 31, 926 blogs, which contained  12, 954 new entries.

BlogPulse went on to innovate with what at the time was possibly the most popular time series presentation of search results (I don't believe we were the first, but we were copied, in at least one case with an exact replica of our HTML showing up on a competitors site).

Time series of topics, keywords extraction and trending, they are both still with us and going from strength to strength. Of course, the media has changed, though I find acclamations of the death of blogs to be generally exaggerated.

It is interesting to note that unrest in the middle east is still a key focal point of all media. It is ironic that as the frequency at which we live our lives, and the frequency with which we produce, consume and mine social content continues to increase, the big picture topics are constant: free speach, british troops, american troops, rubber bullets, ...

I Work For a Giant

It's interesting to see an article on BlogPulse's top news stories for yesterday about Intelliseek/BuzzMetrics (and by implication, BlogPulse). It's also interesting to hear that BuzzMetrics is referred to as a 'giant' in this space:

To capture the chatter, Nielsen BuzzMetrics, a giant in the industry, uses software that collects hundreds of thousands of comments a day. The technology can scan for specific companies, products, brands, people -- anything searchable. It can slice data into a range of categories to quantify the number of times a subject was discussed online, the individuals who mentioned it and the communities where it appeared.

Hundreds of thousands is probably an underestimate. Also the 'anything searchable' is an undervaluation of our technology. The expectations of search are far weaker than what true text mining, nlp and categorization technology can do. Anyway, nice article.

Under the Radar

The BlogPulse team believes very much in nurturing researchers in the area of blog analysis. This is why we made a data set of blog content available in conjunction with the upcoming Workshop on Weblogging Ecosystems. Recently, there have been a couple of projects launched that use either BlogPulse or Intelliseek data. One is BlogsLikeThis - a system that helps you look for blogs on certain topics. The second is a piece of art/data visualization called The Dumpster which is generating quite a bit of buzz. I find the conversation around the later to be quite fascinating. Having observed the effort required to create the data that backs the project, the superficial 'this is cool' commentary that it receives is, quite probably, the online equivalent of the few seconds an art museum visitor spends on each piece as they walk, without pause, through the Louvre.

ARC Featured in the Post-Gazette

There is a nice article in today's Post-Gazette about Intelliseek's Applied Research Centre - of which I am a member. It weaves a pretty good story about our origins as well as the main proposition behind our products: listening to what the world has to say via online content. My only quibble is the focus on blogs - certainly a worthy component of what we do, but dwarfed by the volume found in other online user generated media sources such as boards. That's me on the far left.