Data Mining: Text Mining, Visualization and Social Media

Farewell To BlogPulse

Today, according to the announcement on the BlogPulse homepage, is the day that Neilsen will shutter the longstanding blog analysis and search site.

BlogPulse was originally envisioned by Natalie Glance, myself and other colleagues at Intelliseek as a way to track popular blog posts, people and phrases in the blogosphere. It launched with a sparse set of features in 2003 with the following design:

including key phrases which, at that time looked like this:

The site evolved to become a full blog search engine (I was actually against this at the time preferring to focus on analytics) and launched what was my favourite feature in 2004 : trend search. By that time the site was looking far more mature:

In that year, BlogPulse was named a 'trendsetting product' by KMWorld.

On top of the evolving set of capabilities, we started to publish longer analytical articles including this analysis of the Swift Boat story:

And these views of the 2005 Tsunami:

While (I assume) these assets will no longer be available from their adoptive source, the are archived at the wonderful Web Archive, so if you are feeling nostalgic for a simpler, slightly less caffeinated social media world, head on over and take a look.

One final note: I've always felt that social media analysis, just as it gets comfortable with a particular form of content, gets distracted by a new, shiny channel, abandons the legacy data and starts furiously reinventing its wheels. This certainly happened with the blogosphere when Twitter and Facebook showed up and consumers started to believe that 'real time' was something the blogosphere couldn't provide. From where I'm standing, blogs are alive!

If you have any fond memories of BlogPulse, please drop them in the comments to this blog. If you have generated some interesting trend graphs, send me a link and I'll summarize them in an update to this post.

For the record, the number of discovered blogs reported by BlogPulse right now is 182,397,015.

Unrest in the Middle East, Social Media, Text Mining and Trending Topics

In the early 2000s, after the collapse of WhizBang!Labs, I joined Intelliseek as part of the group of researchers who formed the ARC. In a fit of hacking and inspiration, we created a simple blog tracking site called BlogPulse. We actually launched the site without informing our bosses, which led to some interesting dialgoue.

Back in those days, blogs were thin on the ground (I remember first hearing of them from an early advocate - Fernando Pereira). Instead of crawlling RSS feeds, we simply crawled the front page of the blog and did a diff on the content to extract the new material. We then took that content and ran it through some simple but (generally) elegant keyword and phrase extraction technology to determine, on a daily cadence, the hot topics that people were spending their attention on.

The internet archive's first capture of the site is dated Apirl 8th, 2003. Keyphrase from that day were as follows:

On that date, we  crawled 31, 926 blogs, which contained  12, 954 new entries.

BlogPulse went on to innovate with what at the time was possibly the most popular time series presentation of search results (I don't believe we were the first, but we were copied, in at least one case with an exact replica of our HTML showing up on a competitors site).

Time series of topics, keywords extraction and trending, they are both still with us and going from strength to strength. Of course, the media has changed, though I find acclamations of the death of blogs to be generally exaggerated.

It is interesting to note that unrest in the middle east is still a key focal point of all media. It is ironic that as the frequency at which we live our lives, and the frequency with which we produce, consume and mine social content continues to increase, the big picture topics are constant: free speach, british troops, american troops, rubber bullets, ...

Nielsen/Blogpulse Lose Week's Data

I was playing around with Blogpulse today to look at trends around recent movies and found that 1 week of data (from the 13th of November to the 22nd) is missing. This can be seen in the graph below. The week period with a perfectly straight line indicates that there are no data points between those dates.

 
@Blogpulse - what happened? The charts cycled on the sites home page don't appear to be missing this data. Was there a disc/backup failure?

End of Year Data: Facebook Currently Leads

Like me, John Battelle was underwhelmed by offerings from Twitter summarizing a year of tweets.

Nah. After reviewing them, it's clear that Twitter's first trends release is, well, a bit predictable. But I am sure there is really interesting data locked behind that rather obvious facade....we just can't see it. Yet.

John has similar things to say about YouTube’s offering:

Everyone's doing it! YouTube's most watched came out this morning. Again, nothing that eye opening. Wish they'd dig deeper.

Facebook, on the other hand, while not quite showing the same sort of data, is not afraid to dig deeper. The data teams post includes:

• Proportion of Minorities on Facebook in the U.S.
• Ethnic Breakdown of Facebook Users in the U.S. Named ‘LEE”
• Facebook U.S. Population by Ethnicity
• Relative Saturation of Ethnicities on Facebook in the U.S.

All of these are trended over time :-)

Clearly, the intended audiences are different, but hats off (again) to the Facebook data team.

For those of you who’d like to see trending over topics, here’s a sample from BlogPulse:

 

Who cares about the pandemic, some guy left his flies open!

Debugging BlogPulse

I love to use BlogPulse. I always get a kick out of seeing trends like this:

Or this:

These graphs often have a straightforward story behind them, allowing for a reasonable comparison between mentions of different words. Note, of course, that the graphs show the percentage of blog posts that contain a term, giving a normalized view.

But, what could explain something like this:

Here we see a term 'movie' which appears to have some sort of seasonal trend, dipping in autumn and rising again in the winter. However, the term 'guitar' appears to have a very odd shape, with a dramatic and sharp increase in the winter. Looking at this term on its own, we see:

There is a reason for this. If we look at links to weblogs published on MySpace, we see a matching pattern.

The reason that there are changes in the number of blog posts which link to MySpace blogs is that BlogPulse (Nielsen Online) is adjusting its crawling strategy over time (the above suggests that there was an increase in July, a decrease in September and another increase in October).

So, while I continue to believe that BlogPulse, and the trending tool in particular, are very useful, one has to be careful (and informed) regarding the base data that these analytics are built on.

 

Live Earth Reactions

Some trends associated with the recent Live Earth concerts. Interesting to note the relative association between Live Earth and the bands, brands and celebrities that helped promote the event.

The graph below suggests that Philips, via their promotion of efficient light bulbs, got the biggest jump. Note that the absolute numbers are pretty low, so we can't read too much in to this.

Below we see that bands, obviously, get far more attention than brands.

Finally, the main man:

      

BlogPulse Founder Moves to Google

Some exciting news: Natalie Glance, whom I worked with at WhizBang!Labs, Intelliseek and BuzzMetrics, has taken a new position at Google's Pittsburgh location. Natalie was the driving force behind BlogPulse - a portal to the blogosphere. BlogPulse has been responsible for many innovations (some of which have become mainstream features in a number of other sites) and provided some analytical tools which illustrated the power of social media analytics.

BlogPulse was definitely a labour of love for all those involved. BlogPulse ultimately provided significant value to both our employer and to the blog community in general and it is a testament to Natalie's vision that this skunk works project has continued since its inception over 3 years ago.

Google is lucky to have Natalie on board, and I wish her all the best in her new position.

The News Width

One of the basic properties of old world programmed media was the fixed time slot. This was a particularly defining feature of the news. There had to be exactly 30 minutes (or whatever) of news to fill the slot. If that day was slow, the news was padded with light stuff, if that day was hot, then the light stuff was dropped as were other moderately important features.

You would think that that approach to media would be one of the first to drop online. We may, perhaps, be forgiving of the online presence of mainstream media - there is a fixed real estate on their front page. However, having become aware of the issue, I'm less forgiving of Web 2.0 aggregators including: memeorandum/techmeme, TailRank and even our own BlogPulse. Take techmeme for example. Stories are ranked according, in part (I'm guessing) to how many citations they get from which bloggers. However, on one day, a story with citations from A and B may appear on the front page whereas on another it may not - depending not on how important that story is absolutely, but how important it is relative to all other stories.

I say that we should be looking for interfaces to information that reflect how important that information is and which don't persist artifacts of the very media that we are (apparently) trying to escape.

A Well Stocked Bookshop

An Inconvenient Truth

Technorati's tie in with Paramount to promote Al Gore's 'An Inconvenient Truth' appears now to be active. The news and blogs page contains snippets from blog posts about the movie. Interestingly, the posts appear to be ranked by something other than time (some model of importance/influence). Drilling down it looks that all of the posts mention the name of the film, some link to the youtube version of  the trailer, and others link to the home page of the documentary. As for the issue of dealing with appropriate content, one of the blogs currently linked to from Paramounts site for the movie goes by the title 'Americunt'. I certainly don't have any prudish issues with that sort of thing, but I wonder if the name is palatable for others. Here is a sample from the live page:

BlogPulse provides some additional perspective.

Note that the posts that didn't mention documentary or "an inconvenient truth" could mention other related terms like "global warming". What we would hope to see is the effect of this documentary on the conversation about global warming. Currently, that is not yet visible:

It is interesting to note, however, that in terms of percentage of authors, usenet is the place to be for this issue - not blogs. The graph below shows the percentage of posts in blogs, boards and usenet which mention the phrase "global warming".

Finally, the PR News Wire service has distributed the press release for the Technorati, Paramount relationship, but I've yet to see it show up on the Technorati site. On the reciprocal end, Technorati is currently serving ads for Lions Gate's Peaceful Warrior - expect to see Paramount advertising in this space in the near future. As I mentioned earlier, Technorati recently revised their home page to give more real estate to adverts - in anticipation of the Paramount deal going live.