Data Mining: Text Mining, Visualization and Social Media

Tracking Microsoft Buzz with Blogs, Twitter, Bitly and Videos

Microsoft is an incredibly diverse company. I've just celebrated 5 years here and still don't have a full appreciation of the breadth and depth of products and innovation that the corporation generates. After BlogPulse was unplugged, I felt something of a hankering to continue to follow the buzz around Microsoft, partly as a way to better follow what the company is doing and how it is perceived in the online world.

I'm a big fan of TechMeme, but it has some challenges when it comes to tracking news and trends around a specific company. Firstly, I don't know the sources that are used and the ranking mechanisms in place, so it is hard to really understand quantitatively what it represents. Secondly, with limited real estate, while a big story may be happening for a company of interest, it can be crowded out by other events. Thirdly, I can't help but think it has a strong valley culture bias. Fourthly, it hasn't evolved much in the years that I've been visiting it.

So I've put together an experimental site called track // microsoft which follows a few blogs, clusters posts that are related and uses Bitly and Twitter data to rank the articles and clusters of stories. In doing this, I observed that many posts in the blogosphere about Microsoft would contain videos (be they of Windows 8 demos or the latest research leveraging the Kinect platform).

The site has three basic columns. The first contains established stories, represented by clusters of articles. The second represents a more timely view of posts. Both of these columns use Bitly and Twitter statistics to rank, with a bias to recency. The third column shows videos which have been embedded in posts multiple times.

Thus far, I find the stories and videos that surface here to be very interesting. This is where I first learned about:

• Microsoft's site for the new Hunger Games movie.
• The dynamics of Nokia's presence in the UK cell phone market.
• The release of the iPlayer on the Xbox (how cool is that!)

The nice thing about track // microsoft is that it should be easy to repurpose it to follow any topic - all you need is a list of feeds. I'm hoping to put together pages for taiko, skiing, Kauai, ...

Please take a look at track // microsoft and let me know what you think.

Techmeme's Retrospective

The Techmeme blog has an interesting retrospective of the <somehow ranked> stories of 2009. I've plotted a selection of them via queries to BlogPulse.

A couple of stories that were missed here relating to Microsoft are the launch of Windows 7 (peaking higher than Google Chrome) and the Bing iPhone app (peaking higher than the Crunchpad fizzle).

Spam Kills Pings?

I’m not sure when this content came up on Technorati, but it is interesting to see how the rate of garbage in pings forced them to switch to simply poling the blog feeds directly (which is exactly the behaviour that pings should have addressed).

Welcome to the former ping page

As of the relaunch of Technorati on October 14, 2009, we are no longer using pings; you no longer need to inform us each time you update your page. We are now automatically checking sites frequently to find new posts, without anything (except the post itself) needed from you.

Why the change? More than 90% of the pings we received were spam and non-blogs. A significant amount of our energy was devoted to attempting to find, without fail, the valid blog pings obscured by the torrent of junk. This endeavor demanded an unreasonable amount of resources (both automated and human) while remaining fragile and error-prone.

A pure polling method cannot ensure real time data (assuming that standard crawler politeness is adhered  to). Perhaps CloudRSS and PubSubHubub will really be the future.

Comparing Twitter and Blogosphere Latency

With all the bzzz around Twitter, and its apparent ability to walk on water right up to the brass ring of search, I’m still looking for the definitive study comparing discovery times in Twitter with those in other forms of social media. The study is pretty easy to perform:

1. Grab some Twitter data and extract some URLs
2. Use Twitter search to find the first mention of those URLs
3. Use a blog search engine to find the first mention in the blogosphere
4. Vice versa

This news article Defiant N Korea launches rocket first appeared on Twitter here 5 hours ago, but the earliest I can find it in the blogosphere is here 4 hours ago.

Endangered right whales appear to be on the rebound appeared a full day in the blogosphere before Twitter.

Neither the blogosphere nor Twitter cares about Ice bridge ruptures in Antarctic, and it seems neck-a-neck for Russia to unveil spaceship plans.

Of course, one would have to do a bit more work to really see what is going on. Twitter search doesn’t dereference URLs, so you can’t really search for the first mention of something, never mind the whole can of worms called URLs (the BBC usually has 2 URLs per story…).

And which stories do you want to check for anyway? Those that snowball into big piles of links may be less important (some signal will reach the user anyway).

Twitter search’s trends currently include ‘North Korea’, but then again, Google’s blogsearch has a top cluster for the same topic (and using the title of a post ‘North Korea launches rocket’ is more informative). Twitter also includes ‘DSI’, but Google has ‘Console Review: Nintendo DSi’.

Blog Hosting Statistics

Pingdom has published some interesting statistics regarding the use of different blogging platforms and technologies within the Technorati top 100. There are interesting not just due to the basic distributional data (Wordpress accounts for 27 of the top 100, MT for 8, Typepad for 16, Blogger for 3) but also in what the report reveals about how Technorati computes the top 100. For example, Weblogs, Inc. accounts for 14 of the top 100, Wired for 9 and Gawker for 9. I'd be interested in looking at the sister linking between sites in these networks.

Another part of this signal is to do with comment spam. The more popular the blog, the more important it is that the authors have control over spam.

Google Blog Search Changes

I’ve been noticing some weird behaviour in Google’s blog search recently that makes me suspect that are testing new features and possibly a new back end or index. Firstly, I noticed changes between different attempts to search for the same item. There was a significant reduction in the number of results with many legitimate posts missing. Secondly, I’ve started to notice hits for search terms that are not in the post. For example, a search for “political streams” brings up this post from Blog About Stats which doesn’t mention the phrase, but which (as of the time of writing) has it in a title of a post under it’s ‘Recent Posts’ list. The strange thing is that this feed isn’t partial. I had originally thought that Google was attempting to fill in partial feed data and getting it wrong, but this doesn’t seem to be the case.

Interactive Visualization of Blog Search

Briefly, an interesting blog (via Twingly) recording progress in a project to create a new user experience for blog search.

Blogging is alive and well

This month’s Wired has an article title ‘Kill Your Blog’ – it’s a great article. Great, that is, as an example of poor writing, logic, journalism, etc. It’s written by Paul Boutin for Valleywag, so it may be complete fiction. The basic theme of the article is: blogging has been overwhelmed by corporate content (that, or blogs have become corporations), so little-guy blogging is a waste of time – you will never be heard.

Keeping with the great tradition of blog-o-journalism, Paul uses examples of Jason Calacanis and Robert Scoble as evidence of what is happening in the blogosphere. This is like comparing Larry Ellison’s mega-yachts to a second hand row boat you are thinking of buying on cragslist – i.e. not an exemplar for the population.

Boutin also uses the Technorati 100 list as evidence for something being wrong with the blogosphere, and random, obnoxious comments left on posts as reasons to stop writing. The attitude seems to be – let’s have everything stay the same and wait for bit rot to set in then claim the space to be done with.

Personally, I found the article to be a perfect list of reasons to do something really useful with social media: again, Google’s blogsearch is the thin end of the thin end of the wedge, as is Political Streams.

Sigh, I know Wired’s there to sell copy and one way to do this is to be crazy and controversial – but this is just lame.

Animated Weblog Diffusion

As part of my presentation at the Personal Democracy Forum 2008, I showed an animation of a simulation of diffusion through the blogosphere. Anthony and Guilhem (also presenting at PDF) have just released a version of this on real data over on their Presidential Watch 08 site. They talk about it here, and the two demonstrations can be found here and here.

The visualization is an augmentation of the existing graph based view with a new display/control showing the attention over time to the video that is the subject of diffusion.

Google Blogsearch Update

I wrote on Monday about Google Blogsearch and their graphs. It looks like some events do produce more interesting temporal signatures. Below is the graph relating to the divorce of Madonna and Guy Richie: