Data Mining: Text Mining, Visualization and Social Media

TailRank, Spinn3r, TechMeme and TechCrunch: New Attention

Kevin has released new versions of TailRank (2.5) and Spinn3r (2.0). A new version of TailRank mean a new dose of Arrington's criticism. What is going on here is the use of one piece of technology (TechMeme) to make assumptions and criticize another (TailRank). Michael Arrington compares TailRank with TechMeme in the following way:

• Assume that they are trying to do the same thing (surface news that is hot right this minute)
• Assume that they are using the same algorithm to compute this
• Compare the performance of the back end (grabbing data from the blogosphere in real time)

Ok - now I'm making assumptions, but my point is that TailRank - while it is obviously doing something similar to TechMeme - is not TechMeme. Look at the name: TailRank - it is ranking articles based on the attention given to them by the long tail. Much of this attention is due to interest sparked by the cycles that happen in the blogosphere and how things are passed along (compare this with TechMeme's approach which is based more on immediacy and weights given to 'important' blogs).

To Michael's criticism that the news is old, Kevin points out that if attention is being given to any news, what matters is that that attention is being given right now, not what the date of the news is. For example, if the blogosphere suddenly linked to an archived story about the launch of Sputnik, that would be old news but new attention.

I actually wrote about this issue along time ago when TechMeme initially appeared with features similar to (but different, of course) those we had developed on BlogPulse.

The News Width

One of the basic properties of old world programmed media was the fixed time slot. This was a particularly defining feature of the news. There had to be exactly 30 minutes (or whatever) of news to fill the slot. If that day was slow, the news was padded with light stuff, if that day was hot, then the light stuff was dropped as were other moderately important features.

You would think that that approach to media would be one of the first to drop online. We may, perhaps, be forgiving of the online presence of mainstream media - there is a fixed real estate on their front page. However, having become aware of the issue, I'm less forgiving of Web 2.0 aggregators including: memeorandum/techmeme, TailRank and even our own BlogPulse. Take techmeme for example. Stories are ranked according, in part (I'm guessing) to how many citations they get from which bloggers. However, on one day, a story with citations from A and B may appear on the front page whereas on another it may not - depending not on how important that story is absolutely, but how important it is relative to all other stories.

I say that we should be looking for interfaces to information that reflect how important that information is and which don't persist artifacts of the very media that we are (apparently) trying to escape.

Understanding Alexa Reach Stats

I recently posted an observation about TailRank's reach statistics. Kevin corrected me on the interpretation of the Alexa data confirming that TailRank had been experiencing significant month over month growth.

Greg Lindon recently posted visitor stats for Findory. We can compare this graph with the Alexa reach graph.

Alexa's reach stats are per million - delivering a normalized account of the number of visitors, essentially reporting the proportion of visitors for a site from the total online population as predicted by their sample base.

However, the graphs still don't seem to add up. The last quarter for Findory has seen less growth than previously, however, I doubt it would result in the clear downward trend in reach as displayed by Alexa. The Findory stats are hits, not visitors, and include robots. If we assume robots are a constant and that the number of pages viewed by each visitor is constant, then we can assume that the graph does track the growth of users. This does not fit the Alexa data, which can only be explained if, in Q1 2006, there was sudden growth in the online population with a bias to those not inclined to visit Findory.

So what's going on? Is Alexa reliable?

Popular Video Ranking on TailRank

Kevin, in a comment on this blog, points out that TailRank actually had video ranking first, they just didn't publicize it well. Have a look at http://tailrank.com/posts/filter/handle/_video. There are some noticable differences. TailRank looks at more than just references to YouTube and includes at least Apple and Google Video - though the data is swamped by YouTube. Technorati's service has a shorter window giving a view of the last 48 hours compared with TailRank which has a window of several days (at least 5).

Looking at the current results:

TailRank	Technorati
The Simpsons Movie - Teaser	PeCa bio hazard highlight
danah on O'Reilly Factor	Real Life Simpsons Intro
Pictures of Assholes	guitar
Brandon Hardesty's Re-Enactment Series	El Koala - Yo Via Jase Un Corra
Gamers Getting PWND	Leprechaun in Mobile, Alabama
President Bush's Nephew Interviewed in Iraq (what if?)	Bush Was Right
Leprechaun in Mobile, Alabama	Unreal Gamer
The Promise - Trailer	dani california
Bill Gates: "We need microformats"	Nintendo Sixty-FOOOOOOOOOOUR
United 93 - Featurette	Gintama first commercial

It looks like there are considerable differences in the data - probably due to different crawling strategies - only two of the videos are found in each list.

[Previous post on this topic is here.]

TailRank Update

TailRank released some new features on the 24th of March, most importantly, sub-sections on specific topics. Can you guess which ones? Of course, tech and politics - the core duopoly of the blogosphere. Looking at TailRank's Alexa numbers, it looks like the honeymoon is over and it is going to be interesting to see how Kevin transitions it from the next new thing to a major player in the whatever-this-space-is-called space.

A similar decline can be seen in memorandum's reach stats - what does this suggest for the space?

Note that digg.com does not have the recent decline that these two sites show.

TailRank Update, Part 2

I've just played with TailRank's auto configure feature for weblog importing (see previous post). It certainly did something, however it looks like it imports blogs at a higher level than I'd like. For example, included in the list of detected blogs was 'http://typepad.com', 'http://myspace.com' which, as far as I can see, means that I then filter on any blog from these hosts. That is way to general.

Hopefully this isn't a limitation from the technology side and something that can be fixed or controlled in the future. My guess is that it is simply a bug.

TailRank Update

I'm not sure what is more satisfying - the fact that TailRank just updated with a load of very cool, social media features, or that fact that I predicted it would using my proprietary 'important people don't blog when they are doing something very interesting' blog monitoring method.

I've not yet checked out all the updates, here is a summary of the details on Kevin's blog:

• Weblog Auto Configuration: previously, TailRank asked you to import an OPML file of blogs you visited. Now you can ask it simply to monitor which blogs you read and it does all that for you. Kevin has attacked a key barrier to unlocking some of the potential of TailRank, viz: the need to upload files that you may not even know how to create.
• Weblog Recommendations: one of the biggest gaps in the blog portal space - finding blogs, not just posts. This is based on the population of the data gathered above.
• Recommended Reading Lists: sounds similar to the above - not quite sure yet what the difference is.

TailRank Update?

Based on a research method I first shared with Jonathan Carson, I'm wondering if TailRank is gearing up for something big - I have'nt seen much on Kevin's blog recently...

HeadRank

When I first heard about TailRank, I thought that it was going to be an application that drew attention to the tail, rather than one which used the tail to rank other content. It turned out that TailRank really is a neat site, but I wonder if it could be turned around: partition the blogosphere into the head and tail, and then use links from the head only to rank posts or blogs in the tail.

TailRank

Kevin Burton has been hard at work on TailRank for a while now. Not only has he been hard at work developing and improving what is turning into a very novel application of the social dynamics of the blogosphere, but he's also been hard at work inventing a new approach to micro-funding a startup.

I have to admit, for a while I wasn't really grokking TailRank. This was, I believe, in part due to the user interface, which was a little scattered. Recently, however, Kevin has done a couple of things that make the whole concept a lot easier to deal with. Firstly, the interface has changed a lot. It is far cleaner and easier to look at. Secondly, there is now a neat little bookmarklet which lets you share things on TailRank while browsing.

I'm still holding off on a deep dive, but wanted to write about the main feature that I currently use : the link recommendations. There is not a huge amount of difference here with other systems like del.icio.us, however, I find the interface just a little more approachable - clicking on the share button next to a link to add tags and share them.

Something that would be interesting to see would be an ability to track how many times a link is found in TailRank - or any other recommendation system - and then blogged about. This might be done by the use of a specific tag, or better, the ability in the TailRank interface to open a post editor for your blog with the link already created (a little like TypePad's QuickPost capability). TailRank could then measure its contribution and influence on the blogosphere.