Data Mining: Text Mining, Visualization and Social Media

Chris Anderson and the LogNormal Tail

KDD 2007, which I very much enjoyed, ended with a great invited talk from Chris Anderson. The talk was essentially a summary of his book, The Long Tail, but with a bias towards describing the details of the data rather than the consequences of the vision. One key update that was present in his talk was a discussion of the relationship and differences between power law distributions and lognormal distributions. In fact, Chris admitted that he wasn't really sure if the phenomena he observed and described in his book as having power law distributions really did have these behaviours or if they were actually better captured by a lognormal model. However, he recovered graceful by pointing out that for 50% of the distribution (from the head) there wasn't much difference and so much of the conclusions in the book would hold.

Chris has actually posted on this topic in detail on his blog (a while back).

ICWSM is full of papers with power law distributions in them. However, it should be noted that it wasn't uncommon for presenters to be questioned on the fit of the data to such a law and to have it be suggested that they may in fact be lognormal distributions.

The Economics of Anticipation

This post is really a fraction of a longer story (which, in true blogosphere tradition I may never write). The basic thought is that movies rely on building anticipation and getting a pop in the first two weeks to satisfy their investment (or investors). The pop is important as it minimizes risk - or rather, it is highly informative as an indicator of getting a return and how big that return is going to be.

What could disrupt that model is: less marketing, fewer screens but longer runs. This would challenge a key element of the pop model in that some percentage of viewers are there because of an information asymmetry - the information that is hidden is essentially the quality of the movie. Longer runs could only be achieved by better quality.

Below is a graph of two recent sleepers (ok, the first - The Illusionist - is a true sleeper whereas the second - The Painted Veil - might be a sleeper). I've also graphed the discussion around the dvd for The Illusionist as it comes to market today. I suspect that discussion around a dvd while a movie is still at the cinema is potentially an indicator of reaction to the movie as it may capture individuals' intending to buy the dvd after seeing the film.

As a final thought. Discussion about hits and the coming or going of the hit often mistake the pop for a hit. In addition, they also mistake specific channels and metrics on those channels as the indicators of a hit. To me, a hit is something that is widely popular regardless of when or how it is consumed. Marketing can force a hit, or even suppress one if it aims for a pop.

Don't Get Confused by The Long Tail

Glenn Fannick of Factiva posts about the long tail of the blogosphere:

Factiva Insight is offering content for our clients to mine from nearly 2 million of the most-read blogs (among lots of other content). We often have the discussion about the value of continuing to add more blogs from the end of the long tail. One argument goes that adding blogs that are read by EVEN FEWER people can't be good business. (It costs money to process more content and there has to be a point of diminishing returns.) But at least one person, Chris Anderson, who has authored a blog and a new book called The Long Tail, and an article by the same name in Wired, would disagree. He argues that there's gold in that tail. (Listen to him on NPR's ATC.)

Models like Netflix and Amazon show that there is money to be made when you continue to add lots of esoteric content because there are buyers for seemingly every last title. (Something like 98% of Netflix's huge list of titles have been rented at least once.)

While I respect Glenn's appreciation of the long tail phenomenon, I think this post is somewhat confused. I find it convenient to divide the blogosphere up in to 2 major categories: the head, which contains blogs like BoingBoing, Slashdot and probably a few thousand others; and the rest. When someone from the head posts about a bad experience with Dell, Dell needs to read the signal and react. This is a matter of influence.

The mining application of the rest is somewhat different. There are two important areas: early rumour detection, alerting etc.; and aggregate mining. It is of huge value to get all those facts about why people dislike Hummers, or like Pepsi, or Coke or what have you - the more the merrier, there is no diminishing return. The more data, the better the analytics. It is why we can produce charts like the following and make assertions based on them

Chris Anderson talks in his book about many examples of long tails. There is a particular kitchen gadget that is usually only sold in 2 colours - the company discovered that online there was demand for 50 colours. A long tail of colours. The blogosphere is not like that - there is no analogy. There is not a ranking of bloggers going off into the sunset, each representing a different metaphorical colour of blog, each with its own niche consumer (aka reader). It is a network with participants, structure and complexity.

This confusion is also why I believe there is a problem with the iconic 'long tail' distribution: see my earlier post on the topic.

Visualizing the Long Tail

Part of my write-up review of Chris Anderson's 'The Long Tail' will include a discussion on visualizing the long tail (current delay: looking for my copy of 'New Rules for the New Economy' by Kevin Kelly). One of the intriguing things about the book is the popularization of a concept which is essentially a distribution. However, I believe that the simple hockey-stick-on-its-side, visualization - while a convenient iconic reference - actually fails to capture the long tail. The long tail, is actually doomed to be off stage, to the right.

In playing around with the data I used for my post on blog traffic, I realised that the humble treemap is very good at visualizing the long tail. All that stuff to the bottom right (globally and hierarchically) is the long tail. There is no need to scale, zoom or crop the image - everything is represented.

Careful with Those Stats, Eugene

On page 187 of Chris Anderson's The Long Tail (I'm talking about the book but linking to the blog), a familiar graph appears. It is Technorati's ranked list of sites showing a mixture of MSM and blogs. The claim here is that there are some blogs (BoingBoing, Engadget, PostSecret, Dialy Kos) whose popularity rivals the MSM sites (New York Times, CNN, etc.)

There is a fundamental problem with this stuff. A while back, most likely due to issues with RSS, Technorati switched from claiming how many blogs they crawled, to how many 'sites'. The data that Anderson presents suggests that these blogs rival the MSM sites in terms of counts of inlinks from 'other sites.' Does he mean web sites as we know them or these new 'sites' that Technorati counts?

For example, the New York Times as, according to Google, around 14 MM references. Technorati counts approximately 0.5 MM. BoingBoing, on the Technorati graph, is bordered by Wired (G: 1MM, T: 0.1 MM) and boston.com (G: 5MM, T: 0.08 MM). BoingBoing itself has 0.07MM hits on Technorati and 0.4 MM on Google.

So when is a site a site a site? (snowclone!)

Are Movie Hits Coming or Going?

Chris Anderson discusses the hit that is Pirates of The Caribbean 2.

Well, first of all I've always said that Hollywood was going to be the last bastion of the blockbuster (most of the hits-in-decline examples I've been focusing on are in music and TV), so I'm not surprised that the hit-making machine still manages to fire off a few sparks.

If you take BoxOfficeMojo's data ranking movies by their weekend takings and plot the number of hits per year, you get a view of the data that suggests hits are not simply a phenomenon that is clinging on, but something that is increasing rapidly.

This graph actually shows the number of hits per month per year. I've normalized the values to account for the fact that we are only half way through this year.

The graph is a little tricky to interpret as it represents the number of entries in the ranked list of weekend takings and so may simply be reflecting the number of movies released or some other bias (which I actually don't think it does). Chris points out that revenues on movies are in decline. If this is the case, and record breaking weekends are on the increase, that means that Hollywood is getting more and more accurate at hit making.

There are plots of different data which may clear up this apparent contradiction - please suggest what they might be.

Finally, here is the obligatory BlogPulse graph:

Some additional context: Mixing Structured and Unstructured Data, Lion v Ape. Note the percentage of blog posts on the different movies from these older posts.

The Power Law Bible

A few days ago, Chris Anderson offered copies of his new book 'The Long Tail' for review. I was lucky enough to snag one, so I'm looking forward to receiving and reviewing it. I've written earlier about the long tail metaphor. See also this post on the power law present in community size distribution in the blogosphere.