Data Mining: Text Mining, Visualization and Social Media

Free, Blogs

The Blog Herald comments on the Free skirmish between Gladwell and Anderson by pointing out

The blog is alive and kicking, if nothing else but because it is hard to pick critics and arguments to pieces in 140 characters or less.

I wrote quite a bit about the book when it was being formulated. Now, I’m looking for the (free) online version but haven’t found it yet. Chris/Wired is known for erring on the side of sensationalism to move units (as is appropriate in the media business), so I wonder what his investment in this new thesis is.

Measure, Don’t Guess – Growth in the Blogosphere

Charles Arthur writes a piece about the slow demise of the blogosphere. Arthur asserts that bloggers are a fading breed, and that

they've all gone to Facebook, and especially Twitter.

Arthur claims to have come to this conclusion via a mixture of anecdotal evidence, and data provided by Technorati. Let’s do our own experiment to see if the blogosphere is fading. Let’s take a very mundane search term – one that we expect to be a constant background in the sea of celebrity death buzz, hi-tech launches and liver transplants : ‘car repair’. As we can see from the Blogpulse graph, it is pretty stable with a few blips here and there:

Blogpulse plots the percentage of all blog posts on this topic. If the blogosphere were dying, the absolute counts would also be slowly reducing (even if the percentages were staying the same as the graph shows).

On Jan 4th, 0.026 of posts were on the term ‘car repair’. This translates to 142 posts (Blogpulse allows you to click through to see the number of hits). On June 21st, where there were 0.027 % of posts on the term, Blogpulse registers 144 hits. Ok, I don’t really see any slacking off there. What happens when we look at more data points? If we do this for ‘car repair’ and ‘birthday’ we get the results below. Here I’ve normalized the values by the percentage of posts (count/percent - the trend shows values for 1% of all blog posts as an artifact of using percent * 100). To my eyes, this looks pretty flat – there is a very slight downward trend, but it could easily be in the noise.

Was Charles Arthur going for a Wired-esque sensational piece?

The Long Tail of Text Mining

In systems that execute inferences via a pipeline of steps, every step is an opportunity for failure. Therefore, it is imperative  that implementers focus attention on the details of every step. For example, in text mining, systems have to

1. Import and parse documents – did you get the title? did you recognize the footers? did you strip out the page numbers?
2. Identify sentences and words – is the document in a latin alphabet language? are there word separations? are you dealing with acronyms? how is your unicode-fu?
3. Provide part of speech tags for the words – is the text an example of the type of data that the POS tagger trained on?
4. Identify entities – are you prepared to identify unusual names like Barack Obama?
5. etc.

I’ve seen a couple of attention to details bugs surface in the past few hours. The first was reported by Danny Sullivan, in which Google (and this is still the case at the time of writing) thinks that Michael Jackson the writer is the most salient person with that name.

The second is visible on WeSmirch, in which the system fails to identify the title of Lisa Marie Presley’s blog, naming it ‘Create Free Blogs & Online Journals on MySpace Blogs’:

Attention to detail will always be a killer feature!

Steve Irwin, Michael Jackson

When Steve Irwin, the famous crocodile hunter, was killed by a sting ray, 5.5% of the posts on that day in the blogosphere (September 05, 2006) mentioned his name. Yesterday, mentions of ‘Michael Jackson’ topped out around 3%. Things are heading north from there today – currently around 8% – but note that the more immediate statistics come with a higher margin of error due to the lower sample size.

Twitter Trending Terms – Could Do Better

Twitter has oodles of data – millions of tweets a day. They have smart people working on this data, and they make all the right noises about social search. However, I’m looking at the trending topics on the site just now and I see these: MJ’s, Rip MJ, RIP Michael Jackson, Farrah Fawcett, #iranelection, Pop, Thriller, MTV, Iran, #michaeljackson.

I don’t get it. There are a number of problems here:

1. These aren’t topics, they are words or phrases. There are only 4 topics present here (Michael Jackson’s death, Farrah Fawcett’s death, the Iranian Election and MTV).
2. The ‘phrases’ present in the terms are pretty lame: ‘MJ’s’? Earlier today the phrase ‘Did Michael Jackson’ was a trending topic.
3. There is no attempt at normalization (RIP MJ == RIP Michael Jackson)
4. They actually are not at all interesting – anyone out there not know about Michael Jackson, Farrah Fawcett or Iran? Perhaps the MTV thing is a little more obscure.
5. They aren’t trending – ok, the RIP stories are, but the Iranian election? that’s been top of mind for many days now.

Twitter’s featuring of these topics and the immaturity of the technology in spite of the promise of the data does not paint a good picture for their prospects.

Search User Interfaces: Marti Hearst

Marti Hearst - pre-eminant in the fields of text mining and user interfaces - yesterday published her book on Search User Interfaces. I intend to write up a review of the content later, but for now heres a summary.

The book, which is available freely online,  describes itself thus:

This book presents the state of the art of search interface design, based on both academic research and deployment in commercial systems.

and covers topics including the design and evaluation of interfaces, and visualization of search results.

Coincidently, the launch of the book is timely with the recent launch of Bing. A big part of the strategy behind Bing has been to provide a better user experience around search, offering affordances to help the user succeed in their task right there in the interface. Marti's book, which was completed prior to Bing's launch, doesn't cover this new search engine, but I'm sure the second edition will!

Marti is blogging about the book at SearchUpTicious (!)

Social Business Design

Kate Niederhoffer, and her colleagues have posted about her groups move into a space they term Social Business Design. In a sense, what they describe is a reaction to the siloing of thinking around social media brought on by the term ‘media’ itself. This is, perhaps, an extension of my dismissal of the term ‘consumer generated media’ due to its origination in the world of marketing and advertising. Kate and friends are making the point that the social part of the future of online ecologies is not just about ‘media’ per se, but about the entire end-to-end, front-to-back dissemination, assimilation and synthesis of information and its integration with business processes and models.

While this is an interesting direction, it reminds me of terms like ‘democratization’. As anyone in a large organization can attest, capitalism doesn’t scale through democracy. Good companies succeed via Philosopher Kings. The idea that social effects will permeate a large organization without accommodating hierarchy is a little far fetched.

Niederhoffer et al’s thesis centres on four core notions. As Kate puts it:

• Ecosystem - a community of connections
• Hivemind - the socially calibrated mindset of individuals
• Dynamic Signal - the constant multi-faceted means of collaboration
• Metafilter- a method of finding signals in vast amounts of noise

The later two are clearly about information flow. Hivemind (which I’ve always taken to mean the aggregate psychological process of a group, not that of an individual moderated by social context) is also about information diffusion and the assimilation of social cues in moderating that information and republishing.

Perhaps the key to understanding what this topic is about is to understand the scale at which is intended to be implemented. I’m excited to follow where their thinking leads and what impact it will have on reshaping existing businesses and forming new ones.

Bing, Google: The Features are there, the users don’t know it

Danny Sullivan writes a sympathetic piece describing how Bing’s feature set has prompted Google to more formally promote its long list of similar features. The reality is, of course, that Bing has helped surface features that have been in Live Search for a while as well: on hover vide preview, Farecast price prediction, …

Mashable also covers this angle.

The Realities of the Search Audience

Son: Have you tried out the new Microsoft search engine?

Father: I didn’t know they had one.

Mother: Oh – I knew – it’s called Chrome!

Son: …

Father: I don’t think we are on the cutting front of this stuff.

Mother: Don’t you mean the cutting edge?

Son: …

I overheard this conversation – during a personal phone call.

Blind Search

If you’ve not yet taken a look, visit blind search – a site created by (Microsoft employee) Michael Kordahi. It allows you to query Bing, Google and Yahoo! and to see the results tabulated without knowing from which provider they are from. There are a few posts out there (e.g.) that talk about the relative merits of the stats that the site generated (they are not available just now). However, if you want to take the brand out of the equation, it is interesting to see how often a non-Google result looks the best.