Data Mining: Text Mining, Visualization and Social Media

Visualizing Lexical Novelty in Literature

Novels are full of new characters, new locations and new expressions. The discourse between characters involves new ideas being exchanged. We can get a hint of this by tracking the introduction of new terms in a novel. In the below visualizations (in which each column represents a chapter and each small block a paragraph of text), I maintain a variable which represents novelty. When a paragraph contains more than 25% new terms (i.e. words that have not been observed thus far) this variable is set at its maximum of 1. Otherwise, the variable decays. The variable is used to colour the paragraph with red being 1.0 and blue being 0. The result is that we can get an idea of the introduction of new ideas in novels.

In the first book - Austen's Sense and Sensibility - we can see two things. Firstly, the start of the book keeps a pretty good degree of novelty for the first few chapters. Secondly, each chapter introduces something new.

The second book - Stevenson's Kidnapped - shows a different pattern. While it starts off with reasonable novelty, this then dies out for most of the book with spurts of interest here and there.

What is surprising to me (if we take any real meaning from this approach) is that Austen's Emma - the third book - is strong out of the gate (the first 18 chapters) but fails to break the 25% novelty ceiling thereafter.

[Note that these results are preliminary and I'm going to do more validation and testing.]

Update: see below the original visualization for an updated version with more accurate results.

Update: After looking at the above results I drilled down on the strange behaviour in Emma. It turns out that Emma as multiple volumes within which the chapter counter reset to I. Consequently I was picking up chapter titles (I, II, III, VI, V, etc.) as novel terms the first go round and this was driving the visualization. I've since modified the algorithm to firstly ignore text blocks (paragraphs) with fewer than 5 words and secondly, given it a more dynamic colour scheme.

This improvement still highlights some key differences (again, in as much as the algorithm is correct). However, these differences are now somewhat changed from the first set of observations. Note also that the threshold for novelty has been decreased to 0.1.

Lexical Growth in Various Books

An interesting evaluation of any text is the probability of encountering a new word as one progresses through the writing. In the chart below, I compare Romeo and Juliet, Pride and Prejudice, War and Peace and A Child's History of England. It is interesting to note that War and Peace has a very similar trajectory to Romeo and Juliet, that Austen is clearly below this curve and that a book aimed at children is the one in which one is most likely to encounter novel terms. This later insight might be attributed to the fact that historical documentation likely includes a continuous stream of new characters and locations where as fiction tends to focus on a limited number of both.

In the below chart, the vertical axis indicates the size of the vocabulary and the horizontal axis represents progress through the book (i.e. words read).

Received: Pete Warden, Data Source Handbook

Pete has shared a copy of his new e-book with me: Data Source Handbook:

If you're a developer looking to supplement your own data tools and services, this concise ebook covers the most useful sources of public data available today. You’ll find useful information on APIs that offer broad coverage, tie their data to the outside world, and are either accessible online or feature downloadable bulk data. You'll also find code and helpful links.

Review to follow.

Received: Frank Rose, The Art of Immersion

Frank Rose, who writes for Wired magazine (and with whom I authored this article on information flow in the blogosphere) has just published a new book called "The Art of Immersion":

Not long ago we were spectators, passive consumers of mass media. Now, on YouTube and blogs and Facebook and Twitter, we are media. And while we watch more television than ever before, how we watch it is changing in ways we have barely slowed down to register. No longer content in our traditional role as couch potatoes, we approach television shows, movies, even advertising as invitations to participate—as experiences to immerse ourselves in at will. Wired contributing editor Frank Rose talks to the people who are reshaping media for a two-way world—people like Damon Lindelof and Carlton Cuse (Lost), James Cameron (Avatar), Will Wright (The Sims), and dozens of others whose ideas are changing how we play, how we communicate, how we think. 

The Drunkard's Walk

I've just finished reading The Drunkard's Walk by Leonard Mlodinow. Initially, I was concerned that this was yet another lightweight review of the basic concepts of statistics and probability, but Mlodinow skillfully delivers the conclusion in a way that leaves you feeling you have a deeper understanding of everything from personal choices to media hype.

The key topics that are discussed are:

• The history and sometimes uphill struggle of the mathematics of chance - a branch of human understanding that had to battle with the tyranny of superstition and religious control.
• The basic tools and big ideas of statistics and probability, from the law of large numbers to the normal distribution to Bayesian inference.
• The many demonstrations of our evolutionary bias towards finding patterns when in fact there is only randomness (confirmation bias, survivor bias).
• The profound importance of understanding that with large experiment sets, even with random outcomes someone will always appear to do well and 'beat the odds' (the hot-hand).

In addition to bewailing the lack of data literacy in our media and more importantly the part of the population that consumes this media, I'm now going to bewail the lack of randomness literacy (although surely there is a better name for it).

How to Measure Anything

I'm very much enjoying reading How to Measure Anything by Douglas Hubbard. The book

• is practical in its approach to measurement, showing that one has to define a thing in order to measure it and that many of the things that people think are hard to measure are actually not well defined.
• defines measurement as a reduction in uncertainty - thus measurement happens when we increase our knowledge about the subject (by either increasing our confidence in the bounds or reducing the bounds at a certain confidence level).
• describes how to calibrate individuals so that they become accurate measurement tools themselves by learning the skill to estimate with specific confidence levels and bounds.
• provides recipes for simple statistical methods for interpreting sample data.

Measurement is a key aspect of development, especially when one is developing solutions for the acquisition and publication of data.

• How accurate is the data that one intends to harvest?
• What impact will this new data have in the application area?
• How much effort should we put in to measuring the quality of the data?

One of the drawbacks of the book is that while it gives some very interesting mechanisms for working with samples, it doesn't actually provide the statistical underpinnings explaining them, resorting instead to instructions involving looking up tables and reading distribution graphs.

Overall, the message is: you can measure (reduce the uncertainty about) any quantity you can define with a lot less effort than you think.

If I Read This Book, Will It Make You Happy

Picking up Connected: The Surprising Power of Our Social Networks and How They Shape Our Lives by Nicholas Christakis and James Fowler, you might be forgiven for thinking ‘another book about goldfish jumping from one bowl to another?’ I’m currently through the first chapter, and while the authors don’t go far enough in terms of setting the scene for their work in terms of basic social network theory and where it comes from (Barabasi is not mentioned until page 265), they do offer some interesting anecdotes about the type of information that passes between the socially connected.

With a clear focus on avoiding the obvious and well trodden paths of those that have gone before them, the authors pull their stories from their own work and other accounts of things like cascades of emotion (someone 3 degrees away from you can cause you to be happier or sadder) and epidemics of laughter (mass psychogenic illness).

I’m enjoying this book (provided by the publisher) more than the previous book that I attempted to review. This may be entirely due to my frame of mind, or being a better fit for the audience. Or – perhaps someone three degrees away from me read it and liked it.

I’ll write more when I’ve finished.

Inside Larry & Sergey’s Brain - Review

(@FTC – I received this book from the publisher)

Richard Brandt’s new book outlines the rise to success of Larry Page and Sergey Brin – the Google founders. The title of the book is, however, misleading. It provides no real insight into the psychology or radical thinking of the subjects, but is rather a flattering account of their trajectory coupled with a sympathetic account of their origins. Missing is any comparative analysis of their competitors, or any criticism of business decisions that would appear un-brilliant (like trying to sell their baby to the first taker before realizing what they were really onto) – missing too is one of my favourite topics: survivor bias

The positive and uncritical approach to the subjects leads to some light weight accounts of their success. For example, page 170 reports:

Google has figured out how to make money off all these new forms of entertainment, while every competitor is struggling with that issue.

Considering YouTube, the first statement is hard to swallow (read technovia’s summary of the $470 million that the site will lose this year). Considering the second, plenty of other companies are making money from social sites, blogging platforms and so on.

One of the distinct advantages of being so slow to finish a book for review is the benefit of reading the reviews of others. Strangely, I can’t find the correct page on Amazon – the page I link to above is to the book with the additional text (Spanish, Japanese, Mandarin Chinese and Portuguese Edition). This page has only 1 review which is a collection of review snippets provided by the author.

Traffick’s review has this interesting paragraph:

As someone who has watched the ad auction and the search products, in particular, emerge, I'm struck once again by just how far behind and how dismissive Yahoo and Microsoft were at various stages of innovation, on key areas like how the paid search auction worked, but also, in how much to prioritize search and paid search in overall company priorities.

This seems to be missing the innovation that was going on at Overture, well known to be the inspiration behind quite a bit of Google’s approach. As TechUser.Net summarizes:

Google always had excellent search engine indexing technology, but Google's search technology by itself never generated profits for the company. Google's profitability comes from its search technology combined with text ads and an ad placement mechanism that allows advertisers to bid for the placement of their ads (bid-for-placement mechanism). From a profitability perspective, the bid-for-placement mechanism is as valuable as Google's indexing technology. In the absence of the bid-for-placement mechanism, ad pricing can at best be inefficient. The bid-for-placement mechanism frees up extensive resources that would otherwise be required to set ad prices, and it allows Google to charge ad sponsors in proportion to the value Google is delivering to the sponsors.

The bid-for-placement mechanism was pioneered by Overture, a paid search specialist company. In July 2001, the US patent office issued Overture a patent covering the mechanism. Patent 6,269,361 also known as the '361 patent was bad news for Google: it threatened Google's core business model. It was imperative for Google to have access to the '361 patent, but Google never managed to negotiate a satisfactory licensing agreement with Overture. Consequently, in April 2002 Overture sued Google over patent infringement.

Brandt has (page 96):

A company called GoTo.com (later renamed Overture) had come up with the idea of an online Yellow Pages system, where users would type in search words and be taken to advertisers who bid to have their ads appear when people searched for those words. (In 2002, after Google started showing the way, Microsoft and yahoo both considered buying Overture. Yahoo won the bid).

Silicon Beat has very positive things to say about Brandt at the expense of Janet Lowe’s Google Speaks.

As a result, Brandt gets a number of things right. Most notably Brandt, manages to avoid easy cynicism, which is the first trap of any newbie reporter or book writer who takes on the company as a subject. And as a long-time technology writer, he knows the difference between cloud computing and client computing and doesn’t let technical details get in the way of his story.

In contrast, the newly published “Google Speaks,” by Janet Lowe, appears to be based almost entirely on other articles, which led the veteran author to such bizarre assertions as the patently wrong claim that Flickr was bought by Google and the more puzzling statement that Hewlett Packard once owned the Altavista search engine. (Flickr was bought by Yahoo, and Altavista was created by Digital Equipment Corporation, which was sold to Compaq in 1999, which was sold to Hewlett-Packard in 2002. In 2003 Altavista was sold to Overture which was then bought by Yahoo. So Lowe is technical correctly about the chain of ownership while simultaneously appearing totally confused about which facts are relevant in Silicon Valley’s complicated corporate histories.)

Well, now I’ve read a good number of reviews of the book (by searching for it on Google, of course) and the strange thing is, like the book on Google, I can’t find anything but glowing, uncritical comments. With Occam’s razor in hand, I have to assume that this is something to do with me, not the book! Perhaps I’m the wrong audience, or perhaps I took too long to read the thing. At any rate, I’ll make two final comments. The first is that I’d recommend John Battelle’s book The Search: How Google and Its Rivals Rewrote the Rules of Business and Transformed Our Culture for those interested in a (slightly dated) account of how we got here. The second is the final paragraph from Brandt’s book:

With Google and their sudden wealth at their disposal, Larry and Sergey now have enormous power to [change the world], and they will continue to do so for decades to come. They’re like Harry Potter after he discovered he was a wizard and got his wand. you can expect great things from them.

Yup.

The Numerati - Stephen Baker

I've just started in on The Numerati by Stephen Baker. It promises to dive into how our lives expose more and more of our characteristics online, and how those bits are being mined for the benefit of the consumer economy. Having read the first chapter, it looks like Stephen writes in a very narrative and evocative style.

The Long Tail Arrives

My copy of Chris Anderson's The Long Tail arrived today - stay posted for a review.