Data Mining: Text Mining, Visualization and Social Media

Exploring Time Series Data

In a couple of recent posts, I've included some experiments on using SVG and javascript to render time series data. I've been doing a little more in this area recently, and put together a site which provides access to a small sample of official UK statistics and provides mechanisms for exploring this data using time series graphs.

In addition to the rendering and interactions shown in my original post it provides a few more options to manipulate the data (e.g. hiding certain time series, smoothing the data, summing and averaging).

The site is called d8taplex. Take a look and let me know what you think. Note that currently it only supports Chrome and Safari.

The Drunkard's Walk

I've just finished reading The Drunkard's Walk by Leonard Mlodinow. Initially, I was concerned that this was yet another lightweight review of the basic concepts of statistics and probability, but Mlodinow skillfully delivers the conclusion in a way that leaves you feeling you have a deeper understanding of everything from personal choices to media hype.

The key topics that are discussed are:

• The history and sometimes uphill struggle of the mathematics of chance - a branch of human understanding that had to battle with the tyranny of superstition and religious control.
• The basic tools and big ideas of statistics and probability, from the law of large numbers to the normal distribution to Bayesian inference.
• The many demonstrations of our evolutionary bias towards finding patterns when in fact there is only randomness (confirmation bias, survivor bias).
• The profound importance of understanding that with large experiment sets, even with random outcomes someone will always appear to do well and 'beat the odds' (the hot-hand).

In addition to bewailing the lack of data literacy in our media and more importantly the part of the population that consumes this media, I'm now going to bewail the lack of randomness literacy (although surely there is a better name for it).

What are the Odds?

I'm walking down a street in Bangkok and I bump into a colleague from my place of work in the UK. I'm hiking in the hills near Kathmandu and meet someone from my birth town. A friend take a flight and ends up sitting next to the mother of someone in his daughter's class at school. What are the odds?

When we think about calculating the probability of this happening, by factoring in things like 'what is the probability that the mother was taking this flight', 'what is the probability that she would have the seat next to me', etc., we often think that we are dealing with extremely small probabilities. But like the famous birthday pairs question, this overlooks an important part of the coincidence.

The odds to be calculated are not the odds of the specific incident, but the odds of something remarkable happening. In other words, I would be equally likely to report as 'amazing' sitting next to the mother of my child's classmate, sitting next to someone from my home town, sitting next to someone who bought a lamp at my garage sale, etc. The space of things that we would consider remarkable is actually far larger than we think, especially when we are confronted with a specific case.

NationMaster

Briefly, Search Engine Watch points to NationMaster, a stats service. How about slurping this in to Swivel? A key question: copyright around statistics - how does it work? As far as I know, statistics are facts and thus can't be copyrighted.

Careful With Those Stats, Ms Stewart

Sometimes I keep posts hanging around in my feed reader for ages - I intend to write something on them and they eventually fade away. One of these which I am now finally getting to is this post from the Freakonomics blog regarding the occurrence of middle names in reported crimes. I bookmarked the post as I thought that the reasoning behind the post was flawed. Essentially, the post states that there appear to be many crimes reported and then published (in a newspaper) perpetrated by people with the middle name Wayne. The tongue in cheek conclusion to the post is - tell your daughters to reject any romantic advances from people with this middle name.

This is a very unfortunate post to be turning up on the Freakonomics blog, and is a good example of the way in which folk statistics get passed around in the public forum with little real backing, explanation or analysis.

The data reported is a list of crimes perpetrated by people with the middle name Wayne. What is missing is the total number of crimes reported and the popularity of the Wayne as a middle name. And that is just for starters. I'm still puzzled as to why such a poorly thought out post of a statistical nature would appear on a blog for such an acclaimed author/book.

[Imagine some extreme cases: everyone has the middle name Wayne, 10 crimes are reported out of a population of 10, 000, etc.]

The Danger of Precision

Don't ask me why, but I've just sat through one of Michelle Malkin's amateur news casts (and I use the term amateur in its most commonly accepted sense, not the recent warm-fuzzy sense). She goes on and on about one of her favourite topics: racial profiling is good. The point she attempts to make is that we should all be suspicious of young Muslim males because they are the root of all terrorist activity. Her strategy to do this is to parade a series of terrorist activities before the viewer and point out that they have been enacted by young Muslim males.

Her statements may be precise, but as she makes no attempt to look at the population of terrorist activities or the distribution of ethnicity, gender and age of the suspects, they are meaningless.

Let's imagine that there are 100 terrorist plots. 90 of them by old ladies from Skye and 10 of them by YMM. I could certainly say: look here are 10 plots, all of them by YMM - viewer beware! However, I'd be far more efficient in nailing terrorists if I zipped up to Skye and interviewed all the grannies.

In general, when 'facts' are presented through channels like this, one doesn't have much real hope of finding out if they are true or not. However, one can always inspect the form of the statements and the context in which they are presented and determine if one is being presented with some precise, but highly biased, (potential) information.

(Of course, Islam isn't a race but a religion.)

The Curse of the Quiz

waI heard on NPR that the doomsday book (a unique inventory of England from 11 Century) now has an online manifestation (I'm linking even though the site appears to be unresponsive). The report was followed by the quip '2% of Brits think the Doomsday Book is a novel by Dan Brown [author of the Da Vinci Code].' That got me thinking - how did they come by that? The answer is, of course, that there was some quiz which looked like this:

What is the Doomsday Book? Is it

1. A census of England from the 11 Century?
2. A rule book for a board based role playing game?
3. A novel by Dan Brown Author of 'The Da Vinci Code'?
4. A deth metal band?

When you here such an assertion, you really need to hear 'of the self-selecting respondents to an online multiple-choice question which had no 'other' category, N%...'

Update: My guess isn't so bad, according to the Free New Mexican:

The book is one of Britain's best-known documents, but a poll commissioned by the National Archives suggests not everyone is sure what it is. While 80 percent of respondents had heard of the Domesday Book, 13 percent thought it was a chapter in the Bible _ and 2 percent thought it was a book by Dan Brown, author of the hugely popular "The Da Vinci Code."

One of the cool things about analysing online social media is that one isn't trapped by such silliness - we can just read what people say.