Data Mining: Text Mining, Visualization and Social Media

Where Does Weblog Traffic Come From? Email

By looking at referrer statistics of weblogs, we can get some idea of what drives traffic to blogs - see my previous post on this topic which includes a description of the data. It is interesting to mine this data for referrals from web mail programs. Here is a quick look:

This graph was produced by extracting yahoo mail, google mail and hotmail URLs. Following that, all other URLs which mentioned 'mail' in certain places were extracted and the host part of the URL (http://foo.com) were ranked by frequency. The most frequent domain (comcast) was then included in the data. The most obvious result of this is the dominance of Yahoo's web mail system. It is also interesting to note that Comcast, a cable company, is competitive with Microsoft for webmail. Note that this data doesn't include mail referrals from the live.com domain.

Search Doesn't Work: Story 3

I come from a small town - a very small town - in Scotland called Elie. A search for 'elie, fife, scotland' on Google Maps produces a very strange result - a point in the middle of Elie Road outside St Andrews and a long way from the village. The same search on Google produces a page about the village. The mapping system's assumptions about intention - even though it interfaces a far more structured data set - appear to be wrong whereas the combination of relevance and static ranking in the document search system produces an appropriate response.

The problem here is that search engines (or Google at least) has no real ability to interpret queries. The paradigm of the main stream search engine is to treat everything as tokens (strings of characters) and to rely on behaviour of such things as page rank and statistical models of relevance to conjure results that appear to be meaningful. Because the interpretation in the mapping system and the document system are different, in this case it fails to produce equivalent interpretations of the same query (and the same intention).

The solution lies in (can you guess) natural language processing.

See here for more on search problems.

Real World Text

The great thing about working in NLP in an industrial setting is the exposure to real world data as opposed to the more academic 'John loves Mary' type of thing. In reviewing posts on the horrible Edelman/Wal-Mart story, I came across this I'm-not-making-this-up sentence:

Certainly, if you don’t post about it, I’m personally not going to not respect you for not doing so, however others may.

Duncan Riley is saying something here - not quite sure what.

Sentiment mining in the social media monitoring space requires high precision and recall. Recall is important because a single post from a respected source needs to be captured accurately. In other words, it is not the number of false negatives, but who they are that really hurts you.

Social Structure and Opinion Formation

I've recently read the above titled paper authored by Fang Wu and Bernardo A. Huberman. The paper explores the activity on a network under a set of update rules designed to simulate opinion formation. Opinions are either black or white (in the model, colouring the nodes in a graph either black or white). Opinions are updated by a function which takes in to account the colour of neighbouring nodes.

While this paper has some interesting theoretical points and results, I think that the assumptions which link it to opinion formation in the real world are very weak. Firstly, an opinion is not really defined. This is a problem common in the literature. Is an opinion a linguistic construct? Is it a belief that requires no articulation or resultant action? In the written world, especially the blogosphere, it is trivial to pick up some snippet of text and regurgitate it with little or no consideration or analysis - is that a person's opinion? Secondly, I'm not convinced that the influence of other's on my opinion can be modeled in such a direct manner - by examining the opinions of my neighbours and updating my opinion. However, I think it would be simple to update the model with more complexity (e.g. factors that captured a notion of trust, or alignment which would impact the potential to influence).

The paper is certainly worth a read, but the search for a theory of opinion goes on...

Just Because You Can Doesn't Mean You Must

Update: On TechMeme and TailRank (better coverage).
Update: Jonathan Carson, CEO of BuzzMetrics has a post on the topic.
Update: BoingBoing gets in on the story.

David Armano of Logic + Emotion brings up the issue of the request for no blogging at the recent Nielsen BuzzMetrics social media event. I find this line of thinking severe. Even more extreme are some of the comments, including this from Graham Hill:

Puzzling isn't it. And just a wee bit arrogant. Who do they think they are?

The customers present should just ignore NBM and simply blog what they hear if they really think it appropriate.

Some of this talk is starting to sound almost fundamentalist: it is your right to blog any and all conversations that you are in. This attitude ignores the value and intimacy of private conversation.

US Population Density Visualization

I saw  this graphic  in Time magazine on this morning's flight and was happy to see that it was also available and interactive on their site. 3d histograms are always a little tricky - the problem being the artificial prominence that perspective gives to 'nearer' locations. However, aesthetically, this is a pleasure to look at.

The End Of Influence

In his post summarizing day 2 of the Blog Business Summit, Mitch over at BuzzLogic brings up the issue of influence measurement and how tricky it is. We've been working with influence for a while at BuzzMetrics and one of the most important issues that we've discovered in this process is that of naming. If you are doing link counting, you should call it link counting, not influence. If you are doing traffic measurement, you should call it traffic measurement not influence. Etc.

Like any problem, influence can stumble at the definitional stage. The general approach to this problem is to decompose - what are the elements of influence? I suggest that a good place to start thinking about this is to pull apart at least:

• Authority: the expertise level of the individual.
• Credibility: the trust of the readers for the individual and the manner of presentation.
• Network measurements: link counting, traffic stats, graphical analytics.

I've just rattled these off, and they require further exploration, but the point is that by decomposing the issue in to smaller measurement problems, we can reduce the confusion around terms like 'influence.'

Enron Explorer

Charles Armstrong, CEO of Trampoline Systems,  pointed me to Enron Explorer - a technology demonstration. Email data is hard to publish and share due to obvious issues of privacy. Consequently, it is hard to create data sets that can be used openly for text mining, natural language processing and other language and social research and that can be passed around the scientific community. Thankfully, the boys and girls of Enron came to our rescue and, via their prosecution, inadvertently provided public access to a huge amount of email data.

Since then researchers have been annotating this data, running experiments on it and publishing papers.

Enron Explorer provides a number of interfaces into the data. One can browse the mailboxes of individual employees, search the data set, explore the data visually and (one of the most interesting applications) provide commentary on the juicy data you've found.

The visualization, shown below, provides a localized graph view of an individual - showing their immediate social network as recorded by the emails they have sent and received. The graph uses the thickness of the arcs to indicate the strength of relationships.

I like the visualization for the way in which it doesn't try to show the entire network - which wouldn't allow one to view the detailed information about an individual. However, there is a little too much animation going on in it for my taste, When the user clicks on a new node (person), the whole thing whizzes around finally landing in a new configuration. The transition could have been less complex and consequently less confusing.

Interactive Post Browser

Neoformix has just posted an interesting applet which allows the user to browse the link structure of the blogosphere via an interactive visual map. Worth checking out - though I found it a little sluggish on my machine. This is in some sense complimentary to my post on an interactive map of the blogosphere.

How Not To Approach The International Market

Technorati has been producing a video blog for a week or so now. I haven't commented on it yet, but this edition forced me to say something. Before I say that thing, I would like to say upfront that the video blog has been clearly improving day by day as they find out what they want to do, discover their own style etc (e.g. learning that cutting off the top of Aaron's head is probably not good style).

Anyway, in the vlog entry for October 20th 2006, Aaron talks about the somewhat dated story regarding Bo Stefan Eriksson (I say dated as I read about it a couple of months ago in the print edition of Wired). For some reason, Aaron thinks it appropriate (witty?) to read the blog in some sort of non-English (German?) accent. Now, I'm not a prude when it comes to cultures and humour (at least, that is what I tell myself when I have to listen to people attempting a Scottish accent here in the US), but I'm pretty sure that a company that wants to embrace the European market in the social media space is probably not going to want to do it in a fake accent as Aaron does here.

Ok - now back to watching Borat on YouTube...