Data Mining: Text Mining, Visualization and Social Media

Identity - The Philosophical Challenge For the Web

I work in local search at Microsoft which means, like all those working in this space, I have to deal with an identity crisis on a daily basis. Currently, most local search products - like Bing's and Google's - leverage multiple data sets to derive a digital model of the world that users can then interact with. In creating this digital model, multiple statements have to be conflated to form a unified representation. This can be extremely challenging for two reasons. Firstly, the system has to decided when two records are intended to denote the same real world entity. Secondly, the designers of the system have to determine what real world entities are and how to describe them.

For example, if a business moves is that the same business or the closure of one and the opening of another? What does it mean to categorize a business? The cafe in Barnes and Noble is branded Starbucks but isn't actually part of the Starbucks chain - should is surface as a separate entity or is it 'hidden' within the bookshop as an attribute ('has cafe')?

Thinking through these hard representational problems is as much part of the transformative trends going on in the tech industry as are those characterized by terms like 'big data' and 'data scientist'.

Another example can be found in my recent interaction with spotify where I have the option of the following albums from Rush:

The only distinction I can see here is, upon drilling down, the tracks appear to have been published by different sources. They are, as far as I can tell, identical in all ways. There is a slight variation in the cover art that, if you squint a little you may be able to perceive.

As the online world continues to move towards knowledge the Ph in PhD will become more and more useful.

Search By Metaphor

I wonder if there is a place for search by metaphor. For example, consider the following:

• techmeme for skiing : I want new stuff about skiing, ranked by attention from skiing peeps
• youtube for opera : I want video on all things opera and a dedicated community
• facebook for pets : I have too much time on my hands (actually, there is an app on Facebook called Dogbook...)
• etc.

One of the distincations that will come in to play as search evolves on the web will be a lessening of a reliance on useful bit opaque mentions (that is to say, these scenarios will only work if someone actually referred to Dogbook as a 'Facebook for pets') and more of a reliance on actual derived or inferred facts (which requires an understanding of what type of thing Techmeme is and how to recognize that type of thing elsewhere).

msnNow Offers Real Time Trending, Recalls BlogPulse

Microsoft has just pushed out a new product under the MSN brand called msnNow. It is a combination of data mining for trends and topics and editorial content. Surfacing trends has long been a staple for the real time web / social media, but it is an area that I feel has never really been done well. Twitter trends, for example, tend to be less than informative and more of a reflection on the constraints and structure of the twittersphere. msnNow has the potential to take trending seriously, especially if there is commitment behind the product and the contribution of the editorial team.

Parts of the product remind me strongly of the recently decommissioned BlogPulse, for example this module:

 

Take a look!

 

Survivorship Bias in Twitter 'News'

Mashable has a post getting excited about twitter 'breaking' news stories more immediately than main stream sources. This kind of article is misleading in that it gives no consideration to all the other 'news' first reported on twitter which turns out to be incorrect. The main stream media is responsible for fact checking and this idea that it is behind the times because it didn't 'report' something sooner rather than later is annoying.

What is more irksome is the fact that Mashable wrote an article 2 days ago about the unsubstantiated rumour on twitter that the North Korean leader was dead.

In these situations, twitter should be thought of as a sensor network in which some sensors may be noisy.

I note that there is some clever punning latent in the relationship between this post's title and the topic of rumoured death on twitter...

The Mercurial Nature of Reuters News Articles

There were two big news stories that flowed through Reuters and many other media channels in the last 48 hours. The first was the death of singer Whitney Houston, the second was the election process in Venezuela. I noticed something interesting in both stories - the title and contributors attributed by the agency were altered at least once for the content pointed to by the URL.

In the case of Whitney Houston, the first version, which was associated with the name Xavier Briand, was titled 'Singer Whitney Houston dies'. This was later changed to 'Whitney Houston found dead in hotel, age 48' and associated with Mary Slosson, Bob Tourtellotte and Eric Beech.

In the case of the election, the article was originally titled 'Venezuela's Capriles set for opposition primary win' but now goes under the title 'Venezuela's Capriles to run against socialist Chavez'.

While it is of some use to have the URLs recycled so that visitors will always get the most up to date version of the story, I can't help but think that there might be a better way to capture the evolution of the topic in a more explicit manner.

[Note that it is actually a little tricky to demonstrate that this is in fact going on. I have found, however, that with the Venezuela story there are other sites which report the title in the original form with the link that has the altered title. Note also that I'm not stating there is anything wrong with this, it is more of an opportunity loss for the reader. Note finally that it confuses the hell out of Google's visual summary mechanism - the text snippet, image snippet and highlighting conspire to demonstrate how the data changes over time.

Here, for example, is the image cache/preview feature for this Reuters URL from Google:

and here is an screen capture of the same page:

It is practically a different article.

Update: here is another great example - 'Peru says Shining Path rebel leader found dead' replaced by 'Peru captures wounded Shining Path rebel leader'.

High Velocity News in Venezuela (Updated)

Interesting to see how in the past 31 minutes, the news of the political situation in Venezuela has jumped up in terms of bit.ly clicks.

This puts it in the super hot category on the d8taplex mobile news page.

Update: this is the most interesting type of news event, when a change in exposure gives a story a second wind:

Tweets and News - Some Observations

The current version of the d8taplex news site provides a mapping between news article authors and their Twitter accounts. In addition, I've been experimenting with mining all the tweets that link to a specific news article. This has lead to a couple of observations.

1. Reuters contributors with Twitter accounts don't tweet very much. I can quantify this in a later post, but if you look on the d8taplex site at any time, you will rarely see the icon that indicates a journalist has recently tweeted ().
2. Twitter users who post links to news articles rarely contribute any useful hashtags. The version of the site that I have running locally accumulates all the hashtags associated with an article and they invariably tend to simply be the main nouns in the article title. I was hoping for some sort of magic here but it doesn't look like it's there. For example, an article with the title 'Exclusive: Alibaba plans to take Hong Kong unit private' has the most frequent hashtag 'alibaba'.

The next stones I plan to look under include: aggregate analysis of tweets to better understand the people who tweet news links and crawling conspicuous blogs to see if anything interesting happens when 'influential' bloggers link to source news material. Stay tuned.

Mobile News Page on d8taplex

In a fit of agility, I added a page for mobile devices to get some of the d8taplex news info: d8taplex.com/m/hapaxPage.html. This shows the articles crawled from Reuters which are getting a reasonable amount of bit.ly juice. Note that currently you have to navigate there explicitly.

The page is extremely simple.

Check out Jeremy Singer-Vine's page as well for top headlines. In comparison to using click data to determine 'top' articles, it uses the editorial position of articles on the news sources it crawls.

News Aggregator News

For some reason, a number of projects are coming out of the closet this week. I mentioned Reuters 'Social Pulse' briefly already (not to self: write post describing how while many Reuters journalists have twitter accounts, no-one is tweeting...). Here is another : topheadlin.es (via Nieman Journalism Lab).

The site is, reportedly, a side project by Jeremy Singer-Vine, from The Wall Street Journal. The site, designed for mobile form factor, aggregates 'top' headlines from a small number of sites. From a motivation and design perspective, I really like it - keep it simple, tell me what I need to know. However, from an analytical side, it suffers from not really understanding the content that it is aggregating.

For example, right now it looks like this:

Did the Giant's win (something)?

My experimental site - d8taplex news - attempts to avoid this problem by (currently) sourcing from a single news agency, and by using some simple clustering. That being said, I should probably work on a mobile version of the page if I want to compete!

Reuters Social Pulse

Briefly - I just saw that Reuters launched a site which connects the world of journalism with that of the social web: Reuters Social Pulse. This has similarities with what I've been experimenting with at the d8tplex news page which leverages bit.ly data and identifies over 500 Reuters journalists' Twitter profiles.