Data Mining: Text Mining, Visualization and Social Media

Google+ Is Like 401K For Search

I'm trying to figure out which camera to go for: the GoPro or the Contour. When I search on Google for help in finding some way to compare, it was suggested to me to ask on Google+:

I then waited for someone to answer. I got nothing. Of course, I'm sure if I had the right set of connections I would have found a sweet expert opinion that could have helped with my decision and so, in a sense, it is my fault for not cultivating my network.

In the USA, a major part of the population's strategy for retirement is the 401K account. This is an account which provides the benefits of tax sheltered investment for retirement. The only catch is that it is up to you to manage your funds to ensure that you will have any money at all when you retire. There is nothing stopping you from making a bad bet and loosing it all. This is in contrast to the more staid approach of entrusting your retirement to some sort of national scheme that guarantees a pension and puts the onus on the government to ensure it.

I'm not an expert in fund management.

If Google were smart, it would have some ability to predict if my connections had the remotest chance of producing an answer (if not, why suggest that I ask the question?). Or else, perhaps it could suggest to me someone directly to ask in Google+?

Finally, let's not forget the hilarity that ensues when one plays with this:

Google Provides Links to Journalists' G+ Profile

Google New - which aggregates, clusters and ranks news articles from many sources - recently started including links to the Google+ profiles of journalists.

Here we see an example of a link to Shayndi Raice who, with Randall Smith, wrote this article on the Facebook IPO published by the Wall Street Journal.

However, the recall of the links is very low and, in fact, this link to Shayndi Raice is the only one currently visible on the Google News front page.

There are a number of sites which track and accumulate social identities for journalists. MuckRack, for example, has around 160 twitter identities for Reuters journalists - including Felsix Salmon who has around 50k followers - and many more profiles for the Wall Street Journal.

Over on the d8taplex news page, I've included twitter identities for Reuters contributors. I started out by simply using the twitter identities made available from the Reuters pages of the journalists. However, curiosity got the better of me and I ended up mining twitter for journalists. This has so far yielded over 500 Reuters journalists' twitter profiles (not all of which are currently available on the site, note).

In looking at these accounts, a large number are not active at all on twitter, while others, like Sam Youngman, provide commentary and even photos associated with the area of news that they cover.

Facebook Likes Burgers

I found this data mining error on Silo Breaker somehow amusing:

The article does mention McDonalds in comparison to Facebook, but Silo Breaker didn't figure out that the article wasn't about McDonalds per se.

Bitly Users Vote with their Clicks on Vatican Scandal

By far the biggest story right now according to clicks on the bit.ly links to articles published by Reuters (as shown on d8taplex) is:

This story doesn't make it on to Google News front page or even their page of world news. It's not on the BBC's front page or in its European news section. Could be quite simply due to the fact that Reuters got the news late and everyone has moved on, but I see an article on HuffPo on this (from the same source) posted 3 hours ago.

 

Super-hot News

A simple list of articles may tell you that the ones at the top are 'more something' (relevant, popular, etc.) than those lower, but not by how much.

I just noticed on the d8taplex news page that this article, entitled 'Starbucks to sell alcohol in some U.S. cafes' just hit 11, 430 clicks on bit.ly. This is an order of magnitude more than most articles get to in terms of bit.ly clicks.

Journalists' Tweets Provide Additional Story Context

I'm watching the bit.ly counts for Reuters story on the RIM guard change continue to jump up on the d8taplex news page.

Because the d8taplex page links to the Twitter accounts of the journalists when they are available, I took a look at what Alastair Sharp was sharing on Twitter. Alastair provides a couple of tweets on the topic, including a link to a video interviewing the new CEO.

I'll figure out a way to integrate recent tweets from the journalists into the interface in the near future.

 

Visualizing the ROI of News Articles

By filtering on a contributors name (in the example below, I've filtered on the enigmatic Cynthia Johnston) the d8taplex news page can instantly show to what degree the contributors articles are getting attention (according to bit.ly stats).

Here we see that from the pool of recent articles with CJ's name on, one has made it to the hot column, while nine are still in probation looking for some clicks.

Over time, we will likely see articles move from the right to the left while new ones populate the unloved column.

By filtering on country names, or other keywords, one can get a sense of where attention is being paid.

Exploring News

In experimenting with news aggregation and mining on the d8taplex site, I've come up with the following questions:

1. Why are some news articles picked up and others not? News sources such as Reuters create articles that are either directly consumed or which are picked up by other publications and passed along.
2. Who are these people writing these articles? What are their interests, areas of expertise and personalities?
3. What is the role of the editor and how do they influence the selection and form of the content produced by the news machine?

The next round of experimentation with news aggregation has resulted in the current new site. It has the following features.

Firstly, it presents two lists of articles. On the left, are those articles which are currently getting a reasonable amount of buzz (as determined by bit.ly clicks). The second column, on the right, presents those articles which are not yet receiving much attention. As you read the left you will probably realise that you are already familiar with these stories.

The amount of bit.ly juice is indicated by the number prefixed by 'B:' The article block consists of the title, the time stamp of the article, the link and the list of contributors.

Secondly, by providing the dynamic filtering one can very quickly get a view of which unloved articles are present relating to those stories that are already getting attention. For example, in the screenshot below we can see that one article about Gingrich is getting buzz while four others that mention him are languishing on the right.

Thirdly, it connects the user with the personality of the author and editor by providing, where possible, various facets of information such as job title, twitter account, email address, phone number, role, languages spoken, etc. Some of these are gleaned from the Reuters site while others are mined from various social network sites and so on.

Please take a look at the site and let me know what you think. There are many things that I'd like to do with it, such as surfacing social metrics for the authors, predicting the potential upside for an article based on the historical bit.ly scores of the authors, etc.

If you click through to an article from the site, you will be contributing to its bit.ly score!

Did Web Search kill Artificial Intelligence?

The most commonly referenced definition of Artificial Intelligence (AI) is probably the Turing Test which avoids the tricky questions like 'what is intelligence' or 'what does it mean to think' and replaces them simply by the test of recognizing an anonymous agent as a fellow human (and, therefore, intelligent).

A key aspect of this is, of course, that the interrogator is communicating with an equal in terms of the mode, pace and structure of dialogue (it is possible that a computer could succeed at this deception but reveal itself by being implausibly smarter than a human, but that is a question we can enjoy a little later).

As we have since learned, creating AIs is extremely hard and requires a very large portion of the smart people coming out of a broad spectrum of disciplines including software engineering, robotics, psychology, cognitive science and linguistics.

Another enabler is industrial context which brings the pragmatics of real world problems, and the scale and funding that is often not available to academics and if so doesn't benefit from stark focus required to make progress that is provided by industry.

However, we currently have the following:

• Search engines that don't understand language and which attempt to mediate between people (searches by people and documents by people),
• The best and the brightest coming to work for document oriented web companies.

I can't help but wonder where the AI project would be today if web search (as it is currently envisioned) hadn't gobbled up so much bandwidth.

Farewell To BlogPulse

Today, according to the announcement on the BlogPulse homepage, is the day that Neilsen will shutter the longstanding blog analysis and search site.

BlogPulse was originally envisioned by Natalie Glance, myself and other colleagues at Intelliseek as a way to track popular blog posts, people and phrases in the blogosphere. It launched with a sparse set of features in 2003 with the following design:

including key phrases which, at that time looked like this:

The site evolved to become a full blog search engine (I was actually against this at the time preferring to focus on analytics) and launched what was my favourite feature in 2004 : trend search. By that time the site was looking far more mature:

In that year, BlogPulse was named a 'trendsetting product' by KMWorld.

On top of the evolving set of capabilities, we started to publish longer analytical articles including this analysis of the Swift Boat story:

And these views of the 2005 Tsunami:

While (I assume) these assets will no longer be available from their adoptive source, the are archived at the wonderful Web Archive, so if you are feeling nostalgic for a simpler, slightly less caffeinated social media world, head on over and take a look.

One final note: I've always felt that social media analysis, just as it gets comfortable with a particular form of content, gets distracted by a new, shiny channel, abandons the legacy data and starts furiously reinventing its wheels. This certainly happened with the blogosphere when Twitter and Facebook showed up and consumers started to believe that 'real time' was something the blogosphere couldn't provide. From where I'm standing, blogs are alive!

If you have any fond memories of BlogPulse, please drop them in the comments to this blog. If you have generated some interesting trend graphs, send me a link and I'll summarize them in an update to this post.

For the record, the number of discovered blogs reported by BlogPulse right now is 182,397,015.