Data Mining: Text Mining, Visualization and Social Media

ViewScore - Consumer Facing Social Media Analytics

TechCrunch points to ViewScore, a site which aggregates reviews and review metrics. The cool thing is that for reviews that don't contain a metric (e.g. a star rating), the site infers one based on 'semantic analysis'.

ViewScore has developed an innovative search engine and product ranking system to simplify the complexity of product evaluation. First, ViewScore's crawler (Site Owners) searches the Internet and identifies product reviews by distinguishing them from text in which the product is simply mentioned or referred to. ViewScore's filed patent-pending technology, TextScore™, then takes the stream of unstructured and often incomparable product review data, elicited from hundreds of data sources, and converts it into a single, consistent and coherent product recommendations score. TextScore's artificial-intelligence and NLP (natural language processing) technology parses each product review and assigns it a normalized numerical value that represents the author's opinion of the product under review. These individual scores are weighted by the relevance of the review and the reliability of its source and then aggregated to yield a single product score for each product. The system then ranks the products by their final product score and presents final recommendations to the user.

This is doubly interesting to me as it combines sentiment analysis with a consumer facing social media search application. We are going to see more and more of these over the next few years as the quality of these interpritive technologies increases, and the volume and influence of social media continues to rise.

Note that I haven't yet explored the site in any detail and don't yet have any clear idea of the quality of the results. For example this review of the Canon EOS 30D gets a ViewScore of 98 out of 100. However, the text of the review ends with

As always, I strongly recommend trying the EOS-30D and its competitors before you drop the big bucks on a digital SLR!

This sounds cautionary in tone, whereas 98/100 sounds like a very strong recommendation. Other reviews (like this one) get a score of 100/100.

There are some very interesting challenges in automating this problem. CNet's review system (see here for a review of the Canon Powershot A610) allows users to comment and give their own feedback. The Canon got an average of 9.0 from 59 users who commented on the camera, but ViewScore gives a 74/100 score.

Personally, when I've been looking at reviews of products with the intention of determining which out of a few to go for I've been disappointed by the variance in reviews. One might see 3 reviews - 2 good and 1 very bad. Statistically, this information is not helpful. However, there are ways to weight the data. 2 good reviews from trusted and consistent reviewers should easily counteract 1 bad review from a one off user. On the other hand, if that one of reviewer never writes reviews and has been so disappointed in a product that they have clambered over many barriers to post the review...

Kurt Schrader of Squishr points out that he's been there before:

This is actually a bit of deja vu for me, as we originally had our technology working on gadgets, but we moved to music because we found that gadget reviews just didn't end up that useful in the grand scheme of things. People tended to say the same thing over and over again in their gadget reviews (good/bad interface, good/bad battery life, etc), and let's face it, if you're going to buy an mp3 player it's going to be an iPod.

I do agree with Marshall when he says that there's room for a lot of entries into this space.  Things should be interesting.

What I love about his post is that there is a comment which reads:

I don't buy iPod because it's crap.

Now, if you think about the aggregation of that type of content - unsolicited opinions - and its analysis: that is what we do.

[See other posts on sentiment mining.]

Does Google Own The Blogosphere?

How do readers get to blogs? There is plenty of work looking at the structure of the blogosphere as a stand alone partition of the web, but what is it's relationship with the web at large? I've collected a sample of referral statistics from 1, 500 of the top weblogs. The data was collected between 29th of July and 26th of August 2006 and resulted in 890, 469 referrals. The sample was collected by capturing the 20 most recent referrals at 10:00 EST.

Of this data, 539, 593 were of unknown origin.

Of the remaining 350, 516 referrals, the break down by host is as follows:

host	count	%
google.com	82, 203	23.4
images.google.com	13, 817	3.9
search.yahoo.com	11, 492	3.3
google.co.uk	9, 119	2.6
search.msn.com	5, 684	1.6
google.ca	5, 020	1.4
bloglines.com	3, 765	1.1
google.com.au	3, 597	1.0
google.fr	3, 435	1.0
google.de	2, 840	0.8
michellemalkin.com	2, 786	0.8

Note that these are per host statistics, not per domain (if we look at referrals by domain we will see, for example, that bloglines and typepad are in the top 10).

What these numbers suggest is that main stream web search (MSW search) is a major channel of traffic to the blogosphere. In fact, Google domains (i.e. google.com, google.co.uk, images.google.com, etc.) account for 45.4% of referrals. Google traffic breaks down as follows:

host	count	%
google.com	82203	23.452
images.google.com	13817	3.9419
google.co.uk	9119	2.60159
google.ca	5020	1.43217
google.com.au	3597	1.0262
google.fr	3435	0.979984
google.de	2840	0.810234
google.co.in	2284	0.651611
images.google.co.uk	2015	0.574867
google.es	1889	0.53892
images.google.de	1430	0.40797
images.google.fr	1398	0.398841
google.nl	1308	0.373164
images.google.ca	1245	0.355191
blogsearch.google.com	1101	0.314108
news.google.com	1015	0.289573
google.it	982	0.280158
google.com.br	948	0.270458
google.be	891	0.254197
images.google.com.au	838	0.239076

Some thoughts:

• Are users arriving at blogs from Google and aware that they are on a blog?
• Are users intending to find data on blogs and going to Google to search for it? I suspect not. I believe that these referals represent users who are searching for information in general and happen to land on a blog.
• Does this bias represent an opportunity for blog specific search? Absolutely. I see this partly as an educational problem. If blog search engines play their cards right, these searchers will eventually be using blog specific search engines as they begin to understand what blogs are and what their value is. The key to enabling this change is that blog search be better than general search. Given the the fundamental difference in blog data and MSW data (particularly timeliness) this is not an unreasonable expectation.

Caveats of the data: note that this data isn't a straightfoward sample. Due to the way in which it is gathered, it doesn't represent a snapshot in time. In addition, there is this mass of unknown referrals. I am assuming here that there distribution is the same as that for the known referrals, but there is no way to actually determine that to be the case.

And - this wouldn't be a data mining post without some graphic, so here is a treemap of the data.

Update - I've located a glitch in the data. The implication of the problem is roughly that the basic shape of the data is the same but the precise numbers are not accurate. I'm working on getting a cleaner version of the data and will repost this analysis and more when I do.

Fact Checking in Reverse?

The online version of Business 2.0's article Blogging for Dollars makes passing reference to some Technorati statistics:

As for blogs, they've exploded: There are 50 million of them, and two new ones are launched every second, according to blog search engine Technorati.

However, the paper version includes a hockey stick graph with a title (something like) new blogs are growing exponentially (I'm quoting from memory). Given the discussion in the blogosphere about the validity and interpretation of Technorati's statistics, it is disappointing to see them both swallowed whole and misquoted. Sifry never claimed that the rate of new blogs is exponential, only that the blogosphere was growing exponentially (which is a dubious statement at best).

We know about the blogosphere fact checking the MSM, but what about the other way around?

Getting the Name Right: Farecast Update

I recently got an email from Farecast - the site that predicts the movement of air fares and advises the user to buy or wait for future travel. The cool thing is that they now include Pittsburgh in the list of 55 cities they offer.

I recall hearing that the C*O of Netflix indicated that the company was not called 'DVDs by Mail' as their business is about getting movies to people, not about mail based DVD distribution. One day we will be streaming movies from our Netflix accounts. Farecast could potentially target any market where the price of the product fluctuates over time. In particular, they could target the usual basket of things that any travel site offers, including hotels. However, their name is shrink wrapped to transport ticketing which may require an name change in the future.

MoFo Snakes on a Graph, etc. - Converted and Unconverted Movie Buzz

Max parked a link to the graph below for various movies:

Notice the use of AND ("movie" OR "film") for just the pirates term. I posted on a related issue a little earlier: Ambiguity In Comparative Search: A Text Mining Puzzle.

Da Vinci scored $77MM on its first weekend, Pirates $135MM, and Snakes $13MM. So let's look at another graph - this time using the constraints used for Pirates.

This seems like a picture closer to the the receipts. Buzz is correlated with receipts, but the question is - what buzz? From a research point of view, the problem is not to study the correlation between a movie and its earnings, but to create a classifier that cuts out the subset of the data that gives the best correlation. The rest of the buzz may be thought of as unconverted buzz. Advertisers and marketeers might wonder what to do with it or how to analyse it.

For completeness, here is another take:

See also: Are Movie Hits Coming or Going?

Now, About Those Photos

The blogosphere is familiar with the Reuters photoshop scandal in which a major news agency distributed images of the Israeli-Lebanon war which were doctored to create the impression that attacks by Israel were more photogenic than they actually were. Here's a graph of discussion on this topic. One line shows links to the Little Green Footballs' post unearthing and tracking the topic and the other the volume of conversation in general.

Of course, what this type of graph fails to show is the structure of conversation on the topic.

The next graph shows (a sample of) the tree structure of the conversation. Each dot represents a blog post and each line indicates a citation.

Initial posting on the 5th was followed by a peak on the 6th, just as in the first graph. However, what we can see in this graph is the later pick up on the 15th and 18th and the consequent secondary amplification. The first half of the graph represents what we might call the first epoch of diffusion. This is then followed by a second.

Now, what this graph doesn't show is how that happens. We can see that the citation structure is there, but what we can see the intention behind the post nor the actions of the blogger that led them to discover the post. This post is not going to dig through the data to summarize what is happening precisely in this case, but we can consider a few hypotheses:

• Late discovery and amplification in smaller community.
• Cross-over to another language/geographic location which is not generally concerned with the western media stream.
• Use of the story to support or illustrate a second story.

I find the last explanation particularly interesting as it suggests the narrative structure around a theme rather than a single story.

Sphere is Growing

Sphere, the blog search engine, is looking for new people. Given the number of people in the original team (a small number) and the number of positions advertised, this suggests a pretty significant increase in personal which is a good sign for the company.

David Sifry and the Spiders from Sweden - Finale

This is the last post on this topic - I promise. Firstly, Eric Lunt of FeedBurner assures me the Sweden/Russian problem is fixed - thanks Eric. Secondly, having been away from the blog for a week I can now see exactly what crawls my feed when I'm not posting - Technorati is doing this. The Technorati Search Bot hit me between 2 and 5 times per day over the week when I wasn't posting. I'm assuming that the bot that uses that name is from Technorati, BTW.

To be clear - this isn't any sort of criticism of Technorati, it makes a lot of sense to be crawling feeds as insurance against problems with what can be a shaky ping infrastructure. My original post was simply published as a description of what appeared to be happening and my interpretation of those observations. That, and I thought the post title was kewl.

Back in the Saddle

This post is here to book-end my previous post and to let me get back into blogging. I've been on vacation with my family at the New Jersey shore. A couple of comments that relate this trip to the themes of this blog:

• Mapping software - when you use online mapping tools to get driving directions they look at the requirements (get me from A to B) as an optimization problem (as quick as possible). However, a trip from Pittsburgh, PA to Cape May, NJ isn't really like that. Firstly, the trip is structured by sub-goals (get me to the highway, get me across this state, take me between these two major locations). These sub-goals are never present in the instructions and yet they would provide structure to the text which would make reading it and understanding the intention behind the directions far easier. I believe that someone could write something which takes the existing output from these systems and rewrites it organized in such a way. Secondly, there is a lot more to the trip than minimizing time spent. For example, I find it more tiring to drive on the highway than on local roads, so I like to drive on smaller roads at the end of the day; what about a scenic route option? (we did this ourselves on the way back and passed through York, New Oxford, Gettysburg - passing that most bizarre rustic establishment, the BYOB strip shack). In other words, not all drivers are alike.
• Seasonal satellites - when I look at satellite images I'd like to be able to control for seasons. The beaches we visited on this trip had people on them - armed with colourful umbrellas. Imagine looking at such an image and seeing which beaches were crowded?

A Tour of Data Mining

As I am about to hit the road for a week with no expected connectivity, I thought I'd put up a post which briefly captures this blog. The topics listed below represent the basic content areas that this blog covers. With each topic, example categories/tags are linked to to provide a starting point for anyone interested in those areas.

• Natural Language Processing/Computational Linguistics, Text Mining, Sentiment Mining
• Geographic Information Systems/Digital Cartography, Geographic Data
• The Structure of Social Media, Graphs
• Data Visualization
• Web 2.0, Blog Search Companies, Social Media Companies

Statistically, some of the most popular posts have been

• Mapping The Influence of News on the Blogosphere, Are Movie Hits Coming or Going? (according to FeedBurner stats)
• An Interactive Map of the Blogosphere, Mapping The Blogosphere: LiveJournal (in terms of visits to the blog)
• The Narnia Rap