Data Mining: Text Mining, Visualization and Social Media

Umbria White Paper on Spam Blogs

Umbria's new white paper on spam blogs provides some solid insights into the problem of spam in both the search and marketing intelligence spaces. However, it continues the practice, set out in their recent press release on the topic, of using flawed logic and misleading information to attempt to position itself ahead of the competition.

Umbria analyzes the percentage of spam in the leading blog search engines. There is an improvement here compared to the press release in that it identifies the search terms used and the counts of spam found. The are very open in admitting the anecdotal nature of this experiment. My real concern is that they then use these tests against consumer facing search engines to imply that their enterprise marketing intelligence solution is somehow leading the pack.

They claim that the real time nature of the search engine space prevents adequate spam filtering. Although there may be evidence that the state of existing search engines doesn't neccessarily refute that, fundamentally I don't believe that to be the case. There is no reason why blog search engines can't approach the problem with some form of sandboxing for new blogs whose status is not yet determined. In addition there are approaches to true real time spam filtering that wouldn't even require sandboxing.

The most flawed part of their argument is that they imply that the quality of results in Intelliseek's consumer facing product is identical to that in our enterprise marketing intelligence solution. Given their argument that they would have challanges filtering spam in a real time consumer solution, why would they use the same mechanism in a completely different application? I wouldn't, for example, use the fact that their home page presented the spam blog white paper in a buggy manner to conclude that their web hosted enterprise solution was poorly implemented, or that the poor logic employed in their white paper indicates that the results in their customer reports were not to be trusted! Would I?

Perhaps their story is that their product is more useful than, say, trend mining on BlogPulse, or search on Technorati (which are both free).

Consumer Facing Analytics for User Generated Media

Aggregating and analysing user generated media (blogs, boards, usenet, etc.) posts on products is what Intelliseek, BuzzMetrics, etc. do for a living. This level of analysis is not currently available in consumer facing products other than search (blog search, board search, usenet search) and some specialised search applications (trend search and the simple sentiment search of OpinMind). The content universes for enterprise products in this space include generally blogs, message boards and usenet and can extend to review sites and other genres.

On the other hand, there are social content hosting sites, such as riffs.com and amazon, which support research on products via aggregating content only found on their site - what we might call captured content search. These are useful to a degree. However in order to get a fuller picture - one might say a more accurate picture - of what people think about products, or any topic for that matter, a solution is required which is capable of spanning multiple content types.

When considering user generated media, we are forced to consider the boundary between this type of content and professional content. A boundary which has blurred considerably in the last year and will continue to do so. In other words, there is a problem in deciding when to include opinion from the professional end of the spectrum and how. For example, how does one weigh a movie review from a blog, a movie review from rotten tomatoes, and one from, say, Mr. Ebert? An additional issue when providing consumer facing interfaces for this type of data is capturing the product being researched and finding those posts that refer to it. The captured content, especially in sites like amazon, may well have the advantage of a complete taxonomy of products which can be searched against rather than the content of the reviews. In other cases, the user requires assistance in accurately retrieving results for all variations of expressions used to refer to the product.

In this context, it is interesting to note two developments. First is Google's review search, which Kevin pointed to. This appears to aggregate reviews from a number of sources (and types of sources) for movies. It also provides access (in a faceted search type of interface) to positive, negative and neutral reviews (based on segmentation via numeric review scores). Not only that, but it provides common phrases mined from the reviews. This is pretty cool, and brings features found in standard enterprise offerings to the consumer. Note that BlogPulse has had consumer facing features based on similar features for over a year - for example blog bites, which both extract common terms and find snippets which contain terms with significant relationships.

I'll post a deeper dive on Google's review stuff later.

The second new consumer facing user generated media interface is reported by Smart Mobs, and TechCrunch and appears to be a service which will allow users to use their phones to input bar code data on a product and access user generated commentary drawn from blogs.

Is 2006 going to be the year of consumer facing analytical tools over the user generated media space? And how will the enterprise players in this space respond?

Measurability, Measurement and Accuracy

Shel Holtz writes about the frustrations of being in the PR world where measuring the results of actions is a prime focus on selling new ideas, but where the ability to make accurate measurements, and the ability to determine what to measure is of secondary concern.

My feeling is that the blogosphere (actually, the whole gamut of online personal media, or open media) offers something else to measure without there being any inherent features that result in better measurements. This is why consumer facing products, such as BlogPulse, Technorati, OpinMind, while having a great role in attracting people to the space, cannot, without adult supervision, really be used when betting the farm. Will users consider when and how new blogs are added to the index when looking at Technorati's absolute count trends? Will they really take seasonal variation into account when looking at a 30 day window? Will users moderate OpinMind's positive and negative counts with a known background of precision and recall for each category of opinion?

Monitoring buzz, guaging polarity (aka sentiment) of online opinion is, for some time to come, going to require either expert users, or an enterprise service layer appropriate for the cost of decisions being made (here I am thinking of things like 'do I pull the product now?', 'do we re-target our ad campaign to the teen-female segment?', etc.)

On a related note, calls like this for better consumer services indicate another level of disconnect between the users and the pragmatic, technical world of web 2.0. To think that those working in blog search (and, let's face it, everyone working in this space is working the enterprise angle as well) have not been considering comment search is ridiculous. This is precisely why I wrote my When Worlds Collide post earlier this year.

A great enabler of blog search has been the RSS-ping duo. There is only one blog publishing software which currently allows a good RSS based comment stream. Getting to the time granularity of Technorati for comment search is a reasonable engineering challange. Consider this in the face of the scalability challanges that consumer facing blog search engines have faced and overcome this year.

What is needed is not just the right motivation, or someone to grant us the vision, but some platform alignment and modifications to infrastructure. Of course, having said that, there is an opportunity for a blog hosting system that also provides blog search to lead the way here. Such combinations, however, for some reason tend not to be leading the pack in terms of innovation.

Having got there, the problem of understanding when comments indicate an interesting conversation and when they don't is going to add another layer of complexity which will lead in many misguided inferences being made. Many A-list bloggers turn comments off to remove certain types of work load. Are their posts important?

Feedster, Biz360 get Sentiment

Jason Lee Miller of WebProNews writes about a new relationship between Feedster and Biz360 that provides, well, the same type of analysis as all the other systems out there, including phrase finding, association and sentiment. He quotes the press release:

"We're the only company that has automated sentiment ratings toward specific companies, products and issues you're tracking. Also, Point-of-View Sentiment is a machine-learning technology that "learns" a client's style of rating--what's positive to one client may be negative to another, there's no one-size-fits-all when it comes to rating sentiment."

What is interesting about this is that the learning that is going on means user input. In other words, the user - the consumer of the analytics - has to train the system the bias in their interpretation. For many users, this bias will be the same, and so there will be a lot of duplicated effort. My feeling is that for sentiment analysis it is far better to spend the effort on modelling the opinion of the author of the text (were they favourable or unfavourable wrt some product/event/thing). Some form of tuning will be required per domain, but the bias of the researcher should be modeled at a higher level.

Imagine if you had a collection of Amazon reviews for a book. I'd prefer to know what the reviewers thought of the individual elements (plot, character development, pacing, etc.) and then say: character development is important to me, or pacing is important to me, and then weight the measurement of those opinions accordingly.

Christmas Wishlists

Peter O'Kelly points to a New York Times article about Christmas wishlists:

Two of the most-coveted consumer electronics items this holiday season are the Apple iPod music player and the Xbox 360 game console from Microsoft, according to retailers and industry analysts.

That represents a major shift in buying intentions this year.

In 2004, portable MP3 music players did not even make the annual top 10 list of desired gifts compiled by the Consumer Electronics Association. And video game machines were in ninth place.

This year, though, those categories occupy the first and third positions on the list, based on a telephone survey of 1,000 randomly selected adults. Digital cameras are second on the most-desired list this season.

Here are the trends from the blogosphere:

These results are similar to the New York Times information.

Customer Challanges in Text Analytics

A couple of days ago, I mailed Matt Galloway a heads up the Umbria was providing sample reports via a simple registration on its web site. Matt has beaten me to the punch in writing up some thoughts on those reports. Part of Matt's analysis focuses on the issue of demographic analysis and perceptions of the accuracy.

Analysis in the online personal media space faces a number of challanges that academic research on text avoids. A very important issue is the customer's perception of quality. Matt suggests that analytical results, e.g. of demographics, need to be matched with traditional MR numbers in order to provide comfort to the consumer of those numbers. This is misleading. The numbers are going to be different - that is the whole proposition of doing analysis in this space. The sample - that of people expressing their opinions online - selects for certain types of people, certain personalities.

Another issue Matt highlights is the accuracy of gender classification. There is always a tension in providing analytics of this sort which, again, comes down to some very pragmatic user needs: measuring the distribution of a feature over a set of documents (e.g. gender) is very different to identifying the value of that feature for an individual message (or person). This is where perception of accuracy can mislead the customer. For example, if a system can predict the gender distribution of the authors for a given message set with a certain accuracy (an acceptable accuracy) then the value of that analysis is a function of the accuracy and the action that is to be taken.

Customers, however, like to drill down to specific messages or authors to inspect the values of instances of the feature. They are guaranteed to see errors here. A simple fact of customer facing text analytics of this sort is that errors are perceived as being far more troubling than the actual raw statistics describing the accuracy of the analysis. In other words, if you tell a customer that the gender classifier is 90% accurate they will be happy, but if they see 10 messages and one or two have errors in them (errors that look really dumb to the user), their comfort in the perfomance will go down.

The customer is concerned here with precision - the ability of the system to make accurate statements. The distribution analysis has to consider recall - how comprehensive those statements are. This issue - precision and recall and their effects on different presentations of the same algorithmic predictions - is one of the toughest challanges in selling the propositions of an automated solution.

Kaava

Here's another entrant in the Online Personal Media analysis space: KaavaCorp. I'll write up a deeper dive later. For now, I notice that they use the term online dialogue management a lot on their site. This is code for conversation seeding or, to get down to it, trolling.

I'm by no means judging their capabilities in the analysis side of things (that is to say, their capabilities to acquire blog, message board, etc. content and analyse it to deliver insights and monitor brand and product spaces) and find it actually quite natural that a company like this (formerly BBI) should step across from the other side.

Separating the Wheat from the Chaff

Steve Rubel has posted on illustrating what he calls the conversation gap using the trending tool from IceRocket. It is great to see Steve using a trending tool - which we popularised over at BlogPulse and which IceRocket flattered us in duplicating. Running the search Steve uses as an example on BlogPulse produces very similar results. However, I noticed this in a recent entry on Mark Cuban's blog which casts some doubt on the meaning of the results Steve is generating:

Icerocket.com isnt really a blog search engine, although thats the term we use since its the easiest way to   communicate what we are doing. We are a tracking engine. We dont just index and accept pings from    blogs, we index any and every source of information that is updated on an ongoing basis. Forums. Yes. Corporate   RSS feeds. Yes. Better yet, there are some things happening that will enable traditional websites to identify new   information on their sites and ping that to us. This way if WidgetsInc comes out with a new Dallas Mavericks Widget,   they can update their widgetsinc.com website and we will have it covered. It will show up in Dallas Mavericks   topic Im tracking.

Many blog engines gather some large percentage of their data in response to pings recorded by ping servers. The content has then to be binned into legitimate blogs, spam and other unwanted blogs and other material. Most blog engines do this (including BlogPulse). Certainly, the issue of spam is well known, however, the issue of non blog pings has received less visibility. These pings come from almost anything: message boards, weather services, news and other media sites, flickr postings, etc. It seems that IceRocket has avoided this problem by rebranding as a search engine for, well, anything that pings a ping server.

EA on Blog Analytics

Matt Galloway just posted an interesting perspective on EA's engagement with Umbria for monitoring online discussion, based on a conversation with EA's Shawn Conly, VP of Advertising and Consumer Insights. I'm impressed by the amount of detail in there and the willingness of EA to go on the record. A few points I noted when reading:

• Matt reports: "Umbria was able to offer a higher value proposition" Shawn recalled, noting that Umbria's offering was less expensive than Intelliseek's. One thing I observed at WhizBang!Labs: a lower cost to a customer is not the same as the solution costing less. In new product spaces it is very common for smaller companies to offer large early adopters both discounted products as well as a raft of on demand customizations (which soon add up). Shawn states: "I just call Howard and let him know what we need."
• The problem of capturing content on message boards is discussed: I often question coverage of forums and such because of the relative difficulty of reliably extracting speaker identity and text. I'd love to hear Matt's reasoning behind this, though it is interesting to hear Shawn's opinion that I don't have a good sense of coverage outside of the blogosphere, but I don't think Umbria has a lot in forums.
   
• Finally, Shawn's impression is that currently, Intelliseek is better equipped to help clients understand business implication than is Umbria. Generalizing, Shawn said that Umbria seems more technology focused while Intelliseek seems more consultative. My understanding is that Umbria certainly has a different mix of analysts to tech/R&D staff, but this should not be confussed with the absolute numbers of people working on the underlying technology or the manner in which technology appears to the customer.

Anyway - a nice write up.

Umbria Updates Site

Since I last visited, it looks like Umbria has done quite an update to their site. I've not had time to go through everything there yet, but was interested to see the real estate given to a book chapter written by Franco Salvetti. It is great to see Umbria pushing the technology/science side of things. Kamal Nigam and I also share a chapter in the book : Computing Attitude and Affect in Text Theory and Applications (The Information Retrieval Series).

Other parts of the update include registration for a sample report, a 'latest insights' snippet, better integration of corporate news, customer commentary and a more serious look and feel.

Website updates are often precursors to something more substantial at the corporate level - new products, investment news, refined strategy, etc.

On a related note, Natalie Glance and I have been invited to the programme committee of the upcoming AAAI Symposium on Computational Approaches to Analysing Weblogs, organized in part by members of Umbrias R&D team.