Data Mining: Text Mining, Visualization and Social Media

Subjectivity Bibliography

At yesterday's tutorial session on subjectivity, Jan Wiebe pointed to a great resource she has made available: a bibliography of papers on subjectivity (including sentitment, opinion and polarity papers).

Sentiment Mining: The Truth

Nathan Gilliat (o excellent blogger) posts about BuzzLogic's new partnership with KDPaine which will deliver sentiment scores to BuzzLogic's clients. There are a number of approaches to delivering sentiment analysis including many automated approaches and some manual ones. The customers are still skeptical of automated methods and generally more comfortable with manual methods. Paine writes:

"Computers can do a lot of things well, but differentiating between positive and negative comments in consumer generated media isn’t one of them,” explained Katie Delahaye Paine, CEO of KDPaine & Partners. “The problem with consumer generated media is that it is filled with irony, sarcasm and non-traditional ways of expressing sentiment. That’s why we recommend a hybrid solution. Let computers do the heavy lifting, and let humans provide the judgment."

This kind of statement is particularly unhelpful. Let's break it down.

Why Do Sentiment Analysis?

There are a number of reasons for doing sentiment analysis. Firstly, to track the ups and downs of aggregate attitudes to a brand or product. Secondly, to compare the attitudes of the public (that is, of course, the blogging public) between one brand or product and another. Thirdly, to pull out examples of particular types of positive or negative statements on some topic.

The Challenge of Automated Approaches

Sentiment can be characterized as a triple of <author, polarity, object>. Sentiment analysis in addition to figuring out the direction of the sentiment needs to associate the evaluative language with a target and the whole statement with a speaker. Many automated methods do weak jobs of these tasks if they attempt them at all.

In addition to these association tasks, the basic problem of dealing with sarcasm and so on, as Paine rightly states, is hard.

The Challenge of Manual Approaches

While customers are often more comfortable with manual approaches, this comfort is not always well founded. Manual approaches often have to sample (to me, one of the key propositions in the space is being able to listen to every statement that is made, not a tiny fraction). Sampling is hard as it relies on comprehensive data acquisition at least. In addition, you may well be surprised to see the agreement rate between different human labelers. The literature reports agreements rates as low as 40% in some cases!

Application Details

If you are attempting to track the ebb and flow of sentiment, it is very likely that automated methods are fine as aggregate analysis can often be robust to a certain amount of measured error. If you need to surface individual positive or negative comments, you want to make sure that they really are positive or negative in which case using the confidence scores often available with machine learning approaches can be used to rank remarks (though this does introduce an unknown bias). It should also be noted that while there are many obvious problematic areas, the distribution of these problems needs to be understood before a solution that does better or worse on individual examples is evaluated.

In summary - the challenges of the space are more complex than Paine's statement suggests and this comment is likely more a marketing strategy to support comparative statements that all vendors in this space need to make to distinguish themselves from the competition. There is still research to be done in this space (and it is good to see companies like Cymfony hiring scientists in the field of Computational Linguistics) - and the game certainly isn't over yet!

SentiMetrix Update

SentiMetrix, which I've covered before, now has a website. There is a bit of new information available there, including more background on the company. However, the thing I find most interesting, and which has perhaps the biggest impact on the market, are the claims regarding different languages:

Our system was designed from the ground up to support multiple languages. At this time, we can handle 8 languages: English, French, Spanish, Italian, Chinese, Russian, Korean and Arabic. While accuracy is still best for English documents, we have followed French elections and have successfully predicted Sarkozy's victory.

Generally, any company offering a sentiment analysis service is asked (right after 'yes, but does it really work?') 'can you do this in these 5 other languages?' My understanding is that SentiMetrix' technology requires lexical resources, thus they have a one time cost per language to create that resource (not a huge deal perhaps, but requiring native speakers for sure). If they also use grammatical information (something which I believe to be important) then there are additional costs involved.

Anyway - congratulations to the team for getting this far: onwards and upwards.

Sentimetrix: Sentiment Analysis Company

I've been keeping an eye out for information on Sentimetrix for the past 6 months or so. Today I came across this pdf document from UMaryland which gives a little bit of concrete information about the company:

With a University of Maryland College Park technology spurring the idea, a group of local businessmen and scientists have launched a new company that measures a tough to quantify concept – worldwide opinion.

SentiMetrix, of Bethesda, Md., was launched in fall 2006 by university professor V.S. Subrahmanian, graduate student Diego Reforgiato, and two businessmen who spent time with online giant AOL LLC, Vadim Kagan and Michael Rozenman. The company is based on Subrahmanian’s Opinion Analysis SYStem. commonly known as OASYS. It was developed at the University of Maryland Institute for Advanced Computer Studies, and is based on a series of complex algorithms.

“Recent surveys show that the marketing research market, including opinion research and brand monitoring, is growing rapidly, as new technologies get applied to electronic media, both professionally and consumer generated media,” Rozenman said. “We have started SentiMetrix because we believe that the OASYS technology is the best response to what these markets need today: sentiment tracking in multi-lingual data, done in a timely, cost effective way.”

OASYS, which was a finalist for the OTC Invention of the Year Award, is capable of tracking the media on the Internet in many languages, measuring the intensity of sentiment expressed on a variety of subjects. For this reason, it is unique, the businessmen say: most programs detect just “polarity” on a subject (like/don’t like, for example), and most are not multi-lingual.

So far, the founders said they intend to build “an extensive data collection operation” for traditional and consumer-generated media, including mainstream news, blogs and message boards, starting

first with English-language sources. They will then move on to other languages, starting with the most frequented Web sites.

V.S. Subrahmanian

A SentiMetrix customer will use OASYS through a controlled access Web site, with a search engine-like interface available to run queries. A visual representation and quantitative data is available, and a free Web site with limited options will be developed as a market tool, Rozenman said.

Thus far, OASYS has won Computerworld Magazine’s 2006 Horizon Award, which goes to the most innovative pre-commercial technology.

Opinion analysis program leads to new start-up company

The most interesting paragraph in this text to me is (my emphasis):

OASYS, which was a finalist for the OTC Invention of the Year Award, is capable of tracking the media on the Internet in many languages, measuring the intensity of sentiment expressed on a variety of subjects. For this reason, it is unique, the businessmen say: most programs detect just “polarity” on a subject (like/don’t like, for example), and most are not multi-lingual.

Much of the published work on sentiment/opinion fails to really define what sentiment or opinion is (taking the machine learners path of least resistance: a data set, an algorithm and a result). As a customer, I'd firstly want to know what their precise definition of sentiment is and then how they measure intensity. In addition, there are many types of expressions which while not opinions or sentiment still convey important information about topics and products. For example 'my Hummer broke down' isn't a subjective, opinionated statement but closer to an objective reporting of facts. It is still an important class of statement to capture as it reflects on the quality of the product.

One of the challenges of creating a single valued metric for sentiment or opinion is that equally mixed (aggregated) opinion tends to score the same as neutral opinion. In the simple case, if you have 1 person expressing a strongly negative opinion and another expressing a strongly positive opinion, then the aggregate may be some number in the middle of the range (say, 0 in a -1 to +1 scale). However, let's say that for a topic there is no expression of opinion - that too would score a 0. From the description provided here of the system, it seems that OASYS may suffer from this problem.

There are a number of publications available describing the research in OAYS.

Estimating Distributions in Social Media

Thomas Richardson, at University of Washington, emailed me a pointer to a talk that I wished I could have attended: How to Read 100 Million Blogs (and How to Classify Deaths without Physicians). I was travelling at the time, but I read (most of) the paper that Gary King presented. The content of the paper ties in with a key challenge in industrial text mining settings which we encountered again and again at BuzzMetrics.

Let's say you have a system for mining sentiment from blog posts. By using this system (which, let's assume, can also associate the sentiment with a topic of discourse) we can count the number of positive and negative opinions found online for some topic. Using these counts, we can then state the proportion of posts that contained positive, negative or neutral opinions about the topic. However, such a system will have some error rate (both in the sentiment mining, the topic recognition and the association). By investigating the nature of these errors, we can account for the bias in the automation and provide a more accurate estimate of proportions (e.g. if we find 80 negative posts on X and know that our recall for negative posts is 80%, we can predict - with a number of obvious caveats - that we failed to observe 20 posts with a negative opinion of X). In addition, we can provide some form of error bounds on the estimates.

The challenge with the deployment of such a system is that users want to drill down to instances, and when errors are encountered, they get nervous about the results in general.

King and Hopkins' paper is all about being comfortable with proportional predictions. I've not yet read the paper thoroughly and will probably revisit it here when I have done. I did notice that the data they used was provided by BlogPulse.

One thing to note about this general form of analysis: text/data mining is, fundamentally about finding relationships within the data. This is far easier to do when one can annotate features of data instances.

Some Sentiment Mining and Measurement Tools

Geoffrey Wiseman, in commenting on Positive and Negative National Influence, points to Sucks/Rocks - a site which uses search counts from Yahoo! to measure sentiment for search terms. Sucks/Rocks gives a score from 0 to 10 by normalizing the counts. Opinmind, is system which provides a search interface to blogs to provide a similar measure. I thought it would be interesting to compare the two. The chart below shows the Sucks/Rocks scores on the X axis and the Opinmind scores on the Y (I've translated Opinmind from a percent to a 0-10 score). The points are for the countries mentioned in original post.

Sucks/Rocks uses a simple approach to measurement - it looks for a small set of expressions:

When you enter a search term, sucks/rocks searches the web for several positive and negative phrases using that term. The score is the fraction of positive results to the sum of positive and negative results, normalized to 10.

The negative phrases are: X sucks, X is lame, X is crap, I hate X.

The positive phrases are: X rocks, X is sweet, X is awesome, I love X.

Opinmind, probably uses something a little more sophisticated (certainly their set of positive and negative terms and the way in which they are associated with the search term are both more advanced).

It is interesting to note how different the results are.

Here are some examples of false positives from Opinmind:

• We recommend that the Regulations for Great Britain make clear that the prohibition on discrimination applies to the curriculum and thereby avoid the considerable uncertainty to which the Northern Ireland Regulations have given rise on this question. [positive]
• I'm with Scamp: this little 3 liner from the Miss Great Britain organisation must be one of the best press releases ever ( which is why I'll repeat it here )..... [positive]
• In Japan it is considered rude to talk with mindi in your mouth. [negative]
• they might go undefeated even tho that fuckin pitcher from Japan fuckin sucks. [negative]
  

These examples illustrate some basic problems with an index/search based approach. Firstly, term ambiguity: 'that pitcher from Japan' is not about the nation Japan, 'Miss Great Britain' is not the nation Great Britain. Secondly, association: 'In Japan it is considered rude to talk with mindi in your mouth' - the negative term 'rude' is being associated with the act of talking with your mouth full of mindi (whatever that is), not with Japan.

The association problem, in my opinion, requires at least grammatical analysis of the text and, in the big picture, probably discourse analysis.

However, it should be noted that when the goal is to provide a measurement of sentiment, it is entirely possible that a reliable system may be built that accommodates this type of error via a deeper understanding of the distribution of expressions, term ambiguity and association ambiguity. That being said, relying on knowledge of such biases is likely to lead to measurements that cannot be transfered from one domain to another, and so better approaches to the core sentiment mining problem cannot really be avoided.

This is going to be an interesting year for sentiment mining with at least one new company likely to appear - more on that later!

Modality In Expressing Opinion

David Weinberger commented on my recent post on sentiment analysis. He adds:

[H]ow about this example, drawn from a review of the Su Hong restaurant: "When we moved from the area, my wife's most frequent complaint was that no one made orange peel beef like Su Hong." As Matthew points out, it's a positive review in the form of a complaint.

I love the idea of capturing the modality (complaint) of the sentiment (which is positive). In fact, this example is even more significant as there are multiple topics. It is negative about the implied new locale and positive about Su Hong. I suspect that this is in itself a form of comparative (X is better than Y). Comparatives tend to challenge current formulations of sentiment analysis as they indicate that opinions can be (perhaps should all be considered) relative, not absolute.

The Psychology of Influence

I've been working recently on polarity/sentiment analysis. In the initial phase of this work, I'm creating labeled data. I've been looking at a few data sets, one of which being expressions of evaluation concerning actors. Yesterday I completed labeling 5, 000 instances of 'Edward Norton'. Things like "I love Edward Norton" all the way up to "I've not yet seen a bad movie staring Edward Norton", "I'm gay for Edward Norton", as well as many, many people dreaming of doing certain things to and with him. Anyway, after reading all of this overwhelmingly positive stuff I'm pretty sure it has had a clear effect on my own judgment of the actor. I was just briefly over on our Netflix account and before I knew it I was searching for Edward Norton movies. He totally rocks, or so I'm told.

OpinMind is Hiring

In the past 7 days, I've seen adverts for 2 positions at OpinMind, the consumer facing polarity (sentiment) search engine. These ads have been circulating on LinkedIn and on at least one NLP mailing list. I'd link to the openings on their site, but I can't find them and their blog appears to be down.

What does this mean? Well, perhaps they are getting some capital to ramp up. If this is the case, then it is another instance of investment in NLP in the search area.

OpinMind faces a fundamental challenge: turning a toy-like application into something that can deliver a reliable measurement of sentiment associated with some search term. It is one thing to be able to say there are 90 statements positive about X and 10 negative, but what does one do with that? How does that help the user? The real value is beyond the ability to produce this result. An actionable exit point from the search process needs to be identified and ultimately monetized.

Real World Text

The great thing about working in NLP in an industrial setting is the exposure to real world data as opposed to the more academic 'John loves Mary' type of thing. In reviewing posts on the horrible Edelman/Wal-Mart story, I came across this I'm-not-making-this-up sentence:

Certainly, if you don’t post about it, I’m personally not going to not respect you for not doing so, however others may.

Duncan Riley is saying something here - not quite sure what.

Sentiment mining in the social media monitoring space requires high precision and recall. Recall is important because a single post from a respected source needs to be captured accurately. In other words, it is not the number of false negatives, but who they are that really hurts you.