Data Mining: Text Mining, Visualization and Social Media

Raveable: hotel appraisal

Raveable is a new review aggregation site which specializes in hotels. It claims to provide information for 55 thousand hotels in the US culled from more than 35 million reviews. The site mines these reviews to automate the summarization of opinions from guests for the hotels.

The data

Firstly, where does the data come from? Paging through a number of hotel sheets, it initially appeared to me as if most of the data comes from TripAdvisor. The site states that the data comes from a number of sources, including TripAdvisor, Expedia, Travelocity, etc. as well as individual blogs. Browsing the site brought more of these sources into view – in fact, the experience suggests that some hotels get more data from one source than the other. It would be interesting to know the demographics of those sources and to consider a more representative summary taking the demographics into account. Has Raveable formed relationships with those sources?

The UI

In general, I found the UI clean and pleasantly put together. One of the most important aspects of presenting data for a choice like hotel booking is to get some understanding of the ranking of the candidates and the coverage of the system (how many of the competing hotels are in the system). The UI goes some of the way here by providing an overall ranking:

However, it would be nice to get a bit of additional geographic context that shows other hotels near by. In addition, with the added layer of appraisal (summaries of reviews) one could get a picture of the general quality of the hotels in the neighbourhood. Am I looking at a good hotel in a forest of good hotels? A good hotel in a sparse neighbourhood?

The Interesting Part

Of course, the interesting part about this whole thing is the automated summarization and extraction of opinions written in natural language. With 636 reviews available, on average, per hotel, there should be good opportunity to deal with accuracy issues by doing something sensible with summarization and presentation (note that I’ve not yet seen a hotel with > 600 reviews so I’m not sure if all the reviews are assigned to hotels in the system).

My expectations for the extraction of opinions was that the positive stuff would generally be pretty accurate, but negative comments – always harder than positive to recognize – would show some amount of error. Some anecdotal examples (pulled without too much hunting):

• Our room was on the 3rd floor we were offered a room at the back or the front, we chose the front whats the point of being in new york with out the noise. [Casablanca Hotel]
• The rooms are not large but are more than adequate. [Casablanca Hotel]
• You can see the old bank vaults in the basement and the registration area on the first floor is where the tellers used to be. [Drury Plaza Hotel]
• Housekeeping was called and eliminated the two problems. [Camelback Inn]

These false positives are interesting in terms of what they tell us about the algorithms that compute the sentiment. There is some background value associated with certain terms (noise in the first example triggered a negative tag; large in the second and problem in the third). The trouble comes with the valence associated with the terms by the surrounding context.

I don’t feel that these problems are really a weakness in the system. There are always going to be errors. I believe there are two ways to combat this problem in general. The most obvious is figure out a way to introduce another level of summarization in the UX. There is high level summary for each major category. One could try something simple, like a tag cloud, that summarizes within each category in a way that would allow the user to get a finer grained, but still aggregate summary of the content.

ACL 2009 Accepted Papers

The list is up. There are a few papers on sentiment mining. Among them, I find this title to be the most interesting: Mine the Easy and Classify the Hard: Experiments with Automatic Sentiment Classification, by Sajib Dasgupta and Vincent Ng.

CIKM Workshop on Topic-Sentiment

The workshop that I mentioned briefly in an earlier post now has a CFP.

1st International CIKM Workshop on Topic-Sentiment Analysis for Mass Opinion Measurement (Hong Kong Nov. 6, 2009)

News

03-22-2009: Workshop site is up

Scope of the Workshop

This workshop seeks to bring together researchers in both computer science and social sciences who are interested in developing and using topic-sentiment analysis methods to measure mass opinion, and to foster communications between the research community and industry practitioners as well.

The increasing amount of user-generated content on the Internet and social media and the digitization of large number of government and institutional documents provide new opinion-rich data sources for researchers to examine individual and group perceptions on products, organizations, and social issues at a large scale, and thus contribute to the research and practice in the areas of political science, social policy, communications, and business intelligence.

On the other hand, researchers are tackling the problem of processing large amount of opinion-rich data using various approaches. The increasing number of relevant publications in top data mining, information retrieval and natural language processing conferences (KDD, SIGIR, ACL, WWW, etc.) has witnessed the growing interest in automatic opinion analysis. Both TREC and TAC (Text Analysis Conference) have set up individual tracks for opinion retrieval and analysis tasks.

In recent years topic detection and tracking techniques have been well developed to identify the issues discussed in a large text collection. Sentiment analysis is catching up to detect the polarity of opinions expressed in texts. However, many times real-world applications have to take into consideration of both topics and sentiments for precise opinion measurement. Topic and sentiment alignment is crucial for opinion retrieval, extraction, categorization, and aggregation on various issues. Topics and sentiments could also have sophisticated interactions. For example, the choice of topics and the attention distribution among topics might bear hidden opinions as well.

How do we build synergistic topic and sentiment models for text documents? How do we tackle the domain-dependency problem of sentiment analysis? How do we identify users' needs and integrate them into the design of opinion analysis systems? What are the successful applications of topic-sentiment analysis for mass opinion measurement? What lessons have the pioneers learned? How do we evaluate the automatic mass opinion measuring tools with regard to the reliability and validity? This workshop solicits submissions to address these problems and more.

We hope this workshop can advance research in topic-sentiment analysis, make connections between research community and industry practitioners and encourage development of high performance tools and systems that can work at the web scale for real world applications.

Topics of Interest

Suggested topics include, but are not limited to:

• Opinion retrieval, extraction, categorization, and aggregation
• Topic and sentiment alignment in opinion analysis
• Applications of topic-sentiment analysis, e.g. corporate reputation measurement, political orientation categorization, customer preference study, public opinion study
• Issues in using topic-sentiment analysis as a new research method for mass opinion estimation, such as reliability, validity, sample bias, etc.
• Sentiment identification and filtering at various text granularity
• Domain-dependency of sentiment analyzers
• Evaluation methodologies
• Performance issues, scalability and efficiency
• Web-based system demonstration
• Novel algorithms, tools and systems
• Construction of benchmark data sets

Important Dates

• Individual workshop papers due: July 20, 2009
• Notification of Acceptance: August 10, 2009
• Camera ready: August 15, 2009 *(hard deadline for publication in proceedings)*
• Early registration deadline: August 15, 2009
• Workshop: November 6, 2009

Submission

• All workshop papers will be up to 8 pages, double column in the ACM format. No extra pages can be purchased for all workshop papers.
• Please ensure that your paper is formatted by using the ACM templates. Papers must be submitted in PDF files.
• At least one author of each accepted paper must register for the workshop. Registration must be done at the time when the author sends the camera-ready copy of the accepted paper to the workshop chair.
• Workshop proceedings will be printed along with the CIKM proceedings by ACM. Thus, the timeline to print proceedings must strictly follow with the CIKM proceedings schedule.
• Submission website will be up soon ...

Organizers

Co-Chairs:

Bei Yu, bei-yu@northwestern.edu, Northwestern University, USA
Maojin Jiang, jianmao@iit.edu, Illinois Institute of Technology, USA

Program Committee

James Allan, University of Massachusetts Amherst (USA)
Shlomo Argamon, Illinois Institute of Technology (USA)
Claire Cardie, Cornell University (USA)
Michael Gamon, Microsoft Research (USA)
Natalie Glance, Google (USA)
Xiao Hu, Riverglass (USA)
Matthew Hurst, Microsoft Live Lab (USA)
Panagiotis G. Ipeirotis, New York University (USA)
Maojin Jiang, Illinois Institute of Technology (USA)
Stefan Kaufmann, Northwestern University (USA)
Thomas Y. Lee, University of Pennsylvania (USA)
Qiaozhu Mei, University of Illinois at Urbana-Champaign (USA)
Ana-Maria Popescu, Yahoo! Labs (USA)
Richard Sproat, University of Illinois at Urbana-Champaign (USA)
Veselin Stoyanov, Cornell University (USA)
Stuart W. Shulman, University of Massachusetts Amherst (USA)
John D. Wilkerson, University of Washington (USA)
Waigen Yee, Illinois Institute of Technology (USA)
Bei Yu, Northwestern University (USA)
Chengxiang Zhai, University of Illinois at Urbana-Champaign (USA)
Xiaojin Zhu, University of Wisconsin-Madison (USA)

Topic-Sentiment Analysis Workshop, CIKM 2009

Briefly, a workshop proposal to CIKM 2009 entitled "1st International CIKM Workshop on Topic-Sentiment Analysis for Mass Opinion Measurement" has been accepted (I'm on the PC). Congratulations to Maojin Jiang and Bei Yu - this workshop represents a level of specialization that this important area deserves.

This year, CIKM is in Hong Kong (which is an wonderful place). See you there?

Twendz: Twitter and Sentiment

Waggener Edstrom (who do plenty of work for Microsoft) has released a Twitter topic/sentiment tool called Twendz (other coverage here and here). The site shows both a tag cloud of related terms and attempts to surface positive, negative and neutral posts on the topic. Thus far, I’m not too impressed with the sentiment analysis on a per tweet level. Here are some postitive false positives tweets about Jim Cramer:

• The most important video you'll watch this year: Jon Stewart ends Jim Cramer's career Video
• oh wow, John Stewart probably ended Jim Cramer's career last night, it was amazing.
• Please take 21 minutes and 12 seconds to watch this video. http://is.gd/nahG Jim Cramer and John Stewart discuss financial markets.

Here’s some of the negative false positives for Jim Cramer:

• Jim Cramer is not at fault! He'll agree to be blamed, but YOU PEOPLE are the ones investing! Be accountable! (still love J. Stewart, though)

Here’s some negative false positives or watchmen:

• Fair point But that is intensely faithful to the original graphic novel. It's dark, dark socio-political commentary. #watchmen
• back to designer mode and crazy to see Watchmen!
• Saw #Watchmen. Must watch for anyone interested in politics/visual effects. The complex philosophical plot kept me at the edge of my seat.

It is interesting to see this in the context of recent posting about the twittersphere making things easier for sentiment mining. While the system should be doing the right thing with valence shifters (negation, etc.) these examples also demonstrate the problem of topic association. In addition, the examples with Watchmen demonstrate the problems between the nature of the plot and the evaluation of the film. And, of course, like nearly every tool in this space, by providing a direct link to the raw data, the errors get surfaced and the user begins to question the value of the results.

Summary: I’m really excited to see that WE is building tools in this space. However, it looks like they are starting at the shallow end when it comes to sentiment mining.

Sentimine, Twittrratr

Briefly - a couple of new to me sentiment related things: Sentimine (for sentiment in general) and Twitrratr for twitter (c.f. flixpulse.com). Not only is the sentiment/opinion mining space starting to fill up, but so is the name space: Sentimine, Sentimetrix, SentimentMetrics, ...

Measuring Phenomena in Sentiment

I’m currently reading Logan Dillard’s Masters thesis on sentiment mining titled: “I Can’t Recommend This Paper Highly Enough”: Valence-Shifted Sentences in Sentiment Classification. It’s an interesting read, approaching the space in exactly the right way: looking for types, measuring their distribution and then using that information to improve existing methods. One of the key observations is captured in the table below. 15% of the sentences in the corpus studied contains valence shifters. Of those, 49.4% were valence shifters affecting verb phrases.

VS Type	Corpus	VS Sentences
negated verb phrase	7.4%	49.4%
negated noun phrase	3.9%	25.9%
negated adj phrase	2.9%	19.3%
modals	1.2%	8.1%
total	15.0%	100.%

In addition, it was observed that valence shifters are more often used in negative expressions than positive.

Note: valence shifters are lexical items which later (generally invert) the appraisal orientation of an expression; for example ‘not’ in ‘not good’.

ScoutLabs offers 30 day free trial

Having completed their beta phase, ScoutLabs has decided to roll in a 30 day free trial period to their offering. ScoutLabs blog posts is here, and CNet covers the story here. On their plans page, ScoutLabs lists the type of data they analyze: Blogs, open social networks, photo and video sharing sites, Twitter. It is interesting to see that they don’t include any message boards – where a lot of the gold is.

A while ago, I predicted that with the acquisition with some of the major players in the space (Cymfony, BuzzMetrics, Umbria), feature innovation would be seen from the less encumbered entrants. This is exactly what I’m seeing with ScoutLabs (see their feature list page).

To Simplify, Complexify

I remember an interview – years ago – with Benoit Mandelbrot talking about how he arrived at his famous fractal set. He quoted advice he got from his mentor, Julia:

To simplify, complexify!

This was, of course, in reference to imaginary numbers. By casting real numbers into their two dimensional imaginary counterparts, an extra degree of freedom is introduced. I’ve been taking this advice to heart when considering analysing social media for expressions of appraisal (the text mining problem formerly known as ‘sentiment mining’). Much research in this space tries to view the problem in a simple way  - positive and negative words, classifying whole documents, and so on. In this field, complexifying really means working back to a model – a rich description of social context, discourse context, linguistics and psychology – which aims to describe how these expressions of attitude end up in documents. Pushing for this model is already a big win. However, complexification has another great advantage – by proposing a rich model, one can find a principled component of that model to focus on and make valuable, incremental contributions to our understanding of the space.

The appraisal framework is a great place to start, and the work of Argamon et al, and Taboada are great examples of a more principled approach to this space.

Something that I’m having fun with just now is the relationship between appraisal and word sense. While much work in this space centers on building lists of ‘positive’ and ‘negative’ words, examples like ‘strong smell’, ‘strong candidate’, ‘strong personality’, suggest that the adjective (in these examples) requires a fine, sense based analysis.

Update – I found the quote from Mandelbrot via Google’s book search:

GeeYee : Opinion Mining

Bing Liu, a professor at the University of Illinois, Chicago, is one of the founders of GeeYee:

GeeYee is the industry leader in quantifying unstructured text, enabling you to accurately discover and compare a product's or subject's collective features.  With our solution, you can drill down to the most granular level, the positive, negative, and neutral opinions on these features, in their entirety, directly from the online source. 

(note – all companies are ‘the industry leader’, I never know why people write that stuff).

Bing is well known in the sentiment mining community and has done particularly interesting work in the space of discovering which features of a product attract expressions of opinion. For example, discovering automatically that it is the battery life of the camera which people care about.

I’m trying to find out when the company started (I’ve heard Bing speaking quite recently at a number of events and never heard him mention the company; his website hints at a startup ‘still in stealth mode…’). The domain for geeyee.com was registered in 2006, according to whois. They seem to have a demo up here (id required).

Others in this space include Lexalytics, Jodange, Sentimetrix, and Sentiment Metrics.