Data Mining: Text Mining, Visualization and Social Media

Topical Sentiment Workshop Programme

The 1st International CIKM Workshop on Topic-Sentiment Analysis for Mass Opinion Measurement has published the programme for the day's event. The papers are listed below, and I've linked to the papers where they are available online.

• Patterns in the Stream: Exploring the Interaction of Polarity, Topic, and Discourse in a Large Opinion Corpus    Julian Brooke (University of Toronto, Canada)    Matthew Hurst (Microsoft Live Labs, United States)
• Topic-dependent Sentiment Analysis of Financial Blogs    Neil O'Hare (CDVP, Dublin City University, Ireland)    Michael Davy (Dublin City University, Ireland)    Adam Bermingham (Dublin City University, Ireland)    Paul Ferguson (CLARITY: Centre for Sensor Web Technologies, Ireland)    Paraic Sheridan (Dublin City University, Ireland)    Cathal Gurrin (Dublin City University, Ireland)    Alan Smeaton (Dublin City University, Ireland)
• Scary Films Good, Scary Flights Bad: Topic driven feature selection for classification of sentiment Scott Nowson (Appen Pty Ltd, Australia)
• Evaluating Question Answering Systems on Opinion Queries    Alexandra Balahur (DLSI, University of Alicante, Spain)    Ester Boldrini (Universidad de Alicante, DLSI, Spain)    Andrés Montoyo (University of Alicante, Spain)    Patricio Martínez-Barco (University of Alicante, Spain)
• Automatic Creation of a Reference Corpus for Political Opinion Mining in User-Generated Content    Luis Sarmento (Universidade do Porto, Portugal)    Paula Carvalho (University of Lisbon, Faculty of Sciences, LASIGE, Portugal)    Mário J. Silva (University of Lisbon, Faculty of Sciences, LASIGE, Portugal)    Eugénio Oliveira (Faculdade de Engenharia Universidade do Porto - LIACC, Portugal)
• Domain-specific Sentiment Analysis using Contextual Feature Generation    Yoonjung Choi (KAIST, Korea, Republic of)    Youngho Kim (KAIST, Korea, Republic of)    Sung-Hyon Myaeng (KAIST, Korea, Republic of)
• Weakly Supervised Techniques for Domain-Independent Sentiment Classification    Jonathon Read (University of Sussex, United Kingdom)    John Carroll (University of Sussex, United Kingdom)
• Clues for Detecting Irony in User-Generated Contents: Oh...!! It's " so easy" ;-)    Paula Carvalho (University of Lisbon, Faculty of Sciences, LASIGE, Portugal)    Luis Sarmento (Universidade do Porto, Portugal)    Mário J. Silva (University of Lisbon, Faculty of Sciences, LASIGE, Portugal)    Eugénio Oliveira (Faculdade de Engenharia Universidade do Porto - LIACC, Portugal)
• Beyond the Stars: Exploiting Free-Text User Reviews for Improving the Accuracy of Movie Recommendations    Niklas Jakob (Technische Universität Darmstadt, Germany)    Stefan Hagen Weber (Siemens AG, Germany)    Mark-Christoph Müller (Technische Universität Darmstadt, Germany)    Iryna Gurevych (Darmstadt University of Technology, Germany)
• Aspect-based Sentence Segmentation for Sentiment Summarization    Jingbo Zhu (Northeastern University, China)    Muhua Zhu (Northeastern University, China)    Huizhen Wang (Northeastern University, China)    Benjamin Tsou (City University of Hong Kong, Hong Kong)
• Locally Contextualized Smoothing of Language Models for Sentiment Sentence Retrieval   Takayoshi Okamoto (Kobe University, Japan)    Tetsuya Honda (Kobe University, Japan)    Koji Eguchi (Kobe University, Japan)
• Sentiment Analysis of Movie Reviews on Discussion Boards using a Linguistic Approach    Tun Thura Thet (Nanyang Technological University, Singapore)    Jin-Cheon Na (Nanyang Technological University, Singapore)    Christopher S.G. Khoo (Nanyang Technological University, Singapore)

Visualizing Moods on Twitter

Sune Lehmann, of the Barabasi Lab, pointed me to some recent work on detecting and visualizing ‘moods’ in Twitter: Twitter Mood:

We estimate the mood of all tweets for which the author has provided a location. The mood (valence) of each individual tweet is based on the Affective Norms for English Words (ANEW) data set. ANEW is a set of 1034 words, previously identified as bearing emotional weight (e.g. abuse, acceptance, accident). The mood of each word was estimated in a study at the University of Florida, where participants (college students), were shown lists of isolated words and asked to grade each word’s valence, arousal, and dominance level on an integer scale of 1-9. The mood of a tweet is calculated as the average valence ANEW words included in the tweet (if any).

I don’t see any information on the site regarding the accuracy of the mood detection algorithm. My guess would be that there are plenty of words out there that are not in the ANEW lexicon – it’d be interesting to see that study. The visualizations are both temporal and geographic:

This is, of course, similar to the work on tracking ‘moods’ in LiveJournal done at MoodViews.

For visualizations like this, I’d be interested in seeing some discussion as to how to ‘read’ the presentation, and for evidence of its predictive power.

At any rate, I’m looking forward to seeing how this develops.

Sentiment Papers at KDD 2009

There were a few of papers at KDD this year on sentiment mining:

• Sentiment Analysis of Blogs by Combining Lexical Knowledge with Text Classification, Prem Melville, IBM; Wojciech Gryc, ; Richard Lawrence, IBM, USA
• Entity Discovery and Assignment for Opinion Mining Applications, Xiaowen Ding, Univ of Illinois at Chicago; Bing Liu, UIC; Lei Zhang, UIC
• OpinionMiner: A Machine Learning System for Web Opinion Mining and Extraction, Wei Jin, North Dakota State University; Hung Hay Ho

Raveable: hotel appraisal

Raveable is a new review aggregation site which specializes in hotels. It claims to provide information for 55 thousand hotels in the US culled from more than 35 million reviews. The site mines these reviews to automate the summarization of opinions from guests for the hotels.

The data

Firstly, where does the data come from? Paging through a number of hotel sheets, it initially appeared to me as if most of the data comes from TripAdvisor. The site states that the data comes from a number of sources, including TripAdvisor, Expedia, Travelocity, etc. as well as individual blogs. Browsing the site brought more of these sources into view – in fact, the experience suggests that some hotels get more data from one source than the other. It would be interesting to know the demographics of those sources and to consider a more representative summary taking the demographics into account. Has Raveable formed relationships with those sources?

The UI

In general, I found the UI clean and pleasantly put together. One of the most important aspects of presenting data for a choice like hotel booking is to get some understanding of the ranking of the candidates and the coverage of the system (how many of the competing hotels are in the system). The UI goes some of the way here by providing an overall ranking:

However, it would be nice to get a bit of additional geographic context that shows other hotels near by. In addition, with the added layer of appraisal (summaries of reviews) one could get a picture of the general quality of the hotels in the neighbourhood. Am I looking at a good hotel in a forest of good hotels? A good hotel in a sparse neighbourhood?

The Interesting Part

Of course, the interesting part about this whole thing is the automated summarization and extraction of opinions written in natural language. With 636 reviews available, on average, per hotel, there should be good opportunity to deal with accuracy issues by doing something sensible with summarization and presentation (note that I’ve not yet seen a hotel with > 600 reviews so I’m not sure if all the reviews are assigned to hotels in the system).

My expectations for the extraction of opinions was that the positive stuff would generally be pretty accurate, but negative comments – always harder than positive to recognize – would show some amount of error. Some anecdotal examples (pulled without too much hunting):

• Our room was on the 3rd floor we were offered a room at the back or the front, we chose the front whats the point of being in new york with out the noise. [Casablanca Hotel]
• The rooms are not large but are more than adequate. [Casablanca Hotel]
• You can see the old bank vaults in the basement and the registration area on the first floor is where the tellers used to be. [Drury Plaza Hotel]
• Housekeeping was called and eliminated the two problems. [Camelback Inn]

These false positives are interesting in terms of what they tell us about the algorithms that compute the sentiment. There is some background value associated with certain terms (noise in the first example triggered a negative tag; large in the second and problem in the third). The trouble comes with the valence associated with the terms by the surrounding context.

I don’t feel that these problems are really a weakness in the system. There are always going to be errors. I believe there are two ways to combat this problem in general. The most obvious is figure out a way to introduce another level of summarization in the UX. There is high level summary for each major category. One could try something simple, like a tag cloud, that summarizes within each category in a way that would allow the user to get a finer grained, but still aggregate summary of the content.

ACL 2009 Accepted Papers

The list is up. There are a few papers on sentiment mining. Among them, I find this title to be the most interesting: Mine the Easy and Classify the Hard: Experiments with Automatic Sentiment Classification, by Sajib Dasgupta and Vincent Ng.

CIKM Workshop on Topic-Sentiment

The workshop that I mentioned briefly in an earlier post now has a CFP.

1st International CIKM Workshop on Topic-Sentiment Analysis for Mass Opinion Measurement (Hong Kong Nov. 6, 2009)

News

03-22-2009: Workshop site is up

Scope of the Workshop

This workshop seeks to bring together researchers in both computer science and social sciences who are interested in developing and using topic-sentiment analysis methods to measure mass opinion, and to foster communications between the research community and industry practitioners as well.

The increasing amount of user-generated content on the Internet and social media and the digitization of large number of government and institutional documents provide new opinion-rich data sources for researchers to examine individual and group perceptions on products, organizations, and social issues at a large scale, and thus contribute to the research and practice in the areas of political science, social policy, communications, and business intelligence.

On the other hand, researchers are tackling the problem of processing large amount of opinion-rich data using various approaches. The increasing number of relevant publications in top data mining, information retrieval and natural language processing conferences (KDD, SIGIR, ACL, WWW, etc.) has witnessed the growing interest in automatic opinion analysis. Both TREC and TAC (Text Analysis Conference) have set up individual tracks for opinion retrieval and analysis tasks.

In recent years topic detection and tracking techniques have been well developed to identify the issues discussed in a large text collection. Sentiment analysis is catching up to detect the polarity of opinions expressed in texts. However, many times real-world applications have to take into consideration of both topics and sentiments for precise opinion measurement. Topic and sentiment alignment is crucial for opinion retrieval, extraction, categorization, and aggregation on various issues. Topics and sentiments could also have sophisticated interactions. For example, the choice of topics and the attention distribution among topics might bear hidden opinions as well.

How do we build synergistic topic and sentiment models for text documents? How do we tackle the domain-dependency problem of sentiment analysis? How do we identify users' needs and integrate them into the design of opinion analysis systems? What are the successful applications of topic-sentiment analysis for mass opinion measurement? What lessons have the pioneers learned? How do we evaluate the automatic mass opinion measuring tools with regard to the reliability and validity? This workshop solicits submissions to address these problems and more.

We hope this workshop can advance research in topic-sentiment analysis, make connections between research community and industry practitioners and encourage development of high performance tools and systems that can work at the web scale for real world applications.

Topics of Interest

Suggested topics include, but are not limited to:

• Opinion retrieval, extraction, categorization, and aggregation
• Topic and sentiment alignment in opinion analysis
• Applications of topic-sentiment analysis, e.g. corporate reputation measurement, political orientation categorization, customer preference study, public opinion study
• Issues in using topic-sentiment analysis as a new research method for mass opinion estimation, such as reliability, validity, sample bias, etc.
• Sentiment identification and filtering at various text granularity
• Domain-dependency of sentiment analyzers
• Evaluation methodologies
• Performance issues, scalability and efficiency
• Web-based system demonstration
• Novel algorithms, tools and systems
• Construction of benchmark data sets

Important Dates

• Individual workshop papers due: July 20, 2009
• Notification of Acceptance: August 10, 2009
• Camera ready: August 15, 2009 *(hard deadline for publication in proceedings)*
• Early registration deadline: August 15, 2009
• Workshop: November 6, 2009

Submission

• All workshop papers will be up to 8 pages, double column in the ACM format. No extra pages can be purchased for all workshop papers.
• Please ensure that your paper is formatted by using the ACM templates. Papers must be submitted in PDF files.
• At least one author of each accepted paper must register for the workshop. Registration must be done at the time when the author sends the camera-ready copy of the accepted paper to the workshop chair.
• Workshop proceedings will be printed along with the CIKM proceedings by ACM. Thus, the timeline to print proceedings must strictly follow with the CIKM proceedings schedule.
• Submission website will be up soon ...

Organizers

Co-Chairs:

Bei Yu, [email protected], Northwestern University, USA
Maojin Jiang, [email protected], Illinois Institute of Technology, USA

Program Committee

James Allan, University of Massachusetts Amherst (USA)
Shlomo Argamon, Illinois Institute of Technology (USA)
Claire Cardie, Cornell University (USA)
Michael Gamon, Microsoft Research (USA)
Natalie Glance, Google (USA)
Xiao Hu, Riverglass (USA)
Matthew Hurst, Microsoft Live Lab (USA)
Panagiotis G. Ipeirotis, New York University (USA)
Maojin Jiang, Illinois Institute of Technology (USA)
Stefan Kaufmann, Northwestern University (USA)
Thomas Y. Lee, University of Pennsylvania (USA)
Qiaozhu Mei, University of Illinois at Urbana-Champaign (USA)
Ana-Maria Popescu, Yahoo! Labs (USA)
Richard Sproat, University of Illinois at Urbana-Champaign (USA)
Veselin Stoyanov, Cornell University (USA)
Stuart W. Shulman, University of Massachusetts Amherst (USA)
John D. Wilkerson, University of Washington (USA)
Waigen Yee, Illinois Institute of Technology (USA)
Bei Yu, Northwestern University (USA)
Chengxiang Zhai, University of Illinois at Urbana-Champaign (USA)
Xiaojin Zhu, University of Wisconsin-Madison (USA)

Topic-Sentiment Analysis Workshop, CIKM 2009

Briefly, a workshop proposal to CIKM 2009 entitled "1st International CIKM Workshop on Topic-Sentiment Analysis for Mass Opinion Measurement" has been accepted (I'm on the PC). Congratulations to Maojin Jiang and Bei Yu - this workshop represents a level of specialization that this important area deserves.

This year, CIKM is in Hong Kong (which is an wonderful place). See you there?

Twendz: Twitter and Sentiment

Waggener Edstrom (who do plenty of work for Microsoft) has released a Twitter topic/sentiment tool called Twendz (other coverage here and here). The site shows both a tag cloud of related terms and attempts to surface positive, negative and neutral posts on the topic. Thus far, I’m not too impressed with the sentiment analysis on a per tweet level. Here are some postitive false positives tweets about Jim Cramer:

• The most important video you'll watch this year: Jon Stewart ends Jim Cramer's career Video
• oh wow, John Stewart probably ended Jim Cramer's career last night, it was amazing.
• Please take 21 minutes and 12 seconds to watch this video. http://is.gd/nahG Jim Cramer and John Stewart discuss financial markets.

Here’s some of the negative false positives for Jim Cramer:

• Jim Cramer is not at fault! He'll agree to be blamed, but YOU PEOPLE are the ones investing! Be accountable! (still love J. Stewart, though)

Here’s some negative false positives or watchmen:

• Fair point But that is intensely faithful to the original graphic novel. It's dark, dark socio-political commentary. #watchmen
• back to designer mode and crazy to see Watchmen!
• Saw #Watchmen. Must watch for anyone interested in politics/visual effects. The complex philosophical plot kept me at the edge of my seat.

It is interesting to see this in the context of recent posting about the twittersphere making things easier for sentiment mining. While the system should be doing the right thing with valence shifters (negation, etc.) these examples also demonstrate the problem of topic association. In addition, the examples with Watchmen demonstrate the problems between the nature of the plot and the evaluation of the film. And, of course, like nearly every tool in this space, by providing a direct link to the raw data, the errors get surfaced and the user begins to question the value of the results.

Summary: I’m really excited to see that WE is building tools in this space. However, it looks like they are starting at the shallow end when it comes to sentiment mining.

Sentimine, Twittrratr

Briefly - a couple of new to me sentiment related things: Sentimine (for sentiment in general) and Twitrratr for twitter (c.f. flixpulse.com). Not only is the sentiment/opinion mining space starting to fill up, but so is the name space: Sentimine, Sentimetrix, SentimentMetrics, ...

Measuring Phenomena in Sentiment

I’m currently reading Logan Dillard’s Masters thesis on sentiment mining titled: “I Can’t Recommend This Paper Highly Enough”: Valence-Shifted Sentences in Sentiment Classification. It’s an interesting read, approaching the space in exactly the right way: looking for types, measuring their distribution and then using that information to improve existing methods. One of the key observations is captured in the table below. 15% of the sentences in the corpus studied contains valence shifters. Of those, 49.4% were valence shifters affecting verb phrases.

VS Type	Corpus	VS Sentences
negated verb phrase	7.4%	49.4%
negated noun phrase	3.9%	25.9%
negated adj phrase	2.9%	19.3%
modals	1.2%	8.1%
total	15.0%	100.%

In addition, it was observed that valence shifters are more often used in negative expressions than positive.

Note: valence shifters are lexical items which later (generally invert) the appraisal orientation of an expression; for example ‘not’ in ‘not good’.