Data Mining: Text Mining, Visualization and Social Media

Twitter in Web Search

It is interesting to see how twitter users are ranked in search engines. By searching for 'site:twitter.com' on live search, the top 5 accounts are:

1. Barack Obama 562k  followers
2. Jonathan Ross (Wossy) 169k
3. Biz Stone 248k
4. Pete Cashmore (mashable) 300k
5. ev 339k

Google has quite a different story. A search for 'site:twitter.com' produces the following accounts ranked from the top:

1. Andy Tarczon 213
2. Nadia W. 51
3. Ryan Stewart 4, 500
4. Zna Trainer 5, 281
5. Matt Haughey 3, 491

There appears to be a clear difference in ranking criteria. Googles' top ranked Twitter users are 2 orders of magnitude less 'popular' in terms of followers.

It is also interesting to look at the latency in the crawl. We can do this by looking at the cache pages for the accounts. Jonathan Ross' cached page on Live Search is from Feb 17th, Google's cache of Andy Tarczon is from March 2nd.

Web search engines are going to have to figure out what to do with Twitter. Certainly it is of huge value as a feature for ranking other content. Is it useful to index individual tweets? or accounts?

How to Improve Twitter Search

I’m starting to get Twitter as a publication mechanism, and I’ve always understood the value of mining the aggregate of tweets for any number of reasons. However, I don’t totally get Twitter search. In trying to, I’ve been playing around with it a little more. Here’s what I though:

• Why no link search? Most urls on Twitter go through TinyUrl or something similar, and yet these are not dereferenced. 23 % of tweets contain a url and so there is a big space of search targets for url search. I did a search for links to a blog post that I’d shortened and no results came up.
• Language classification doesn’t work. Of course, language classification is harder the shorter the text is, so Twitter is either asking for trouble by having it at all, or has some very cutting edge thoughts on how to do it. Plenty of languages can be bucketed by character encoding analysis (e.g. Korean, Japanese, Thai). However, do a search for something obscure and you’ll hit the language classification problem. E.g. a search for Elie Fife, my home town, produces no results (in the default English category), but a search over all language produces only English language results.
• Why no search counts? When results come back, we don’t know how many there were. For any type of rudimentary tracking, this count is important – at least for relative analysis.
• Add in time series. Time series in the results would be awesome. Some sites (like Twist from Flaptor) are attempting to provide this, but without the full data set, it is not clear how good or accurate these results are.
• Authorial search. Wouldn’t it be fine to search limited by features of the author? Certainly location is a no brainer.

I’m pretty sure that the lack of these features has nothing to do with the team’s ability to deliver them. It is likely a set of pragmatic issues to do with the value of these features in the context of their yet-to-be-decloaked business model.

Twendz: Twitter and Sentiment

Waggener Edstrom (who do plenty of work for Microsoft) has released a Twitter topic/sentiment tool called Twendz (other coverage here and here). The site shows both a tag cloud of related terms and attempts to surface positive, negative and neutral posts on the topic. Thus far, I’m not too impressed with the sentiment analysis on a per tweet level. Here are some postitive false positives tweets about Jim Cramer:

• The most important video you'll watch this year: Jon Stewart ends Jim Cramer's career Video
• oh wow, John Stewart probably ended Jim Cramer's career last night, it was amazing.
• Please take 21 minutes and 12 seconds to watch this video. http://is.gd/nahG Jim Cramer and John Stewart discuss financial markets.

Here’s some of the negative false positives for Jim Cramer:

• Jim Cramer is not at fault! He'll agree to be blamed, but YOU PEOPLE are the ones investing! Be accountable! (still love J. Stewart, though)

Here’s some negative false positives or watchmen:

• Fair point But that is intensely faithful to the original graphic novel. It's dark, dark socio-political commentary. #watchmen
• back to designer mode and crazy to see Watchmen!
• Saw #Watchmen. Must watch for anyone interested in politics/visual effects. The complex philosophical plot kept me at the edge of my seat.

It is interesting to see this in the context of recent posting about the twittersphere making things easier for sentiment mining. While the system should be doing the right thing with valence shifters (negation, etc.) these examples also demonstrate the problem of topic association. In addition, the examples with Watchmen demonstrate the problems between the nature of the plot and the evaluation of the film. And, of course, like nearly every tool in this space, by providing a direct link to the raw data, the errors get surfaced and the user begins to question the value of the results.

Summary: I’m really excited to see that WE is building tools in this space. However, it looks like they are starting at the shallow end when it comes to sentiment mining.

Tracking Twitter with trackingtwitter.com

A colleague here at Live Labs pointed me to a new service by Electric Artists called Tracking Twitter. The site provides rankings for a number of categories (brands, media and celebs). The site uses only structured data to do the tracking. It tracks the number of followers for the online identity of each category. Zappos is currently the top brand-with-a-twitter-identity with 179, 257 followers, which puts it below the 7th ranked celebrity.

While followers is certainly an interesting and necessary metric to track, the site doesn’t tell you anything about what people are talking about. This is perhaps not surprising as one would need more data than is available in the public timeline to really keep on top of this.

Update – Marc Schiller at Electric Artists says:

[I]f you click on the little grey speech bubble next to each feed name you will then see what people are saying on Twitter about that specific brand. We are making changes to the Tracker every day, and in a future rev we will highlight real time comments even more than it is now.

Temporal Information Retrieval

Greg posted a great review of Introduction to Information Retrieval by Manning, Raghavan and Shutze. I asked him if it covers any of the temporal issues that social media introduces to the IR space: answer no. At the same time, there is this post by John Borthwick about the threat that Twitter posses to Google. Coincidence?

Twitter Venn Diagrams

[Update: I was having issues with this app due to corporate proxy rules – it’s all good from home.]

Jeff Clark over at Neoformix has come up with a really simple but compelling application/visualization. For 2 or 3 keywords, it displays the venn diagram of search results over Twitter for the sets and their intersections. Of course, there is no reason why it needs to be pointing at Twitter – the data could come from anywhere.

Unfortunately the app is down right now (not sure if it is the client or Twitter search that is broken), but you can get a good idea of what it does from Jeff’s blog posts here and here.

Twitter DataViz for Marketing Orange Juice?

I find the concept of using a data visualization over tweets for marketing orange juice a little more interesting than the visualization itself, but what the heck, I’ll embed!

Twitter In and Out

David Winer posts about different user models for twitter in terms of followers, follwed and posts. This graph, taken from Why We Twitter by Akshay Java et al, provides a good description of the range of users.

Technorati Indexing Twitter

I just did a search for ICWSM on Technorati and it surfaced a tweet by Dave Sifry from his Twitter account. David is giving an invited talk at the conference!

Some Perspective on the Twitter Paradigm

A referral in my log lead me to this BBS page which (as far as I can make out) contains links to a number of Chinese Twitter clones:

• fanfau.com
• jiwai.de
• taotao.com
• v2ex.com/ing
• weazone.com/say.do
• wulog.com

I'm guessing this is just the time of the iceberg for Chinese Twitter sites - I wonder if there is an index of all Twitter clones, just to get an idea of how big this new paradigm is. Turns out there is some information here: Twitter Fan Wiki; Wired wrote about some others way back in May 2007.

A cursory eyeballing might suggest that there are certainly 10s of micro-blogging platforms out there, possibly more than 100?. It seems clear that aggregating and analysing this data is going to be a requirement for social media analysis firms throughout 2008 and beyond.