Data Mining: Text Mining, Visualization and Social Media

Twitter Spam - What is it?

Abdur over at twitter has an interesting post about Twitter's fight against spam. The information in the post is summarized by this graph:

The problem with Twitter spam (other than the existential one) is that because Twitter is such a great platform for novel communication behaviours, it has blurred the definition of spam. For example, say I visit a site which is offering me free downloads of music (sweet!). To get at the goodies, I have to allow the site to post on my Twitter acount a message saying that I downloaded a certain track from that site. That message goes out to my network, with the intention of getting them to visit the site as well.

Is that spam?

Spam is also dependant on the mode of consumption. If I follow a certain brand or outlet and it feeds me tweets about deals, we may both benefit. If I search and find nothing but links from thousands of accounts that the brand has set up to push such information is that spam? It sure looks like it.

While Abdur's graph says something about the fight with spam, what it doesn't say - i.e. what constitutes spam in Twitter and how such a taxonomy break down across tweets - is perhaps more interesting. Twitter does have some information here about their spam reporting system. I wonder if we could see a break down of the fight by the types mentioned there.

I think that with the complexity of interactions that Twitter allows, spam is more an experience than a fixed type of data.
 

End of Year Data: Facebook Currently Leads

Like me, John Battelle was underwhelmed by offerings from Twitter summarizing a year of tweets.

Nah. After reviewing them, it's clear that Twitter's first trends release is, well, a bit predictable. But I am sure there is really interesting data locked behind that rather obvious facade....we just can't see it. Yet.

John has similar things to say about YouTube’s offering:

Everyone's doing it! YouTube's most watched came out this morning. Again, nothing that eye opening. Wish they'd dig deeper.

Facebook, on the other hand, while not quite showing the same sort of data, is not afraid to dig deeper. The data teams post includes:

• Proportion of Minorities on Facebook in the U.S.
• Ethnic Breakdown of Facebook Users in the U.S. Named ‘LEE”
• Facebook U.S. Population by Ethnicity
• Relative Saturation of Ethnicities on Facebook in the U.S.

All of these are trended over time :-)

Clearly, the intended audiences are different, but hats off (again) to the Facebook data team.

For those of you who’d like to see trending over topics, here’s a sample from BlogPulse:

 

Who cares about the pandemic, some guy left his flies open!

Are You Looking Through the Twitterscope the Wrong Way?

Abdur has written up a post summarizing the year in terms of what people were tweeting about. For example, here's a list of 'news' topics:

News Events
1. #iranelection
2. Swine Flu
3. Gaza
4. Iran
5. Tehran
6. #swineflu
7. AIG
8. #uksnow
9. Earth Hour
10. #inaug09

Reading this, I feel somewhat disappointed. Twitter represents one of the most interesting data sets out there for text mining. Here's some thoughts on what he could have published:

1. term clusters: words that are strongly associated with each other (e.g. 'bush' and 'shoe')
2. terms associated with links: when people share a link, they are annotating that content, what are the popular links and what are the annotations?
3. long term trends: remember a trend can be up or down or any shape. Were there any terms or topics that were decreasing over time? how about terms that were slowly burning over the year and had a final flourish right now (avatar?)
4. diffusion patterns: were there topics that had very broad diffusion? topics that had very narrow diffusion? topics that reached a lot of people very quickly? topics that took longer?
5. cross cultural linking: how about the nature of following across country/language borders? do a lot of Scottish people follow Obama? how about French people following Armstrong?
6. what was the uptake of businesses using Twitter for promotion?
7. how about spam? spam is an arms race, any patterns destroyed? new ones emerging?

Let's not forget - in social systems, things are popular because they are popular, so frequency is not always the best thing to look for.

The Many Dimensions of Real Time

Update: A less abstract response to the Twitter as Cronkite post.

With posts like this driving the debate over the value of real time ‘news’ we should stop and think a little about the nature of time, information, the networks through which information is propagated and the values we associate with various aspects of this complex. I tend to think of the temporal aspect of data with the following timestamps:

• Ideation: the time at which the event occurred (or at which the author had the idea to write something to be published through their network)
• Authoring: the time at which the document was created
• Publication: the time at which the document was transmitted
• Consumption: the time at which the document was read

One can debate real time in terms of the difference between these time stamps. Often, the time between publication and consumption is really what is being discussed (and here the world has decided to leave blogs behind for the new shiny world of microblogs even though one can get pretty close to real time via pings and RSS). There is an assumption that this time difference is identical to that between ideation and consumption, but this is where it is important to pause and consider another key aspect of time and information:

• Content value

Content value increases wrt the time between ideation and consumption in inverse proportion to the probability that the information is accurate. For example, if experiencing an earthquake and reporting it has a low chance of being inaccurate and thus increased value. On the other hand, if I make a snap judgment about the stock market and report it as fact the content value decreases. Or, perhaps a better example, if I am sent a document reporting the behaviour of the president during his youth at an airforce base, then immediately broadcasting that may not be the best move.

Understanding the value of the content doesn’t stop there. There are two other factors:

• Relevance to the consumer
• Ability to take action

Information about Tiger Wood’s inability to control an oversized motor vehicle is of no interest to me, so I couldn’t care less if the information took an infinite amount of time to reach me. When it does reach me, what action can I take? Absolutely nothing. If there is an earthquake in Edinburgh, however, there is very direct action that I would take.

When I worked at Intelliseek/BuzzMetrics, many customers wanted what they called real time alerting. In working with these customers it became apparent that this meant daily updates. The reason being that their ability to take any action was moderated by at least 24 hours lag time anyway.

As is typical in the blogosphere, and especially when it comes to reading the tea leaves of the technologies of the future, I can’t see any real thoughtful debate on issues like those I’ve outlined about. I’m not saying that my framework is correct, but it does attempt to pull apart the issues. Having said that, analysing the space with this perspective may not actually help us predict where the money will be made, or what behaviours will ultimately prevail. Attentions spans are contracting at alarming rates and it is not Adam Smith’s job to look out for our shriveling minds.

Twitter, Bing and Google

Readers of this blog will have figured out that I’m swamped with shipping something right now. I just wanted to resurface and point enthusiastically to Bing’s launch of a Twitter search vertical. There are basically 3 features in this vertical: search results which show matching tweets, the tag cloud of hot topics (similar to what Twitter already has) and the list of top links from tweets that include a term – a sort of auto-digg driven by Twitter data.

The first two are required but to me not especially interesting. The third is really what I think the basics of real time search are all about – except that I would call it attention search: searching what people are paying attention to.

At any rate, I think the significance of Bing producing something like this is important. In addition, it has been produced in a manner that perfectly fits the data. Pushing out of the standard search experience to deliver something that is driven by the data model is a direction that Microsoft is clearly making with all things Bing. This should be contrasted with Google’s hot-on-the-heels announcement of their Twitter integration:

Given this new type of information and its value to search, we are very excited to announce that we have reached an agreement with Twitter to include their updates in our search results.

If the implications of this are that the Twitter data will be somehow blended into the SERP then Google is being a slave to its simplicity at the expense of leveraging the nature of the data. We’ll see when it surfaces. TechCrunch also points to this, although with what look like different expectations:

It’s worth noting that while Bing’s Twitter search is live today, it sounds like Google will be working Twitter’s data into its search results over the next few months.

It is also interesting to take a look at the media reaction. Search Engine Land gives a reasonably comprehensive overview. Techmeme is currently showing Bing getting a lot more attention on this (with a real product) compared to Google (with no real product). Of course, the real issue of attention is: how are Google, Apple, etc. going to rain on our parade tomorrow…

Gmail, Twitter Borked

I try to log in to Gmail – server error. I go to Twitter and find gmail is a ‘trending topic’. I click the word and Twitter tells me there are no results for that term.

Problems with the Twitter/Blog Hookup

I generally try to resist the temptation of putting up a link to each new blog post in my twitter feed. Of course, I generally fail to resist this temptation (one of the reasons I try to resist this is that my tweets appear on my blog and having them be a bunch of links to my post is not pleasant). Using the excellent backtweets.com, I can get a pretty good picture of who is tweeting my posts, who is retweeting and so on. This has lead me to notice some worrying issues that require some pretty detailed attention.

The title of my blog is ‘Data Mining: Text Mining, Visualization and Social Media’. When you bring up the page for a post, Typepad creates a title for the web page which is the blog title appended to the post title. I really don’t like this, but have thus far failed to find out the right lever to pull to disable this behaviour.

For example, my post drawing an analogy between Twitter and free electricity has the title ‘Data Mining: Text Mining, Visualization and Social Media: Twitter: Free electricity for All’. What a mess!

There are certain cases where people pass this link on in Twitter. Unfortunately, there is almost no way for any of the real meaning of the post to survive. I see things like:

• Data Mining: Text Mining, Visualization and Social Media: Twitter ... http://bit.ly/vHNry
• Data Mining: Text Mining, Visualization and Social Media: Twitter ...: In explaining the popularity of Twitter t.. http://bit.ly/19Xwq1

I see lots of the first and plenty of the second. Who would click on that? There’s no real information about what the article is about.

However, when people retweet my tweet about my blog posts, I see the content of my tweet – which I now try to make a concise representation of the post or at least the title. Thus:

• RT @matthewhurst Twitter is like free electricity: http://tinyurl.com/nvua9c
• Twitter is like free electricity ~ http://tr.im/wZHt

The latter is perhaps more like a stolen retweet, but you get the idea.

This is all pretty messy – and you may not care – but if you are setting up a blog at Typepad, be aware of the issue with blog/post title concatenation. If you want to better own the message of your blog post, you may have to give in and tweet all your blog posts as well.

Anyone else fumbling around with this stuff? Anyone from Typepad like to help out?

Twitter: Free Electricity for All

In explaining the popularity of Twitter to a new colleague, I suggested the following analogy:

A company starts providing free electricity. At first, only a few friends hook up to the service, but slowly neighborhoods and whole cities are using this amazing resource. Wired, Business Week and a thousand other popular publications feature the CEO of the company on their cover – the most successful new company of the year, amazing growth!

Twitter is like free electricity.

The Twitter Dichotomy

There are two basic ideas that Twitter pundits are focused on. The first is that Twitter is the centre of a universe of real time updates that will drive the engines of real time search. This is enabled partly by the behaviour of users and partly by the low latency publication and consumption mechanisms of the product. The second is that Twitter is just a load of "pointless babble" and just a waste of time.

The most surprising thing about the second point is that the pundits who write about that are simply saying that they don't understand Twitter - as well as a good chunk of social media and social networking in general. This article from the BBC 'Twitter Tweets are 40% Babble' demonstrates a shameful lack of analysis. It is based on the Pear Analytics study of a very small sample of Twitter data.

Fortunately, danah is on the ball enough to bring clarity to the situationby uttering the magic word - 'phatic'. If you're interested in a bigger picture, you should read Dunbar's Grooming, Gossip and the Evolution of Language in which the idea that language evolved to allow our ancestors to interact socially with better scale and distance (language - or gossip in particular - is has the same social function as pulling nits out of each other's matted hair).

There's another angle to this story that interest me - it suggests that most of the data is of the 'I'm doing this right now' type. That type of thing may be literally useless to someone not in the author's social group - the intended audience - but the value of this type of thing in aggregate is high. So where are the killer products that leverage this? If they exist they are hidden from the users and are being used to drive ranking functions and content discovery in search engines and the like. But what would it be like if this data mining were exposed to the public. Twitter trends gives us the thinnest possible idea of this.

Is there really a dichotomy here - between the pointless babble and the soul of the new real time machine? Probably not - given the volume of tweets, percentage-wise you don't need too much to get what is needed for the real time stuff.

Visualizing Moods on Twitter

Sune Lehmann, of the Barabasi Lab, pointed me to some recent work on detecting and visualizing ‘moods’ in Twitter: Twitter Mood:

We estimate the mood of all tweets for which the author has provided a location. The mood (valence) of each individual tweet is based on the Affective Norms for English Words (ANEW) data set. ANEW is a set of 1034 words, previously identified as bearing emotional weight (e.g. abuse, acceptance, accident). The mood of each word was estimated in a study at the University of Florida, where participants (college students), were shown lists of isolated words and asked to grade each word’s valence, arousal, and dominance level on an integer scale of 1-9. The mood of a tweet is calculated as the average valence ANEW words included in the tweet (if any).

I don’t see any information on the site regarding the accuracy of the mood detection algorithm. My guess would be that there are plenty of words out there that are not in the ANEW lexicon – it’d be interesting to see that study. The visualizations are both temporal and geographic:

This is, of course, similar to the work on tracking ‘moods’ in LiveJournal done at MoodViews.

For visualizations like this, I’d be interested in seeing some discussion as to how to ‘read’ the presentation, and for evidence of its predictive power.

At any rate, I’m looking forward to seeing how this develops.