Data Mining: Text Mining, Visualization and Social Media

AAAI Opens Access to Digital Library

I've just heard that AAAI (the parent organization of ICWSM and many other outstanding conferences) has completed the opening of access to online versions of all papers. This is a great step to make and I believe that all will benefit from it (authors, conferences and AAAI).

Some key conferences:

• ICWSM 2008
• AAAI 1980 - the first AAAI conferece!
• KDD 1996 - the first KDD conference while still at AAAI

Congratulations to everyone at AAAI!

Twitter Maths

There seems to be a little feeding frenzy going on regarding how much cash Twitter burns. After reading Paul Boulin’s original post, I was going to write up some thoughts. I see, now, that the conversation has progressed with Paul posting some details of the feedback he’s been getting. However, there is still something bugging me about the write up. It’s this bit:

Finally, the biggest problem of all is the network effect of Twitter’s popularity. As the number of people using a network goes up, the number of connections between them rises much, much faster.  Most Twitter users now have several times more followers than they did a year ago.  As a result, every update sent to all of a user's followers costs the company more to send to everyone who’s subscribed by phone.

It’s an exponential function: The more people use Twitter, the faster the number of messages grows.

Let’s imagine you have 4 people in a network. How many connections are there between these people? Each person can have at most 3 connections, so we can approximate the max size of the connections as 4². In general, the max number of links in a network of size N is approximated by N².

The word ‘exponential’, like the word ‘literally’ seems to be losing its meaning. Backing its invocation with a link to Wikipedia doesn’t really help.

[If you are interested, the term ‘literally’ is used to remove the metaphorical or idiomatic meaning from a phrase. Perhaps the best comment of this is a search on Google, which appears to turn up more articles about the word’s misuse than anything else!]

How Old Are You Really?

Bryant Smith sent me a link to a tool he's developed which uses a game-like mechanism to gather age perception data from users: How Old Are You?While it is interesting to compare your guess with the ground truth, I think the real opportunity here is in getting the data in the hands of a number of groups including psychologists and machine learning people.

I've spoken with Bryant about this and he's certainly open to the idea.

The Knol Mystery

Silicon Alley Insider posts asking why Knol, Google's Wikipedia-ish content service, still survives while many of Google's experiments and acquisitions have fallen by the wayside. I think there are two ways to answer to this question. The first is: the only thing that has surprised Google in the past few years has been Wikipedia. Who would have thought that a single site would end up appearing on a huge fraction of search result pages (I can't find an exact statistic on this, even using Google...). The second answer, relating to the persistance of the project, is to do with focus. There are many projects within Microsoft which are (or have been) considered core strategic plays and thus have had consistent investment because a results has to be delivered. Today's trial-by-fire, show-me-the-numbers, has-to-be-measurable approach to innovation is quite alright, but sometimes the (top down) knowledge and intuition of the leader(s) must be heard! Perhaps there is a kill Wikipedia mandate at Google?

Data, Statistics and Unbelievers

I was pretty impressed with Obama's inauguration speech. He almost slipped by without using the word 'terrorist' or any of its relations (he used 'terror' once) and he gave respect for science and rationalism a shot in the arm by appealing to data and statistics:

These are the indicators of crisis, subject to data and statistics. Less measurable, but no less profound, is a sapping of confidence across our land; a nagging fear that America's decline is inevitable, that the next generation must lower its sights.

He almost did right by the rest of us by referring to those of us for whom faith is not a concept, although he used a term which centres on faith. One can be sure that every word in the speech had a great deal of thought behind it, and that every word was balanced with its synonyms. We can infer, therefore, that Obama's team considers atheist as being in some way wrong. It is arguably the most accurate term, and so it must be wrong for political reasons. Perhaps 'unbelievers' is a way of being inclusive while still indicating his theological position.

Naturally, Neoformix has the whole thing in perspective.

Social Media Search is Different

Twingly (certainly a company to watch in this space) today announced their microblog(twitter, jaiku, etc.) search engine. While I think it is great to see more activity in this space, I can't help but be puzzled by current approaches to searching social media. We are faced with a very different type of content, with a very different ecology and yet we throw archaic interaction paradigms around it.

Take a search on the Twingly search engine for 'inauguration day". I get back a set of results with the following:

• Happy Inauguration day!
• Twitter is going to be so bogged down today. I already have been unable to see pics for 24 hours. Happy Inauguration Day everyone!
• Watching Inauguration DayLIVE on MSNBC. <3 GOBAMA 2009.
• Wish I was in DC; happy inauguration day everyone!
• Happy Inauguration Day Oregon! Where's everyone watching today?

I'm trying to figure out what I learned from this. It seems obvious to me that there is a lot to be learned from microblog data, but search is exactly the way to bury that information. I'd much rather see

• Trends: show me how these terms are changing over time
• Associations: show me aggregates of terms associated with the query
• Author information: tell me something about the distribution of demographic features of the authors.

Photosynth and the 44th President

Photosynth – a Microsoft product that was incubated in Live Labs – has joined up with CNN to create a synth of Obama’s inauguration.

We’re partnering with CNN to gather thousands of your photographs to create an immersive experience of the moment when President Obama takes the Oath of Office. From the vast sweep of the crowd to a close-up on the President’s hand on the Bible, every part of this historic scene will be frozen in time and presented in 3-D as only Photosynth can.

Well placed CNN photographers will capture lots of detailed shots for this synth, and if you’re there you can help make this an even more amazing experience by adding your perspective with your digital camera or camera phone.

Check out CNN’s page dedicated to this collaboration: http://cnn.com/themoment.

Single Cell Web Search

A search on Google’s blog search for ‘hudson crash video’ brings up a post I wrote asking where the citizen journalist video of this event was. The same search on Google’s main search includes some horrible spam/malware (on the first SERP). While web search has come a long way, these examples really make me believe that we have so much further to go.

Marissa Mayer famously stated, then retracted, that web search was 90% done. In terms of understanding the complexity of the problem, we are approaching it like Minsky and his summer project to solve computer vision. In terms of (expected) utility, we are so hugely underestimating what could be done with the data available online. The percentage-complete attitude makes a strong assumption about what is possible.

What is confining our vision? The user interface.

Where's the US Airways Hudson Crash Video?

With all the talk about how social media, twitter, etc. is going to revolutionize reporting - or rather, the collection of potentially newsworthy information - where is the video of the US Airways crash on the Hudson? I don't mean the video of it sitting in the river, or videos of other water crash landings. Huff seems to be collecting plenty of content, but as yet the video-most-likely-to-get-viewers-to-your-screen has not surfaced. Of course, that hasn't stopped SEO types and spammers from trying to make you think that they have it.

It's also an interesting challenge for search. I don't want a video on a page about the crash, or a video of some other crash, ...

Blog Hosting Statistics

Pingdom has published some interesting statistics regarding the use of different blogging platforms and technologies within the Technorati top 100. There are interesting not just due to the basic distributional data (Wordpress accounts for 27 of the top 100, MT for 8, Typepad for 16, Blogger for 3) but also in what the report reveals about how Technorati computes the top 100. For example, Weblogs, Inc. accounts for 14 of the top 100, Wired for 9 and Gawker for 9. I'd be interested in looking at the sister linking between sites in these networks.

Another part of this signal is to do with comment spam. The more popular the blog, the more important it is that the authors have control over spam.