Data Mining: Text Mining, Visualization and Social Media

A Tale of Two Bells

An image search on Google for 'bell' produces (can you guess?) a multitude of pictures of the type of bell that makes a pleasant ringing sound when one calls for afternoon tea, not to mention the more coarse cousins to be found in ships and belfreys.

A similar search on Live's search engine produces (can you guess?) an array of images of people with the name Bell.

Ask follows Google's bias whereas Yahoo does mix object and people, though the pictures of people are not, shall we say, quite what you might expect.

What is interesting about these differences is that there are obvious biases in the search engines in terms of how they disambiguate terms (is it an object or a person that the user intends to find?). However, none of the search engines really acknowledges the ambiguity, nor do they assist the user in dealing with it. Perhaps the best attempt here is in Ask's interactive query completion, however this is far from perfect as typing 'bell' precludes any suggestions that start with a term other than 'bell'. Thus someone with the surname 'bell' is not going to surface at this point in the interaction.

 

Communities: The Third Age

If Usenet represents the first age of online communities, and web based message boards are the second age, then it may be that we are about to step in to the third age. TechCrunch reports a new product to come out of Meetro which brings a dose of Web 2.0 user experience to the creation and management of message boards. Perhaps more importantly, however, this new system will bring far more exposure via up front thinking wrt search engines and indexing.

A single product doesn't define a new age, and so it is worth also mentioning Six Apart's reported steps in the same direction: Movable Type Community Solution.

Hopefully, these new pushes into the forum space will include ping like notifications and perhaps even improvements to RSS.

Game Visualization

I posted recently about an article in Wired which presented player state at different time slices in Halo maps.

I'm currently playing Half Life 2: Episode 2 and recalled that Information Aesthetics had linked to similar maps from that game. SteamPowered has a nice set of stats for Valve games. For example, it shows the distribution of completion times for HL2: EP1 and HL2: EP2 (the modal time seems to be 4 hours for EP1 and 5 for EP2). The maps, of course, are the most interesting - showing the distribution of deaths by location. This is the map I'm currently playing:

Identity, Health and Socialism

A quality objective news source (e.g. the BBC - or do I mean i.e. the BBC) still has plenty of leeway for expressing opinion via juxtaposition. The BBC's news website has something of a tradition of reporting objectively on Bush's presidency while poking fun via the use of awkward photographs. Recently I heard on NPR an article regarding the next $50 billion spending spree on the Iraq war (that is to say $50 billion to take us to next March) followed closely by an article on the average income in India ($1, 000 per annum - couldn't we do something better with the $50 billion?).

Another great example this morning again from NPR was the presentation of report on a recent set of discussions that health care journalists have had with presidential candidates. This was followed by a discussion on the benefits that technology could bring to health care records and the privacy concerns that always ensue. What is the relationship of this to the topics on this blog? Well, it seems to me that the fears of privacy are associated in part with the fact that capitalist health care systems are all about exploitation, and that 'customers' would be right to feel exploited. Perhaps systems like Sxip, in conjunction with real, socialized medicine will be the future? The other future is online management of health records being coordinated with adverts trying to push drugs based on the mining of your records. Nice.

On Read Write Culture - Nothing New?

Below are some rough thoughts inspired by Lessig's recent talk at Microsoft. I've not had time yet to synthesize them into a coherent piece, but I'll risk posting them rather than risk never getting round writing them up properly.

Reuse, Recreate, Remix

The notions of reusing, recreating and remixing are nothing new - they are, one could argue, the driving forces of cultural evolution. Shakespeare: many of his plays are based on existing stories already, at the time of writing, produced as contemporary plays; Classical music: many grand symphonic pieces are based on interpretations of folk/country music, not to mention the whole genre of 'variations on a theme'.

Barriers to Entry, 'Democratization'

Digital technologies improve access to the tools that support the creative process and the distribution channels - but what were the barriers to entry for Elizabethan poets, composers, musicians, etc? Paper and ink were not fantastically pricey by any means (ref), the simple matter was that people had to spend a larger chunk of their time ensuring that there was food on the table. Thus patronage was the great enabler (either via a wealthy individual or via cooperatives that created and performed pieces of culture).

The Medium

A big mistake made by social media pundits is that there is a one to one relationship between a medium and the application of that medium. YouTube isn't a long tale of movie makers - many (most?) of YouTube is an application of the visual medium to soap boxing - that is to say, video blogging. Blogging itself has several applications, the two most noticeable being topical blogging and journalling.

Debate Transcript Analyzer

Neoformix has been posting a number of interesting applications recently. Check out Jeff's analyzer for debate transcripts.

Refer to the image below. The top section shows the distribution of some selected words within the text across a 'timeline' which goes from left to right. Each speech segment is the same width and the height of the small white bars show the number of occurences of that word for that segment. You can add new words with the text box in the top right corner or you can remove existing words by clicking on them.

Right below the word distribution graphs is a similar coloured set showing a spectral decomposition of the text based on who spoke and how much was said. In this case the bar heights give the amount of text for each segment. Click and drag the mouse left to right to move along the timeline and show the actual text for 3 consecutive segments. Mousing into this lower region will cause the blocks to expand and show more text.

On The Lessig Method

Lawrence Lessig visited Live Labs yesterday and gave a fascinating talk about the future of cultural artifacts and practices. This was my first time to see one of Lessig's presentations in person and I was very interested to see how the Lessig presentation method worked in practice. Subscribing broadly to Tufte's point of view with respect to the manner in which presentation packages like Power Point force a particular cognitive style, I believed I was going to enjoy the format.

While the content of the presentation was very good, informative and stimulating, I was surprised to feel that the presentation style itself was distracting. Much of Lessig's method amounts to displaying on the screen the exact words that he is speaking - in some sense the slides and the script (it has to be tightly scripted to work) are something like a performance rather than loosely coupled as many presentations are. Due to this redundancy, it was less of a burden to simply look at the presenter than it was to try to take in both streams - almost identical, but not quite resulting in more cognitive effort than if they had been clearly different.

Part of Tufte's reaction to the style of Power Point is to the use of slides as documents to be handed out for post-talk consumption. Interesting, although I no longer feel strongly in favour of the Lessig method per se, I think it may make for better post-talk consumption. Take a look at this presentation (utilizing the Lessig method) by Dick Hardt.

[Thanks to Alexey Maykov for the link to Hardt's talk.]

Trendpedia

Trendpedia is new to me, but not new. A beta product from Attentio - who is in the social media analysis space - it has a few similarities with BlogPulse (that is to say, it has a trend search component). It has a number of other features, such as a pie chart of the percentage of data from each trended query.

Lijitimate

You may have noticed a search box on the right hand side of this blog. Two things resulted in this. Firstly, Sundar Kadayam, the former CTO of BuzzMetrics, was curious why I didn't have a search box for my blog. I hadn't actually thought of it, but realised how useful it might be - even just for me. Secondly, I met Leslie Osborne and Tara Anderson recently at Defrag. They were demoing the Lijit.com product.

Lijit is not just a search box, however. In addition to providing me with stats regarding visitors and searches, it also gives visibility to my connections across a number of social services. Hitting the explore button below the search box overlays the below graphical description of relationships mined from the sites that I've entered in the configuration. I've not fully grok'd all the features that Lijit provides, but I does appear to be both fun and useful. (Note that I haven't turned on anything like the full compliment of Lijit features - check out Tara and Leslie's blogs for more stuff).

ICWSM 2008: Submission Site Live

The paper submission site for the International Conference on Weblogs and Social Media 2008 is now live!