Data Mining: Text Mining, Visualization and Social Media

Twitter Maths

There seems to be a little feeding frenzy going on regarding how much cash Twitter burns. After reading Paul Boulin’s original post, I was going to write up some thoughts. I see, now, that the conversation has progressed with Paul posting some details of the feedback he’s been getting. However, there is still something bugging me about the write up. It’s this bit:

Finally, the biggest problem of all is the network effect of Twitter’s popularity. As the number of people using a network goes up, the number of connections between them rises much, much faster.  Most Twitter users now have several times more followers than they did a year ago.  As a result, every update sent to all of a user's followers costs the company more to send to everyone who’s subscribed by phone.

It’s an exponential function: The more people use Twitter, the faster the number of messages grows.

Let’s imagine you have 4 people in a network. How many connections are there between these people? Each person can have at most 3 connections, so we can approximate the max size of the connections as 4². In general, the max number of links in a network of size N is approximated by N².

The word ‘exponential’, like the word ‘literally’ seems to be losing its meaning. Backing its invocation with a link to Wikipedia doesn’t really help.

[If you are interested, the term ‘literally’ is used to remove the metaphorical or idiomatic meaning from a phrase. Perhaps the best comment of this is a search on Google, which appears to turn up more articles about the word’s misuse than anything else!]

Animated Weblog Diffusion

As part of my presentation at the Personal Democracy Forum 2008, I showed an animation of a simulation of diffusion through the blogosphere. Anthony and Guilhem (also presenting at PDF) have just released a version of this on real data over on their Presidential Watch 08 site. They talk about it here, and the two demonstrations can be found here and here.

The visualization is an augmentation of the existing graph based view with a new display/control showing the attention over time to the video that is the subject of diffusion.

Weblog Activation Simulation

This post is partly to show the animation that I presented at PDF2008 and partly to test Vimeo (thanks to Jeff Jarvis and Jake for the recommendation).

This graph animation illustrates the reach over time of a blog post. When a blog is activated (when a node is selected by the pointer) other blogs that link to it have some chance of referring to the injected post. The animation is intended to give an impression of how information might spread in the blogosphere.

Firstly, I activate a few peripheral blogs – these don’t have many connections and so we don’t see much spread. Then a small community in the south west corner is activated – information spreads within the community but doesn’t break out. Finally I look at a few blogs in the central core. Here we see information spreading through the core.

Note that the graphical data is real – this is a visualization of the core of the blogosphere circa 2007; the spread of information is only a (simulated) illustration. Note also that there are many other (better) ways to model diffusion – I’ll publish more animations as and when I start looking into this area!

Weblog Activation Simulation from matthew hurst on Vimeo.

The weblogs that I use in this example are:

• Living Catholisism
• RevGalBlogPals
• Who Tends The Fires (defunct)
• Her Bad Mother
• Gateway Pundit
• Engadget
• BuzzMachine

Twingly: Innovating The (Blog) Search Interface

I really like the vision behind this interface to the blogosphere that the six Twingly interns are working on. It manages to push all the right buttons: blogosphere, graphs, zoomable interfaces and fun.

Large Graph Visualization

Information Aesthetics points to a nice collection of large graph renderings by Yifan Hu at AT & T Labs. There are many images to browse here - enjoy!

Networks in Technology Review

MIT Technology Review has a photo essay on network visualization which includes some of my work.

Last.fm Viz

Anonymous Prof posts some viz work representing the relationships between users of Last.fm.

 

Evaluating Graph Visualization Systems

One of the most frequent questions I get from readers of this blog is: can you recommend a graph visualization package? In general, I've always liked to roll my own - many of the layouts available in packages out there are much alike, so if you are interested in novel visualizations you would probably have to do this anyway.

I've not had time to really look at all the possibilities out there and really find one that I recommend. That being said, my most common answer is to take a look at GUESS, a system created by Eytan Adar at the University of Washington (and co-chair at ICWSM 2008).

If you do feel inclined to dig a little deeper in your evaluations, here are some things I would recommend looking out for:

• Does the package allow you to provide a simple per-line file of data points describing the graph? This would be the quickest way to evaluate anything?
• Does the package allow you to abstract the data layer and, for example, implement a database backed graph?
• Does the package provide multiple layout options?
• Does the package allow a clean API for adding your own layout managers?
• How quickly does it layout the data?
• Does it produce the same layout in repeated applications of a layout manager and the same data? Many layouts use some amount of randomization, but repeatability is key to good data visualization, so if a layout produces different results when applied repeatedly to the same data, it is a dud in my book.
• How much data can it handle? Of course, this will be determined by your application, but intuition suggests that if it can handle large volumes of data it will be better engineered.
• Does the package allow for graph computations (e.g. distance between two nodes)?

Six Degrees of Netflix

Eytan pointed me to this interesting post over at Netflix about using his graphing software to visualize the computed similarities between movies. Details can be found there.

Janice Caswell

I've been following the work of Janice Caswell  for a while now. Janice creates some really amazing pieces which communicate and explore the nature of interconnectivity and memory with a graphical flare. Her works are both installations and paper mounted.