Data Mining: Text Mining, Visualization and Social Media

Visualization of Google Plus Graph

I took a sample of the Google Plus graph and loaded it up in to the excellent Gephi graph rendering package. Due to the simple manner in which I crawled the data, this doesn't represent a true breadth-first crawl of the graph (rather it captures the random set of in and out links that appear on a user's profile page). However, what is interesting is that there is a clear tightly connected component that is distinguished from a more dilute area. I suspect, without any real investigation, that this is the core area of Google employees and some of their immediate connections as well as alpha users. This is complete conjecture, though.

In capturing this data, I also observed the complete in and out degree of each node. The top ten that I came across where (in order): Mark Zuckerberg, Vic Gundotra, Robert Scoble, Tom Anderson, Matt Cutts, Markus Persson, Pete Cashmore, Thomas Hawk, Evan Williams and Om Malik with links ranging from 456k to 33k.

Finally, I noticed that 48% of users haven't yet posted.

Visualizing Directed Edges in Graphs : Don't Use Arrows

I just came across A User Study on Visualizing Directed Edges in Graphs by Danny Holten and Jarke J. van Wijk. The problem they look at in this paper is how to represented directed edges in visualization of graphs, e.g. to represent A is a part of B or A likes B, etc.

Interestingly, their results indicate that what I suspect might be the most commonly used approach (placing an arrow head at the end of the graph) is not the best. Rather, they suggest, using the tapered elements illustrated below is better.

The paper was published at CHI in 2009.

Malaysian Blogosphere Division

(Briefly) I've been working with some new data that our team has produced and created the view below. What I find striking about this visualization of 6k blogs is the clear division between two major clusters. A very limited drill down on the data suggests that all of these blogs are Malaysian in origin (and most are on Google's Blogspot). I don't yet have enough insight into this component to understand why there is a split - perhaps more to follow.

Visualizing Network Evolution

New to me is this work by Bergstrom and Rosvall on discoverying and visualizing changes over time in networks. The basic idea is to track nodes as the join and migrate between clusters in a network.

Some additional visualization are available here.
 

TechNet August 2009

TechNet’s August 2009 edition features on of my network visualization on the cover.

Twitter Maths

There seems to be a little feeding frenzy going on regarding how much cash Twitter burns. After reading Paul Boulin’s original post, I was going to write up some thoughts. I see, now, that the conversation has progressed with Paul posting some details of the feedback he’s been getting. However, there is still something bugging me about the write up. It’s this bit:

Finally, the biggest problem of all is the network effect of Twitter’s popularity. As the number of people using a network goes up, the number of connections between them rises much, much faster.  Most Twitter users now have several times more followers than they did a year ago.  As a result, every update sent to all of a user's followers costs the company more to send to everyone who’s subscribed by phone.

It’s an exponential function: The more people use Twitter, the faster the number of messages grows.

Let’s imagine you have 4 people in a network. How many connections are there between these people? Each person can have at most 3 connections, so we can approximate the max size of the connections as 4². In general, the max number of links in a network of size N is approximated by N².

The word ‘exponential’, like the word ‘literally’ seems to be losing its meaning. Backing its invocation with a link to Wikipedia doesn’t really help.

[If you are interested, the term ‘literally’ is used to remove the metaphorical or idiomatic meaning from a phrase. Perhaps the best comment of this is a search on Google, which appears to turn up more articles about the word’s misuse than anything else!]

Animated Weblog Diffusion

As part of my presentation at the Personal Democracy Forum 2008, I showed an animation of a simulation of diffusion through the blogosphere. Anthony and Guilhem (also presenting at PDF) have just released a version of this on real data over on their Presidential Watch 08 site. They talk about it here, and the two demonstrations can be found here and here.

The visualization is an augmentation of the existing graph based view with a new display/control showing the attention over time to the video that is the subject of diffusion.

Weblog Activation Simulation

This post is partly to show the animation that I presented at PDF2008 and partly to test Vimeo (thanks to Jeff Jarvis and Jake for the recommendation).

This graph animation illustrates the reach over time of a blog post. When a blog is activated (when a node is selected by the pointer) other blogs that link to it have some chance of referring to the injected post. The animation is intended to give an impression of how information might spread in the blogosphere.

Firstly, I activate a few peripheral blogs – these don’t have many connections and so we don’t see much spread. Then a small community in the south west corner is activated – information spreads within the community but doesn’t break out. Finally I look at a few blogs in the central core. Here we see information spreading through the core.

Note that the graphical data is real – this is a visualization of the core of the blogosphere circa 2007; the spread of information is only a (simulated) illustration. Note also that there are many other (better) ways to model diffusion – I’ll publish more animations as and when I start looking into this area!

Weblog Activation Simulation from matthew hurst on Vimeo.

The weblogs that I use in this example are:

• Living Catholisism
• RevGalBlogPals
• Who Tends The Fires (defunct)
• Her Bad Mother
• Gateway Pundit
• Engadget
• BuzzMachine

Twingly: Innovating The (Blog) Search Interface

I really like the vision behind this interface to the blogosphere that the six Twingly interns are working on. It manages to push all the right buttons: blogosphere, graphs, zoomable interfaces and fun.

Large Graph Visualization

Information Aesthetics points to a nice collection of large graph renderings by Yifan Hu at AT & T Labs. There are many images to browse here - enjoy!