Data Mining: Text Mining, Visualization and Social Media

Mapping The Influence of News on The Blogosphere

Inspired by a number of comments from The Hyperlinked Society Conference, I'm posting some preliminary results of including mainstream media nodes in a map of the blogosphere. The full image can be found here. These results are very preliminary and are aimed at simply getting a feel for what is going on.

In the graph, as in earlier work, the circular nodes represent blogs and the edges represent citations between blogs. The square orange nodes represent press nodes (the data is that published for WWE 2006 which contains classification of citation links including 'Press' - thanks to Natalie Glance).

The cluster at the centre of the image is the familiar core of the blogosphere. The links emanating from the core show clearly the importance of MSM to those and other blogs. The large blue cluster on the left of the core represents LiveJournal. A possible result of this analysis could be that MSM is less important to LiveJournal blogs than to others (though we don't need a graph to discover that).

It is interesting to see the smaller groups of non-core blogs that cluster around MSM sites.

Until a more analysis is carried out on this there isn't much point in making any claims - I just wanted to get this out there. Note that Yahoo! News is noticeably absent suggesting some issues with the process I am using, though it could be explained simply by the thresholding I am using to only include links which have a strength of 10 or more (i.e. a link is only present if a citation has been made at least 10 times from blog A to blog B, or blog A to MSM site B).

Cartographic Silence and Network Effects

Among the questions and responses to our panel on Mapping Lines of Influence (Hyperlinked Society), two stick out. The first was a comment from David Weinberger who pointed out that there is a great mass of the blogosphere that is not represented by the type of graph that I used to map the blogosphere. It occurred to me that different algorithms for projecting graph data have similar consequences to different projections for physical maps. I haven't yet stumbled on a solution to this problem. The mass of the blogosphere that is missing is missing due to their low linkage.

The second was a comment from Jeff Jarvis about where discussion of influence should be concentrated. Jeff referred to a post he made on 'the magic middle', aka 'the big butt' of the blogosphere. While I basically agree that this area is extremely interesting, it still remains that the network effects between the head and the middle give tremendous influence to the A-list (a term which is, perhaps, losing currency). The graphical analysis that I've presented here before, and which I used in the panel, shows clearly that the magic middle (the communities outside the blogosphere's core on topics like knitting, cooking, etc.) are very much aware of the core.

One thing I have to do to update my analysis is to include directionality in the display of the link structure. I am still making an assumption about the directionality of links in the map (i.e. that the peripheral communities link in to the core). It should be relatively straightforward to fix this. If my assumption is incorrect, then we should see something quite fascinating: communities that the core refers to but which do not give any link-love back.

Hyperlinked Society

I've just returned from participating in the Hyperlinked Society Conference at The Annenberg Public Policy Center at the University of Pennsylvania. I'll summarize my thoughts on the day in a later post. For now, I'm noting 2 to-dos:

1. Generate a map of linking in the blogosphere which includes links to news media sites.
2. Put the 'YouTube has more viewers than Adult Swim' debate to sleep. Jack Wakshlag declared that Adult Swim had more viewers than YouTube in response to a question I asked him regarding the fragmetting of media bundles (aka the loss of programming). AdultSwim stats are available, and appear to indicate around 1MM viewers. YouTube also has some stats floating around out there. Not the least of which is its Alexa numbers (ok, Alexa is not reliable, but I'm assuming in the top 100 the numbers are a reasonable indicator): ranked 23 with 30, 000 viewers per !MM web users. Paul Kedrosky links to this story from earlier in the year which puts YouTube's numbers at 35MM videos viewed in April 2006. More work is needed to figure out how to compare these numbers.