Data Mining: Text Mining, Visualization and Social Media

24 Hours of Blogs, Part 2

Taking the domains of all the posts and sorting by posting frequency according to the 24 hour dataset, the top 20 domains are as follows:

146920 msn.com
  62179 blogspot.com
  53041 search-now130.com
  40225 findallarticles.com
  38509 search-now140.com
  28478 look4articles.com
  23049 findactions.com
  13375 find4news.com
   4472 wefindforyou.com
   3919 livejournal.com
   2633 persianblog.com
   2604 blogdrive.com
   2232 blogfa.com
   2001 cocolog-nifty.com
   1627 typepad.com
   1141 kotobabooks.com
    989 canalblog.com
    987 choose-champagne.com
    665 myblogsite.com
    639 blog.com

It is interesting to see how the simple time series provides some pretty clear insights. Here is a graph showing the posting frequency for the total data set as well as those for msn.com, blogspot.com and findallarticles.com. Where as msn.com (blue) and blogspot.com (green) follow the global trend, findallarticles.com (red) shows a very steady flat trend. Not surprisingly, this is a spam blog. Note also how there are a couple of outage points for the spam blog which are reflected in the global time series as dips in the data.

It is also remarkable that blogspot's posts are far less organized than msn's:

I'm not going to guess why that is just now, but it is interesting to say the least. In fact, looking at this graph suggests that another interesting graphic is going to be the timeseries of posts with msn's removed to see how much it accounts for the overall trend.

24 Hours of Blogs

Recently I have been spending some time working on geocoding blogs. In gathering data for this work, I have become interested in the volume and location of Chinese blogs, particularly hosted on Microsoft Spaces. I thought it would be an interesting experiment to take a snapshot of 24 hours in the blogosphere and look at the posting patterns globally. To make things simple, I have just used Weblogs.com as a source of updates and archived update information for the 28th of July 2005. During this time period, there were 816, 704 updates reported by Weblogs.com which have the following distribution:

What is interesting about this is the clear rise in the middle of the graph (which shows counts of updates per minutes throughout the day). This seems to occur around 11am. Of course, we would like to know which timezone this is in. Unfortunately, Weblogs.com doesn't really go into too much detail about this. And for those paying really close attention, I have had to untangle a number of inconsistencies and errors in the Weblogs data. Firstly, I used the html data as the xml data doesn't seem to be archived properly (if in fact it can be called xml - but that is another story). Secondly, the data for any hour includes updates for the last minute or so of the previous hour - something that is important to note when grabbing data for a 24 hour period.

Anyway, back to the graph. My guess is that the rise in the centre indicates US bloggers blogging sometime around lunch time. Once I have geocoded the data, it will be interesting to see if this is infact the case. I will also post on other analyses that come out of this data. I think a data set like this has the potential to be a good reference set for any number of experiments. For example, imagine using the derived set of blogs as a test set for link search evaluation.

TalkDigger

TalkDigger - perhaps the first tool that could make the investigation into blog search engine result counts a little less painful. Fredrick Giasson (profile) has put together an interface which queries Bloglines, Technorati, BlogPulse, PubSub, Feedster, BlogDigger, MSN Search and Google. The results are displayed in a table together with some extra stats recording the trend over the last 7 searches. This makes it easy to compare searches on the same engines, but not searches between engines, though the intention was originally to let you track search result trends on each engine - according to the description:

Talk Digger will ask to major search engines who links to a specific URL. The results will then be processed and displayed on Talk Digger. There are 3 specifics things that will appear when you dig for a link:

1. Result. This is the number of links to that URL. If you click on that blue number, you will be redirected to the result page of the search engine and be able to know who links to you
2. Trend. This is an arrow that will shows you if the number of results for that search is higher, lower or the same as the previous one. This is really effective when you wake up the morning and that you need to instantly see if someone as talked about your blog during the night
3. 7 last digs trend graph. This is a graph that shows you the evolution of the results returned by the search engines in the last 7 search requests

One thing that the tool shows as a side-effect of its live polling of the search engines is the time taken to respond. It is interesting to see how the horses come in (I'm assuming a multi-threaded implementation of course).

Gawker Media: Out with the old, in with the new

Nick Denton's stable of professional blogs offers a great set of sources for studying the whims of the linkfest that is the  blo-go-sphere. While we are all churning up the water with studies about the quality and comprehensiveness of blog search engines, we might at least have some confidence in aggregate trends over time, regardless of our opinions of any isolated results. Blogpulse's rank trends, appearing in the profiles section, shows how the rank of a blog changes over time, where rank is computed from inlink counts.

What is interesting about Gawker media blogs is that they can be used to illustrate some very clear types of trends. Gawker and Wonkette have very similar rank trends which show a clear decline:

Rank trend for Gawker:

Rank trend for Wonkette:

One the other hand, newer additions to the stable such as Kotaku, Screenhead, Gridskipper and Lifehacker show a clear ascending trend. Here is the rank trend for Kotaku (one of my favourites):

Finally, there are the other blogs in the set which show more varied or turn-around rank trends. Here is the rank trend for Defamer:

While one might be cautious to draw too strong a conclusion from these samples, it might be remarked that the older blogs are waning while the newer ones are still enjoying a flush of popular buzz.

Mixing and Filtering RSS Feeds

I came across FeedShake this morning:

Feedshake tool helps you to generate new feeds by merging, sorting and filtering existing online RSS feeds.

It has two modes allowing for various features including sorting, filter by keyword (inclusive and exclusive), re-titling, etc. Matt Siegler pointed me to this Lifehacker (profile) post, and Michiel Dethmers over on the backstage mailing list pointed me to another similar site: sux0r.

UPDATE: Eric Williams at backstage pointed to another mixer: aggrssive.

The Map Room: Comparing Online Mapping Services

Jonathan Crowe at The Map Room (profile) provides a good summary of links to comparisons for online mapping solutions. The post contains a link to Ryan Jonasson great page providing a side by side comparison of Google and Microsoft's web hosted maps (thanks to Matt Siegler who pointed me to this originally).

Reviewing the “big four” in online maps — Google Maps, MapQuest, Yahoo! Maps and MSN MapPoint (yes, in that order) — is problematic: they rely on the same map data and routing engine for directions, according to Webmapper, so it basically comes down to the user interface and extra features layered on top. But that apparently makes a big difference; via Webmapper again, a report concludes that people believe Google Maps is more reliable because of its user interface. Presentation matters, in other words; data along isn’t enough. It may be that Google’s clean pages, lacking the cruft of other services whose ads drown out a comparatively small map window, matters more to the general user than the hackability and other virtues we, as map enthusiasts, focus on.

It is interesting to note that Google and Microsoft share the same map data (something which Ryan's page really highlights. I noted earlier that Google at least uses Landsat imagery (all be it somewhat cleaned up).

Here is the BlogPulse view:

Technorati Feature Roll Out

I just spotted Steve Rubel (profile) testing out Technorati's mobile search on my blog access log (link to search). This is the second new feature that Technorati has rolled out in the last couple of days, together with language filters on search results.

Seconds after the search pops up here, I see Steve's post.

Here's the link to Niall Kennedy's post about m.technorati.com.

User Behaviour

Steve Rubel at Micro Persuasions (profile) says Blog Search Will Soon Be Extinct:

What I want is to be able to sort all my Web results by date as well as relevance and link popularity. Right now I can't. Doing so, as Chris Were describes, is no easy task. But increasingly, it will become what many searchers will crave.

Actually, Microsoft and Yahoo have both played with interface elements that allow for more control over the ranking of results. There are two aspects to dealing with the issue. Firstly, the right data needs to be associated with the documents in the database. For blogs, this is easy as this is pretty much where we started - lots of meta data around very small documents. For the web, it is a bit harder as this requirement was not foreseen (though it may well be that Google's data base does store ingest dates at least). The second, and hardest part of the problem is user behaviour - or user interface design depending on your point of view. Lots of power users want these seemingly obvious features and will have no problem using them. For most of the users of search inputting their on average 2.4 word queries, such multi-dimensional controls will be a barrier.

This explains why Google has stuck fast to their simple interface and why there are still so many AOL users. It is a simple truth of developing interfaces to rich data sets - the interface can only really provide access to 1% of the features available in the data simply due to the incredible difficult user interface problem.

That being said, I think it is the blog engines that are going to lead the changes here as they provide a place for users to come and have their expectations and behaviour changed (in a good way!) over a simpler and intuitive data set. The data is simpler and more intuitive due to its homogeneity. The innovations made in this space will then pass over to the general web space. The notion of search provides the bridge between Web 1.0 and Web 2.0, but it is analytics and user control that are going to reform what we do within these engines.

Forbes.com Best of the Web

Forbes.com has released its Best of the Web for Summer 2005. The Meta Blog category breaks down like this:

• Technorati: first to the finish line followed by
• Blo.gs, BlogPulse, Feedster, PubSub, and completing the list of nominees
• Bloglines, Findory, NewsGator, Syndic8, Weblogs.com

It is great to be listed here, but I am a little confused by the Meta Blog category. Blo.gs (now Yahoo) and Weblogs.com act as lists of blogs which have new content and are only really machine readable; Technorati and BlogPulse are at least search engines but also offer analytics of one sort or another; Bloglines and NewsGator are more in the RSS feed aggregation space...

Perhaps this points to the continuing gray area between RSS and Blogs. An RSS feed is not the same as a blog so applications that are built on them, while capable of reading some blogs, are also capable of reading news, flickr feeds, etc. In fact, this issue is bleeding rather rapidly over into the ping model where there is no mechanism to declare the type of a feed. One question that comes out of this is: is there value to the user in determining automatically the type of an RSS feed? The answer is, IMHO, certainly yes. Spam is probably all the reason you need, but in addition, you may or may not want comments, flickr discussion threads, news, etc. showing up in your search feeds.

Feed Annotation Stream mashups already?

AJH over at Micro-info (profile) has posted some thoughts on Feed Annotation Streams that suggest a new layer for consumer applications (that is to say, consumer of annotation streams);

I wonder if there is any merit in going a step further and having an annotationMix (or annotationPatch?) (that may or may not be an annotationStream?) that:

1) contains one or more <useAnnotationStream annotationURI="URI-for-annotation-service" /> tags, and

2) defines which bits of those various annotations from each feed should be used in the final annotationMix?

This extension needn't go as far as defining an XSLT to remix the final feed, but would be useful as a config file for a mixer of several feeds, perhaps, or setting up an XSLT to generate the final annotated/mixed feed?

It will be interesting to see what other communities might pick up on RSS annotations, because I have a feeling this idea might run and run....

I'm still working on the specification and hope to get it out soon. The spec needs to be presented with some form of ping service which is going to take a bit more time to set up. I am happy for others to develop ideas based on my original proposal - which I want to keep as simple as possible to allow for maximum exploration of possible application layers.

I'm going to use the tag 'fastreams' to label postings on this topic. It is not ideal, but a check on Technorati for other candidates (fas, annotations) showed that they were taken.