Data Mining: Text Mining, Visualization and Social Media

Industry Standard's Top B-Z Blogs

The Industry Standard has produced a list of B-Z blogs of note.

These are the blogs you won't see on the Techmeme Leaderboard, Technorati's Top 100 blogs, or the CruchBase BloggerBoard ... at least not yet. They include VCs, entrepreneurs, coders, experts, and observers, and they bring a delicious mix of insight, experience, and passion to their blogs. While they may not have the right amount of link love, they need to be on your radar screens.

I'm very happy that this blog has been included on the list!

The TechMeme Bikini

Some stats regarding the distribution of headlines on TechMeme between A-listers and others seems to be getting some attention. Attention, but little real thought. The basic observation is that while 70% of the headlines on TechMeme are accounted for by the top 100 ranked sources (according to the leaderboard), 30% is from the long tail.

For the sake of argument, let's assume that the data is static - that is to say, that the leader board 100 is always the same (it isn't). Let's also assume that TechMeme crawls 10k weblogs (I don't know that it crawls that many). Let's make some more assumptions: that every weblog posts 1 post a day and that there are 10 headlines per day on TechMeme. Thus, there are 100 posts per day from the top 100 sources and 7 of them will appear on TechMeme. Thus, each of the top 100 sources has a 7% chance of producing a headline on any given day.

The other 3 headlines come from the remaining 9, 900 sources. Thus, if they are also producing 1 post a day, each source has a 3/9, 900 =  0.03 % chance of getting noticed. So while the 2:1 ratio of A-listers to others sounds good, for any individual, it actually translates to 233:1 odds (7/0.03).

Of course, the assumptions above are a little rough and there is absolutely no accounting for how network effects really get things done in the blogosphere. The point is, there is a 2 orders of magnitude difference in these numbers between what an individual can expect and what the groups (A-listers/others) can expect.

Sphere acquired by AOL

TechCrunch reports that Sphere will announce acquisition by AOL tomorrow. There is a great story here: Tony Conrad entered the weblog search space pretty late, altered course to provide a related content matching algorithm that he sold to main stream media and then exited. The only other acquisition in this space was IceRocket by Think Partnership - but didn't that fall through?

Congratulations to Tony.

Map of Iranian Blogosphere

Nathan points to this interesting paper on the structure and topics of the Iranian blogosphere. I've not yet had time to read it, but wanted to pull out this visualization for starters.

Wired - The Secret Life of a Blog Post

The current edition of Wired has an article which Frank Rose and I wrote describing what happens to a blog post once you hit the 'publish' button. The article has now made it to the web in the form of a snappy flash graphic which allows you to move around the diagram to focus on the details.

I've been chatting off and on with Frank for a while about the social media space and it's great to see something make it to the magazine.

IceRocket Relaunch?

I'm sure I'm very late in noticing this, but just after I saw Technorati's relaunch as a memetracker, I noticed that IceRocket is starting to look different, with a new (to me) looking front page, ranked video, movies and news. According to Compete, Technorati's shift looks like it may have resulted in an upswing of visitors, while IceRocket is still languishing.

Of course, searching in blogs for mentions of an IceRocket redesign is hopeless. A search on IceRocket just brings up spam, a search on Technorati brings up anything with an IceRocket tag (even a search for "icerocket new design -"icerocket tags"" doesn't seem to help). Perhaps no-one noticed.

Interestingly, a search on Google unearthed this:

None of the Twitter search engines appear to have any posts indexed mentioning IceRocket (Twitter is hard to crawl comprehensively). 

A New Life for LiveJournal

My colleague Alexey Maykov has just alerted me to the news that LiveJournal is to be acquired by SUP:

Six Apart, the world's leading independent blogging software and services company, today announced that SUP, an international media company, has acquired LiveJournal (LJ), the pioneer of social networking communities online used by millions of people around the world to connect through personal journals and topic-based communities. SUP has launched an American company, LiveJournal, Inc., to manage and operate LiveJournal globally.

To a certain group of individuals, the first question to be asked is: what will happen to LiveJournal updates, currently a part of the SixApart atom stream?

The big picture here is that firstly LiveJournal is a huge part of the blogging scene in Russia (Moscow, essentially) and secondly, it is not clear quite how to keep growing the thing in a way that brings in the big bucks. Brad Fitzpatrick, the founder of LiveJournal, will be speaking at ICWSM next year in Seattle.

Trendpedia

Trendpedia is new to me, but not new. A beta product from Attentio - who is in the social media analysis space - it has a few similarities with BlogPulse (that is to say, it has a trend search component). It has a number of other features, such as a pie chart of the percentage of data from each trended query.

The Pocket Cruiser Blogosphere - Lost at Sea?

An experiment in discovering niche blogospheres: what are the best blogs that focus on pocket cruisers and pocket cruising? Wikipedia says:

A pocket cruiser, microcruiser or pocket yacht is a small, often lightweight sailboat with a cabin, which is intended for recreational cruising (either overnight or for extended periods) of the owner's chosen waterways. Pocket cruisers often can readily be loaded on a trailer and towed by most passenger automobiles. Both commercially made and designs for home built pocket cruisers are readily available.

Firstly, how do the major search engines help us? A search for "pocket cruiser blog" on Google produces, as most relevant, a blog post on German pocket battleships. The blog itself is entitle 'World Association of International Studies' so not really on topic.The rest of the results include

• A link to a blogspot blog that doesn't actually exist
• A page reviewing a dvd on pocket cruiser trailers
• A link to a blog that mentions a pocket cruiser, but doesn't seem to be about pocket cruisers.

Searching for "pocket cruising blog" provides no results, but "pocket cruising" brings up a link to the Wooden Boat Forum as well as BoatDesign.net.

Searching on Google's blog search isn't really much help. "pocket cruising" brings back a mix of stuff about cruising and pockets (pocket computers). Google's blog search defaults to sort by relevance, but I don't really think this feature works in this case.

Searching on Technorati for "pocket cruising" produces no results. So, now, assuming the error is with the user, perhaps the problem is the search term. Perhaps pocket cruising isn't a term used by people who actually go pocket cruising, or who own pocket cruisers.

Searching simply for 'sailing' on Technorati, and displaying the 'blogs about sailing' (i.e. blogs that use the tag 'sailing') provides me with 1, 877 blogs. But here we run in to the problem either with tags, or with Technorati's indexing of them. The first blog 'about sailing' is actually nothing to do with sailing at all and while The Thinking Christian employs some nautical metaphors, I don't see any topgallants there. The third on the list, however, looks more promising: SailBlogs appears to be a blog hosting site for blogs about sailing (Technorati seems to confuse this with an actual blog...). SailBlogs does seem to include a section on cruising, so perhaps this is going to be it. Further down Technorati's results are real sailing blogs - though a mixture of types of sailing.

Going back to Google's blogsearch and searching with 'sailing' does produce better results (including some cross over with Technorati's results). Interestingly, a search for 'sailing blog' on the main Google site also delivers some good results.

Any conclusions? The simple task of finding blogs on a topic seems to be under supported. I was trying to find blogs on a specific sub area of sailing and essentially failed. This may be due to my ignorance on the topic, but I'm surprised - given how many people do this type of sailing and how much money is in it - not to have found more weblogs on the topic.

Historical Blog Data

TechCrunch describes an interesting interaction between a Technorati user and Technorati in which it appears that the blog search company has elected to archive data that is more than 6 months old. Most of the discussion around this appears to centre on the impact that might have on search, however there are a couple of more significant aspects.

Firstly, historical data is important for computing statistics and features of authors. Of course, one can create a theory of influence which only requires 6 months of data though this is not going to help you do longitudinal research.

Secondly, if Technorati really is in the business of deriving business and marketing intelligence from their huge weblog archive, they will certainly need access to this historical data. A big part of this analytical space is the ability to compare some current phenomenon (e.g. a new product launch) with others in the category - several years of data are required for this.