Data Mining: Text Mining, Visualization and Social Media

Historical Blog Data

TechCrunch describes an interesting interaction between a Technorati user and Technorati in which it appears that the blog search company has elected to archive data that is more than 6 months old. Most of the discussion around this appears to centre on the impact that might have on search, however there are a couple of more significant aspects.

Firstly, historical data is important for computing statistics and features of authors. Of course, one can create a theory of influence which only requires 6 months of data though this is not going to help you do longitudinal research.

Secondly, if Technorati really is in the business of deriving business and marketing intelligence from their huge weblog archive, they will certainly need access to this historical data. A big part of this analytical space is the ability to compare some current phenomenon (e.g. a new product launch) with others in the category - several years of data are required for this.

Technorati Brings Charts Back

I'm happy to see that Technorati has brought back the display of charts showing the number of blog posts matching a search term over time. Below is the chart for "south africa".

Redefining Conversation

While everyone is enjoying the hoopla around a new web based implementation of a very old approach to presentation, Peter Hirshberg over at Technorati has written a long post on the topic of 'conversational marketing.' This is going to take a while to get into. However, one initial perspective - in fact, something that has been bugging me for a while - is this notion that weblogs are conversations. Let's review the facts: weblogs are not conversations.

A weblog is a means to broadcast (distribute identical content to many consumers). Conversations are symmetric interactions (first you, then me, then you, ...). Weblogs are a new form of interaction - a hybrid of dialogue and monologue presented on a platform which by intention and accident has evolved into a hybrid of channels (do you reply on a comment, in an email, on your blog, ...?) Unfortunately, this new form of interaction has not been adequately modeled.

Bloggers must be very sensitive to the effects of lexical imperialism. Terms like 'consumer generated media' establish an us and them perspective of the intentions behind the content that we generate. Similarly, Doc Searls' statements are worrying:

• The purpose of conversation is to create and improve understanding, not for one party to "deliver
  messages" to the other. That would be rude.
• There is no "audience" in a conversation. If we must label others in conversation, let's call them
  partners.
• People in productive conversation don't repeat what they're saying over and over. They learn from each other and move topics forward.
• Conversations are about talking, not announcing. They're about listening, not surveying. They're about paying attention, not getting attention. They're about talking, not announcing [sic]. "Driving" is for cars and cattle, not conversation.

'There is no audience in a conversation.' I agree with this, however there is an audience for a blog. Labeling people in a conversation a 'partner' suggests equality. But as this applies to marketing it is the wrong suggestion. A partner doesn't try to get you to buy stuff you don't need/want. The implication that the blogosphere is a conversation; that we are all partners; therefore people marketing to us in this 'conversation' are our partners is creepy. Another point to note is that there is a backchannel in the blogosphere. Many of us get emails requesting this that or the other get some exposure. Conversations are transparent to all participants.

Those interested in style may be entertained by the 'conversations don't repeat...' followed by the repeated structure of 'Conversations are about A not B', not to mention the actual repeating of 'they are about talking, not announcing.'

Technorati has tried on and off for a while to figure out what their real business is. This post hints at a model based on 'conversational marketing.'

Technorati Time Travel

Ah - remember back in 1970 when blogging was just beginning? No? Well, Technorati can take you there. I just did a search for http://www.resourceshelf.com and found that TechCrunch had posted about the blog 13, 770 days ago (in Technorati time). That is over 37 years ago. 1970 is a special date for computers, so it looks like the Technorati is failing to get dates computed properly and defaulting to zero (which translates to the moment 1970 began).

Technorati's Real Time Blogscroll

I'm a little late getting to this, but I just noticed Technorati's new home page: a live scroll of blog posts bucketed by broad topic (e.g. politics). I like it. I've not yet dug down to see how well they are doing this (e.g. how real time is it? how accurate is the topic classification?), but certainly an interesting step.

What Happend To Technorati Charts?

One thing I'd like to know? Why did Technorati pull their charts in their latest release? Well, they didn't actually remove the service, but there aren't any links to it - at least, that I've seen. One possible reason for this is that their index appears to be slightly temporally challenged. Here is the chart for the term 'today' which you would imagine would be a pretty steady type of trend.

It looks like Technorati's recent memory is ok, but the long term memory isn't what it used to be...

So my guess is that either they removed the charts as they would have exposed this type of problem, or that they were planning on reducing the temporal width of their index for scalability reasons and the charts had to go as a consequence.

While the recent memory is more important for certain aspects of blog search - the long term index is something that can be used more aggresively for things like 'influence' or 'authority' (whatever they are).

Technorati Update

Technorati launched a significant update yesterday. While I'll let others discuss the merits (or lack thereof) of the new design, I'd like to lament the new scoring that Technorati offers bloggers. Previously, there was a ranking score which produced a number from 1 (the highest) to <unbounded> the lowest (or, effectively, unranked). With that form of metric, you could look at the number and get an idea as to where you were and track any changes you were making. With the new 'authority' score, we have a range from <unbounded> (the highest) to 0 (the lowest). Consequently, you can no longer look at your score and figure out where you are in the grand scheme of things. You would have to at least know the highest score.

This, to me, is a poor design of metric.

Interestingly, as the comments on the TechCrunch post indicate, reaction to the ticker at the top of the page has been somewhat mixed or negative. If you look at them closely, you will see that it isn't a real time stream of terms, but a cycle of terms repeated over and over (refreshed at some point I assume). In other words, the ticker nature of it is a presentation decision, not necessitated by the fact that it is (could have been) a real time stream of data.

In the big picture, I think the new design is an overall improvement, and the search speeds to appear to have improved.

Technorati Graph Hack

When you do a search in Technorati for a term, it provides a neat little graph of the volume of posts per day containing the search term. However, if the search is for a URL - no graph is produced. It turns out that you can create a graph of link volume by using the following URL:

http://technorati.com/chartimg/<URL>?days=<DAYS>&size=<SIZE>&type=url

The key thing is the last argument above. <SIZE> can be one of (at least) t, s, l and URL is host.domain.tld (I've not yet managed to get it to work with paths).

Here's the graph of links to the Discover Magazine (guess when they switched to the new URL): http://technorati.com/chartimg/discovermagazine.com?days=365&type=url&size=m

Here's Powerset's attention:

BlogPulse never made the distinction between searching for URLs and searching for regular terms - I suspect the Technorati feature just hasn't been publically described for priority reasons as it seems to work fine. I'd love to see them develop their graphing feature further - including options at least for multiple lines and for normalization. I don't really care that that would be a complete copy of ours as they bring a longer window of data in to play which is of interest in itself.

[Via: I stumbled across this somewhere and don't recall where I found it!]

The State of the Tea Leaves

Technorati's David Sifry has published a new state of the blogosphere - which he is now calling the Live Web, though I don't believe he is including the larger space of social/UG media. On January the 12th, I posted a couple of items in a wish list for his next account:

• An account of the US or English language blogosphere and
• An account of trends in terms of active blogs (not all blogs ever encountered).

David commented on this post, saying:

Great points. I'll work with the team developing SoTB (yes, there's now a team!) to incorporate your suggestions. Keep up the great blogging!

While the new report does have some language breakdowns, it doesn't attempt to address either of the issues that David said he would try to incorporate. In addition, while the graph of the posts per day appears to me to be showing a clear downward trend, David says that this shows a slowing in the growth in the rate of posts. Assuming that he intended only to include a single derivative here and meant to say that growth was slowing, I think he is missing the point. It looks to me as if the blogosphere that Technorati indexes is actually shrinking. David does talk about some numbers that do indicate continued growth, but the chart he uses seems to contradict these claims.

We also see a slowing in growth in the rate of posts created per day; while there are spikes in blog posts during times of significant world crisis -- for instance, last summer’s conflict between Israel and Hezbollah -- the overall trend is that posting volume is growing more slowly, at about 1.5 million postings per day. That's about 17 posts per second. In October 2006, Technorati was tracking about 1.3 million postings per day, about 15 posts per second.

Compare this with the graph:

I wonder why the data is being spun this way. Could it have to do with TechCrunch's rumour about acquisition?

David has done a great job historically of creating thought leadership via this quarerly documentation. However, I wonder if this may become a liability in the future as they look for exit strategies that require evidence of continued growth. Note again that these numbers are the blogosphere that Technorati crawls. Crawling the blogosphere is a hard problem and so estimating its size and growth is tricky.

Technorati WTF : Update

My original post about Technorati's WTF application was shallow. Clearly I didn't really get a lot of what was going on. With Natalie's help, I've now explored a bit deeper and it does look like this could actually be quite a significant feature. The important thing to note about the system is that WTF blurbs are served in search results.

For example, if you search for ICWSM on Technorati, you will get Natalie's blurb at the head of the results list. In the screen shot below, you can see the results above Danah Boyd's post about her invited talk at the conference.

Now it is time to don the grey hat! Why don't you help ICWSM get some visibility by contributing some WTF blurbs to the ICWSM thread.