Scoble (profile) has blogged about Mary Hodder's article on search evaluation with, can you guess?, another set of anecdotal search results. I will now attempt to use anecdotes to dissuade others from using them. The methodology is as follows: pick a blog, go through the archives and select the first post for each month (Jan-July), count the hits on the search engines.
Battellemedia (profile) gives the following:
| Blogdigger | Bloglines | BlogPulse | IceRocket | Technorati |
| http://battellemedia.com/archives/001224.php | ||||
| 0 | 0 | 6 | 0 | 0 |
| http://battellemedia.com/archives/001291.php | ||||
| 0 | 0 | 11 | 0 | 4 |
| http://battellemedia.com/archives/001383.php | ||||
| 0 | 0 | 3 | 0 | 1 |
| http://battellemedia.com/archives/001480.php | ||||
| 0 | 0 | 0 | 0 | 0 |
| http://battellemedia.com/archives/001586.php | ||||
| 0 | 0 | 0 | 0 | 0 |
| http://battellemedia.com/archives/001672.php | ||||
| 0 | 0 | 0 | 1 | 1 |
| http://battellemedia.com/archives/001726.php | ||||
| 0 | 0 | 0 | 0 | 0 |
Hmmm, all that seems to demonstrate is that Battellemedia is not on fire on the first day of the month. Same goes for Instapundit (profile). Hang on, what about the Scobelizer?
| Blogdigger | Bloglines | BlogPulse | IceRocket | Technorati |
| http://radio.weblogs.com/0001011/2005/01/01.html#a9089 | ||||
| 0 | 0 | 0 | 0 | 0 |
| http://radio.weblogs.com/0001011/2005/02/28.html#a9488 | ||||
| 0 | 0 | 5 | 0 | 0 |
| http://radio.weblogs.com/0001011/2005/03/31.html#a9771 | ||||
| 0 | 0 | 5 | 0 | 0 |
| http://radio.weblogs.com/0001011/2005/04/30.html#a9961 | ||||
| 0 | 0 | 4 | 0 | 0 |
| http://radio.weblogs.com/0001011/2005/05/01.html#a9971 | ||||
| 0 | 0 | 0 | 0 | 0 |
| http://radio.weblogs.com/0001011/2005/06/01.html#a10291 | ||||
| 0 | 0 | 6 | 0 | 3 |
| http://radio.weblogs.com/0001011/2005/07/01.html#a10526 | ||||
| 0 | 0 | 0 | 0 | 1 |
Not much difference there, but we have learned something from this experience:
- If you follow a methodology, the results will be sparse. This is okay because the methodology should be comprehensive.
- Going back and forth between search engine interfaces is painful - evaluation needs to be done programmatically via an API.
- Those with vested interests should not run evaluations. I came up with the the experiment without thinking of the results. The inadvertently showed that BlogPulse is doing pretty well (even with the sparse data). Half way through I contemplated throwing out the post but decided not to as the experimental procedure came before the results (as it ought).
- Don't trust link search - I'm not convinced that all those zeros indicate lack of data, they could point to different styles of input for link search.
- Finally, you might not even trust my results - I haven't double checked them. Another reason for following the programmatic approach.
Comments