I posted briefly about the Technorati spider behaviour suggesting that it didn't just crawl in reaction to pings and secondly that it was crawling from different international locations. David Sifry was kind enough to leave a comment denying both of these. So please consider the following, which I just captured from my FeedBurner account:
Exhibit A - a record of the number of times a robot identifying itself as Technorati Search Bot hits my feed per day.
Exhibit B - an indication of where this bot is crawling from.
Now, we can't all be right here, so either
- This isn't the Technorati bot - in which case, David might want to get in to some legal action for being misrepresented.
- FeedBurner is making an error and putting the Technorati label on a different bot.
- FeedBurner is right about the location but wrong about the number of pings (in which case they need to fix a bug).
- FeedBurner is right about the number of pings but wrong about the location (in which case they need to fix a bug).
- FeedBurner is wrong on both counts (in which case they need to fix a bug).
- I'm not understanding what I am reading in which case I need to RTFM.
- I'm photoshopping these images (which I'm not, BTW).
My guess is that the location may be off, but the crawling behaviour is correct and that it is the Technorati bot, but I'd love to hear from both David and FeedBurner on this one.
Update: I just spotted the Technorati bot crawling this feed 4 times in a row - possibly as a response to publishing the post. This time it was identified as being from the Russian Federation.
Matthew, I just sent you an email. I'd like to work with you to make sure FeedBurner's not misidentifying user-agents or messing up on the geolocation stuff.
Eric Lunt
CTO, FeedBurner
Posted by: Eric Lunt | August 18, 2006 at 10:54 AM
Does feedburner supply the IP addresses? If so, would you publish them or send them to us?
thanks,
ryan
Posted by: ryan king | August 18, 2006 at 12:32 PM
Could the reason for multiple bot visits be because some of the places you ping also ping technorati? Do you see 4 vists from every time you post? Would you still see 4 if you posted and only pinged technorati?
Posted by: quadszilla | August 18, 2006 at 02:51 PM