BoardTracker Update

I've been monitoring the statistics published on BoardTracker for a few days now. My hope had been to see the number of forums and the number of threads showing a steady, monotonic increase. However, as the graphs below demonstrate, something quite different appears to be happening:

The second graph, showing the number of forums tracked over time, is probably the key here. The fact that it appears to flip between certain values suggests not that there is a gradual purging of, for example, dead boards, but that there is some issue with BoardTracker's infrastructure.

What is interesting also is that during the period at the centre of the second graph during which there appear to be not additions to or removals from the database of boards tracked, the number of forums shows a steady increase: the natural growth of the forums on a closed set of boards.

What's up?

Comments

[Ron Kass]

Just to shed some light on the recent thread/forum count changes.
Those changes are in fact purging related.
If you notice on your graph, on mid 29th and mid 30th there were two purging actions on forums. One was on a removed board, and the other was on a dead board.
BoardTracker does not leave in it's database such removed/dead boards and their data. Also, when robots.txt directives to block a forum/board is detected, the removal is immediate (full thread data removal follows not too long after). Equally when a robot exclusion is removed, a board/forum could be re-added shortly after. Threads might need to be re-spidered if the block was in place for a long period.

Following the forum purges, after not too long a period of time the threads themselves are also purged. Very soon after in fact (compared to traditional search engines at least). Notice on your graph a drop in threads on the middle of the 30th and the middle of the 31st.

Fluctuations on forums may on other times have little effect on thread count when those are part of reorganization of an existing board's forums or a drop/addition of a board with many forums but little threads.

Putting things in perspective though, your recent reports on BoardTracker's sizes were:
October 06, 2005: 11,000 forums (if memory serves me correctly, thread count was about ~6M)
October 24, 2005: 12,563 forums (~7M threads)
November 19, 2005: 16,610 forums (~11M threads)
Numbers for Today: 23,132 forums (~13.2M threads)