Understanding Alexa Reach Stats

I recently posted an observation about TailRank's reach statistics. Kevin corrected me on the interpretation of the Alexa data confirming that TailRank had been experiencing significant month over month growth.

Greg Lindon recently posted visitor stats for Findory. We can compare this graph with the Alexa reach graph.

Alexa's reach stats are per million - delivering a normalized account of the number of visitors, essentially reporting the proportion of visitors for a site from the total online population as predicted by their sample base.

However, the graphs still don't seem to add up. The last quarter for Findory has seen less growth than previously, however, I doubt it would result in the clear downward trend in reach as displayed by Alexa. The Findory stats are hits, not visitors, and include robots. If we assume robots are a constant and that the number of pages viewed by each visitor is constant, then we can assume that the graph does track the growth of users. This does not fit the Alexa data, which can only be explained if, in Q1 2006, there was sudden growth in the online population with a bias to those not inclined to visit Findory.

So what's going on? Is Alexa reliable?

Comments

[Greg Linden]

Hi, Matthew. The data used for that Findory graph does exclude robots. It is hits, but the page view numbers show similar growth Q1 2006.

Yes, I think the problem is that Alexa is generally unreliable. Alexa stats only come from people who have installed the Alexa toolbar, a tiny sample that is heavily biased.

As you can see from the chart you cite, Alexa data is extremely noisy. The Findory website does not see wild daily fluctuations that appear in that Alexa data. The most likely explanation for those fluctuations is a small and biased sample.

Alexa is also trivially easy to manipulate. There are many techniques and tools available for manipulating Alexa ranking. Simply asking a couple people working at your startup to install the Alexa toolbar seems to be sufficient to yield a large jump in your website's Alexa rank.

Alexa probably is accurate comparing sites with an order of magnitude difference in traffic, but I would be careful about using it to make conclusions about small differences.

[Max Kalehof]

Matt -
Alexa is a great free tool -- it's better than nothing, but it is the furthest thing from real measurement projections. It's accurate of one thing: people who install the Alexa toolbar. Who are those people anyway?

[Ron Kass]

Alexa is not an accurate tool, but it does reflect real life changes to some degree.
It is also important to realize that robots traffic is not a constant but a changing factor. Usually a rapidly growing one in the early days a site, and a growing one as number of pages on a site grows.
It is also important to remember that, unless using image verification for logging (which is many times not the case with internal stats) self log-analyzing trends are not accurate and are not able to keep out all robots activity since some of it is unreported (through user-agent).
Also, it is true that alexa is heavily fluctuating, however that is true mostly for smaller sites, since as sample usually go, accuracy is in high frequency numbers.
Manipulation of alexa is possible, but again, to a degree..

That being said, it is important to realize that Alexa is a statistical projection rather than a true measurement and comparison tool. One should not trust alexa when estinating traffic pattern changes.
But one can certainly assume longer term trends are corrent.

Things that might be considered when judging alexa's graph compared with internal stats:

1. Robots traffic IS growing.
2. Many robots are stealth (including google verification for checking for cloacking), this might be nistakenly accounted in intenal stats as surfer traffic. A suggestion would be, trying to measure statistics with image verification (which most robots don't follow)
3. Views per user may grow as a site mature. Average per user may go from 2.5 pages per session to 3.5 for example. this is a very big change that might change reach greatly in alexa.. a thing that doesn't show in internal stats.
4. Your visitors' demographic matters. Ans so are other characteristics of your visitors. Some populations differ in the aspect of alexa toolbar installation.

[John Cass]

I would not trust the alexa results for numbers, but I think the trend might be used as a basis of information.

[Account Deleted]

You can also try http://www.estimix.com – a free tool that provides a nice summary of the website performance. The estimation provided by estimix is the result of a complex analysis based on factors like: the age of the website, the demographic structure of the traffic, the countries where the website is popular and sources of the traffic.