Data Mining: Text Mining, Visualization and Social Media

Edgeio Update

In a comment posted on this blog, Keith Teare stated that Edgeio (the listings aggregation site) listings automatically expire in 90 days. The graph below shows the number of listings declared on the home page - curiously it seems to be increasing with no clear dips. Are listings really being expired, or is the number an accumulative numbers of all listings that have appeared in Edgeio even though they may have expired? (To be honest, I do see one dip - can you spot it?)

Edgeio Update

Edgeio appears to have achieved some fundamental change. This graph shows the number of listings appearing on the site:

It appears that around the end of April, some changes have occurred that greatly impact the number of listings. There doesn't seem to be any real indications of what this might be on the Edgeio Blog, though there is a great post there regarding Google's Microsoft-like tactics in owning representational spaces wrt their GData formats.

Interestingly, the number of web sites being tracked (as indicated on the front page) hasn't changed since the end of March, so the sudden increase can only be accounted for by other channels of listings information. In a comment on an earlier post, Keith Teare gave some indication of how this works:

edgeio was architected for any RSS enabled publisher to upload their catalog automatically. And for updates to be uploaded in real time. We expect that alongside bloggers, ther will be many larger publishers who choose to do that.

Edgeio Update

A quick update of Edgeio data.

Listings over time:

Cities over time:

Notes:

• The listings appear to be accumulative - how are listings removed once a sale has been made?
• Listings are growing linearly - no big jump yet.
• There is a clear isse of some sort with the location information causing a big drop in the number of cities. As this doesn't correlate with a drop in listings, I'm guesing it is due to a fix in the location parsing system.

Edgeio Update

Keith Teare, founder and CEO of Edgeio responded to my initial analysis on the first few days of the company's life with a detailed comment on some of the issues I brought up. Here is the comment (thanks, Keith, for the insights):

Hi there

Great that you have pulled this together. I can shine some light on the jumps for you.

It isn't caused by bug fixes but by single large listings owners deciding to support edgeio. This is particularly noticeable in the Jobs and Real Estate tags.

edgeio was architected for any RSS enabled publisher to upload their catalog automatically. And for updates to be uploaded in real time. We expect that alongside bloggers, ther will be many larger publishers who choose to do that. The spikes in cities and listings are when they do so.

This is a kind of non linear growth, but because it is very early it looks spikey. If it continues, then within a few months edgeio will grow to several million listings (it will continue :-) ), and be in tens of thousands of cities.

This is the power of a bottoms up publisher/submission approach. It is really hard for centralized submission based services, or scrapers, to get to this kind of scale (Craigslist is only in 100 or so cities due to its top down approach to geography).

After a week we are thrilled that we have listings from almost 1500 cities, even though each city (with some exceptions) has only a small number of listings. The approach will scale very well and quickly. Imagine the power of a system where a small town in China can have its own classified marketplace simply because people start to use the local cybercafe to come and post items for sale or wanted. and it will all be in Chinese because edgeio is publishing what the users write (both the text and the tags). It's hard for a top down system to have this power.

On your point about the need for feedback. We do confirm receipt of a listing - usually via trackback. See here - http://listings.teare.com/?p=134#comments - for an example. If trackbacks are switched off then only the publisher sees the trackback, so it is a private communication.

We will introduce some new features for less technical bloggers over the next 2 weeks, and also some tools for those without a blog, or who want a special blog for just their listings. Mike Arrington described some of that in a podcast - http://blog.edgeio.com/?p=19 - today.

Best regards
Keith Teare
founder/ceo/edgeio

Edgeio Update

Edgeio provides some stats on its home page which indicate the number of listings it has gathered and the number of cities associated with those listings. These have both been increasing, and it is probably too early to use this data to judge the success of the system, but there is not yet any clear rapid growth. Here is the graph for listings.

And here the one for cities.

It is interesting to note firstly the large jump in both listings and cities - a sudden increase of data in the database, perhaps associated with a bug fix and recrawling or reparsing cached data. Secondly, there is a smaller increase in cities which has no associated jump in listings. Perhaps a problem with the location identification code which had no impact on the number of listings discovered.

The Alexa reach statistics show some growth, though very spikey:

The shape of this graph seems to be mirrored in the mentions of the URL as found in BlogPulse:

What we are looking for as a measure of success is non-linear growth in the number of listings (indicating that sellers are using the system and that the system is doing a good job of gathering the data) as well as similar growth on Alexa. One of the problems with the basic idea of Edgeio is that the passive manner in which you add listings provides no confirmation - sellers would have to go to Edgeio to discover if there stuff was picked up or not. Anywhere there is such a barrier, users will be dropped.

Listings from the Edge

There will be plenty of posting today about Edgeio, which crawls social media sites for information it then renders as classified ads. One thing to point out - the listings are not just from blogs. My first experience, browsing the for sale|auto category, was to arrive at listings which originated from message boards.

Edgeio

There is a buzz abuzzing around Edgeio, which, according to this entry [Via /Message], will use tags and blog crawling to gather classified ad information right out of your blog and manage it online to help connect you with buyers.

Ad listing in blogs? Who does that? Today, almost nobody. And that's why this idea could work at all: Teare said the tag "listing" is found only about 10 times a day on millions of blogs, so it's an ideal, clean tag with which to create a unique index of "listings from the edge."

There appear to be two risks in this model (I haven't yet ready any critical descriptions of the system - everyone blogging about seems to have been positive and/or had some relationship with the company). Firstly, no-one owns tag space. This is one of the biggest failings of tags (and one of the most predictable). By not having a formal system for meta-data, there is no control over meaning. So now the tag 'listing' means 'I want Edgeio to crawl me, interpret somehow what I have on my blog and then offer that to buyers'. The argument against formal spaces in tags (a la XML name spaces for example) is that they require either more work on the part of the user or integration into blogging platform as a standard (oops - didn't Technorati achieve that with their ownership of the tagging mechanism - perhaps that would be too meta).

Secondly, the value of such a system to sellers requires pretty good coverage. I'm guessing that Edgeio only crawls RSS enabled blogs. This means that there will be some percentage of bloggers out there who cannot list on Edgeio. In addition, given the unstable state of the ping/RSS duopoly, Edgeio will have all the challenges of any other blog search engine in keeping things up and running. This risk may result in value - more reason to improve the ping/RSS stability and coverage, but it may not.

It's going to be interesting to see how this stuff pans out. Good luck to them! Somewhat Frank has a more detail explanation of the service.

BTW - imagine what would happen if someone wanted to put out a DOS attack on Edgeio by flooding the blogosphere with the listing tag. Will we enter a new era of spam functionality?