Data Mining: Text Mining, Visualization and Social Media

Sphere Version Next

Briefly, Sphere has updated its home page and, it appears, its primary focus. The success of the SphereIt embedded widget has resulted in a new data source for the site which allows it to essentially operate as a memetracker.

You've Been Sphered

Looking at my referrals, I see a visitor from TechCrunch. Looking on the TechCrunch site, I don't see any link to my blog. However, looking at the results delivered by the Sphere It widget (shown below), I can see how this happened.

So, Sphere is driving blog to blog traffic and monetizing it via placing adverts in the pop up. This is different to the Sphere It approach used on media sites which drove traffic directly to the Sphere site itself. The real challenge here - and the real need for a more in-depth review - lies in the quality of matching algorithm. These are not link based matches (though there will be some amount of that of course), but (intended to be) meaning based. The post that is mentioned above is not about the TechCrunch article, nor is it about the company being discussed in the article. The TechCrunch post talks about a new image search application, and my post is about a search for image search systems - pretty removed, though there is some sort of connection. My first guess, therefore, is that Sphere's technology is looking for significant phrases in the posts (words and phrases that differentiate this post from others) and then cross indexing them. It's probably doing this as well as a number of other things.

Sphere is Growing

Sphere, the blog search engine, is looking for new people. Given the number of people in the original team (a small number) and the number of positions advertised, this suggests a pretty significant increase in personal which is a good sign for the company.

How to Build Traffic for Blog Search

How do you build an audience for a blog search engine? You can be good, you could even be better than the most popular engine. But what it really takes is smart business relationships. Reading Steve Rubels blog via Bloglines, means that you are reading his feed served by FeedBurner, which means that Steve can add FeedFlare elements to the feed, which means that you could hit the handy 'Related Posts from Sphere' button to see, well, related posts served from the Sphere site.

You can't just be good, you have to spend effort (which means time and money) on relationships that don't have a clear input and output (where the output is of type MONEY). Technorati was always very good at that, and stealing Technorati's position means building these relationships to ultimately create a portal that is synonymous with blog search.

Sphere Launch

Sphere launches this morning. Tony Conrad, the CEO/Founder, highlights the following features:

• custom range histogram;
• sphere it bookmarklet that makes it easy to find blog posts that relate to what you're reading on the Web;
• top queries this hour/ this week;
• RSS for search results;
• community feedback buttons throughout the site;
• expanded related media verticals including podcasts (Yahoo! API), books (Amazon API), and photos (CNET Webshots API);
• expanded related news articles coverage (over 50 mainstream news sites such as CNN, NYT, ESPN, Fox, USA Today to name a few);
• expanded profiles;
• featured blogs for 12,000+ keyword search terms covering over 500 broad topics including baseball, anime, food, web 2.0, tech news, knitting, culture, etc.; and
• a brand new back end that scales.

Custom Range Histogram

This feature, similar in appearance and interaction to MeasureMap's date range selection element, has two roles. Firstly, it shows the volume of posts for a search. Secondly, it allows for the customization of date ranges for a search. Like Technorati, the charts show message volume, not normalized message volume (which is the BlogPulse model). The chart below shows the volume per day for the term 'bush'.

This could be improved by adding more date information on the x-axis, giving it a y-axis and providing some information in the graphic indicating that it comes from Sphere (useful when it is grabbed like this to illustrate a post). For comparison, below is the same graph from Technorati:

It is interesting to see the differences. It is possible that the Sphere graph indicates the ramping up of coverage prior to launch. Problems with dates associated with Technorati data are well known - the timestamp reported for search results is often inaccurate. A minimal inspection of results on Sphere have not yet uncovered a problem in this area, though it should be noted that dealing with time in blog search is a very difficult problem: the use of standard time formats is shaky at best.

For comparison, BlogPulse and IceRocket take a different approach to trending, reporting volume over time as a normalized value. This indicates the percentage of the blogosphere that mentions a term. Here is the BlogPulse chart for 'bush' for the same period:

And IceRocket (note it is for a shorter period):

Sphere It Bookmarklet

The bookmarklet:

is a free bookmarklet that makes it easy to find blog posts that relate to what you're reading on the Web.  It may change your life.

This is a neat feature. It certainly changed the life of Technorati which, as of a few hours ago, put up it's Technorati This! bookmarklet to:

see who's blogging about any web page!

I've put up similar bookmarklets for BlogPulse in the past, and here is another for BlogPulse Search.

Top Queries

I suspect that this is the feature I spotted a little while ago. Similar to Technorati's 'What's happening right now?', but with the extra per week feature. Currently there is no overlap in the results, but it will be interesting to see if there are similarities in the future - if not it would indicate differences in the audience. Note that a feature like this has two roles. Firstly it helps us see what others are looking for - it is a window into the stream of consciousness of the searchers. Secondly, it provides another route for search queries - this is important for monetization. Sphere doesn't currently run ads (Technorati does).

Community Feedback Buttons

These are great. Basically, it allows for feedback (e.g. suggesting a blog to be a featured blog) that will help to improve the service.

Related Media/Related News

By using a number of web service APIs (Yahoo, CNET, and Amazon), a search can be expanded outside the blogosphere to include photos, news, books and podcasts. The photos appear to be supplied by Webshots. Using APIs like this provide, with little extra cost, some great potential for the user. However, by using APIs, the service loses control of quality issues.  For example, a search for the phrase "text mining', provides me with good blog results, but images of Greek kittens (with no mention of text mining on the native Webshots page). I like kittens - but they aren't relevant.

Related news is nice (an interesting inversion of Yahoo!'s blog search which hangs off its news search). It could be slightly enhanced by providing links back into blogsearch so that a search for 'hybrid' provides links to news articles and then I can essentially 'Sphere It'.

Profiles

Profiles were in very early on the Sphere platform and similar to BlogPulse's profiles feature. Currently, there appear to be some glitches in the extended version of Sphere's profiles - I see a largely blank page with a few stats in the right margin and along the bottom. It looks like something (some charts?) are intended to appear in the centre of the page. This may be an issue with using the Firefox/Linux combination.

Featured Blogs

I like this. For a search, hitting the featured blog button brings up a list of blogs on this topic. This has been a big hole in the blog search world for a long time. Google and Technorati have both attempted to fill this void but the results have never been too good. It looks like Sphere is going for an editorial approach (compared with Google's automated ranking approach and Technorati's tag based approach). It is, at least for me, more often the case that looking for featured blogs on Sphere produces no results, so there isn't really an opportunity just yet to evaluate the results. But hopefully once this is built out, we will see some quality here. I said I like this feature - the thing I like is, actually, the interface. It is not on the search results page (like Google) and is very simple and clean.

Interface

Right from the get go, Sphere has never had any real problems with the look of their interface. It has always been clean, simple and intuitive.

Summary

The existence of Sphere indicates, of course, that the race is still very much on to find the winner in the blog search space. Sphere itself has clearly gone down the path of a feature rich, highly engaging blogosphere platform (like Technorati and BlogPulse). Interestingly, this version of Sphere indicates a more pragmatic attitude to the space than the original declared intentions, which appeared to be more about the problem of finding blogs, not finding blog posts. As noted above, this feature is now somewhat sidelined as a partially complete, beta function that shows promise but is not yet ready for evaluation.

Coverage wise, the histogram of posts over time quoted at the top of this post suggest that Sphere users may have to wait a while to get the same sort of coverage as other players in this space provide. Note that Sphere's default search mode is by relevance over a one week period, not a time ordered list of result. This is one way in which they will become known for 'better' search results - by slightly changing the goal posts (not that there is anything wrong with that).

The big question is, of course, how are they going to monetize? Are they going to chase ads (in which case they probably aim to be a Technorati killer)? or are they going to offer data mining services (as Technorati is rumoured to be doing and as BlogPulse's mother ship does for a living)?

TypePad Widgets: OpinMind, Sphere

TechCrunch writes about TypePad's soon-to-be-released widgets. There are a couple that really interest me. Firstly, OpinMind - you will be able to include a widget that mines your blog for snippets that represent the things that you have expressed opinions about. Secondly, there is blog search provided by Sphere.

Sphere Update

Not much has been heard or said about Sphere of late. However, I did notice something today on their main page. It looks like (a messed up version of) some sort of 'popular searches' or something of that ilk. I say messed up as it doesn't appear to have been rendered in their stylish design.

Note the French terms. Are the playing with non-english data, or considering launching a French version? Of course, it could all just be some sort of bug that is exposing search terms that land on the sphere home page. Other weirdness is that if you reload the page, you get a different set of terms, but there only appears to be two distinct sets of terms. Who knows?