Data Mining: Text Mining, Visualization and Social Media

Bloglines Updated UI (one more time)

Now that Bloglines has some serious muscle behind them, I hope they can get some better UI design. Steve Rubel recently blogged about the keyboard shortcuts. A the same time (approximately), Bloglines added an improved account of how many posts were in a folder, separating the number of unread from the number of read thus: [u:r]. They have now made another change, to (u)(r). What they need to do is to put these numbers on the left hand side. For most of my folders, the numbers are not visible with the default settings. To make it worse, a bug in Firefox on linux makes it hard/impossible to resize frames.

The Relevance of Relevance

I've been reading some reviews of Google Reader as well as playing with it myself. therssweblog provides some interesting comments - but what sticks out the most for me is the suggestion that Google seems to have missed the fact that relevance for the web is different from relevance for blogs and RSS feeds:

It seems clear that PageRank is a hindrance in Google Reader, forcing the delivery of high-ranking results with low usefulness as feed-discovery destinations.

This is something that I feel is apparant in Google's blog search offering, where relevant blogs are sometimes served at the top of the results page - the results aren't really that impressive. Blog search engines - that is to say, real blog search engines - have been struggling with the fact that adding a time dimension to the relevance problem changes everything.

With 8 billion web pages out there, the idea that there is a single relevance algorithm is starting to look a little silly. The huge constraints that the simple search box and list of results puts on the ability of the user to express their intentions and the ability of the search system to express the complexities and ambiguities of the results surely talk to how the paradigm has long since served its purpose. Portal plays offer a far more natural starting point for well integrated vertical search than Google's over simplified interface and lack of real integration.

On a related not Battelle has just posted about Yahoo's relevance slider. Having recently read his book (The Search) I was really taken aback by this. The slider has been around for at least 6 months. I wonder if he will soon be posting about Microsoft's search builder feature?

Two new RSS Readers of Note

SearchFox have just launched their RSS reader (via TechCrunch). SearchFox describes the new RSS reader:

SearchFox RSS is an RSS reader that aggregates and personalizes feeds. It uses machine learning     technology to automatically rank and personalize incoming feeds to reflect each reader’s unique interests.

Having looked at tour reel, which highlights the most significant features (of which ranking incoming posts from RSS feeds via a personalized model of what is interesting and important to the reader based on historical behaviour is the killer) I was reminded of a post that Greg Linden made a while back basically describing this type of feature. On going to Greg's blog to look for the post, I found that Findory has also just launched an RSS reader with a feature that sounds almost exactly the same as SearchFox's personalization. (I also found this post which I don't think was exactly the one I was looking for, but contains similar sentiments.) Greg says:

Unlike all other RSS readers, there's no need to laboriously click and skim, click and skim. It's easy to find the most interesting articles. Try clicking "My Top Stories" to see recommended recent articles from your favorite feeds picked based on the articles you read in the past. It's the most important, most recent, most interesting articles surfaced for you from your favorite feeds.

Excellent!

How should blog portals react to Google?

How should existing blog portals react to Google's move?

• Lead By Innovation:  what does Google's blog search look like? It looks like Google's regular search. Blog content is, structurally, very different in many ways from your average web page - or rather, blogs are more coherent as a class of structured documents. In addition, the relationships between the document elements, the intentions of the search/user group and the strong community aspect of the content creation component necessitate new paradigms of interaction and access. This is exactly why blog search tools have lead to some incredible innovations that make the simple Google experience almost look out of place in the blogosphere. Google has tried to interface to the blogosphere on its terms - the first movers have all been about making interfaces to the blogosphere that include search, but so much more.
• Forget Blogs: blogs are an instance of a whole class of data - Online Personal Media (to create a new term for what is sometimes referred to as User or Consumer Generated Media - more later on why I prefer this new terminology). The play for those in this space (which includes blogs, message boards, flickr, social tagging - all that good stuff) is to translate the hype around  blogs into access into a vast space of content which simply cannot be adequately serviced by a simple search interface.

I'm excited by Google at last moving into the space, but on reflection, my expectation is to see Google reacting to the realities of the space rather than to see the likes of Technorati, BlogPulse, PubSub and so on panic and turn into Google clones over blog data.

Some other considerations:

• Google will bring more traffic to blogs (I've seen a sliver of this already in my log).
• Blogger will become less of a hot-bed for spam (one can dream!).
• The pressure is on for full content - leading to acceleration in the feed advertisement space.
• Google brings more awareness to the space - have a look at Alexa's view of the last day: more traffic to Technorati and BlogPulse:

(BTW, if you are wondering why I prefer Online Personal Media to Consumer Generated Media, I realised I can't think of people stuck in a war or disaster zone uploading images to Flickr and blogging via their cellphones as being consumers! 'Consumer' is a role for an individual - it is the individual that blogs, sometimes as a consumer, but often not. The next step in terminology is to figure out something that captures this and which can capture feeds - 'Feed Media'?)

 

Is Prospective Search different from Search Feed Subscription?

I've seen a number of posts talking about how PubSub's system is fundamentally different because it does prospective searching, unlike the stodgy old backwards facing search that Technorati, BlogPulse and others provide. But how is prospective search any different from having a subscription to a search feed? Technorati and BlogPulse (not to mention everyone else) has this as part of a standard feature set. Cindy Sullivan says:

PubSub is a prospective matching engine, meaning it's only concerned with information you want to be alerted to from this very moment forward.  I'm by no means the expert here just really fascinated with what PubSub is doing.

Isn't that a description of a search feed? I have to be missing something here.

PubSub describes a system that would require a representation of all queries in a form that can be efficiently matched against each new post (or rather the content of an RSS feed):

The heart of the PubSub service is a powerful, proprietary Matching Engine that makes it possible, for the first time, to match millions of search queries against thousands of new pieces of information every second.

A naive implementation of this would simply place the new data in the index and then run a time restricted search. A more interesting implementation (theoretically interesting) would be one which keeps no object data state, just query state. The criterion that drives this is the time taken from a post being published to notification of the search subscriber. This time becomes more and more critical given the importance of the issue and/or the prominence of the poster. PubSub have worked on the later issue with their linkRank system. I don't believe any of the free public services allows for a request like 'let me know if someone says anything libelous about my product.'

 

IceRocket blasts KBCafe

Following the discussion on spam filtering, I read KBCafe's reaction to Mark Cuban's post on using email confirmation to validate pings (via Kevin Burton's FeedBlog). If IceRocket has really black listed KBCafe simply due to this piece of text:

Mark Cuban: If blogging is supposed to be a personal medium, I dont know why we cant use an email confirmation for blog posts.  We do it for comments to keep out comment spam. Why not do it for blog posts?

Randy: Let me get this straight. When I ping IceRocket, I'm gonna get an email with a link that I need to click to validate the ping. At which point, most of us will stop pinging IceRocket. I'm sure Sifry (the competition) likes this idea!

as Mark Wilson claims:

Update: It would seem our friend Mark Cuban, wasn't impressed with this post. IceRocket informed me that I'm no longer welcome to meta-aggregate their feeds, quoting the above as final reason.

then that has to be the weirdest thing. KBCafe's RSS blog is reasonably well cited (according to both PubSub and BlogPulse - Technorati was taking a break when I ran the query) and Mark Cuban isn't doing himself any favours by doing things like this. Can you believe I actually hesitated about posting on this due to the negative image I got of IceRocket from this thread? But I feel confident that they like me as they created a nice homage to BlogPulse's trend search right on their own site...

Feedback on PubSub's LinkRank Graphs

Bob Wyman recently slipped a mention of the new LinkRank analysis from PubSub into the discussion of top N blog lists, and now the analysis is live. In general, I have liked PubSub's approach to profiling. They solved an interesting problem (how to put all the information on a single page) in a reasonably elegant manner. The latest addition essentially adds an extra column to the table (with two new fields per cell) and an additional graph (again, with two new variables tracked).

As Wyman mentioned in his initial post, the new graph displays the LinkRank and the percentile of the blog. For illustration, here is the graph for this blog as of today:

It took me a while to really grok the graph. Upon reflection, I realised that I was used to seeing percentile information expressed as a shaded region on a graph (as percentile in the contexts I have seen implies the range of instances that are subsumed by the instance under consideration - thus a child in the 80th percentile in height is taller than 80% of all other children of his age). In addition, as in the child example, the number is expressed as the inverse of that used by PubSub. My understanding may be entirely incorrect here, but it is the reason I had some difficulty at first.

Another issue is the range of values. I range from 25, 000th in LinkRank to somewhere around 500. For blogs of a more stable rank at the top of the list, it is hard to make at any fluctuations in the small numbers, this is even more the case for the percentile value. Here is the graph for Battellemedia.com:

The small number issue is interesting as it illustrates the relationship between the metric and the pragmatics of presenting the metric in an intuitive and consumable graphic. It may be that for this metric, a non-linear scale would produce a better presentation.

As is discussed elsewhere, Wyman's metric gives more weight to recent links. This means that regular posting is required to keep a high rank. For me, this highlights an important part of the whole ranking discussion: what is the meaning of the rank? Wyman's rank conflates volume/time and in-links and so, if it is used to judge quality, or authority, it means that more posts more frequently are better than fewer. I'm not sure if this is how I judge the quality of a blog. Blog posts are time stamped, but not all posts contain time-constrained information.

I'm not going to comment on rank algorithms here, but will say that there are many types and uses for blogs. Just as tags might have been a good idea, it is easy to see how they have become noisy and spamable - becoming things in their own right rather than descriptions of object data. If ranking algorithms like this do the same thing then we will artificially value features of blogs which perform well in such rankings.

Anyway, I have to stop here as I need to write another five posts today to keep my rank up...

BTW, the link to 'learn more' on the site is broken.

The scoop on CustomScoop

I couple of weeks back I signed up for a free trial with CustomScoop. This trial period is now coming to an end, so I'm going to try and capture some of my impressions here.

This is what CustomScoop has to say about itself:

A leader in the online news clipping service industry, CustomScoop provides your organization with a daily, customized news briefing. Our proprietary technology reviews news sites around the globe to gather the latest information 24 hours a day, 7 days a week and deliver it to your desktop. Our pre-screened database of online news outlets includes not only the major wires and daily papers, but also TV and radio stations, smaller daily and weekly newspapers, magazines, trade journals, and new media publications.

The trial was set up in the following way: up to 5 keywords or phrases are provided and a set of source types is specified. The source types include:

• U.S. Online News (5,000 sources)
• International Online News (2,000 sources)
• US & EU Government Web Sites (7,000 sources)
• Policy Web Sites (1,200 sources)
• Blogs (6,000 sources)

Clearly, the service is very much news oriented.

I chose 'xbox', 'playstation', 'electronic arts' as my keywords and the two news and the blogs sources. Note that the blog sources have a very low count suggesting that they are hand picking blogs in some way. I haven't seen any information on their site about how they are getting their sources - is it via business relationships or do they do the crawling by themselves?

CustomScoop delivers daily clippings and provides a web served interface for exploring the data generated by your configuration. Currently, the daily email reports are not particularly useful for me as they are too long. This is not much of a criticism as clipping services are better suited to a low stream of daily results. So the comments that I will make are on the web based interface.

Perhaps the most interesting feature that the interface provides is a breakdown of results by the geographical region of the source of the clipping. In other words, CustomScoop has annotated all of their feeds with the location of that feed allowing them to break down results along this dimension. This is a great feature. The interface, however, is a little cumbersome. It lists all regions including all US states, all countries, and other breakdowns in a single long list. It would be far more useful to see only those regions which had results, together with a count of the results rather than this long list (possibly with the option to see the empty categories).

The interface provides a number of graphing and charting facilities. Here is an example from the top level report:

Initially, I've found this breakdown confusing. For example, under 'Radio' I have 3 results. Clicking through I can get to a display of all the clippings and their sources. I then find that the content is actually online content on a site for a radio company (e.g. here). In a sense, all the text is online, this is a breakdown of the type of media company whose online presence, or online version is being harvested.

The Charts and Graphs section can show me histograms of clipping volume along a number of axes including the source (URL), keyword, and date. This goes part of the way, but what I really want in order to get some value is the ability to intersect these things. In particular, I want to see keywords over time, locations over time and so on. Of course, it is not surprise that their fee based enterprise version appears to contain the usual trend charts like those that I describe.

In summary (and I want to be up front with the fact that I haven't given enough time to really looking at this service) there are a number of modifications that could be made to the interface that wouldn't compromise the teaser strategy that CustomScoop is going for with this free demo, particularly the rearranging of the navigation by region. The geographic location break down by sources is, I believe, novel in free tools - though this is only free for 2 weeks as a demo. The mixture of source types is confusing. The fact that there are 6, 000 blogs - which blogs are they? how were they picked? - doesn't present the user with enough information in order to judge what to make of the data they are seeing.

I'm sure there will be many regular updates to this tool as CustomScoop gets more and more feedback, so I hope to get back there sometime to give it another go.

RSS Changes Everything

Consider two metaphors:

• The church door metaphor: someone has something to say, so they post a document on the church door. Word spreads about the document and people come to the church door to read it.
• The pamphlet metaphor: someone has something to say, so they stand in the street and pass out (an infinite supply of) pamphlets.

The first is Web 1.0 content, which uses URLs to point to content. In fact in many cases, the URL means the content and people pass around the URL. The second is RSS (and other feeds) where it is the content which is passed around. Clearly, the URL is in trouble. It is going to change from a nearly unique reference to content to a many to one reference to content. Search engines want to deliver pages (URL) with the most appropriate content, but what the user wants is simply the most appropriate content. This is where permalinks become so important. In RSS 2.0, permalink is defined as an optional (default true) attribute on the guid element (which is also optional).

Auto-podcasting

One of the potential values of podcasting is drive-/walk-/run-time access to blog posts. However, a good blogger does not necessarily a good radio voice make. I've been playing with Cepstral's text to speech (tts) synthesis demo and found it to be remarkably good at speaking short blog posts. Here is an mp3 (wav) of a recent post.

This demo is of the Cepstral system with no blog specific mods of any type. Certain structures (in this case parenthetical text) could be dealt with in a slightly more listener friendly manner, but as an out of the box demo, this is really great.

Now imagine being able to listen to posts from your morning read away from the computer. Systems like Cepstral's offer multiple voices, and many configurations providing lots of potential to vary the voice per post, have quoted material spoken in a distinct voice and so on.

Note that speech is another example of an annotation stream!