Data Mining: Text Mining, Visualization and Social Media

Virtual Earth

I've been watching my Bloglines search feed for "virtual earth" get swamped over the last couple of days. Far too many posts to read (whoever comes out with an RSS reader that can rank or otherwise filter posts is going to be onto something pretty hot). In the head to head with Google, who is winning recognition? The basic trend graph below gives some indication:

Virtual earth has a bit of a way to go. Another interesting thing to look at is how separable mentions of the two systems are. We can look for mentions of one that don't contain the other.

This suggests that until recently Google Maps was mentioned without discussion of Virtual Earth.

On the other hand, perhaps not surprisingly, Virtual Earth is mentioned half the time with one of Google's mapping products.

Finally, here is how links to these systems are trending:

Trending Multiple Content Types

I'm trying to put together a post on the reaction to BlogPulse's recent launch which includes a site makeover and the introduction of a number of powerful new features. However, in getting some comments together on the context in which the launch took place (IceRocket's pending launch of 'BlogScour', the visibility of two new services - BIRG and CustomScoop, discussion of Technorati's service, Umbria Communications site makeover - no doubt anticipating something new coming from them) I got held up reading this post from InterAdvocacy (profile).

Chip Griffin (found partner of CustomScoop) wrote of CustomScoop's trending demo:

Basically, it's a free service that let's visitors see how two search terms do in head-to-head competition in the online media (including newspapers, mags, blogs and more).  Obviously, it serves as a teaser for the full CustomScoop service, but it also generates interesting results in its own right -- not unlike Intelliseek's Blogpulse Trends, except that we have a more balanced mix of coverage where they are exclusively blogs.

It is always encouraging to see how people are taking the notion of trend search (which BlogPulse popularized, though we can't claim to have invented the idea of the time series ;-) and applying it in different areas. However, the idea that you can take a set of different types of data and count results over them to produce some insightful and actionable intelligence has more than a few problems.

Firstly, absolute count versus normalized count. Showing a graph of absolute count can tell you little. You have to factor in the volume of posts on the day (or whatever the granularity is).

Secondly, the type of document being counted has a huge implication as to the interpretation of volume from that content source. A single post in the New York Times is quite different from a single post in a LiveJournal blog.

Trend graphs have great potential, but they have to be handled with care. In addition, the type of data being counted and the way in which you mix the results has to be very transparent otherwise the results can not be used. Yes, it is true that CustomScoop is getting data from a number of different sources, but there is considerable value in knowing that BlogPulse's graphs are exclusively over blogs.

It will be interesting to see how free, self service portals like BlogPulse and CustomScoop deal with the complex issues of multiple data types. These are already being handled by fee based products (like Umbria's Buzz Report and Intelliseek's BrandPulse).

Microformats offer innovation opportunity

Bud (The Community Engine) makes a lot of sense. The opportunity to innovate that search and other consumer interfaces to blogs (and other online public media such as boards, usenet, reviews and so on) presents provides a way for smaller outfits to push ahead of web engines that are not focused on microformats. Analytics, link based and otherwise, are unique to these engines' intimacy with the data format.

Popularizing Advanced Technology

Something very significant happened with Google's promotion of its Keyhole acquisition. Keyhole had taken a new approach to GIS (geographical information systems): they created a fast intuitive interface and they made the technology relevant to consumers (not enterprises) by making it a fun exploration tool with a strong sense of grass roots community. You didn't have to have any goal in mind to get value out of the tool. Google then came along and used its mega-brand muscle to get this technology to a huge user base. One could debate whether the inclusion of local search and directions impacted the hype and excitement that was generated.

To put it in perspective, the Google mapping buzz was in the ballpark of that for the launch of the Sony PSP, a major event in the huge handheld gaming industry:

Shortly after its launch, Sony sold somewhere in the region of 500, 000 units.

There has been some discussion in GIS circles that this was a missed opportunity for the major GIS firms (particularly ESRI). However, the consumer market has never been the focus of these corporations and so the opportunity involves a huge consideration wrt user support, deployment, server infrastructure and so on - something which a major search engine is swimming in but which enterprise software vendors are not.

Ok, now given the above description of Google's impact on GIS, what other technologies will be popularized in similar ways, how will the be made relevant to consumers and how will they impact the search space. Don't forget, Google Earth is a departure from Google's simple light weight interface.

I see this as breaking down into two broad categories. Firstly, there are those technologies which will give greater power to the user but not change the search interface significantly. Take the area of corpus annotations. Imagine being able to search for nouns, verbs and combinations of these. That would be immensely powerful. Secondly, there are those technologies that would require a departure from the the generic search interface. Specifically, I see the potential for a number of technologies that deliver analytics. Imagine if Google provided trends over spans of text that accurately identified person names, or product names.

This second category is where I see the world of online personal media (OPM) portals as being trailblazers. The most obvious example being blog search and analysis engines. The reason for this is that the volumes involved, in terms of posts, is huge (giving a user similar experiences to web search in general and making data driven analytics such as trend mining meaningful), but there is a very strong constraint in terms of the type of document that is covered (simple and generally short messages, most often created by a single author) and, importantly, there are very clear meta data which users can understand (comments, citation links, trackbacks, etc.).

Bringing these things together has the potential to generate a huge shift in user expectations and behaviours when it comes to how they interact with online data. The hardest part about moving forward with search is not the existence of powerful technology, but the popularization of that technology in a way that can alter consumer behaviour.

Evaluating Blog Search Engines

There have been a number of posts recently that take a look at evaluating blog search engines. Not surprisingly, this is a lot harder than it looks. To understand the problem in more detail, it is useful to try and throw out some definitions.

Firstly, what are blogs? A year or two ago, it would have been relatively easy to answer that question. However, the definition has been getting more and more muddied over time. Problems include:

• The location of the blog. If a blog is defined by a URL and I place the blog on the home page of my corporate web site, then are links to that home page links to the blog, or links to the site?
• RSS gave blogs their big break, and the use of ping servers and the aggregation of updates that they provide were a win for everyone - sort of. There is nothing in the structure of these systems which indicates if the pinger is a blog or not. Consequently, if you listen to a ping service, you will get non-blog content - e.g. discussion threads on flickr.

Any definition of blogs will also include models of link structure (trackbacks and citations), comments, blogrolls and so on. All of which affect the expectations of the user (searching for posts with comments? searching for comments on a topic?)

Secondly, what is search? Or rather, what is the intention of the searcher? For blogs, there is a large proportion of search focused on figuring out who is talking about the searcher - so called vanity searches. Due to the long tail, this type of search is very sensitive to coverage issues, which is not the case if you are looking for information. In addition, as there are types of search which are variations to those of major search engines, or entirely novel in form, the semantics of the search terms presented to blog search engines are still evolving. What does it mean to enter a URL into a search engine? Are you looking for citations of that URL? Are you looking for posts from that blog?

Thirdly, what are the results. We have all been led to expect the type of results that major search engines deliver - a ranked list of web pages. However, there are other factors that are important to rank as well as other ways in which blog data can very naturally be delivered. One obvious issue is the distinction between a search for a blog (find me a good blog on this topic) and the search for a post (find me posts that match my criteria). This underlines the fact that blogs results are often at the granularity of sub-page documents. In addition, blogs are far more timely than web data in general. Consequently, time is an important factor (witness the several blog search engines that now provide trend graphs as search results).

Fourthly, what are the quality issues for search results? These must include (at least):

• Segmentation - the separation of blog posts from the blog template and peripheral data.
• Deduplication - the filtering out of multiple copies of the same post (something you will never see on a web search engine).
• Spam filtering - the removal of spam blog data.
• Time - the accurate representation of the time of a post.
• Relevance - the boosting of results that are more relevant than others.
• Speed of query execution - how fast the results come back.
• Comprehension - how complete the coverage is.
• Time to index - how long it takes for a post to become part of the search engine's index.
• Repeatability - if I issue the same query (immediately) do I get the same result?
• Result count estimation.

In addition, the search engine must be judged on a number of service issues such as:

• Ability to request the inclusion of a blog.
• Ability to remove a blog and all posts from an index.

So how do we go about testing blog search engines? Ultimately, the quality of a search engine can only really be measured by the success or failure of a user to achieve some task, a task in which the search engine is a tool and the search results are not the final goal of the task. This means that anecdotal tests (in which one determines that one engine is better than another because they return more hits) are completely out. It also means that a representative set of tasks needs to be captured and translated into realistic queries. Note that, interestingly, the willingness of the market to provide more interesting tools than the monolithic one dimensional list of ranked results means that those who innovate with tools (like trend mining, etc.) that are accessible to users have an advantage.

Who should test search quality? Anyone involved with blog search, myself included, can not be asked to provide a complete criteria for testing. If asked I would certainly either intentionally or otherwise give a bias to the things which I think are important and which are, of course, in the system I am involved with. What is needed is an external market observer, and no doubt some money to fund the whole thing (or at least the ability to derive revenue from the process).

UPDATE: discussion continues here and here.

Mixing Structured and Unstructured data

The following graph shows the trend for three major motion pictures released this year: Return of the Sith, Batman Begins and War of the Worlds:

Star Wars opened on May 19th, Batman Begins opened on June 15th, which corresponds to the first
major peak. War of the Worlds opened on June 29th.

Through its first Sunday, Star Wars grossed $158, 449, 700; Batman Begins $72, 896, 986, and War of the Worlds  $100, 561, 125. Normalizing by the Star Wars gross, these numbers become 1.0,  0.46,
0.63. If we take similarly normalized percentages of blog posts (1.30, 0.80, 1.15), we get 1.0, 0.62, 0.88. Although the trend chart matches the overall ordering of grosses, the relationships are not linear. This observation is mirrored in the trend for the box office receipts (as reported by the excellent BoxOfficeMojo):

However, what we really want to do is to be able to predict some aspect of the movies receipts from the pre-release online trends. In addition, we want to be able to qualify the type of trends. In this example, both Batman Begins and War of the Worlds show weekly dips and peaks - not so Star Wars which shows no valleys between weekends. This lack of valleys does not correspond to a similar pattern in the receipts (as reported by BoxOfficeMojo) and so indicates not the volume of people seeing the movie but the level of excitement around the movie - in other words, the post movie chatter is more prolonged than for the other two.