Data Mining: Text Mining, Visualization and Social Media

BoardTracker gets Trendy

BoardTracker, the message board search engine, is now displaying time series of the number of boards and forums that it tracks. I've been monitoring these numbers for a while and it is interesting to compare different views. Here is BoardTrackers graphs of forums and threads over time:

The data I've been accumulating is over a wider time window:

and

My graph for forums suggests that BoardTracker updates its index of forums on a weekly schedule.

BTW, an additional stat I'd like to see is the number of actual boards that are crawled (not just the number of forums).

Listings from the Edge

There will be plenty of posting today about Edgeio, which crawls social media sites for information it then renders as classified ads. One thing to point out - the listings are not just from blogs. My first experience, browsing the for sale|auto category, was to arrive at listings which originated from message boards.

Geocoded User Generated Media

Steve Rubel points to the next step in Topix' use of geocoded forum posts:

t's not enough that marketers/PR pros know what's being said about our clients/brands, what language this commentary originated in, how loud it is and if the speed is accelerating or decelerating. We also need to be able to pinpoint just where the conversation is strongest or weakest. Topix.net has taken the first step I have seen this direction. (Disclaimer: Topix is a CooperKatz client). They have created a real-time map that shows where the contributions from citizen journalists are coming from inside their walls. I think we'll see tons more innovation in this area this year. This is data that marketers die for.

It is a pretty cool map (see below). This is not the first effort to geolocate user generated media (not by a long shot). I agree with Steve, however, that this is certainly a must-have feature for analysts of this type of user generated data.

Topix Discloses Poster Location

Members of message board communities disclose plenty of information on their profile pages, as do bloggers. In all cases, they have the option to omit or hide certain fields. Topix now discloses the location (based on IP lookup) of posters contributing to their forums. This gives a fascinating insight into what topics are important to people from which countries, however it also bypasses a very standard level of privacy - how can contributors opt out of this?

Via BuzzMachine

BoardTracker Update

I've been monitoring the statistics published on BoardTracker for a few days now. My hope had been to see the number of forums and the number of threads showing a steady, monotonic increase. However, as the graphs below demonstrate, something quite different appears to be happening:

The second graph, showing the number of forums tracked over time, is probably the key here. The fact that it appears to flip between certain values suggests not that there is a gradual purging of, for example, dead boards, but that there is some issue with BoardTracker's infrastructure.

What is interesting also is that during the period at the centre of the second graph during which there appear to be not additions to or removals from the database of boards tracked, the number of forums shows a steady increase: the natural growth of the forums on a closed set of boards.

What's up?

Message Board Search

I've posted a few times on this blog about the message board search space. In this post, I'd like to bring together a summary of the main tools that are out there.

• BoardTracker : search over message boards only, search for threads or authors, results page directs traffic to the original board. Results can be sorted by relevance or by date. Author profile information is available, linked from the search results page.
• Itch search over message boards, news headlines and blogs, search for threads, results page directs traffic to original board.
• OMGILI : search over message boards only, search for threads, results page directs traffic to original board.
• BoardReader : search over message boards only (note that they used to have a blog search tab but it was never delivered and has now been removed), search for posts, results page directs traffic to original board.
• Lycos Discussion Search : search over message boards only (including Yahoo! Groups and MSN Groups), search for posts, results page directs traffic to original board.

In terms of volume, precise comparison is difficult. However, BoardTracker is reasonably far ahead of the rest, all of whom are reasonably comparable.

All of the sites except for BoardReader have some form of advertisement in their search results.

In addition to standard features, BoarderTracker offers a number of subscription based capabilities that allow for more detailed analysis and tracking of conversation in the board space. BoardReader can be contacted for advanced solutions which are not described on their site. These two sites also offer features for message board hosts that increase the value returned via the search engine results page and in terms of content acquisition.

In addition, GYM each have some form of group/discussion search over their own message board formats.

• Google Groups
• Yahoo! Groups
• MSN Groups

Consumer Facing Analytics for User Generated Media

Aggregating and analysing user generated media (blogs, boards, usenet, etc.) posts on products is what Intelliseek, BuzzMetrics, etc. do for a living. This level of analysis is not currently available in consumer facing products other than search (blog search, board search, usenet search) and some specialised search applications (trend search and the simple sentiment search of OpinMind). The content universes for enterprise products in this space include generally blogs, message boards and usenet and can extend to review sites and other genres.

On the other hand, there are social content hosting sites, such as riffs.com and amazon, which support research on products via aggregating content only found on their site - what we might call captured content search. These are useful to a degree. However in order to get a fuller picture - one might say a more accurate picture - of what people think about products, or any topic for that matter, a solution is required which is capable of spanning multiple content types.

When considering user generated media, we are forced to consider the boundary between this type of content and professional content. A boundary which has blurred considerably in the last year and will continue to do so. In other words, there is a problem in deciding when to include opinion from the professional end of the spectrum and how. For example, how does one weigh a movie review from a blog, a movie review from rotten tomatoes, and one from, say, Mr. Ebert? An additional issue when providing consumer facing interfaces for this type of data is capturing the product being researched and finding those posts that refer to it. The captured content, especially in sites like amazon, may well have the advantage of a complete taxonomy of products which can be searched against rather than the content of the reviews. In other cases, the user requires assistance in accurately retrieving results for all variations of expressions used to refer to the product.

In this context, it is interesting to note two developments. First is Google's review search, which Kevin pointed to. This appears to aggregate reviews from a number of sources (and types of sources) for movies. It also provides access (in a faceted search type of interface) to positive, negative and neutral reviews (based on segmentation via numeric review scores). Not only that, but it provides common phrases mined from the reviews. This is pretty cool, and brings features found in standard enterprise offerings to the consumer. Note that BlogPulse has had consumer facing features based on similar features for over a year - for example blog bites, which both extract common terms and find snippets which contain terms with significant relationships.

I'll post a deeper dive on Google's review stuff later.

The second new consumer facing user generated media interface is reported by Smart Mobs, and TechCrunch and appears to be a service which will allow users to use their phones to input bar code data on a product and access user generated commentary drawn from blogs.

Is 2006 going to be the year of consumer facing analytical tools over the user generated media space? And how will the enterprise players in this space respond?

Message Board Decline?

I spoke recently with Ron Kass, founder of BoardTracker. He imparted a clear passion for the communities that grow up around message boards, contrasting them with other forms of online chatter such as blogs and what he terms 'talkbacks' (think - the commentary hanging off a BBC news article posted in by readers). I think Ron is right when he underlines the differences (and different values) of blogs vs boards. Boards encourage a different level of social responsibility than the soap-box world of blogs.

With this in mind, it is worth reading this GreatNexus' post on the topic:

I agree with Pinyo’s argument in the blog The Death of Discussion Forums? where he questioned the rational behind starting an SEO discussion forum in this day and age. I will put forward an even more radical argument, questioning the rational of starting any forum at all this days, forum for any conceivable topic already exists in their hundreds. Some webmasters even predict the slow death of discussion forums and the rise of blog. I don’t quite agree that blogs will completely kill off discussion forums, but I am sure it will greatly weaken the power and popularity forums enjoyed in the last few years.

I am a new convert to blogging and liking it very much, in fact much better than discussion forums. Before discussion forums start their inevitable decline, one may as well put the old forums to good use. One of the good uses of forum was bringing people together. I have started a thread which I hope will bring bloggers together at my webmaster forums  called blog exchange This will work in a similar way to the way forum owners use post exchange to help one another populate their forums.

This is a classic scenario of an aging technology making way for a newer one, some concepts from the dying technology are often transferred to the newer one so, a slow goodbye to “Post Exchange” and a warm welcome to “Blog Exchange”.

Before anyone sounds the death knell, it is worth considering two important factors. Firstly, the number of boards and volume of posts is huge - far bigger than the blogosphere. Secondly, just as the two forms of interacting with those online are different, they attract different people.

I believe that message boards are far from over and are probably maintaining their presence if not increasing. I also believe that there are valid hybrids, such as the Slashdot blog/forum structure, or Ain't It Cool News' discussion fora, which further underline the basic need for explicit community models, something which the blogosphere lacks.

A Source for Everything, Everywhere

I really like this article about the XBox 360 currently running on the BBC news site. As with most next gen gaming console releases, it details the queuing over night of hardcore gamers, the out of stock stores, the anticipation for release games and so on. However, this time, the report is derived in part from simply reading articles found in online message boards.

While some struck lucky, many others were left disappointed.

"Thought I'd wake up at 5am then sneak down to Toys 'R' Us in Nottingham. Got there at 5:50am. No joy though, over 20 queuing up!" wrote one gamer in a posting to a forum on the Xbox Scene fan site.

"I went to Telford Curry's this morning, 7:20am. I was 5th in the queue. Opened at 8am only to announce they had four. Four! Gutted," said another posting.

This ties in nicely with some comments I posted about the different forms of online personal media in 'The Blogosphere is not a Walled Garden.' But here is the kicker, a search for combinations of terms in the quotes on both Google and BoardTracker (a board specific search engine) turned up either no relevant hits, or links to this BBC article. Interestingly, the search engine on the specific site mentioned is down now. However, by paging through the thread, I managed to identify the messages reported.

The article clearly indicates that this form of content has the potential to play an important role, but that  is currently not taken seriously by search general web search services - credit goes to BoardTracker to actually have a service in there. They are still growing their index, hopefully they will get to critical mass soon.

The Blogosphere isn't a Walled Garden

I posted recently about the Murdoch/Vivsimo story reported in the Pittsburgh Post-Gazette. This post on a web based message board (Yahoo! Message Boards: LOOK) generated a most of traffic to that post. As we think more and more about the conversational structure of the blogosphere, rather than thinking about it as a walled garded, we need to think more and more about how it integrates with other spaces on the web.

I've been saying for a long time that those interested in analysis over the blog space are going to be knocked out when they see the potential of other similar spaces, most noticably the message board space. They are going to see topics of conversation which are more richly represented in other spaces, they are going to see different (and stronger) community structures, and then they are going to have fun integrating analytics from both.

Here are a couple of examples. The first trends 'murdoch', the second trends 'xbox'.

Notice that while there are echoed trends (the large peak to the left), there are also trends which are unique to each source. For example, Usenet appears to show an upwards trend to the end where as Boards and Blogs show a steady flat trend.

Here we can see that a higher percentage of Boards are focused on the XBox. In addition, there is an initial peak - an anticipatory burst - which preceeds the main launch peak. Notice too the leap in Usenet posts.

To give full credit to everyone involved in the conversation, visualization like those posted on Anjo's blog  (just been updated I see) need to include links at least to external (non blog) pages. The fact that a message board concerning LookSmart is discussing a story I'm interested in (Vivisimo) is certainly a thread I'd like to follow.

(These images are copyright Intelliseek).