Data Mining: Text Mining, Visualization and Social Media

Future of News at Princeton

I'm back in Seattle after attending the excellent Future of News workshop hosted by David Robinson and Ed Felten. There were many contributing factors to the success of the workshop, not least of which was the cross disciplinary nature of speakers, panelists and attendees. In addition to participants in my line of work, or with similar areas of interest, such as David Blei (former colleague at WhizBang!Labs) and Kevin Anderson (whose career spans both the BBC, with some involvement with Backstage, and the Guardian, an organization that is not caught with with fretting about the past; and who blogs on Corante, for his sins) I met representatives (and survivors of) traditional newsprint organizations, individuals centrally involved in transforming news in the Web 2.0 world, academics with impressive access to the entire trajectory of media evolution, hackers and so on.

Given the rich range of voices, while we may not have solved any specific problems (despite Ed's closing remarks) I certainly feel as if we aired a reasonably good sample of them. On reflection, I stand by yesterday's summary: there is plenty of pessimism around old media structures and plenty of optimism around the opportunities that new sources and new forms of information, combined with new ways to filter, analyse and aggregate this data presents. Bridging the two positions, there is concern around issues of quality and value with respect to the nature of the content (that is to say, a contributor's ability to provide transparent and supportable content) - will the new information ecology support reasonable ideals for news? Note, to me, those ideals centre on making the reader better informed and more efficient at selecting content.

For other coverage of the event, a great starting place would be Kevin's posts (starting here with an account of Paul Starr's opening talk). Also Steve Boriss, Tim Lee, and Jack Kemp. Note that the presentations and discussions will soon be available online.

Workshop on Web 2.0 Patterns

The every increasing population of Web 2.0 technologies and applications is ripe for deeper analysis. The Web 2.0 Pattern Mining Workshop plans to do just that: bring together Web 2.0 practitioners to discuss and explore common patterns that define the space.

Web 2.0 features are now commonplace—blogs, wikis, RSS feeds, social bookmarking and the like are almost everywhere you look online.  Now that these technologies are maturing, what are their common problems and challenges? How are these problems being solved? What similar challenges do Web 2.0 developers face, and how can they leverage the most common solutions?  Here’s your chance to gather with other professionals facing the same issues and work together to identify solutions.

Maps of Edinburgh

If you are traveling to Edinburgh for WWW 2006, you may be looking at some online maps of the city. It is interesting to compare the results from different services. Below I have placed some samples of maps from Microsoft, Yahoo, Ask and Google. While the Yahoo result is pretty much useless, it is clear that the other 3 services have similar data from which they render their maps. What I notice most in this comparison is the different design decisions that have been taken. Ask, for example, has opted to highlight the green urban areas as well as the industrial dock areas. Google, on the other hand, has opted for a simple bold colour coding of different route types (A, B, C roads in the UK code for progressively smaller and lower volume routes).

Microsoft Live

Yahoo

Ask

Google

Interface Updates

One thing about the web, especially web 2.0, is the fast, fluid iterations on interface design. Something that I really like about the Corante hub is the fact that they are constantly improving the web page. The same goes for TailRank. Some of these changes are suble/blink and you'll miss them types (Corante's latest is the use of resizing content rather than fixed width), others are more significant, high impact changes (TailRank flips its left and right panels to great effect).

How To Get What You Want

There are 3 things you can do with a good business idea:

1. Think about it - do nothing.
2. Talk about it - do nothing.
3. Execute.

The launch of 3Bubbles, suggested that now, in the world of web 2.0, and subject to the type of cosmic laws that Douglas Adams would approve of, you only need to get to step 2 to ensure that the company will exist in the very near future. 3Bubbles is a good example of this: a colleague and friend, in a brainstorming session, came up with (1) and described to me (2) exactly the 3Bubbles idea before any news of it had hit the blogosphere. Hey presto, now we know it is going to be out this month.

Can you guess what I'm thinking of now?

Edgeio

There is a buzz abuzzing around Edgeio, which, according to this entry [Via /Message], will use tags and blog crawling to gather classified ad information right out of your blog and manage it online to help connect you with buyers.

Ad listing in blogs? Who does that? Today, almost nobody. And that's why this idea could work at all: Teare said the tag "listing" is found only about 10 times a day on millions of blogs, so it's an ideal, clean tag with which to create a unique index of "listings from the edge."

There appear to be two risks in this model (I haven't yet ready any critical descriptions of the system - everyone blogging about seems to have been positive and/or had some relationship with the company). Firstly, no-one owns tag space. This is one of the biggest failings of tags (and one of the most predictable). By not having a formal system for meta-data, there is no control over meaning. So now the tag 'listing' means 'I want Edgeio to crawl me, interpret somehow what I have on my blog and then offer that to buyers'. The argument against formal spaces in tags (a la XML name spaces for example) is that they require either more work on the part of the user or integration into blogging platform as a standard (oops - didn't Technorati achieve that with their ownership of the tagging mechanism - perhaps that would be too meta).

Secondly, the value of such a system to sellers requires pretty good coverage. I'm guessing that Edgeio only crawls RSS enabled blogs. This means that there will be some percentage of bloggers out there who cannot list on Edgeio. In addition, given the unstable state of the ping/RSS duopoly, Edgeio will have all the challenges of any other blog search engine in keeping things up and running. This risk may result in value - more reason to improve the ping/RSS stability and coverage, but it may not.

It's going to be interesting to see how this stuff pans out. Good luck to them! Somewhat Frank has a more detail explanation of the service.

BTW - imagine what would happen if someone wanted to put out a DOS attack on Edgeio by flooding the blogosphere with the listing tag. Will we enter a new era of spam functionality?

Novelty versus Currency

In Fact versus Opinion I discussed a fundamental dimension of content which has implications for the personal value of news media, web 2.0 media, etc. In looking at digg.com, tailrank, etc., I realised that there is another fundamental issue at stake: Interestingness versus Timeliness, or Novelty versus Currency to use slightly more pleasing terms.

There is an article at the top of the digg list right now regarding hyper-drive experiments that the US military is considering - reach Mars in 3 hours. I remember reading about this quite a while back, possibly last year when the research was originally published. The thing is, it's very interesting. There is an assumption that socially driven link promotion (and this includes BlogPulse) mixes currency with some degree of interestingness. Is it true that we want to mix them? Is it true that we can?

Bloglines Update

I recently posted about Bloglines - how their features were stuck and how their quality was starting to go down hill. The Bloglines team posted this on their blog a couple of days ago:

We're not going to beat around the bush about this. Bloglines performance has sucked eggs lately. Why? In short, Bloglines has been busting at the seams like the Incredible Hulk.

 

All of us here at Bloglines have been foregoing sleep and social lives over the past several months to keep Bloglines running and preparing for our move to a new access center (with bigger britches and a very elastic waistline).

 

So hang tight because there's a light at the end of the tunnel. The move will happen soon; we'll keep you posted.

 

- The Bloglines Team

With web 1.0, search engines built businesses out of content that they appropriated from web sites. With web 2.0, the relationship between the content (both providing and accessing) and the content creators is a lot closer. Consequently, their is far more pressure on companies that are making money at either end (e.g. blog services and rss aggregators) to be stable. A melt down on TypePad and Bloglines, for instance, would cut me off to a large degree from the blogosphere.

The question these companies need to be asking themselves is: where is the tipping point at which customers are going to flee like rats from a sinking ship?

How to Build (most of) an Ontology

This is pretty cool. Matt Biddulph shows how to take a list of entity names (e.g. people names), mine Wikipedia for entities using Yahoo's API and produce a map of relationships (via Google Blogoscoped). The results look like this:

 

The reason I say this is not a full ontology is due to what I view as being the essential elements of an ontological structure:

1. A symbol space: it is not clear what type of symbols space is being used in the above example.
2. Structure: the above is a directed graph.
3. A relationship space: defining the relationships between elements in the ontology - what do the arrows mean in the above?

Having said that, I think that this is a really nice example of both the power of current simple natural language processing methods, the web 2.0 environment and the value of open information on the web.

[Trackbacks appear to be broken on Matt's site...]

Greg on Mashups

Greg makes a good point: are mashups really the wave of the future or just toys that will never be more than demos?

I keep hearing people talk about as if companies are creating web services because they just dream of setting all their data free. Sorry, folks, that isn't the reason.

Companies offer web services to get free ideas, exploit free R&D, and discover promising talent. That's why the APIs are crippled with restrictions like no more than N hits a day, no commercial use, and no uptime or quality guarantees. They offer the APIs so people can build clever toys, the best of which the company will grab -- thank you very much -- and develop further on their own.

The BBC recently launched backstage.bbc.com which is designed to be a source of data and a community base for people happy to mix it all up. It is very overtly a place for the Beeb to get good ideas fed back to them (which is good when you remember that the corporation is a public entity). On the other hand, the commercial examples are, as Greg points out, making offerings with no guarantees. In fact imagine the following example: data is made freely available; everyone throws in their idea; whenever a killer app emerges, the data is suddenly no longer free (I believe Alexa has been very open about this strategy). Now what do you do with your users?

Greg:

There is no business model for mashups.  If Web 2.0 really is just mashups, this is going to be one short revolution.