Data Mining: Text Mining, Visualization and Social Media

Map Madness

The world is going map crazy. I'm seeing a lot of posts recently about Google Earth (of course), job openings at Microsoft for their Virtual Earth product, the Where 2.0 conference, Yahoo's API, a9.com's new mapping/local search page (via Greg Linden)...

Eventually, things will calm down and people will start looking more at the value of these systems and (perhaps) less at the eye candy. One thing that is going to be interesting at that stage is the way that mapping and local search work together. A local search which presents results as a list of links has the potential to hide, or make less obvious, mistakes. However, now with Google Earth, when I enter 'london, canada' and it takes me to London, UK, I am presented with a very immediate way to see this error.

All Eyes on Blogs?

Everyone is talking about blogs, but who is reading them? Of all the types of Online Personal Media (which includes blogs, boards, usenet, photos, etc.), blogs are certainly the most hyped, so caution is called for. Are blogs really that big in the grand scheme of things? It turns out that the answer is yes, and no. The blogs at the very, very top of the pyramid (including Slashdot, BoingBoing, Engadget, etc.) get a very large readership. First, a look at the Alexa reach measurement (the percentage of all Internet users who visit a given site - I've taken the 3 month daily average, and it is actually expressed in site users per million internet users).

• Slashdot: 861 per million.
• Fark: 581
• Engadget: 573
• Gizmodo: 335.5 
• BoingBoing: 298.5
• DailyKos: 234.5 
• Instapundit: 116 
• Talking Points Memo: 95.5
• Wil Wheaton: 86.5
• Kottke: 64

We can also look at the visitor stats for a site whenever some sort of site meter (e.g. Site Meter) is present:

• DailyKos:  508,233 (average visitors per day, Site Meter)
• BoingBoing: 228, 723 (boingboing/stats)
• Gizmodo:  average visits per day: 163,380 (Site Meter)
• Instapundit: 151,603 (Site Meter)
• Talking Points Memo: ?
• Slashdot: ?
• Fark: ?
• Engadget: ?
• Wil Wheaton: ?
• Kottke: ?

This gives a very different ranking. For example, DailyKos is gets twice as many visits as BoingBoing whereas the Alexa reach measurement has them almost equal.

Another measurement is the Bloglines measurement. When subscribing to a feed on Bloglines, you can see how many others subscribe to that feed:

• Slashdot: 35, 901
• BoingBoing: 22, 545
• Gizmodo: 18, 022
• Engadget: 15, 957
• Talking Points Memo: 4, 552
• Instapundit: 4, 197
• Fark: 1, 812
• Kottke: 1, 065
• Wil Wheaton: 617
• DailyKos: 511

I'm not sure I trust some of these numbers, especially that for DailyKos. Even so, BoingBoing jumps from 5 (Alexa) to 2,  Talking Points Memo from 8 (Alexa) to 5 and Fark from 2 (Alexa) to 7.

The Alexa numbers put the top blogs on a par with quite substantial local press:

• post-gazette.com: 368.5 (daily paper circulation: 240, 930)
• mercurynews.com: 404
• siliconvalley.com: 120.5

Comparing these numbers with those for online message boards is tricky. Alexa is by domain, so we can only really take measurements of the large hosting systems in some cases. A small illustrative sample (including some of the higher ranked boards from big-boards.com):

• ezboard: 2, 100
• Gaia Online: 271.5 (1 on big-boards)
• delphiforums: 192
• Face the Jury: 191.5 (2 on big-boards)
• honda-tech: 63 (17 on big-boards)
• Offtopic: 21.5

This ad-hoc sample suggest that the biggest board hosting sites (of which ezboard is the grand-daddy - remember others are not represented due to the difficulty of doing Alexa rankings over different hosts or URLs) are still ahead of the top blogs in terms of visitors; smaller sites (honda-tech is a good example), receive traffic similar to the lower end of the elite bloggers.

The numbers above are far from scientific, but they do go a small way to understanding where blogs sit in the bigger picture of online destinations for community and personal media. They also show that different measurements present a very different view of the top bloggers and although the relative changes in position are small, as the measurements are not linear, they indicate quite substantial changes in how the blog might be perceived.

The Politics of Projection

As a follow on to an earlier post on Google Map's projections, I notice that this is still a sensitive issue
for, well, billions of people (thanks to the Map Room).

Writing in India’s Financial Express, Y. R. K. Reddy calls for India to discard the “racist” Mercator projection, which makes “our country look so small on the map,” and advocates a switch to the Peters projection (about which see previous entry). My take is that the Mercator is a lousy projection for a general-purpose world map, for the obvious reasons, but that the Peters is too distorted to be a good alternative. I much prefer the Robinson (see previous entry), but while it’s actually used more by publishers, it’s neither as politicized nor as proselytized. The Peters is first and foremost a political stance rather than a projection.

Google Earth

I'm impressed. No, I'm not talking about Google Earth now, but about Google's support. After failing to install the new free application, I sent a message to their support centre via the online form. I got an answer back in a few hours which solved my problem. Excellent!

I'm going to do a deep dive on the application in a later post - something I am really looking forward to.  For now, let me note a few things:

• Perspective projection: the projection in the application is not simply from the 2d image to a globe, but with perspective (which is why America looks so huge in the load up screen).
• Data streaming: is Google Earth slower at streaming data than the 2d Google Maps counterpart?
• Line drawing: I'm guessing they are doing real time great circle based polygon and line drawing over the projected 2d imagery. However, I do see some gaps in the lines which suggests issues to do with projecting from 2d to 3d so not sure of this one. Real time great circle line drawing is fare better looking but slower.
• The poles: due to the way in which the imagery originates in a 2d projection, the poles look pretty messy, particularly if you manipulate the globe with the pole approximately in the centre of the screen.
• Elie, Fife, UK: Google doesn't know where my home town is, though it did get the more famous Earlsferry (of McDuff/MacBeth fame).
• Cavities: is it possible to get cavities in your eyes from so much eye candy?

Let's face it, the whole thing very slick.

Google Earth is free (so I hear)

According to Philipp Lenssen at Google Blogoscoped, Google Earth is now free. Google is doing a poor job of software support however, when I install the free version, it installs the Plus version and asks me for a username/password. A while back I installed the free demo of Keyhole and I suspect that information is somehow lurking on my machine/in their databases. I have contacted Google about this and look forward to a solution.

Simple List Extensions

Microsoft's Simple List Extensions - it's what everyone's reading...

Tag Ecology

When someone visits a blog via a tag link of some sort, it informs the blogger which tags bring the most visitors. If the blogger wishes to get more eyes on their blog, then they might well use this information to determine which tags to place on a post. This has the potential to end up like using keywords and hidden text in web pages so that search engines index them with certain content simply to increase the number of visitors. This sounds like a bad thing with the potential to suck the value out of any folksonomy.

Another Virtual Google Earth Planet

Slashdot just reported news on Virtual Planet, another interactive interface to the earth and potentially other planets. Looks very nice, which is to say, the same as the other offerings. It is the result of a European Union project and comes at a significant price.

Blogs, Marketing Intelligence on Slashdot

William Bulkeley's article on blogs and Marketing Intelligence was picked up by Slashdot and generated quite a bit of discussion. The article itself suggests the aggregate, metric based, data driven approach to using blogs (and other personal online media) for marketing and business intelligence applications:

Marketers say bloggers' unsolicited opinions and offhand comments are a source of invaluable insights that are hard to get elsewhere. "We look at the blogosphere as a focus group with 15 million people going on 24/7 that you can tap into without going behind a one-way mirror," says Rick Murray, executive vice president of Edelman, a Chicago public-relations firm.

though the one way mirror isn't really a one way mirror as blogs are not a content type but a format type with a certain content bias. Often, this type of article picks up on some rather unstartling insights provided by the interviewees. This is a fact that was echoed in the discussion on the slashdot forum associated with the article. This is not surprising, as companies that engage such services often have restriction wired into their contracts that  restrict the publication of insights.

The WSJ article also gives some insight into the mechanisms that are used to determine demographic information such as generation (X, Y, etc.):

Umbria, with clients including Sprint Corp. and Electronic Arts Inc., says its natural-language analysis can determine blogger demographics based on language, subject matter and acronyms. OMG ("oh my God!") or POS ("parent over shoulder") are expressions defining Generation Y girls, or those ages 10 to 25; FUBAR ("fouled up beyond all recognition") is often used by male baby boomers.

Again, this is a very light-weight description of the methods used, perhaps deemed more suitable for the general WSJ readership.

The discussion on slashdot included some considered and well articulated opinions about mining blog data (both for and against), some brief discussion as to the legality of taking this data as well as some, shall we say more reactionary comments poo-pooing the idea. If you are in the area, reading the commentary is definitely worthwhile.

One takeway from this recent discussion on the topic is that the two different models of mining blogs for consumer opinion and actionable intelligence (data driven text mining, versus influence driven tracking) are not yet well articulated and lead to a certain amount of confusion. Secondly, articles of this nature are often driven by the hype around blogs. When discussing the quality and value of the insights derived, a comprehensive article would compare these with those derived from broader, richer sources of personal online media such as message boards.

Companies mentioned include Umbria Communications, Intelliseek (my employer), BuzzMetrics,  MotiveQuest

 

Google Maps Coverage

Looking at the map view of the world on Google maps, it is interesting to see that Cozumel off the coast of Mexico is missing (though present on the satellite picture). I wonder which other areas have been missed and why.