Data Mining: Text Mining, Visualization and Social Media

Geocoded News

I know - you're asking: how many more of these can we take? I don't know the answer to that. But here you have the next mashup to join the pile of news on-a-map applications: WeReporters GeoNews.

(via Google Blogoscoped)

The spin here is that it tags the news by type (the categories in Google News). You can also read the news in a non-mapped form by selecting a category. I've not looked in too much depth at the accuracy of the geocoding (this often gets missed by the blinding eye-candy effect of this type of visualization). However, a first look at the World results showed a number of errors including the association with Berlin of this article:

Kidnapping wave again sweeps Iraq - International Herald Tribune

BAGHDAD A Christian peace group said Wednesday that the policies of the US and British governments were ultimately to blame for the kidnapping of four of its members over the weekend, among the latest in a new wave of abductions of Westerners.

which links via there site to a completely different IHT article, as well as a number of others whose only connection with Berlin is:

Associated Press reporters David Rising in Berlin, Michael Tarm in Chicago and Rob Gillies in Toronto contributed to this report.

Geocoding really is in the details.

Update: The mislinked article appears to be a problem of IHT's - it has the same title linked to the wrong article on its own pages. One of the kidnapping victims does appear to be German:

Susanne Osthoff, 43, of Germany. Disappeared Nov. 25. On a video made public Nov. 29, kidnappers threatened to kill her unless Germany stops dealing with the Iraqi government. An archaeologist who speaks Arabic, Osthoff works for a German company that helps distribute aid in Iraq.

However, the article is clearly about Iraq, not Berlin - or if it is about Berlin, it is also about the capitals of the other nationals kidnapped as well.

 

What Ever Happened To: RSS Ping

RSSPing makes a lot of sense. How come no-one is talking about it? Google BlogSearch 16, BlogPulse 34, Technorati 48. As far as I can tell, it started sometime in August (judging by the earliest posts found):

RSS Ping combines RSS item metadata with site update information. It enhances the current ping specification, adding information about exactly what has been updated or published, with the notification that something has been updated. RSS Ping reduces the need to revisit or work done by aggregators and search engines when revisiting the publisher site after being pinged, to provide truly real-time updates for information on the Web.

Question Answering: What is the Density of France?

A subtle but powerful feature on Google is question answering. If you type in a question like 'what is the population of UK', you will see an answer at the top of the page:

When I was working at the University of Edinburgh, I was involved in a project that required the extraction of information from tables. To cut a long story short, this train of research resulted in my thesis: Understanding Tables in Text.

Why do I mention this? Google is using table understanding to come up with the answers to these questions. If you take a look at the source in the above example, you can see that it is a view of the data presented in the CIA World Factbook. The page is, in fact, a big table of country names and population statistics. If you browse the other listed sources, you will find that they all contain some element of structured content pertaining to the answer of the question. It may not be an explicit table - there may be, for example, the name of the country and the key value pair population: #.

Once you grok this, you can get Google to do some impressive party tricks. By entering 'what is the religion of nepal?' you can find an answer: Religion: 90% Hindu, 5% Buddhist, 3% Muslim, 2% other from this page. By reverse engineering the table to create a question, you can get Google to answer something very interesting, like: who are the major trading partners of nepal? A neat trick to suggest real intelligence in the machine.

Of course, as Google is doing a light-weight trick which avoids any real semantics, you can get it to give odd answers:

• What is the density of France? 110.
• Who is the Queen? is the female head of a royal family.
• Who is the King? town (1990 pop. 4,059), Stokes Co., North, North Carolina, 15 mi/24 KM NNW of Winston-Salem;
• Where is the end? Country: UK

My guess is that for certain sources (like Wikipedia) which they have directly wrapped, they trust the answers. For other sources, where a loose match is found, they require multiple sources of evidence - what I call the WoodStein inference paradigm (Woodward and Bernstein wouldn't report anything unless they had multiple sources).

Google plans to roll out this feature gradually as it becomes more powerful and as the answers become more reliable. When they step over the threshold of simple text based answers to actual semantics, they will have taken a first step into a world where search no-longer means 'give me a list of documents'.

Data Mining on Corante's Web Technology Hub

Yesterday, Corante launched its first series of hubs: marketing, media and web. The hubs bring together expert bloggers in the various fields, aggregate their feeds and top it off with an editorial blogger-in-chief. I'm very excited to be involved in this new endevour, and - I have to say - somewhat flattered to be counted in such good company.

Personalize Search Smack-Down

Something is going on with Vivisimo. First there was the Murdoch rumour, then I hear them interviewed on NPR (in a story relating to immigration), now I see this paper by Raul (via Battelle). The paper makes a good argument for why personalized search won't happen for underlying technological reasons. I have to say, I find myself agreeing with Raul's point to some extent. I would add that personalization is another example of making the computer smarter (while preserving the sacred simplistic interface) rather than empowering the user. Vivisimo's clustering engine is a great example of extending the interface: giving the user more control and more information.

I put the question: why haven't other search engines gone for this? on the same stack as: why can't we browse news by maps?

All Your Base are Belong to Microsoft?

eBay Strategies has the scoop:

As predicted, it looks like Google Base's launch is going to be followed rapidly by at least one more system designed to help people sell online in new and exciting ways.

Microsoft is evidently in beta with something codenamed Fremont that from what I can gather is a classified system that internally has been called a craigslist-killer.

More Blog Conversation Visualization

I just found this older post by Jon Schull on visualization of conversations on blogs:

Akismet Blog Spam Filtering

Planet Ozh points to Akismet. I'm not sure how I missed this (it looks like it launched in late October). Anyway, just as blog search engines parade their stats, Matt Mullenweg's Akismet has this to say:

441,477 spams caught so far

2,106 so far today

82% of all comments are spam

82 % of comments are spam! This reminds me of a recent comment spam gaff I saw on a friends blog:

I have a ##KEYWORD## site/blog. It pretty much covers ##KEYWORD## related stuff.

I'd love to get me a ##KEYWORD## site/blog too.

BTW, as I leave Akismet, the numbers just clicked up:

441,534 spams caught so far

2,163 so far today

82% of all comments are spam

TalkDigger/BlogBar

Fred has just started alpha testing of a new version of his excellent TalkDigger site (more later). On an (almost) related note, BlogBar is a new embedable search bar that allows you to, well, I'll let the site tell you:

Once included in your blog, BlogBar lets users do their search directly from your blog/website. Technically, Blogbar processes your query through our servers. But that’s just to maintain the system alive and working : we can add new engines, modify the code in case an engine modifies its way of getting a query processed, etc. Once the queries are processed, Blogbar opens your search results into new windows and – if your browser is properly configured – into tabs. Isn’t that cool ? ;)

(Thanks, Natalie).

Christmas Wishlists

Peter O'Kelly points to a New York Times article about Christmas wishlists:

Two of the most-coveted consumer electronics items this holiday season are the Apple iPod music player and the Xbox 360 game console from Microsoft, according to retailers and industry analysts.

That represents a major shift in buying intentions this year.

In 2004, portable MP3 music players did not even make the annual top 10 list of desired gifts compiled by the Consumer Electronics Association. And video game machines were in ninth place.

This year, though, those categories occupy the first and third positions on the list, based on a telephone survey of 1,000 randomly selected adults. Digital cameras are second on the most-desired list this season.

Here are the trends from the blogosphere:

These results are similar to the New York Times information.