Data Mining: Text Mining, Visualization and Social Media

Google Teleport

I was walking down Denny Way in Seattle and turned onto Broadway. Whoosh - thanks to Google's teleport feature I was zapped over to Broadway in Renton! How cool is that!

Actually, what I'm talking about is the difficulty of interpreting geography properly. If you enter Google's street view mode in their mapping product at the crossroads of Broadway and Denny in Seattle you will find that while the imagery is correct (if a little dated), the addresses that are associated with these locations change from Seattle to Renton as you turn the corner.

In other words, this is not Renton:

Here's the live embed to see if the issue is fixed.

View Larger Map

Reducing Geographic Data

A while back, I posted about a very cool site created by InfoMagnet called Glogbal-i. As a result of that post, I had some communication with Matthew Leigh of InfoMagnet. I had stated in the post that taking a set of polygons representing geographic data and simplifying them was 'pretty straightforward.' Matthew was amused by this statement - maybe he knew I was speaking from ignorance.

So I figured I should actually look at the problem. Below is the result of some work I did on reducing a set of geographic data to provide a simplified map. The main goal here is to use fewer data points while maintaining the coherence of the data. It is actually a really fun problem, and I certainly haven't taken the solution as far as I want.

The solution tracks every point in the data set and ensures that polygons that share points also share the transformations of those points - that's the easy part. Making the decision regarding transforming a point is something of an art. I currently look at the variance of the points in a window and make a decision based on a threshold. Regardless of the decision of how or when to transform the points, this approach will miss cases where lines are shared but no points. In this case it is possible for the transformed points to result in very different lines which no longer coincide.

 

Australian GIS Data

The Map Room posted a while back about some newly available Australian GIS data. I'm adding it to my link list of GIS data. I've yet to evaluate it, but when I do I'll post some datasphere images.

Google Maps Coverage, Part 2

Andrew Hitchcock pointed out in a comment to my first post on this topic that Google is also missing Hong Kong. Follow this link to see the Google map. Microsoft has no such problem. If anyone has seen any other map data quality issues, send them over.

Virtual Earth

I've played a little with Microsoft's just released virtual earth. On linux/firefox I see far too many image tile outages (bits of the image are missing to I see black squares every where). However, they did get Cozumel on the map which, strangely, Google messed up. More later.

DIY Virtual Earth, Part 1

Everyone is busy sticking pins in the earth via either Yahoo's or Google Earth/Google Maps' APIs. But what does it take to build an application from scratch? There are three basic components:

• Data - lots of it; satellite data and GIS data.
• Maths - projections (mapping projections and 3D projections).
• Code - you need to build a client server architecture.

For satellite data, you need global coverage and you need detail. Currently, most of the great coverage - i.e. the coverage that produces the 'wow, have you seen this?' reaction, is limited to certain areas of the planet and certain types of locations. Google's data, for example, is heavy on cities, but not so heavy on places that Matt Hurst wants to go on vacation. This makes a lot of sense as the first thing people do is to look up where they live to see it from the sky. Populations are denser in cities, so more satisfied customers.

So, how do you go about getting Google Earth-like data? Landsat is your friend. Compare these two images, the first from Google and the Second from Landsat:

These pictures are comparable in quality. Of course, downloading and storing all the Landsat data would require significant disc space. However, if you have the right architecture, you could potentially use the web interfaces to this data to act as your server. Whatever you want to do, you want to have a look at https://zulu.ssc.nasa.gov/mrsid/mrsid.pl.

Next GIS data - you want boundary information and place name information. The first allows you to draw the polygons around countries, the second allows you to pinpoint cities (and act as a resource for location searching and geocoding). For this, turn to the world of potatoes. Specifically, the country data found via the Centro Internacional de la Papa. This will give you all the country boundaries as well as a goodly amount of admin1 and admin2 level data (i.e. states and counties or similar).

For location data, there is little to rival that provided by the Government of the USA. The Geographic Names Information System (GNIS) provides very large datasets of locations, names, alternate names, etc.

Coming in part 2, pointers to the mathematics needed to quickly play with this type of thing.

Google Maps Coverage

Looking at the map view of the world on Google maps, it is interesting to see that Cozumel off the coast of Mexico is missing (though present on the satellite picture). I wonder which other areas have been missed and why.

More on half-lifing the planet

Greg has a link to Brad about creating the data for use in Google Earth (aka keyhole). This stuff is just too cool. Wait until people are posting pictures on flickr taken in this environment and including their avatars! My next question is: how does this work for beaches? I'm actually interested in improving my ability to make efficient use of my vacation money by choosing the right beach.

UK Data

I few weeks ago, I followed the instruction on this site to request a CD of UK shapefile data from the government. Yesterday, the CD arrived in the mail (international). I haven't yet got into the data as I need to find some time to code up a coordinate translator, but I was impressed that the data arrived, and with not much of a delay.

I am going to put up some links on this site to various data resources, or you can use my del.icio.us page and sift through the GIS or other tags.

Where is this going?

If you have ever played (or watched) the game Half-Life 2, you will be familiar with state of the art urban modeling, physical models in an interactive context etc. Now have a look at some of the images from Google Earth, which includes data describing the shape of buildings, mix in some of Microsoft's Virtual Earth, which has high resolution data of buildings taken from a 45 degree angle and imagine the possibilities (taking a good does of geocoded data, GPS, RFID and so on).

You search for a hotel, a map tells you it is in striking distance of your business meeting or conference, so you flip your rich client in to walk-around mode and walk around the neighbourhood. It turns out there are just too many broken windows, and what is more, the pool of the hotel is constantly in the shadow of  surrounding buildings. So you explore - just walking the streets, guided by your local search which tells you which way to head for other hotels. Finally, you find one you like. You walk up to the door and interact with an online receptionist to book your stay.