Data Mining: Text Mining, Visualization and Social Media

Metric driven Agile for Big Data

Working in Bing Local Search brings together a number of interesting challenges.

Firstly, we are in a moderately sized organization, which means that our org chart has some rough similarities to our high level system architecture. This means that we have back-end teams who worry mostly about data - getting it, improving it and shipping it. These teams are not sitting in the end-users laps and our customers, to some extent, are internal.

Secondly, we are dealing with 'big data'. I don't consider local as it is traditionally implemented to be a big data problem per se, however when one starts to consider processing user behaviour and web scale data to improve the product it does turn in to a big data problem.

Agile (or eXtreme programming) brings certain key concepts. These include a limited time horizon for planning (allowing issues to be addressed in a short time frame and limiting the impact of taking a wrong turn and the 'on-site customer'.

The product of a data team in the context of a product like local search is somewhat specialized within the broader scope of 'big data'. Data is our product (we create a model of a specific part of the real world - those places where you can peform on-site transactions), and we leverage large scale data assets to make that data product better.

The agile framework uses the limited time horizon (the 'sprint' or 'iteration') to ensure that unknowns are reduced appropriately and that real work is done in a manner aligned with what the customer wants. The unknowns are often related to either the customer (who generally doesn't really know what they want), technologies (candidate solutions need to be tested for feasibility) and the team (how much work can they actually get done in a sprint). Having attended a variety of scrum / agile / eXtreme training events, I am now of the opinion that the key unknown of big data - the unknowns in the data itself - are generally not considered in the framework (quite possibly because this approach to engineering took off long before large scale data was a thing).

In a number of projects where we are being agile, we have modified the framework with a couple of new elements.

Metrics not Customers: we develop key metrics that guide our decision making process, rather than rely on a customer. Developing metrics is actually challenging. Firstly, they need to be a proxy for some customer. As our down stream customers are also challenged by the big data fog (they aren't quite sure what they will find in the data they want us to produce for them), we have to work with them to come up with proxy metrics which will guide our work without incurring the cost of doing end to end experimentation at every step. In addition, metrics are expensive - rigorously executing and delivering measurements is a skill required of second generation big data scientists.

The Data Wallow: While I'm not yet happy with this name, the basic concept is that in addition to the standard meetings and behaviours of agile engineering, we have the teams spend scheduled time together walling in the data. The purpose of this is two fold: firstly, it is vital that a data team be intimate with the data they are working with and the data products they are producing - the wallow provides shared data accountability. Secondly, you simply don't know what you will find in the data and how it will impact your design and planning decisions. The wallow provides a team experience which will directly impact sprint / iteration planning.

Related articles

5 Hidden Skills for Big Data Scientists

O Knowledge Graph, Where Art Thou?

The web search community, in recent months and years, has heard quite a bit about the 'knowledge graph'. The basic concept is reasonably straightforward - instead of a graph of pages, we propose a graph of knowledge where the nodes are atoms of information of some form and the links are relationships between those statements. The knowledge graph concept has become established enough for it to be used as a point of comparison between Bing and Google.

Last night, I went to see a performance of Kodo - regarded internationally as the premier taiko group. A search on Bing for 'kodo' produced the following result:

 

Bing showed good results for the web and images as well as a knowledge driven portion of the answer from wikipedia with links to play some of their songs. Not bad - but no mention of the performance.

As Kodo were performing at Meany Hall on the University of Washington campus, I did another search on Bing for the venue:

Here we see something better - the venue is recognized as a venue and consequently joined with the events that are known to Bing, including the concert I was attending. As the event information included a link to the performer (the blue Kodo link in the screen shot) I followed through and found Bing gave me event information.

In these interactions, we can see part of the promise of the knowledge graph, but many areas for improvements. The event node relates the performer to the venue to the event. However the venue information in this part of the graph is isolated from that used to deliver the result for the query purely about the venue (note that the addresses are different - a common problem with campus and mall-like areas). The above experience, I think, shows the true challenge of the knowledge graph proposition - bringing all the isolated data graphs together correctly when the nodes in the graphs are actually representations of the same real world entities.

Note that in exploring this particular scenario, Bing appeared to be doing a little better than Google, though Google had partial event information associated with the Kodo entity.

As these names are possibly taken from the listings information from different sources, the name of the performer is confusingly presented in different forms.

Much of what we see out there in the form of knowledge returned for searches is really isolated pockets of related information (the date and place of brith of a person, for example). The really interesting things start happening when the graphs of information become unified across type, allowing - as suggested by this example - the user to traverse from a performer to a venue to all the performers at that venue, etc. Perhaps 'knowledge engineer' will become a popular resume-buzz word in the near future as 'data scientest' has become recently.

 

Participation and Observation in Search

The early days of web search were essentially about observation. The web search engine observed the web (documents, links and user behaviours) and then delivered results based on those observations.

In recent years we have started to see more of a position of participation in web search engines. Examples of participation include:

• Hosting web sites for businesses - by getting their data on the web more useful targets are provided for user and a short loop is developed with the source of accurate data, i.e. the business.
• Providing feed proxy services (like feedburner) - by providing a service to bloggers, the search engine gets access to valuable user information.
• Hosting content - by hosting news articles and blogs directly, the search engine gets real time updates to content first as well as direct access to user behaviour.
• Exposing data editing tools like map editors - by offering crowd sourcing tools the search engine benefits the community by improving data and is the first to know about and leverage that fresh information.

Participation looks like a core strategy for search.

Better Beaches

Having recently returned from a trip to Kauai where I used my beach search engine with middling success, I've now got a few updates out on the site.

Firstly, there is a full map showing either all the beaches in a location, or all the beaches from a search within a location. This was a pretty obvious missing feature.

Secondly, as this is an active map, you can zoom and pan the map which interactively restricts the set of results.

There are some minor improvements to other elements of the site as well.

Note - something that always interests me is the relationship between back-end data quality and the presentation of the data. By having a complete map of beaches, it highlights cases where there are duplicates in the results (a topic for another post).

If you are heading to Hawaii - give it a try and let me know how you get on.

Related articles

Snorkel* and Surf* in Kauai and Maui

The State of Hawai'i Demands a New Search Engine

Google's Year in Local Search

At the beginning of the year, Google's results page for {restaurants in seattle} looked like this:

Note: the size of the map, the presence and colouration of the main column ad block and the use of the restaurant's web page title in rendering the entity. In addition, the summary of reviews is derived from Google's community.

In March, a number of changes were introduced: the main ad block was removed and the size of the map was considerably reduced to allow more above fold ads to appear on the right rail.

The main ad block was reintroduced mid March.

At the end of April, improvements were made to the naming of entities. In the examples here, the name used for Wild Ginger was originally taken from the web site "Wild Ginger Asian Restaurant & Satay Ba...". The new name is simply "Wild Ginger". This indicates either a change in how the name is populated or a change in how the name is extracted from the web site.

In August, Google's search results incorporate the Zagat review scores combined with the native Google review scores:

August also saw a subtle change in the rendering of the smaller POI in the SERP map:

Through the remainder of the year, there were three additional changes. The ranking of restaurants appeared to undergo a modest change. For most of the year, Wild Ginger was at the top of the list:

However, as we close out the year, it has dropped closer to the bottom of the list:

The second interesting change was the introduction of local into the Google Plus product which saw the reintroduction of a two box approach to search (rather than a single search box, there are two boxes one for the 'what' of the query and the other for the 'where').

Finally, there is the introduction of markers (POI) on the map for promoted content:

Snorkel* and Surf* in Kauai and Maui

When I pushed out the first version of Beach G33k / Beach Ball (I'm not actually sure what the thing is called) - a search engine for beaches - I did as close to nothing in terms of features as possible. This approach to release follows the release early, release often paradigm which leverages user feedback and small iterations to help guide a service according to the user needs.

Thanks to a comment by Viliam Kanis I've now pushed out an update which does use some amount of stemming to improve searches. Thus, if you search for 'snorkeling' you will also find beaches with associated content which mentions things like 'snorkel' and 'snorkeled'.

Below is an example of a search result for 'snorkeling' which surfaces Lydgate Beach Park in Kauai.

If you are heading for Kauai or Maui in the near future, take the site for a spin.

Beach Search Engine Demo

I've written recently about building a perfect beach search engine. Here is a brief example of using the site.

Let's imagine you want to find a beach that offers snorkeling, but you want to find one that is shallow because you have small children with you. A query for 'snorkeling AND shallow' brings up the following results:

(currently the data on the site is limited to Kauai and Maui).

From these, I might decide that I'm more interested in Kauai than Anini (it does appear to have more of the beaches that I'm looking for after all).

Clicking on Kauai, then selecting Anini beach leads me to the following:

Here I can see that there is plenty of content describing the beach. Some of the content contains highlighted terms that lead me to confirm that it is indeed described as a beach that offers snorkeling and that it is shallow. Content from both Fodor's and FourSquare supports this:

A great family park, Anini is unique in that it features one of the longest and widest fringing reefs in all Hawaii, creating ashallow lagoon that is good for snorkeling and following the occasional turtle.... [Fodors]

A nice, shallow place for snorkeling, but can get tiring to swim around if the current is too strong.... [FourSquare]

There are many issues with the site that I'm slowly addressing, but the basic task of finding beaches seems to be up and running. If you are interested, please take a look and let me know what you think - the site is available here.

Techno Punditry with Data

There are plenty of techno pundits out there who are on a mission and rarely dip in to data to determine if reality is or is not cooperating with them. From the outside, one has to be somewhat creative to get at data that reflects consumer trends, but not that creative.

Below is the graph from Google Trends (Google Insights; a great tool) that shows search behaviour (let's call that consumer interest) around some key Microsoft product terms.

What does it suggest? If we use xbox (green) as a baseline (xbox is the best selling gaming console on the planet and the companion of the kinect, the world's fastest selling piece of consumer electronics ever) we might imagine that we see searching around windows 8 (blue) to exceed interest in xbox. Interest in lumia is approaching that in bing and microsoft surface is clearly showing an uptick with now obvious down turn.

Building the Perfect Beach Search Engine

Currently, as I've mentioned in previous posts, beaches are a strangely under-served segment of the local search space. Searches on Google and Bing for beaches are fielded by entities such as resorts and restaurants that happen to be matches for certain beach related terms. If you search for 'beaches in kauai' you will get hits for beach resorts, etc.

There is plenty of content about beaches, from the many dedicated locale sites to general travel related community sites (like Trip Advisor) and editorial sites (like Fodor's). In addition, there are a number of resources that aggregate structural data about beaches. These include open data resources like GeoNames and GNIS but also proprietary resources like Foursquare.

Unfortunately, there is nothing that brings all these things together. There is not product which provides an aggregate view of the set of beaches or the collection of things said or otherwise reported about them.

With an upcoming trip to Hawai'i at the end of the year, I wanted to make sure I was getting the best value for my travel dollars. I've build a prototype beach search engine which provides the following.

• a partly curated set of beach data covering approximately 12, 000 international beaches
• aggregation of beach related content
• search funtionality (so you can search for kid friendly beaches that offer good snorkeling)
• summarization of Flickr images so that an impression of what it's like to be at the beach can be formed

I believe there is plenty of potential for such a system. I've already found some hidden beaches that I wasn't aware of at our destination that I'm excited to check out when we get there. My goal is to make the system public in the next few weeks (my trip will be a forcing function for this!).

For now, here is a screen shot of part of the experience.

How Apple's Local and Mapping Investment Will Lead to Web Search

[I work at Microsoft where I work on projects that drive data quality in our local search experiences on Bing and other clients.]

Most of the civilized world, by this time, has heard about Apple's fumble with their new mapping and local search capabilities in iOS. Apple replaced Google's application - which is possibly the largest investment in cartography, imagery and local data ever made - with a home grown solution reportedly rolled out of maps from a number of providers including TomTom and local data from providers including Yelp.

As Apple has realised, there is a lot to learn for an entrant in this space. The hardest lesson they are learning now is actually not about data sources but about metrics and how to assess the quality of the product - something which they don't appear to have invested in in a manner fitting to their global user base.

Apple will soon learn another lesson. Once the fog has lifted over the state of their entity data set (e.g. fixing the location of cities and ensuring coverage for local businesses), Apple will have to start worrying about ranking search results. When a user asks for {kid friendly sushi in seattle} which of the many sushi places ought they to return. They will be presented with a choice between specialized providers - with whom they will actually be in competition - or creating the resources required for relevance ranking themselves.

A key aspect of providing appropriate indexing and ranking features is the association of content with the entities. Where does this content come from? The web. How is it acquired? Through large scale crawling, understanding and indexing.

Apple will likely find that as they pull on the thread of local search, their scope will have to open up to quite a different world, another world which - like local - they haven't yet the expertise in.