Data Mining: Text Mining, Visualization and Social Media

Social Psychologists in Las Vegas

I've just returned from a brief trip to Las Vegas where I had been invited by Sam Gosling and Kate Niederhoffer to participate in a panel at the annual meeting of the Society for Personality and Social Psychology. I was very happy to participate with Tom Lento from Facebook and Winter Mason from Yahoo Research. The panel topic was Forging connections between Social Media and Social/Personality Psychology and was intended to help bridge the gap between researchers in the social-psych field and those of us in industry working with social and personal data.

Perhaps the most salient point that came out of the discussion was the difference between current social-psych methods and the data scale of the online social world. To help understand this new world, and for social-psych researchers to make the transition, an ability to code and to deal with large amounts of data is required. There was genuine interest in this (provoking questions about the specifics we had in mind, such as working with scripting languages) and I believe a real interest in the opportunity that lies in this data rich online world.

In terms of facilitating interactions, Winter probed the audience regarding knowledge of key conferences for web data analytics, etc. such as WWW and ICWSM. I consider the later to be a perfect forum for this discussion to continue, and with Sam's help I think this can really happen.

The conference itself is run in a manner quite different to that which computer scientists are used to. No papers are submitted and acceptance (which runs high) is based on short abstracts (generally paragraphs). I managed to see one of the poster sessions and was fascinated by the topics (extremely focused accounts of certain behaviours and traits) and the methodologies.

I feel that there is a lot to learn and the the ultimate winners in the social space will be those companies farsighted enough to really embrace this field and who figure out how to integrate such insights about users into their products.

Malaysian Blogosphere Division

(Briefly) I've been working with some new data that our team has produced and created the view below. What I find striking about this visualization of 6k blogs is the clear division between two major clusters. A very limited drill down on the data suggests that all of these blogs are Malaysian in origin (and most are on Google's Blogspot). I don't yet have enough insight into this component to understand why there is a split - perhaps more to follow.

A Different Way to Think About Apps

I use an app to check on the skiing conditions at local slopes. I click on the icon on the iPhone, the app pops up and I see some data. When I'm on a desktop, I do exactly the same thing, except the app I click on is a web page. While Apple claims 140, 000+ apps available for their phone, and others in the space do the same, a good number of these apps are really just thin clients backed by the same sort of data that usually goes to populating a web page.

This prompts an obvious question: how many 'apps' are really just thin clients backed by web servers similar to those for traditional browsers? An initial answer to this might be constructed out of a break down of app categories (games, for example, are less likely to fit this model).

In addition, as we hear more and more about crawling the deep web, what are the rules of engagement for crawling the data services that back these thin client, browser-like apps?

Google Updates Whistler Aerial Imagery with Snow

I just noticed that Google Map's aerial images of Whistler (hosting the 2010 Winter Olympics) is now nice and white. This is definite a change (and other ski locations I visited have not been updated with winter data).

Visualizing Network Evolution

New to me is this work by Bergstrom and Rosvall on discoverying and visualizing changes over time in networks. The basic idea is to track nodes as the join and migrate between clusters in a network.

Some additional visualization are available here.
 

New Google UI Fit and Finish

As I tweeted today, I noticed something wrong with the Google landing page I was getting: the advanced search link mistakenly links back to www.google.com, not the advanced search page. Thinking it through, I thought this might be due to flighting a new UI.

This UI is similar to that which Tom Krazil spotted back in November, but different in colouring. Going via an anonymous proxy, I get the normal fit and finish.

Update: note also that the web options (the left rail facets) are now their by default, whereas in the standard UI, they are collapsed and require expansion by the user.

In addition, the actual landing page has a different design for its buttons and the shadow on the main Google logo has all but gone.

Infostate of Africa

AppAfrica has an interesting post showcasing an infographic describing the state of the internet in Africa: Infostate of Africa. I took the liberty of uploading this to SeaDragon which allows for better browsing and appreciation of this type of resource. Note that SeaDragon doesn't get the centering of the image correct, so hit the home icon to see the full thing.

[HT Nathan]

West Seattle Blog going Mobile

I just noticed that the West Seattle Blog- a well established hyperlocal blog in Seattle - has either rolled out, or is testing a new mobile theme.

 

At the bottom of the page is a button to switch the mobile setting off, but it doesn't appear to have any effect, so I suspect this what we are seeing is the testing of a new mobile site for the blog and that they are experiencing a few glitches. Strangely, when I visit this from my iPhone, I get the non-mobile version of the site.

See my related post on 2010 - The Year of the Neighborhood.

2010 - The Year of the Neighborhood

As the geosocial revolution continues - creating more and more intimate links between the digital space and our physical spaces via mobile devices and data driven services - the word 'neighborhood' is becoming more and more prominent. A neighborhood (in urban terms, larger than a block, smaller than a zipcode) is the perfect granularity to connect with users as we spend a good chunk of our time there.

'Near by' is often scoped by neighborhood, our schools define catchment areas at this level, supermarkets serve neighborhood sized portions of the population.

As we see the rise of geosocial gaming (things like Foursquare, Gowalla, MyTown), and the mechanisms they introduce being adopted by other spatially aware services (Yelp) we are also seeing the rise of the importance of real estate data. It is in no way surprising that Google is interested in the real estate market.

NabeWise, a new neighborhood review site, similar in some regards to both EveryBlock and Centerd. It's just opened its doors with coverage for New York and San Francisco, and its entry points are qualities of neighborhoods (trendy, singles, beautiful people, etc.). It is very interesting to note that the sign up process includes the question 'are you a real estate agent?'

Design-wise, many of these sites have to address the presentation of rich data in an understandable and consumable manner. In this regard, these resident oriented sites have similarities with real estats 2.0 sites (RedFin, Zillow, Trulia) and take advantage of the increased data literacy of a younger, web 2.0 savvy audience.

Key in all of this is the bedrock data set of neighborhoods. As the LA times demonstrated with their Mapping LA project, the definition of any neighborhood is somewhat subjective and borders need to be negotiated. For many cities, Wikipedia keeps rich pages describing neighborhoods and their histories.

From a UX point of view, we can expect to see more interfaces with elements like these, sampled from some of the companies above (NabeWise, Trulia, EveryBlock):

Bing Maps Updated with More Coverage, Apps

Yesterday, Microsoft rolled out an update to our mapping product Bing Maps. The update includes more street imagery coverage, including Vancouver and Whistler:

Two new applications, Events and Destination Maps

Destination Maps allows you to create a fun rendering of the map data for use in invitations, event planning, etc.

With this release, the product is no longer in beta which means that more users will start seeing at by default. This heats up the competition in the mapping space and distinguished Bing by virtue of its platform approach to mapping (ReadWriteWeb and others offer commentary).