Data Mining: Text Mining, Visualization and Social Media

Manufacturing Consent in the Google Age Part 2

I finally bit the bullet and wrote some code to pull news stats from Google News to provide something more attractive than the charts that Google News Archive provides. I should be living up to other posts on this blog and present the data in HTML 5 charts - I'll get to that. In the mean time, here are two charts which capture mentions in the news of some key terms present in the main stream media during the US's invasion of Vietnam. Note, of course, how these terms are getting plenty of press attention now.

 

The y-axis is the number of documents Google reports in its archive. The x-axis is without labels (Google docs seem determined to copy some of the worst aspects of Microsoft Office's charting controls). It represents the years from 1960 to 2010.

Some more context to this thread can be found in my earlier post.
 
 

Two Views of UK Election Trends

Briefly, the two charts below show firstly mentions of the party leaders without collocation with mentions of the other leaders, and secondly mentions of the party leaders without this restriction (i.e. without requiring that the name appear in isolation).

Gordon Brown's microphone gaffe is clearly visible (the second blue peak). The first graph can be thought of as a description of each candidate's recognized identity. Notice how Brown is isolated in the first graph around the unfortunate microphone story.
 
 

Manufacturing Consent in the Google Age

Currently, I'm enjoying (finally) reading Herman and Chomsky's Manufacturing Consent. If you are not familiar with the book, you are probably familiar with many of the concepts. Essentially, it proposes a 'propaganda model' describing the interactions between mass media, corporations and government. Reworking the media model, in part, to one in which the customers are corporations and the product is the audience. At the same time it establishes the idea that while the 'free press' wears the guise of an open information market it is in fact hiding a bias towards the interests of corporations and the government.

The book is structured around establishing the model and then examining in the context of several case studies.

There are two interesting aspects to consider. Firstly, the research that was done to develop the case studies could be done, now, in a very different way. Rather than manually examine a small number of publications, one could leverage far larger corpra via tools like Google News. Secondly, it is interesting to examine the blogosphere ecologically with the same eye towards the overall behaviour of the system rather than the 'good intentions' of the individual participants.

Looking at the first issue, we can re-examine one of the case studies using Google News. The comparison of media coverage over the assassination of Jerzy Popieluszko in Poland (which the authors claim received heavy interest from the US media-complex) to that of Archbishop Oscar Romero in El Salvador (which encouraged far less media 'outrage').

Using Google News we see the following:

In 1984, hits for 'Jerzy Popieluszko" are estimated at 1,380.

In 1980, hits for 'Oscar Romero' are estimated at 286.

Lamentations regarding Google's interface aside, and making some gross assumptions regarding methodology, these numbers are in line with those in the book (the Romero story getting approximately 20% of the coverage of the Popieluszko story).

The book goes further into the content and address the tone and intentions of the authors (were they outraged? did they attempt to 'demand' a response?). The point being illustrated is that client states (like El Salvador) will not receive media attention when acting in ways for which non-client states would generate outrage and hightened interest.

Looking at the second issue, we can think of part of the propaganda model wrt content found in blogs. One of the issues the Herman and Chomsky raise is that of sourcing news content. A news organization must gather news. News 'events' might happen anywhere but there are limited resources. Thus one would like to determine where rich sources of news content are and go there. The government (in the form of press room) and corporations (in the form of press releases) provide just that economy.

In the case of bloggers, what are the sourcing models? Most likely a mixture.

Perhaps the most immediately salient component of the model is the generation of 'flak'. Flak in journalistic terms is either content used to question reports or harass the process of reporting, or distracting information released to divert attention. One only has to take a look at TechMeme to witness the relationship between the corporations and the blogosphere. Today Apple's CEO, Steve Jobs, sources a rare 'letter' to the blogosphere regarding his position on the Adobe Flash product to coincide with the increased attention that the other Apple story - a far more sinister one regarding the attempts to recover the lost iPhone prototype.

Data, Statistics and Unbelievers

I was pretty impressed with Obama's inauguration speech. He almost slipped by without using the word 'terrorist' or any of its relations (he used 'terror' once) and he gave respect for science and rationalism a shot in the arm by appealing to data and statistics:

These are the indicators of crisis, subject to data and statistics. Less measurable, but no less profound, is a sapping of confidence across our land; a nagging fear that America's decline is inevitable, that the next generation must lower its sights.

He almost did right by the rest of us by referring to those of us for whom faith is not a concept, although he used a term which centres on faith. One can be sure that every word in the speech had a great deal of thought behind it, and that every word was balanced with its synonyms. We can infer, therefore, that Obama's team considers atheist as being in some way wrong. It is arguably the most accurate term, and so it must be wrong for political reasons. Perhaps 'unbelievers' is a way of being inclusive while still indicating his theological position.

Naturally, Neoformix has the whole thing in perspective.

The Edited Palin: Can Journalism Be Too Objective?

You may prefer ‘objective’ reporting, or you may prefer character driven news-o-ticians. Either way, one thing I’ve noticed more and more is the use of verbatim transcripts including misspeaking and repairs. Traditionally, Bush has been an easy target for this, but Palin is another example. Take this recent CNN article “Palin says she doesn’t regret Couric interview” which includes:

• "Certainly I should have done the interview. And to attribute I think that interview to any kind of negativity in the campaign or a downfall in the campaign, I think it's ridiculous."
• "In retrospect, in hindsight, I wish I would had more opportunities or that we would have seized more opportunities to speak more,"
• "However, the campaign is over. That chapter is closed. Now is the time to move on and make sure all of us are doing all that we can to progress this nation."

Now, does Palin speak with perfect ineloquence? or does the media report too literally what is said? Couldn’t this have been reported:

• "Certainly I should have done the interview. And to attribute I think that interview to any kind of negativity in the campaign or a downfall in the campaign, I think it's [is] ridiculous."
• "In retrospect, in hindsight, I wish I would had more opportunities or that we would have seized more opportunities to speak more,"
• "However, the campaign is over. That chapter is closed. Now is the time to move on and make sure all of us are doing all that we can to progress this nation."

All Eyes on Obama

Obama has the attention – will he get the vote? Of course he will!

Visualizing Tax Policy

William’s been doing some nice work on visualizing the tax policies of the two US presidential candidates: here, here, and here. For readers not in the US, note that the term ‘socialism’ is considered a scary word over here, and – at least in the mainstream media – the frequency with which a pundit uses it is in indirect proportion to the speakers understanding of its meaning. Note also that there is an idea over here that taxes are a form of punishment (at least, this is how Joe “Wurzelbacher” the Plumber expresses his relationship with this basic instrument of government).

Political Streams Update; ACORN Boating

We experienced a hiccup early this morning, but the site is back up – hopefully this wasn’t too disruptive.

While we don’t provide any supported API’s right now, you can persuade a trend chart out of the site. Here’s one showing the attention around ACORN.

Political Streams: Update

We’ve just rolled out some fixes to Political Streams. Among them

• Improved extraction of news article titles: we do pretty well at this using a generic system, but were failing in some specific cases.
• Fixes to trend graphs: due to some issues with our content store, our trend graphs were a little suspect. The trend graph are now providing an accurate signature for the terms they show.
• Improved classification: a lot of what drives our system is the ability to classify content either by type (news articles, weblogs posts, …) and by topic (politics, etc.). The system is set up to allow incremental improvements moving forward.
• UX improvements – paging on the home page now works with the real time pushed updates.

I’m going to be keeping a sharp eye on the trend graphs now, especially tracking how Obama and McCain attract attention.

Another thing to note: because we use the most excellent Freebase for our semantic content, we can take advantage of updates made there. For example, here is the page for Joe “the plumber” Wurzelbacher.

Animated Weblog Diffusion

As part of my presentation at the Personal Democracy Forum 2008, I showed an animation of a simulation of diffusion through the blogosphere. Anthony and Guilhem (also presenting at PDF) have just released a version of this on real data over on their Presidential Watch 08 site. They talk about it here, and the two demonstrations can be found here and here.

The visualization is an augmentation of the existing graph based view with a new display/control showing the attention over time to the video that is the subject of diffusion.