Data Mining: Text Mining, Visualization and Social Media

Vocal fry is nothing new!

If you are following the discussion about the new 'trend' of vocal fry-ing, consider this graph from Google's ngram graphing thingy:

It looks like the term peaked in the corpus (whatever that may be) in the late 1980s.

One recollection I have when first encountering this when I came to the US over 10 years ago, was the difficulty of asking US females what it was. "So, why do you growl when you talk?"

Tracking Unique Terms in News Feeds

I've put together a simple system which reads news feeds (the BBC, NPR, the Economist and Reuters) in approximately real time and maintains a record of the distribution of terms found in the articles. It then indicates in a stream visualization the articles and unique terms that are observed by the system for the first time within them. The result being that articles which contain no new terms at all are grayed out.

The larger idea here is to build a 'linguistic dashboard' for the web which captures real time evolution of language.

Follow this link to take a look.

Visualizing Lexical Novelty in Literature

Novels are full of new characters, new locations and new expressions. The discourse between characters involves new ideas being exchanged. We can get a hint of this by tracking the introduction of new terms in a novel. In the below visualizations (in which each column represents a chapter and each small block a paragraph of text), I maintain a variable which represents novelty. When a paragraph contains more than 25% new terms (i.e. words that have not been observed thus far) this variable is set at its maximum of 1. Otherwise, the variable decays. The variable is used to colour the paragraph with red being 1.0 and blue being 0. The result is that we can get an idea of the introduction of new ideas in novels.

In the first book - Austen's Sense and Sensibility - we can see two things. Firstly, the start of the book keeps a pretty good degree of novelty for the first few chapters. Secondly, each chapter introduces something new.

The second book - Stevenson's Kidnapped - shows a different pattern. While it starts off with reasonable novelty, this then dies out for most of the book with spurts of interest here and there.

What is surprising to me (if we take any real meaning from this approach) is that Austen's Emma - the third book - is strong out of the gate (the first 18 chapters) but fails to break the 25% novelty ceiling thereafter.

[Note that these results are preliminary and I'm going to do more validation and testing.]

Update: see below the original visualization for an updated version with more accurate results.

Update: After looking at the above results I drilled down on the strange behaviour in Emma. It turns out that Emma as multiple volumes within which the chapter counter reset to I. Consequently I was picking up chapter titles (I, II, III, VI, V, etc.) as novel terms the first go round and this was driving the visualization. I've since modified the algorithm to firstly ignore text blocks (paragraphs) with fewer than 5 words and secondly, given it a more dynamic colour scheme.

This improvement still highlights some key differences (again, in as much as the algorithm is correct). However, these differences are now somewhat changed from the first set of observations. Note also that the threshold for novelty has been decreased to 0.1.

Verbs: The Final Frontier

Named entity recognition – the discovery in text of strings that refer to classes of things like places, people, companies, etc. – has become a standard tool in the text mining world. For many classes (e.g. people) recognition is pretty good. But what are these entities doing? Figuring out the verb associated with a named entity, and the type of association (or the role) is trickier than you might think.

I’ve been playing a little with Thomson Reuters’ OpenCalais – their publicly available service built on top of the ClearForest technology acquired in 2007. To illustrate the challenges that the verb poses, take this sentence:

The Director of National Intelligence, Mike McConnell, has meanwhile implied he suspects the Pakistan-based group Lashkar-e-Toiba was responsible.

Verbs here are has, imply and suspect. OpenCalais analyses this as follows:

First, it works with the named entities, discovering

The Director of National Intelligence, Mike McConnell, has meanwhile implied he suspects the Pakistan-based group Lashkar-e-Toiba was responsible

Not too bad. Note that they are smart enough to pick up the use of ‘he’. In addition, interaction with the tool reveals that they resolve the pronoun to the preceding  entity. Now, with the propositional analysis, things start getting tricky. Rather than recognizing that McConnell implied that he suspected [the …], the system looks for the verb (imply) and then takes the preceding and succeeding noun phrases as the grammatical subject and object, resulting in The Director of National Intelligence implying Mike McConnell.

This example illustrates some of the challenges in the space. It is costly and difficult to do a full parse (which in this case would have discovered the subordinate sentence, and discourse analysis (figuring out what pronouns refer to) is complex. Finally, the lexical resources required to guide the association of the noun phrases that go with verbs is a large investment.

While, to a large extent, one can get away with light weight solutions to named entities that treat the text as linear. Getting the verbs done correctly requires a richer understanding of the structure. You can no longer treat the text sequentially (or, to be fair, you have to at least approximate the grammatical, tree-like structure of the language).

To some degree, certain text mining applications can help here, if designed appropriately. While the above example surfaces a clear error in analysis, if the application relies on the aggregates of this type of analysis, and if in aggregate you tend to get your facts right, then the user can win. However, if at any stage, one has to reveal a specific document, the user is likely to be exposed to errors.

The Edited Palin: Part 2

Earlier this month, I wrote (somewhat tongue in cheek) about the way in which journalists are apt to quote literally what people of note are saying – complete with all the slips and repairs. Well, I’m currently re-reading Steven Pinker’s The Language Instinct (I had originally skimmed it thinking it to be a sort of psycholinguistics text book – how wrong I was!) and found an interesting piece in chapter 7 (Talking Heads) which starts with the reaction to the release of the transcripts of Nixon’s meetings in the White House:

Not everyone was shocked by the unintelligibility of transcribed speech. Journalists know all about it, and it is a routine practice to edit quotations and interviews heavily before they are published…

Journalists’ editing of conversations became a legal issue in 1983, when the writer Janet Malcolm published an unflattering New Yorker series about the psychoanalyst Jeffery Masson.

…

In a closely watched opinion, the majority [of Supreme Court Judges] defined a middle ground for journalists’ treatment of quotations. (Requiring them to publish quotes verbatim was not even considered.)…”deliberate alteration of the words uttered by a plaintiff does not equate with knowledge of falsity.”

Note – if you are ever reading a book on linguistics and think that the examples of branching are silly (the man by the lake on the hill with the ..), take a look at your own writing. In the first para above, I wrote (without thinking about it):

the reaction to the release of the transcripts of Nixon’s meetings in the White House

In summary – subjects can be skewered by literal quotations and certainly things have changed since Pinker wrote his book. I was thinking of delving into the New York Times’ archives to see if a trend could be measured. This seems like too much work, but here is the first quote I stumbled across, from May 26, 1912:

“Why, In Utah,” he said, “we could have stopped instructions for Taft but for a telegram sent by Senator Smoot at the last moment. As it is we will get four of the delegates when the time comes. I am positive of two and pretty  sure of the other two.”

The Trouble With Adverbs

I was reading a book to my daughter. A scene taking place in a school showed a poster on the wall with some carrots and the caption 'eat healthy.' To me (a speaker of British English), this seems incorrect. It makes me want to ask: 'eat healthy what?' - I'd rather say 'eat healthily.' I'm left with the following questions:

• Can adjectives be substituted for adverbs in American English?
  • probably not - I've not heard 'he quick ran to catch the bus.'
• Can adjectives be substituted for adverbs in American English in certain constructions?
  • this seems to be the key questions, a generalization of the next question.
• Are certain adjectives being used idiomatically in adverbially positions in American English?
  • this seems to be at least the case: 'how are you? I'm good (well).'
• Is this nothing to do with American English and universal to all flavours of the language?

I haven't yet found a study online regarding this (though see note at the end of this post). Neither have I come up with some neat way to use search engine queries and resulting counts to help clarify.

So why is this important? Well, one of the fundamental steps in analysing text is the determination of parts of speech (e.g. adjective and adverb). From this analysis, many other inferences can be made, inferences common to text mining and natural language processing applications in general.

Systems that determine part of speech tags are often trained using text more formal than that found in your average piece of social media text. Consequently, expressions like 'eat healthy' can cause problems. There are, of course, a number of straight forward pragmatic approaches to dealing with these problems - particularly when parser rules can be modified manually.

Note that in understanding and accepting change in language is fundamental to text mining. I enjoy reading the work of Patrick O'Brian in which people go around saying things like 'please to sit down' which sounds very odd nowadays.

Here is a fascinating article on this topic. This leads to this post on Linguist List on changes in English:

What about the almost complete U.S. substitution of adjectival for adverbial forms? This is evident locally here in Minnesota and in the speech of a number of national politicians, journalists and educators.

The post goes on:

There seems to be a growing aversion to using adverbs ending in <ly> at the end of sentences[.]

Which suggests this appears in certain constructions, not in all.

World Language Mapping

This excellent page, created by Dr. Stephen Hoffman, has maps representing various language families and their geographic distribution.

It is remarkable to note that the page has several blogosphere links (2 on both blogpulse and t'rati, 4 on google) and yet in the webosphere is shows 0 hits from Google and about 30 hits from Yahoo (for sites that link to the page). The Google hits don't include the blog search hits, but Yahoo does. I counted approx 10 blog hits in the 30 results.