Diva Marketing Blog Interview with Bill Neal

The Diva Marketing Blog has an interview with Bill Neal - a marketing research guru. There are a number of problems with Mr Neal's responses regarding the value of social media anlaysis as well as the sophistication of the technologies involved.

[T]hose out there in the Internet world who are generating their own media are self-motivated to do so and are not representative of any defined population of buyers. And, given the fact that they have taken a public position on a particular product or service, it means that they more often than not have exceptional or non-typical attitudes about those products and services. The information they generate may be true, or not true – there is no way to discern which. Therefore, the information generated by those folks is neither credible nor reliable. So, as researchers, yes, we should be listening, but we must be very cautious and skeptical about its veracity and its usefulness.

One of the fundamental problems with the term Consumer Generated Media, a term created by those firmly on the marketing/advertising side of the fence, is that it forces a view of the data (blogs, boards, etc.) with the assumption that an individual acting as a consumer of a product is creating the media. One can see a trace of this in the above comment: the fact that they have taken a public position on a particular product or service. If I state that I ate a pizoid pizza while watching the Superbowl, what position have I taken about that product? The notion that consumers are writing this stuff specifically to take a position wrt a product is simply wrong. In addition, it misses the aggregate value of a million LiveJournal users mentioning pizoid pizza in passing.

The information they generate may be true, or not true – there is no way to discern which. This statement seems to suggest that other forms of inquiry are more reliable. I don't follow the logic that something I write in a journal would be less true than something I say in response to a market researchers question.

As someone on the outside of this emerging space, Mr Neal makes the following statement:

Let me give you an example. I own two Ford trucks – a 1997 Expedition with close to 200,000 miles on it and a 2004 F-150 with 25,000 miles. In the blogosphere I’ve seen a bunch of postings on Ford trucks depreciating their quality and reliability. Yet, both of the vehicles I own have been exceptional in quality and reliability. I don’t take the time to post those positive experiences, but some who have had problems are very vocal about their supposedly negative experiences.

So what is the truth of the matter? Did these negative experiences really occur? Was it the fault of the manufacturer? Or the Dealer? Or the buyer? Are these generators of consumer media about Ford trucks really being the dispassionate arbiters of truth, or do they have an agenda? That’s the key issue – that information gleaned from the blogosphere is simply not reliable, and in many cases it is not valid.

By carrying out large scale aggregate analysis, we can identify those vehicles which do attract lots of positive mentions, and those which attract more negative mentions. The class of vehicles which tend to attract negative mentions are those which tend to be commodity vehicles (e.g. the Ford Escort). In other words, they are very good performers in their price range and do a reasonable job of getting from A to B. This class of vehicles over indexes on negative comments, but by modeling this expected behaviour we can determine when there is an outlier from the class for which the negative (or positive) signal is abnormal, thus gaining insight.

Finally, there is the following:

Diva:  How about your thoughts on the companies that are jumping on the research social media band wagon i.e., Nielson BuzzMetrics, Fortune Interactive?

Bill: Honestly, I don’t know a lot about them and have not used them in my consulting practice, nor do I address them in our forthcoming book.

But to the best of my understanding, they are primarily counting product/service mentions and, in some cases identifying the major sources of those mentions. The basis for their business model is the belief that consumers have a higher trust of consumer generated media then they have for company generated media.

I think that might be more of a reflection on the stupidity of much of the advertising and promotion that permeates today’s traditional media. I’ve already talked about the problems with simply counting the number of brand “hits” and how that can be so misleading.

And, as consumers mature in their understanding of how consumer generated media can be manipulated by those with less than honorable intentions, I think their trust in those sources of information may wane considerably.

As an expert in this space, Mr Neal owes it to his readers to be more informed in this area. The level of technological sophistication involved in the types of analysis we do at Nielsen require a more mature understanding before they are dismissed. In addition, the statement [t]he basis for their business model is the belief that consumers have a higher trust of consumer generated media then they have for company generated media is incorrect. The trust issue is only a small part of the story.

 

Comments

[Max Kalehof]

Matt,

Thanks for properly calling Bill Neal out here. I agree with everything you say, except:

"One of the fundamental problems with the term Consumer Generated Media, a term created by those firmly on the marketing/advertising side of the fence, is that it forces a view of the data (blogs, boards, etc.) with the assumption that an individual acting as a consumer of a product is creating the media. One can see a trace of this in the above comment: the fact that they have taken a public position on a particular product or service. If I state that I ate a pizoid pizza while watching the Superbowl, what position have I taken about that product? The notion that consumers are writing this stuff specifically to take a position wrt a product is simply wrong. In addition, it misses the aggregate value of a million LiveJournal users mentioning pizoid pizza in passing."

Sometimes that is the case (a position), but moreso, the fact that the content exists is what matters -- regardless of who created it, a consumer wiht a position, or just someone in passing. Regarding your pizza example, there may not be a passionate position implied, but that person surely implied that pizoid pizza is a part of his life, an indirect, passive endorsement at the least -- sort of like product placement in television. And that's a good point...aggregate value.

[Bill Neal]

Folks,

Please permit me to briefly respond.

I would just like to make a few additional observations:

1. Science, and the scientific method are the foundations of all acceptable marketing research. Without the discipline of a scientific foundation, all data, and the information gleaned from that data, represent information that is unreliable – and I am using the classic scientific concept of reliability – the ability to replicate an experiment and get the same results within acceptable bounds.
2. At its very best, CGM research is purely descriptive, and purely qualitative. It is not predictive and cannot ever meet the conditions for predictive research unless there is a well-defined population and a well-validated sampling frame from which that population can be sampled and tested. I strongly suspect that if someone tried to define the sampling frame for current CGM research efforts, they would see the immediate problems.
3. Therefore, one cannot (should not) ever consider using the results of CGM research in its present state as a basis for market segmentation, product/service positioning, product/service optimization, brand value analysis, or consumer/customer acceptability testing. These research-based protocols are the foundation of marketing strategy and tactics.
4. So what is left? A piece of qualitative, descriptive “research” based on a sampling from an incomplete and undefined population. There is no “structure” to the inquiry, so any respondent can say anything they want, and it is up to the researcher to interpret whatever those words mean. It’s akin to asking people to take a survey, but each can make up their own. There is absolutely no control of what is asked and what is responded to.
5. The bottom line is that CGM research cannot be relied upon for much of anything. So what if it includes some “super consumers”, “early adapters”, and “extreme loyalists and detractors?” There is no telling which are which. It may just as easily include those who have never purchased or used the product/service, and simply want to express some opinion based on some unknown motivations.

So, pardon me. Yes, I may be a traditionalist, and not hip to the new world of alternative media. But, science is science, and there is no science here.

[Max Kalehof]

Again, my critique of Bill's rigid framework:

1. CGM Is Not Representative

“In many ways it [CGM] combines the worst elements of non-scientific research - self selection and advocacy - both positive and negative. That is, those out there in the Internet world who are generating their own media are self-motivated to do so and are not representative of any defined population of buyers. And, given the fact that they have taken a public position on a particular product or service, it means that they more often than not have exceptional or non-typical attitudes about those products and services…”

CGM is biased by self-selection and advocacy, and that’s precisely why it’s so valuable! CGM creators are, by definition, a highly defined population of hand-raisers–the ones so engaged with a category or product that they talk about it without prompting. These are the most elusive individuals, though highly valuable. They are your super consumers, your early adopters, your extreme loyalists and detractors, who have disproportionate influence on broader discussion and ideation. The argument that CGM is weak because it happens to be effective at identifying powerful stakeholders outside of the average is without merit. The problem with marketing today is that it too often constricts itself to the average, where findings are diluted, insightful nuances ignored and early indicators forgotten.

2. CGM Is Neither Credible Nor Reliable

“The information they [consumers] generate may be true, or not true - there is no way to discern which. Therefore, the information generated by those folks is neither credible nor reliable.”

Readers, participants and researchers in social media determine CGM credibility, reliability and influence according to a huge range of factors. For example: Am I familiar with the person or his affiliation? Is that person known to be trustworthy? How often does that person speak, and how often do others in the community respond? How often do others seek out someone precisely because that person is perceived as credible? How often is that person linked or referred? Are there norms and benchmarks, and historical indicators of trust?

Moreover, truthfulness often is irrelevant. Why? Because CGM acts just like media! It’s become one of the most prolific sources of online content. It’s attractive to search engines because of its robust text and numerous links, thereby enabling passionate information seekers and speakers to find one another. CGM tends to be compelling, refreshing and believable, and it is successfully competing for attention against all the other messages marketers continue to throw against the wall. Perceived credibility is why e-commerce sales increase when products are juxtaposed with consumer reviews.

3. CGM Includes Viral Discussion Which Marketers May Not Like

“But marketers must keep in mind that a few influencers can generate a great number of product mentions if they decide to feature a particular product or service in their blogs. And these things can get out of hand very quickly, signaling a problem that’s really not a problem to the vast majority of customers.”

What matters is not whether things get out of hand, but that there are influential consumers creating CGM, who spark ongoing, viral dialogue around products or services. This is CGM acting not only as media, but as influential news media.

4. I Don’t Know A Lot About CGM Research Firms, But I Do Know The Problems

“I don’t know a lot about them [CGM research firms] and have not used them in my consulting practice…But to the best of my understanding, they are primarily counting product/service mentions and, in some cases identifying the major sources of those mentions…I’ve already talked about the problems with simply counting the number of brand ‘hits’ and how that can be so misleading.”

CGM research firms are analyzing billions of public discussions to deliver a wide range of quantitative and qualitative research applicable in nearly every marketing situation. These measurements go far beyond product mentions and brand hits, and do dive deep into sophisticated consumer segmentation and insights, social influence mapping, and media measurements.

Moreover, Michael Cornfield, Ph.D., Vice President of Public Affairs, ElectionMall.com and Adjunct Professor in Political Management, The George Washington University, sent me an email and said:

On point #1, researchers like us aren't looking for representative samples, we're looking for ideas, individuals, memes, and diffusion patterns. And on point #2, he stresses how important it is to discover uncredible and unverified information circulating in CGM circles just so that information can be corrected in a timely manner.

Yes...these last two points are notions that could kill a brand. Still, what kills me most is Bill's acknwoledgment that he "[doesn't] know a lot about them [CGM research firms]" and has "not used them in [his] consulting practice." Fortunately, he agreed to speak with me to learn more about what we actually do. I just wish he'd be open to learning before slamming. Just like being open to new people or new food, you should try it before you dismiss it -- because you may find it very valuable. But no matter what you call it -- research or whatever -- the bottom line is that brand managers, who are the core customers of market research, are finding CGM research incredibly valuable and complementary to the traditional research and insights approaches otherwise contained in "science."