My Photo

« TED: Hans Rosling | Main | Attention on Hold »

July 06, 2007


Nathaniel Faery

I don't agree - i see a random inter-labeler error as rebalancing bias in the classification. A sysematic error retains that bias towards incorrect results. Systematic error is fine if we know the nature of the resultant bias - but randomness is better if we don't know it. I don't know enough maths to prove this - perhaps someone can?

Matthew Hurst

Consider the case where a client wants some sentiment data over time. In the first period, labeler A reports a certain result. In the second period, labeler B reports a result. Due to the difference in labeling, there is no real continuity between these periods, and any observation that sentiment is trending up or down is not really reliable.

Peter Turney

This reminds me of another story. I was playing with a French version of SO-PMI (Turney and Littman, 2003), which rates a French word as positive (praising) or negative (criticizing) by measuring its co-occurrence with seven positive French paradigm words and seven negative French paradigm words. I discovered that the word "fort" (the masculine form of "strong") is highly positive, whereas the word "forte" (the feminine form of "strong") is highly negative. Could be just random noise, I suppose, but it seemed revealing to me.

Turney, P.D., and Littman, M.L. (2003), Measuring praise and criticism: Inference of semantic orientation from association, ACM Transactions on Information Systems (TOIS), 21 (4), 315-346.

William Cohen

Thanks for the link, Matt - but it's not just a content problem, it's also a problem of understanding the user's information needs, and deciding what is appropriate to present. Even if I perfectly understand the content of every web page, it's still not obvious what the right answer to the query should be. For the query "jew" do you present the most authoritative source on Judaism, as judged by the world as a whole? or the most most authoritative source on Judaism, as judged by the world of anti-semites?

BTW this isn't an unique case. Try Googling "Hellary Clinton" and note the polarity of the top ranked documents.


I think it raises an interesing question. For instance, I work at Fair Isaac and we get lots of criticism of credit scoring. Yet credit scores replaced human judgment and meant that good credit was determined by your behavior, not how well you knew your bank manager or what color your skin was. People say they want humans to make judgments and not machines but I think that's more an emotional reaction than anything. Automated decisions can offer a lot of consistency and precision but we should not underestimate the need to get people to understand and support the value as their gut reaction is to dislike "machine-made" decisions.
Author, with Neil Raden, of "Smart (Enough) Systems" a book on this topic.

The comments to this entry are closed.

Twitter Updates

    follow me on Twitter

    March 2016

    Sun Mon Tue Wed Thu Fri Sat
        1 2 3 4 5
    6 7 8 9 10 11 12
    13 14 15 16 17 18 19
    20 21 22 23 24 25 26
    27 28 29 30 31    


    Blog powered by Typepad