Hybrid Sentiment Scoring

Matthew Hurst over at Data Mining has an excellent post on sentiment scoring

Sentiment Mining: The Truth

His conclusion? This is not a case where people are better than machines or visa versa. It is a complex problem requiring problem specific methods and approaches.

(His post was triggered by Nathan Gilliat’s Friday roundup of social media news including KD Paine’s announced partnership with BuzzLogic.)

I have to agree with Matthew – in that sentiment scoring is challenging for humans or machines, but done right, machines can successfully score sentiment. MotiveQuest uses high level tools that enable our strategists to build linguistic models for the particular category or question at hand. The model is used to score sentiment in the entire context of the message – not just particular words. This is hard to do well, and our approach is to build sophisticated, parameter driven tools that our strategists can customize to do a great job of automated sentiment scoring in a given context.

We have done side-by side tests with human scoring (Northwestern grad students) and found wide divergence with disagreement among human scorers on ~30% of posts, but when we consider the sentiment of a post to be based on the majority of human scores, we have seen a 96% agreement between automated and human scoring. This kind of side by side testing is required if you are going to rely on results.

Simple automated methods won’t work, and a purely human approach cannot handle the throughput so I propose a new approach – Hybrid Sentiment Scoring.

Discuss . . .




One Response to Hybrid Sentiment Scoring

  1. Tom, I totally agree that the properly choreographed integration of humans and automation holds the key. Hybrid Sentiment Scoring is a good term for this. It’s also what we’re doing with our new service. Human level accuracy at speed and cost that’s much closer to an algorithmic process. (check out sentimine.com)

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s

%d bloggers like this: