Matthew Hurst over at Data Mining has an excellent post on sentiment scoring
His conclusion? This is not a case where people are better than machines or visa versa. It is a complex problem requiring problem specific methods and approaches.
I have to agree with Matthew – in that sentiment scoring is challenging for humans or machines, but done right, machines can successfully score sentiment. MotiveQuest uses high level tools that enable our strategists to build linguistic models for the particular category or question at hand. The model is used to score sentiment in the entire context of the message – not just particular words. This is hard to do well, and our approach is to build sophisticated, parameter driven tools that our strategists can customize to do a great job of automated sentiment scoring in a given context.
We have done side-by side tests with human scoring (Northwestern grad students) and found wide divergence with disagreement among human scorers on ~30% of posts, but when we consider the sentiment of a post to be based on the majority of human scores, we have seen a 96% agreement between automated and human scoring. This kind of side by side testing is required if you are going to rely on results.
Simple automated methods won’t work, and a purely human approach cannot handle the throughput so I propose a new approach – Hybrid Sentiment Scoring.
Discuss . . .