In a recent blog post called The Problem with Automated Sentiment Analysis from Fresh Networks, a social media agency, they evaluated a few sentiment tools and their results are quite similar to what we’ve found in a number of our own experiments:
- About 80% of posts are neither positive nor negative.
- Sentiment tools “accuracy” of 70% to 80% is largely driven by their ability to correctly label neutral posts.
- “In our tests when comparing with a human analyst, the tools were typically about 30% accurate at deciding if a statement was positive or negative”
From the blog comments, it’s clear that the companies in this space are doing their best to obfuscate the truth. To some’s credit, they do state that sentiment alone is not enough information to derive any conclusions.
However it’s NOT better than nothing, it’s actually worse than doing nothing because you are getting INCORRECT information.
With sentiment there is no such thing as accuracy, there is only agreement. The technology can’t become more accurate, it can only agree with people more often. And, “sentiment” does not mean the same thing to all people in all situations. You can’t get more “accurate” at “sentiment” because what you are actually talking about is trying to solve hundreds or thousands of slightly different problems with one tool. Until we can map the human brain into a program or electronic circuits, I just don’t think that is going to happen.
I completely believe that having inaccurate sentiment is worse than having nothing. Here is a good example. In posts about “Blackberry” that have been classified 3 different times by hand, about 32% of posts are positive (with a majority vote). When we take that same data set and have each post classified 10 times, now about 10% of posts are positive (with a majority vote). And, if we only consider the posts we are confident in, only about 3% of posts are positive.
So, which is it: do 30% of people like “Blackberry” or do 3% of people, because that’s a BIG difference. Of course, the answer is probably neither because we aren’t actually measuring how many people like “Blackberry”. Unfortunately, that’s how it can be interpreted. Hence, bad information can be worse than no information.
Marketers need to be ware that a lot of these companies say they do monitoring and provide analytics like sentiment but in reality they are really keyword-focused listening platforms with limited analysis capability. If you really want to go beyond sentiment analysis you need to use semantic analysis. With semantic analysis marketers can better understand the conversations about their brand or product category– here is a white paper that compares Semantic vs Sentiment Analysis and can help you make a more informed decision about when and how to use Sentiment.