Showing posts with label market research. Show all posts
Showing posts with label market research. Show all posts

Thursday, 9 October 2014

Sentiment And Semantic Analysis

                                              
Michalis founded DigitalMR in 2010 following a corporate career in market research with Synovate and MEMRB since 1991. This post was first published on the DigitalMR blog. Explore the blog here: www.digital-mr.com/blog

It took a bit longer than anticipated to write Part 3 of a series of posts about the content proliferation around social media research and social media marketing. In the previous two parts, we talked about Enterprise Feedback Management (December 2013) and Short -event-driven- Intercept Surveys (February 2014). This post is about sentiment and semantic analysis: two interrelated terms in the “race” to reach the highest sentiment accuracy that a social media monitoring tool can achieve. From where we sit, this seems to be a race that DigitalMR is running on its own, competing against its best score.
 
The best academic institution in this field, Stanford University, announced a few months ago that they had reached 80% sentiment accuracy; they since elevated it to 85% but this has only been achieved in the English language, based on comments for one vertical, namely movies -a rather straight-forward case of: “I liked the movie” or “I did not like it and here is why…”. Not to say that there will not be people sitting on the fence with their opinion about a movie, but even neutral comments in this case, will have less ambiguity than other product categories or subjects. The DigitalMR team of data scientists has been consistently achieving over 85% sentiment accuracy in multiple languages and multiple product categories since September 2013; this is when a few brilliant scientists (engineers and psychologists mainly) cracked the code of multilingual sentiment accuracy!
Let’s dive into sentiment and semantics in order to have a closer look on why these two types of analysis are important and useful for next-generation market research.
 
Sentiment Analysis
 
The sentiment accuracy from most automated social media monitoring tools (we know of about 300 of them) is lower than 60%. This means that if you take 100 posts that are supposed to be positive about a brand, only 60 of them will actually be positive; the rest will be neutral, negative or irrelevant. This is almost like the flip of a coin, so why do companies subscribe to SaaS tools with such unacceptable data quality? Does anyone know? The caveat around sentiment accuracy is that the maximum achievable accuracy using an automated method is not 100% but rather 90% or even less. This is so, because when humans are asked to annotate sentiment to a number of comments, they will not agree at least 1 in 10 times. DigitalMR has achieved 91% in the German language but the accuracy was established by 3 specific DigitalMR curators. If we were to have 3 different people curate the comments we may come up with a different accuracy; sarcasm -and in more general ambiguity- is the main reason for this disagreement. Some studies (such as the one mentioned in the paper Semi-Supervised Recognition of Sarcastic Sentences in Online Product Reviews) of large numbers of tweets, have shown that less than 5% of the total number of tweets reviewed were sarcastic. The question is: does it make sense to solve the problem of sarcasm in machine learning-based sentiment analysis? We think it does and we find it exciting that no-one else has solved it yet.
Automated sentiment analysis allows us to create structure around large amounts of unstructured data without having to read each document or post one by one. We can analyse sentiment by: brand, topic, sub-topic, attribute, topic within brands and so on; this is when social analytics becomes a very useful source of insights for brand performance. The WWW is the largest focus group in the world and it is always on. We just need a good way to turn qualitative information into robust contextualised quantitative information.
 
Semantic Analysis
 
Some describe semantic analysis as “keyword analysis” which could also be referred to as “topic analysis”, and as described in the previous paragraph, we can even drill down to report on sub-topics and attributes.
 
Semantics is the study of meaning and understanding language. As researchers we need to provide context that goes along with the sentiment because without the right context the intended meaning can easily be misunderstood. Ambiguity makes this type of analytics difficult, for example, when we say “apple”, do we mean the brand or the fruit? When we say “mine”, do we mean the possessive proposition, the explosive device, or the place from which we extract useful raw materials?
Semantic analysis can help:
  • extract relevant and useful information from large bodies of unstructured data i.e. text.
  • find an answer to a question without having to ask anyone!
  • discover the meaning of colloquial speech in online posts and
  • uncover specific meanings to words used in foreign languages mixed with our own
What does high accuracy sentiment and semantic analysis of social media listening posts mean for market research? It means that a 50 billion US$ industry can finally divert some of the spending- from asking questions to a sample, using long and boring questionnaires- to listening to unsolicited opinions of the whole universe (census data) of their product category’s users.
 
This is big data analytics at its best and once there is confidence that sentiment and semantics are accurate, the sky is the limit for social analytics. Think about detection and scoring of specific emotions and not just varying degrees of sentiment; think, automated relevance ranking of posts in order to allocate them in vertical reports correctly; think, rating purchase intent and thus identifying hot leads. After all, accuracy was the only reason why Google beat Yahoo and became the most used search engine in the world. 

Thursday, 25 September 2014

Thinking, Fast and Slow: The Social Media Research Perspective

Dr. Nicos Rossides is the CEO of Medochemie, an international pharmaceutical company with more than 100 operations worldwide. Nicos is also the Chairman of DigitalMR's Advisory Board. This post originally appeared on the Digital MR blog www.digital-mr.com/blog/

Nobel laureate Kahneman has written a seminal book on the different types of thinking processes we humans deploy. In “Thinking, Fast and Slow” he argues that cognitive biases profoundly affect our daily decisions - from which toothpaste to buy, to where we should go on holiday. He goes on to claim that our decision processes can be understood only by knowing how two different thinking systems shape the way we judge and decide:

"System 1" is fast, instinctive, subconscious and emotional;
 
"System 2" is slower, deliberative, logical.
 
The book delineates cognitive biases, such as how we frame choices, loss aversion, and our tendency to think that future probabilities are altered by past events..  All these can throw light on fascinating facets of human judgement and thought and are posited by Kahneman to be both systematic and predictable.
 
Framing: Drawing different conclusions from the same information, depending on how or by whom that information is presented.
Loss aversion:  The disutility of giving up an object is greater than the utility associated with acquiring it.
Gambler’s fallacy: The tendency to think that future probabilities are altered by past events, when in reality they are unchanged.
 
How is this relevant to research? And why exactly should research agencies and their clients care? Well, I would argue that the basic dichotomy described in the book is critical to the existence of the market research industry. Our ability to generate insights, which in turn can only be gained through the analysis and interpretation of evidence, is key to managing a modern business – it is “system 2” thinking. Of course, one could get some things right by merely relying on intuition or gut-feeling; therein lies the caveat: get some things right. Indeed, chances are that the odds would be heavily stacked against you if you ignore facts and rely on less than rigorous or no analysis. Putting this in a different way, fast thinking is not a good way to raise your metaphorical batting average as a business.
 
One could certainly argue (correctly) that intuition does not occur in a vacuum – in that it often has its roots in prior experience. But testing your assumptions before going ahead with a decision is a way to avoid mistakes. Indeed, examining available evidence to inform decisions is a tried and tested way of succeeding in business. Ask Procter & Gamble  which spends hundreds of millions every year on painstakingly researching all aspects of the marketing mix.
 
We can draw a parallel to P&G’s B2C decision model  (based on “System 1” thinking) in which they established the “First moment of truth” which stated that there were 3-7 seconds from when the customer sees the stimulus to when they react (decide to buy). There was then a second decision (“Second moment of truth”) that customers made after the purchase; based on the negative or positive experience with the product, a decision would then be made as to whether they should continue using it/buy from this vendor again
 
Stimulus -> Shelf (First moment of truth) -> Self experience (Second moment of truth)
 
Google pointed out that the ubiquity of internet access has caused an upward trend in people (in a B2C and B2B context) after they had observed the stimulus, researching about the product/service to obtain more information before they made their first moment of truth decision; they call it the “Zero moment of truth” (ZMOT)
 
Stimulus -> Information (Zero moment of truth) -> Shelf (First moment of truth) -> Self experience (Second moment of truth)
 
Companies are now looking to the web as a solution; this should be done carefully as even using the web as a source of informing decision can lead to systematic biases (searching for what you want to see).
 
By analysing the zero moment of truth (sources from which customers are obtaining their information) and the first and second moments of truth of current/potential customers, (through what customers are saying online), DigitalMR serves to create an objective way to inform choices (the zero moment of truth) of decision makers of a company through tools such as social media listening, online communities and an array of other digital research tools.
 DigitalMR & ZMOT
So, fast thinking is all nice and good, deeply steeped in our evolutionary past, but when it comes to business, “system 2” slow thinking based on informed choices is the way to go, especially when dealing with big ticket decisions.
 
What is your view on systems 1 and 2 thinking? How many of your decisions are rooted in system 1 Vs system 2 Vs both? Please share your way of being part of the conversation during the zero moment of truth.