Semantic analysis and Feedback Management

The use of semantic analysis is increasingly being exploited in customer satisfaction surveys. However, behind the fantasy represented by artificial intelligence, machine learning and other Big Data, its use often generates frustration and disappointment. Here's why.

We'll also look at what you can really expect from semantic analysis as part of a Feedback Management program.

Usefulness inversely proportional to the number of closed questions

The fewer closed-ended questions a questionnaire contains, the greater the usefulness of the open-ended question (and therefore of semantic analysis). In fact, the open-ended question generally serves to cover aspects not addressed by the closed-ended question.

For short (or low value-added) customer experiences, the number of questions can hardly exceed 10, or even 5 in some cases. At this point, the open-ended question and the associated semantic analysis undoubtedly come into their own.

In addition, open-ended questions allow customers to express themselves on any subject. They can return to a subject covered by a closed question (thus highlighting the importance of the concept expressed), or express themselves on a "warning" concept (e.g. threat of legal action, intention to go to the competition, etc.); concepts which, by their very nature, are not covered by closed questions.

TreeMap and word clouds: a short-term wow effect

Reading a TreeMap or Word Cloud report has its effect... the first time you read it. Even if it doesn't come as a surprise, it's when you use it that you realize its lack of operational utility. From one month to the next, the most frequently cited positive or negative concepts remain the same. And it's very difficult to make these elements evolve as a result of corrective actions taken internally.

Why is this? Because of the granularity of the semantic analysis. This is also true of Likert-type questions (Very satisfied, Satisfied, Not very satisfied, Not at all satisfied), where percentage changes are already not very noticeable. Despite technological advances, it's hard to imagine a semantic analysis with more than 3 response modalities: positive, negative and neutral. And it's probably not just a technological problem. When a customer says "The cleanliness of the accommodation was very average", this is clearly a negative tone. But can we objectively go beyond that and assign a score from 1 to 10?

Alert detection: useful and operational

In our experience, semantics really serves a Feedback Management project when used in the context of alert detection. A customer can thus be identified as an alert (and require contact from the organization) if he expresses a so-called alert concept. Even if the expression of such a concept is reflected in the overall satisfaction and recommendation score, some customers will fly under the radar.

Example: Let's imagine that an organization defines its alert threshold when the recommendation score is below 4 out of 10. It is possible (and, according to our calculations, this happens frequently) that a customer may give a rating of 5, but clearly state in his comments that he never intends to come again. They may also say that they are going to complain to "60 millions de consommateurs".

Semantic analysis is thus the only way to identify alert customers who would not otherwise be dissatisfied enough to trigger a more standard alert.

Easy-to-read comments

We say it again and again to all our customers: read the reviews! It's pure, rich information, a mix of practical information and feelings. However, it's understandable that everyone only wants to read the comments relevant to their own area. For example, an Operations Manager has to read all the comments received. But the person responsible for cleanliness may only want to focus on comments that talk about cleanliness. And when an operational unit receives several hundred verbatims per month, semantic analysis can sort out the comments that are relevant to the reader.

Conclusion

Semantic analysis is a fantastic and very useful tool in a Feedback Management program. But from our point of view, its operational usefulness is far more important than its statistical usefulness. And the way in which this tool is to be used must be clearly defined upstream, to avoid disappointment.

Finally, don't forget that it's all the more useful if :