Customer Satisfaction Recognition Based on Facial Expression and Machine Learning Techniques

A R T I C L E I N F O A B S T R A C T Article history: Received: .14 June, 2020 Accepted: 21 July, 2020 Online: 19 August, 2020 Nowadays, Customer satisfaction is important for businesses and organizations. Manual methods exist, namely surveys and distributing questionnaires to customers. However, marketers and businesses are looking for quick ways to get effective and efficient feedback results for their potential customers. In this paper, we propose a new method for facial emotion detection to recognize customer’s satisfaction using machine-learning techniques. We used a facial landmark point; we extract geometric features from customer’s emotional faces using distances from landmarks points. Indeed, we used distances between the neutral side and the negative or positive feedback. After that, we classified these distances by using different classifier, namely Support Vector Machine (SVM), KNN, Random Forest, Adaboost, and Decision Tree. To assess our method, we verified our algorithm by using JAFFE datasets. The proposed method reveals 98,66% as accuracy for the most performance SVM classifier.


Introduction
This paper is an extension of work originally presented in world conference on complex systems (WCCS '19) [1]. We decide to develop this work because, we observe in the last decade, business primarily has an important impact based on the way, how customers observe their products and services [2]. Customer satisfaction can be measured using manuals methods such as the satisfaction survey, the interview, and the focus group. These methods are not efficient and effective to the cost, time and reliability of the data. Facial expressions are used to communicate non-verbally. They are a special way to express our emotions and appreciations. In the context of customer satisfaction, a negative feedback emotion is often related to a lower perceived quality of service. The facial expression contributes 55% to communicate with a speech. More than that, 70-95% of negative feedback can be understood verbally [3]. Companies have long been interested in understanding the purchasing decision-making process of consumers [4]. In this work, we aim at detecting the positive or negative emotions customers from the analysis of facial expressions. In this way, this type of information is useful. For example, we can calculate the statistics about the products as well as the replacement of their exposures. Appreciated products must be brought to the fore, while negatively appreciated products must be replaced or otherwise requested. For this reason, we are so existed to propose a new method for facial emotion detection to recognize customers' satisfaction using facial expression and machine learning techniques.
The remainder of this paper is structured as follows: Firstly, we offer an outline of the related work. Secondly, we introduce our approach that contains a system for customer satisfaction using facial expression and machine learning techniques. We reserved a separate section to the experimental results. In the last section, we sum up our work.

Related work
Currently in the field of marketing and advertising, facial recognition is used to study consumers' emotions in two forms: positive and negative emotions. Charles Darwin was the first to provide robust basis for emotions, representing their significance, their usefulness and communication. Already, many of facial expressions of emotions have an adaptive significance that serve to communicate something, most emotions are expressed alike on the human face nerveless of culture or race [5], his work "Expression of emotions in humans and animals", remains a ASTESJ ISSN: 2415-6698 reference for many scientists. In [6], authors proposed method to classify muscle movement to code the facial expression. The facial movements are determined through action units (AU). Each AU is based on the affinity of the muscles. This method, used mostly in classifying facial movement named Facial Action Coding System (FACS).We can define six universal basic emotions: happy, sadness, anger, surprise, fear and disgust. Various researchers have supported the universality of the expression of emotions. Generally, in reaction to similar stimuli, people prompt similar expressions with local variations [7]. Frequently, we used two methods: motion-based method and deformation of face. The change of face is taken into account for the motion-based method [8,9]. On the contrary, we take into account a neutral image and another image for the deformation-based method [10,11]. In [10], authors use FACS (Face Action Code System) as features to classify emotions. Their model consists to count images, which classified correctly and weighted by the system. They found that, 2% of images (6 image) are failed completely in tracking. They also reported 91% as average correct recognition ratio. Otherwise, in [11] a model which able to use also speech content is developed. They create a system to compute the Human Computer Interaction (HCI). In order to test their model, they utilize 38 subjects of affect recognition approach with 11 HCI-related affect states. Their model a gave 90% for the bimodal fusion as an average recognition accuracy.
In [12] feature extraction contains geometric and appearancebased models. The first model involves feature extraction about eye, mouth, nose, eyebrow…However; the second model covers the specific part of the face. They also evaluate their model by employing SVM, ANN, KNN, HMM, Bayesian network, and sparse representation-based classification. The experiment results report that, sparse representation-based classification is the more performant classifier for their model.
Deep learning algorithms are also used to extract pattern from facial expressions. More specifically, we develop a convolutional neural network to detect features from faces, and to classify these features to different emotions. This type of network contains two parts. The first part contains convolutional layers, which are based on applying mathematic convolution operation. The result of this part is features extracted from faces. The second part contains feed-forward network. This part of network is able to classify features extracted to different facial expression. In [13,14] the based results researches of this algorithm is proposed. For example, authors, in [13], apply KNN to classify emotions. After that, they find 76.7442% and 80.303% as accuracy in the JAFFE and CK+, respectively, which illustrate viability and success of their model. In [14], authors employ also Decision Tree and Multi-Layer Perceptron to classify emotions. However, they find that, Convolutional Neural Network report the greatest recognition accuracy. An Affdex by Affectiva [15] is one of the most implements for visual emotional analysis. Which permits giving the emotional trend of a subject, through detection emotions [16], and the Microsoft Cognitive Services based on the Azure platform. These two studies are able to perceive age, sex and ethnicity and depend on Convolutional and/or Recurrent Neural Network [17]. In addition, they find that, the proposed model is yields performance increases for facial expression recognition using CK+ dataset. A multimodal affect recognition system was developed to determine if the customer exhibits negative affect such as being unhappy, disgusted, frustrated, angry or positive affect such as happy, satisfied and content with the product being offered [18]. Furthermore, authors apply Support Vector Machine an emotion template to evaluate their algorithm. They test the real-time performance, which evaluate the feedback assessment in order to compute accuracy and viability for multimodal recognition. To appreciate customer satisfaction special facial emotion namely" happy"," surprised" and "neutral" is proposed in [19]. They evaluate their model by using datasets as Radboud Faces and Cohn-Kanade (CK+). Output results report that their proposed algorithm a gave an important recognition accuracy utilizing Action Unit (AU) features, Support Vector Machine (SVM) and K-Nearest Neighbors (KNN). In the last years, eye-tracking system has been developed to analyze the research of customer [20]. Companies invest intensely within the advancement and publicizing of products, the return on investment (ROI) must be justified by the organization. Hence, the specific observing of the fascination of a product by the customer amid its promotion, and how to develop their promotion campaigns. Eye tracking is very promising in navigation advertising. In [21], authors developed an algorithm aims to compute sales assistant, which depends specially on emotion detection. Otherwise, [22] present a study, which contains products features impact the costumer's emotional. Furthermore, they used 21 emotional categories of products.

Our Approach
In this section, we present an algorithm to recognize the emotions of customers towards a given product. In order to analyze the facial expressions of a client to derive satisfaction from the product offered, our algorithm is based on three essential steps. The first is to acquire the image of the face and the second allows extracting the expressions in the form of geometric features. To obtain these features, our algorithm transforms the input image into geometric primitives such as points and curves. These primitives are located via distinctive patterns such as the eyes, mouth, nose and chin by measuring their relative position, their width and other parameters such as the distance between the nose, eyes, mouth. Our contribution is therefore to propose a method for selecting the optimal distances that can make the difference between facial expression linked to emotion and customer satisfaction. In the last step, these different distances are used to classify a client's emotions towards a product, into three types; positive, negative or neutral appreciation. These results will be analyzed and provided to the company to understand how customers perceive the product.
Otherwise, the pipeline of our algorithm begin by detecting the face into an input image. After that, we compute the key point of the face by applying a mask, in order to compute the distances between key points. We aim to measure the variation of face's shape in order to classify appreciation of customer's toward a product. For this raison, our algorithm is based on computing face's key points to calculate distances between those landmarks. Finally, we employ a set of classifier to evaluate our algorithm

Features extraction (Geometric features)
In order to represent the emotions of a face, this step consists in locating the facial landmarks (for example the mouth, the eyes, the eyebrows, the nose and the chin). The model presented in figure (1) allows us to locate the face marks. This model has a drawback; the deformation of these marks modifies the facial expressions.

Training and classification
Using the most used algorithms in machine learning, namely: SVM, RANDOM FOREST, Decision Tree KNN, and Adaboost classifiers, the geometric features presented in the previous sections will be classified into three classes according to customer's satisfaction.
• SUPPORT VECTOR MACHINE (SVM): SVM is a discriminative algorithm, which used to develop a supervised model. It used to classify or to predict regression [23]. In this algorithm, we aim to find a separator, which can classify neural, positive, and negative emotion class. This separator is baptized hyperplane. To find a very performant hyperplane, we tested kernel methods. We found that RBF Kernel is the best kernel for our model. We utilized xi, as a vector support, and σ, as positive float, which is, specify a priori, to compute RBF Kernel. Alternatively, RBF ( , ) = ( −‖ − ‖ 2 2 2 ). We used also S as a set of vectors support, which contains xi,b as bias, and Lagrangian coefficient αi, is a vector support, to compute hyperplane equation: (1) • RANDOM FOREST: Random forest [24] is an algorithm constructed by combination between trees. This algorithm used decision trees concept to classify. In decision Tree, we calculate Gini index or Gain ratio to construct the tree. In Random Forest, we compute just Gini index to find the most popular attributes founded by Decision Trees [25]. To compute Gini index, we used the probability, which link between class k and selected case ( ⁄ ). Or, Gini index can written as follow: (2)

Experimental Results
In this work, the Jaffe dataset is used to test the performance of our method. The database contains 213 images of 7 facial expressions (6 basic facial expressions + 1 neutral) posed by 10 Japanese female models. 60 Japanese subjects have rated each image on 6 emotion adjectives. The database was planned and assembled by Michael Lyons, Miyuki Kamachi, and Jiro Gyoba. To use this dataset for our algorithm; we have classified its images into three classes. The first one contains images for satisfied persons, the second is for non-satisfied persons and the third class is for neural persons. Indeed, each face is represented in the form of a vector of 68 elements. Each element is the coordinate of a key point. All key points are calculated using a mask. These operations are applied to all Jaffe images. Next, the distances, shown by Table 1, are calculated. Therefore, the input of our classifiers SVM, RANDOM FOREST, KNN, decision tree and Adaboost classifiers will be vectors, which contain eighteen distances. Alternatively, a distance vector will represent each face. On the other hand, we computed confusion matrix of each classifier. We had also calculated accuracy, sensitivity, and specificity of each classifier. The following equations present the formula of specificity, sensitivity, and accuracy: • Specificity is the ratio between non-satisfied and neutral faces, which are not classified as a satisfied face and the number total of non-satisfied and neutral faces. We also calculated the ROC curve (Receiver Operating Characteristics curve). ROC curve determinate the probability of distinguishing capability of our algorithm, i.e. how much our algorithm is able to separate between satisfied, neutral, and nonsatisfied faces. We found that our algorithm detected 97% of satisfied faces correctly. In addition, our method extracted 91% nonsatisfied faces correctly. Finally, it detected 100% of neutral faces correctly.
After computing confusion matrices of our algorithm, we computed the sensitivity, specificity, and sensitivity of our algorithm. We found the results showing into table 2. We calculate these statistic criteria for all classifier. We calculate the sensitivity of applying our algorithm by using SVM with RBF kernel, KNN, AdaBOOST, and Random Forest. We found that the most performant classifier is SVM with RBF Kernel. Table 2 shows the results of each classifier by using JAFFE datasets. Our algorithm report that, geometrics features a gave a great model to measure satisfaction customers. Our algorithm consists to measure the variation of face's shape, by computing distances between face landmarks. Since, we based our classification on using the different variation between satisfied, neutral, and nonsatisfied faces. For this reason, our algorithm a gave a great model to detect the satisfaction of customers by using emotions of customers faces. Figure. 3.ROC curve computed by applying our algorithm using SVM and JAFFE dataset.

Conclusion
One of the greatest challenges for marketing era is to ensure customer satisfaction. We have proposed a new method based on the classification of various facial expressions, to assess customer satisfaction using different classifiers: SVM, RANDOM FOREST, KNN and AdaBOOST. Distances between facial landmarks points represent the geometrics features. A vector of eighteen values represents customer's face expression. We tested our method on the JAFFE database and we obtained a performants result. In the future work, we will develop a multimodal algorithm, which contains speech and motion recognition.