Human Emotion Recognition Based on EEG Signal Using Fast Fourier Transform and K-Nearest Neighbor

A R T I C L E I N F O A B S T R A C T Article history: Received: 31 August, 2020 Accepted: 28 November, 2020 Online: 14 December, 2020 Human emotional states can transform naturally and are recognizable through facial expressions, voices, or body movements, influenced by received stimuli. However, the articulation of emotions is not practicable by every individual, even when feelings of joy, sadness, or otherwise are experienced. Biomedically, emotions affect brain wave activities, as the continuously functioning brain cells communicate through electrical pulsations. Therefore, an electroencephalogram (EEG) is used to capture input from brain signals, study impulses, and determine the human mood. The examination generally included observing a person's frame of mind in response to a given stimulus where the immediate results were inconclusive. In this study, the associated classifications were normal, focused, sad, and shocked. The raw brainwave data from 50 subjects were recorded by employing a singlechannel EEG called the Neurosky Mindwave. Meanwhile, the assessments were performed while the candidates’ minds were stimulated by listening to music, watching videos, or reading books. The Fast Fourier Transform (FFT) method was utilized for feature extractions, along with the K-nearest neighbours (K-NN) for classifying brain impulses. The parameter k had a value of 15, and the average classification accuracy was 83.33%, while the highest accuracy for the focused emotional state was 93.33%. The Neurosky Mindwave in conjunction with the FFT and KNN techniques is potential analytical solutions to facilitate the enhanced identification of human emotional conditions.


Introduction
The signal of the brainwaves has specific traits and characteristics of the person. The brainwaves cannot be imitated or interpreted by individuals so that there is no chance of resemblance. Identity recognition is essential to identify the features of the person [1].
Identity development is done using biometric technology, which aims at personality-implantation through the use of parts of the body or social behavior-application of Identity Recognition using Biometrics Technology for self-recognition by using parts of the body or human behavior [2][3][4][5][6]. In [7], the author has presented machine-learning algorithms to categorize EEG dynamics according to subject self-reported emotional states during music listening. Support Vector Machine was employed to classify four emotional states and obtained an accuracy of 82.29% across 26 subjects. Vanitha [8] has used Hilbert-Huang Transform (HHT) to remove artifacts and relevant time-frequency features from the EEG signal, and the extracted features are manipulated to detect stress levels using Support Vector Machine (SVM) Classifier on real-time stress detection.
EEG is used as a diagnostic tool by medicine. In comparison, a broader application of the EEG is to analyze a person's mental state, thought patterns, or emotional state [9,10]. The EEG signal's visual observation is real, given the very low amplitude of the EEG signal, and the pattern is very complex. In addition, EEG signals are strongly influenced by various variables, including emotional conditions, health, respondent activity, environment, electrical disturbances from other organs, and external stimuli. [11,12]. Conducting the introduction of the EEG signal requires a pattern of brain activity that is prominent and constant. This pattern emerged due to specific stimulation from volunteers. EEG signals are shown to provide more insight into the subject's emotional processes and responses. The recorded EEG signals will get the waveform, the duration, the orientation of the signal, and the signal rhythm. [13][14][15].
The EEG signal pattern data's shape and position are not yet known because the signal pattern and response speed response of each individual's brain to the received stimuli are different [16]. Therefore, we need a signal processing method EEG that can pattern specific brain activity [17].
In this study, the Fast Fourier Transform (FFT) based spectrum analysis method converts the EEG signal from the time domain to frequency domain. The FFT results were classified by K-Nearest Neighbor (KNN) as a training data set to characterize the brain wave output from the EEG as a material for analyzing differences in brain waves in each emotional state studied.

Neurosky Mindwave Mobile
Neurosky Mindwave Mobile is a brain-computer interface (BCI) device that can be used wirelessly and connected to several devices such as computers and smartphones via Bluetooth connection to collect input from the human brain wave activity [18]. Mindwave provided single-channel EEG in the form of an electrode and placed in the forehead or Fp1 position to measure the brain's electrical voltage and ear-clip on the left ear or A1 position as ground. With Neurosky's ThinkGear and eSense technologies, connected devices can receive a real-time visual representation of brain waves generated from the electrodes [19,20].

Subjects
In this study, EEG data were recorded from 50 subjects aged around 20 -22 years old who were in healthy condition. All of the subjects were undergraduate students and were informed about the purpose of this experiment.

Stimuli
The use of appropriate stimuli is needed to get better brain wave recording results where a psychologist has validated the stimulus used for this study. Respondents were given stimuli in the form of listening to music, watching videos, and mathematical calculations to influence brain wave activity in the subject to produce an emotional condition that was following the object to be studied [21,22].

Procedure of Data Collecting
Before starting data collection, make sure that the respondent is in good health. The data collection process is carried out in a quiet and calm place to make subjects feel comfortable and reduce disturbances affecting data collection. We use the computer laboratory at Ahmad Dahlan University for this research. Data collection will be needed as much as 200 data with 50 respondents and carried out four times with a span of at least two days. The recording time for data collection is 7 minutes. When the recording begins, the respondent will be given a stimulus in instrumental music, video, and math questions [23].

Fast Fourier Transform (FFT)
FFT is applied in various fields,from digital signal processing and solving partial differential equations to algorithms to multiplicate integers in large numbers. FFT is an algorithm for calculating discrete Fourier transforms (DFT) quickly and efficiently. Because many signals in a communication system are continuous in the time domain, we use the Fourier transform to represent the frequency domain [24]. FFT is a reliable method for signal processing in the form of a sine wave as EEG signals, and it was proven in previous studies related to real-time applications because of superior speed compared to other available methods. However, it is not suitable for the analysis of short EEG signals [25]. The FFT formula is defined in (1).

dt s(t)e S(f)~~π ft
where S(f) is a signal in the frequency domain, s(t) is a signal in the time domain, and is a constant value of a signal, f is frequency, and t is time.
FFT algorithm is used to reduce the complex transformations performed by DFT. For comparison, when using DFT, then the transformation complexity is N2. Meanwhile, with the FFT algorithm and faster transformation times, the complexity of the transformation is too decreased to (N) log (N).

K-nearest neighbor
K-Nearest Neighbor is a method for classifying objects based on training data closest to the object. The data is then presented with a distance measure to be processed into a mathematical calculation. In this classification, the calculation is done by using Euclidean Distance. The training data with the closest distance are said to be neighbors and then sorted from the closest distance to the farthest distance. Each neighbor is different from one another or the like. Similar neighbors with the largest number among k neighbors correspond to the object being classified [26]. To find the neighbor, it uses distance metrics like euclidean distance as given in equation (2) .
where D (x, y) is the distance between x and y facilities, xk-yk is the x and y coordinates of variable k (k = 1,3,5,… n). The value of k must be an odd number, and the number of training data is even to avoid ambiguity. This value is necessary to facilitate the achievement of classification results from the number of closest neighbors. When there is a class with the most neighbors, the test data will get that class result [27].

Hardware Design
The block diagram above shows that Neurosky Mindwave Mobile is a major component in this hardware system. Neurosky captures input from brain waves via electrodes attached to the FP1 area of the respondent's head and sends an EEG signal via a Bluetooth connection, which acts as a communication protocol between Mindwave and PC / Laptop.

Software Design
The block diagram above explains that when raw EEG data is obtained from recording brain waves using Neurosky Mindwave on respondents who have been given a stimulus. Raw data taken in the signal pre-processing stage is cut 2 seconds early to minimize noise at the beginning of the recording, due to external interference such as a Bluetooth connection between the recording device and the application, or due to the imperfect installation of the device on the respondent's head. The next stage of feature extraction is carried out using the FFT algorithm to convert the EEG signal in the time domain into the frequency domain and extract all important frequency components from the EEG signal such as Delta waves (0.1 -3 Hz), Theta (4 -7 Hz.), Alfa (8 -12 Hz), Beta (12 -30 Hz), and Gamma (30 -100 Hz) [28].From the results of feature extraction, a huge amount of data is obtained, and data simplification must be done by not including data that is below the average value so that the classification results are more accurate for each emotion. Furthermore, grouping training data and test data are carried out using the KNN classification method in mathematical calculations with the Euclidean formula to calculate the closest distance to the results of the FFT feature extraction. The final step is to determine the accuracy of the KNN classification results on waves using the Confusion Matrix.

Result
In this research, several experimental stages were carried out. The first step is signal pre-processing to reduce noise in raw data, then feature extraction to change the form of RAW data from brain waves to the FFT spectrum. Second, monitoring the FFT results in each emotional state, followed by taking all the highest data from the FFT spectrum with a range of 0.2 Hz. Finally, KNN is used to classify emotions (normal, focused, sad, and shocked) and calculate accuracy using a confusion matrix.

Feature Extraction of Raw Data
As previously explained, FFT feature extraction with a sampling frequency of 512 Hz is used to convert the time domain signal from raw data to the frequency domain. We use MATLAB2016b software to do this process. The results of running the program can be seen in Figure 6 below.
The picture above shows the feature extraction program results, which produces two different forms of signal output with the same raw data input from subjects who have been given a stimulus in the form of listening to instrumental music. It can be seen that the signal in the time domain has data along with 2.13x10 5 ms with the highest amplitude of 631, which is in the range of 1.32x10 5 ms, while the signal in the frequency domain has data with a range of 0 -50 Hz where the highest amplitude value is at a frequency of 2.26 Hz. These results will show a different value for each respondent due to different human brain wave activity against the given stimulus.  It can be seen that the spectrum of the FFT signal in four emotional states, namely Normal, Focus, Sad, and Shocked. When the spectrum is observed, there is a significant difference from the shape of the spectrum wherein the Focused and Normal emotional state, the difference in the form of the spectrum is very recognizable compared to the Sad and Shocked emotional states. Then the amplitude value is compared to prove the truth of the observations. However, each spectrum with a frequency range of 0 -50 Hz produces hundreds of thousands of data so that only the average value from data will be taken. Normal emotions get an average score of 0.2386, Focus 0.8904, Sad 0.3625, and Shocked 0.3044. Figure 8 shows the average value of the FFT spectrum.
From a comparison of the average values on each FFT spectrum, focus emotional state has the highest amplitude and lowest for Normal emotions, between Sad and Shocked emotions there is a small difference in the average results so that it can be proven the truth of observation result on the FFT spectrum.

Collecting Data Based on Spectrum Peaks
A reduction process is carried out at this stage at a frequency of 50 Hz to 10 Hz because the fluctuation of data values in the FFT spectrum always occurs at frequency range 0-10 Hz. Figures  9 to 12 show the results of the reduction process. The Values at the highest peaks of the marked frequency spectrum will be collected from the figure above. The frequency in the range 0 -10 Hz produces 40 data by looking for the highest peak value every 0.2 Hz. Then, from 40 data, each 10 data's average value will be sought to produce four average values used www.astesj.com 1086 as material for the classification process. The results of the average value are shown in table 1.  If the reduction process is not carried out in the FFT spectrum, then based on the previous discussion (4.2), it is explained that the frequency in the range 50 Hz produces hundreds of thousands of EEG data and a minimum average value, which can complicate the classification process. We get between the results of previous observations and the current calculations to get very similar results from this process. It can be proven that the results of recording brain waves with the stimulus used to produce good data quality.

KNN Classifier
In this process, 50 respondents studied and recorded brain waves four times obtained 200 data with 140 training data and 60 testing data. The KNN classification is then carried out to determine the results of the subject's actual emotional state from the 60-testing data. The KNN classification results run through MATLAB2016b are displayed in a scatter chart and the coordinate points. It can be seen that each color at the coordinate point is a representation of the emotional condition of the subject, which forms a group based on the value obtained from 60 testing data. If observed, there are several color points whose coordinate values are closer to other neighboring groups. This indicates a change in the status of the color point. more clear results are shown in Table 2 below The table above shows that the KNN classification results using the k = 15 value are the best choice among k = 7 or k = 23. Sixty Average Value test data produces 50 data that corresponds to the original emotional state status, and 10 data does not match. Then perform accuracy testing using the confusion matrix as in table 3 below. From the table above, the highest average accuracy is obtained from the results of the KNN classification using a value of k = 15 with a result of 83.33%, where the accuracy value of the focused emotional state gets the highest result of 93.33%, followed by a normal emotional state of 86.66%, 80% sad emotional state, and shocked get the lowest accuracy of 73.33%. Differences in accuracy results using the right stimulus dramatically affects the accuracy obtained. Regulations about sad emotions and focus are needed to help improve the accuracy of the results obtained.

Conclusion
This study aims to identify human emotions towards a given stimulus by applying the FFT method as feature extraction from the raw data generated by recording brain waves using the Neurosky Minwave mobile with the KNN method using the variable k = 15 to classify the FFT spectrum. Accuracy in each emotional state studied was performed using a confusion matrix where the best accuracy was obtained for Focus emotions as much as 93.33%, followed by Normal emotions 86.66%, Sad 80%, and Shocked getting the lowest accuracy results of 73.33%. The results of the average accuracy in the classification of brain waves were 83.33%. This research found that FFT and KNN can be used as alternatives in the EEG signal analysis method.