1. Introduction

Motoric skill learning can be trained with repetition, also utilizing stimulants from the surrounding environment [1]. Such motoric skills, for example, are typing, writing, and playing a musical instrument. To acquire dexterous motoric skills, practice with repetitive movements is needed. Moreover, these practices need persistence from the instructors and the learner. On the other hand, a stimulus can be used to augment the learning process.

Visual and audio information are commonly used as stimulants. Learners are showed how to do the skill accurately. In this process, they are played, or listened to, the appropriate sound. A haptic sensation can be applied to the learner using mechanical interaction. Some researchers utilize these stimuli to enhance the method to train the motoric skills, for example, playing the piano [1, 2], writing Japanese characters [3], writing Arabic and Japanese characters [4], and typing the Braille alphabet [5]. However, these systems are cumbersome, and the user finds it difficult to move their fingers.

To overcome these shortcomings, electrical stimulus is used. This information is given to the skin of the user. The electro-tactile feedback system is proven to become feedback used for controlling mobile [6], arm robot [7] also fingers [8], information for haptic sensors [9], enhanced virtual game [10]. To examine the potential usage of electrical signal, a multimedia learning system for a motoric finger is established.  This paper is an extension of work initially presented in the 2019 International Conference on Sustainable Engineering and Creative Computing (ICSECC) [11]

For the experiment, subjects are divided into three groups. They are trained using different methods. Information in the form of audio and visual means are given to the first group to train the finger movements of each subject. In the other groups, the subjects utilized audiovisual means, enhanced with electrical sensations to practice. These groups consist of subjects which do the experiments with active mode and subjects with passive mode. The results show that the group using an electro-tactile feedback system is capable of playing the music appropriately with both active and passive modes. Moreover, they learn more rapidly than the other group. Besides that, the proposed system is simple and easy to use. The haptic sensation also does not cause pain to the subject.

This paper is organized as follows: Section II delivers a brief overview of related works within this field of study. Section III presents the details of system implementation. Section IV describes the experiment, which demonstrates the effectiveness of the system for finger motoric learning. Finally, Section V delivers the concluding remarks.

2. Background

The process or action to achieve an ability to do, or to gain knowledge, is known as a learning process. The actions done in this process require stimulation from the surroundings. Moreover, repetition is needed in the learning process [12]. In this process, one thing can be connected to others, or known as associate learning. Stimulus or information is given repeatedly, resulting in a reaction, until the reaction can be an automatic response and become habitual. This stimulus can be associated with a reaction. Pavlov shows this condition with his experiments, in which the  subject dogs have a new behavior. The dogs are able to associate the sound of a metronome with food [13]. Some physical training can be involved in associate learning, such as finger movements.

 The skilled fingers motions are involved for a specific action, such as typing a computer keyboard or playing the piano. These finger skills are challenging to learn using only verbal instructions. Such training will be effective and efficient if the learner does  a lot of exercises. Some software have been developed to assist people to learn to type in a keyboard or to play the piano. These software are intended to support the user in typing or playing the piano with all fingers. Visual stimulation is used by these software, for example, colors or lights. Other programs utilize audio information, e.g., the sound of the notes. Moreover, some of the applications apply both audio and visual information. However, another sense has excellent potential to transmit data, namely the sense of touch or tactile taste, which uses the skin to deliver stimulation.

Human skin is sensitive to several sensations, for example, vibration, heat, force, and electric signal (voltage or current). The skin has some sensory receptors, namely mechanoreceptors, thermoreceptors, nociceptors, and proprioceptors. These receptors are to accommodate all the sensations. Moreover, these receptors also enable humans to feel texture, paint, etc. [a]. Tactile sense causes humans to have the ability to detect and classify objects through their skin. This sense is sometimes called haptic sensations.

 The term haptic comes from the word ‘hapto’ from Greek. This word means ‘to touch and handle objects’ [14]. Several researchers introduce haptic information utilizing vibration [15], heat [16], force [17], and electric sensations [6]. These information aims to enrich the user to interpret or interact with the application more effectively.

Haptic feedback has been used widely in various applications. A simulator to learn how to perform a surgery [17], remotely controlling a robot’s actions [18], a tactile sensor to sense the roughness of a surface [9] and gaming experience using VR [10] are the examples of the application of this feedback. Also, there are applications for training fingers skills using haptic feedback. For instance, haptic feedback is used for learning to play the piano [1,2], and using a Braille keyboard [5]. Moreover, this feedback is combined with other feedback, namely visual and audio, for writing characters  [3] [4].

One of the rehabilitation procedures for a person with spinal cord injury (SCI) is to learn how to play the piano. For this training, subjects are equipped with a vibrating glove. This glove is equipped with a vibration motor on every finger. The vibrations help the subject to know which finger has to strike the keys of the piano [1]. The subjects are required to move their fingers while training. Such a method is called an active mode. Markow et al. report that this system enables the subject to play an easy song.

Others report that a passive mode training using vibration cues has positive results. The passive mode method is the method in which the subject trains without actually doing the activity about which they learn. The learning process is given using their senses. In [2] the subjects are trained to play a song with a piano using a vibration glove. The audio and haptic feedbacks are given to the subjects while they are doing the activity. Subjects are listening to the song, while the vibrations in their fingers give information which finger has to move related to the song. These experiments show that this information, including haptic feedback, enables the user to play the piano reflexively.

In [5], active and passive modes are used to train blind subjects to type in a Braille keyboard. This system has a keyboard that has the capability to train the user to use audio and vibration. The vibration information is given to the user’s fingers. The subjects are able to use both hands to learn using this system. In these studies, the experiments are done with active and passive modes. The authors argue that both methods using audio and haptic information can be used  to learn to type.

Another haptic information is force feedback. One of the devices able to give kinesthetic feedback to the user is called The Phantom. This device is used to train the learner to write Japanese  [3] and Arabic [4] characters. This device helps the hand of the user to write the characters correctly. Both studies show that the force feedback is able to assist the subjects in training to write several newly-learned characters efficiently.

Mechanical feedback, such as vibration and force feedback,  is able to transmit information for finger skill training. Haptic feedback has a great potential to be a training method, however it is cumbersome and costly. These devices hinder the movement of  users in typing on a keyboard or playing the piano.

One type of haptic feedback is electro-tactile feedback. This applies an electric current to forward information to the skin. This feedback gives sensation to the skin and directly to the nerves. Electrotactile feedback system is simple and comfortable to use. Data from electric sensations can be informed through differences in amplitude or frequency. Thus, electrotactile feedback has various ranges of information that can be transmitted to the user [6]. In the next section, detailed methods to demonstrate the potential of electrotactile feedback are presented.

3. System

3.1. Overview

The proposed system for finger training is a computer with a screen and sound system, in addition to a gesture sensor, also with an electro-tactile feedback system. The electro-tactile feedback has different modes to vary the frequency, amplitude and the width of the electrical pulse. These variations have to be adjust to the subjects wearing this device. This system is comfortable to wear and does not hinder the user’s movements [6-11].

The frequency, amplitude and pulse width have a wide range of options, the frequency was varied between 10Hz to 120Hz. For this experiment stimulus pulses are set at 20Hz frequency (Figure  7). The amplitude of the pulses is adjusted by the user between 40-80V to achieve a comfortable condition. The intensity is controlled by the pulse width. The width of signal is varied, particularly between 10 to 100μs.

This system can be seen in Figure 1. Figure 2 shows the diagram block of the system. The screen of the computer displays the character of the note and which finger(s) to bend, while the corresponding frequency of the note is sounded using the computer’s sound system. The electrodes in the fingers of the subject are transmitting the electric sensation. These sensations are given to the fingers appropriate to the note. The gesture sensor is used for tracking the fingers of the subjects. This sensor is placed near the top of the screen facing down to detect the movement of the subject fingers.

The learning is done by a passive and active method. The audio, visual, and haptic information are given to the subject while learning in the passive method, whereas the hand tracking device with the feedback is used when active learning is performed. In the test period, only a hand gesture sensor is used with the computer.

Figure 1: Arrangement of the system

Figure 2: The Diagram block of the system

3.2. Hand tracking and software

One of the main components of the proposed system is the hand and fingers tracking sensor. For these tasks, a leap motion sensor is used. Leap motion is able to detect fingers and hands accurately from around one meter in three dimensions. The accuracy of this sensor is around 0.1 mm. This device is equipped with two infra-red cameras and three infrared LEDs [19]. This sensor has two modes. The first mode faces upward; the hands are above the leap motion sensor, known as a desktop mode. In the second mode, the HMD, the leap motion faces downward. HMD mode is used for this system. A custom bracket is utilized for placing the sensors. In this position, it is possible to have noise from reflective surfaces, so a dark mat is placed to cover the desk. The mat is shown in Figure 1.

The finger training multi senses system has several modes to train the subjects. The first mode is training using three senses; visual, audio, and haptic. The second mode uses visual and audio information, and the third uses audio and haptic information. The first and the second modes will be used to train the subject with the active method, while the third one is used to train with the passive method. The active mode requires subjects to tap the fingers below the gesture sensor.

The software is able to detect the correct or the wrong fingers. The function of this feature is to test the result of the training.  The GUI of the system can be seen in figure 3. Moreover, the system is able to train the subject to play piano with only the right hand/notes only, or with both hands (notes and simple chord).

Figure 3: GUI of the system

The software is developed to display the characters of the note. Also, this software shows the appropriate fingers that have to bend. Figure 4 shows the fingers as appropriate to the note in a piano for the left hand (Figure 4.a) and right hand (Figure.4.b). The left-hand fingers of the human are placed in the a lower octave than the right hand. The left hand is trained to press the fundamental note of the chord. For example, if we want to play the C chord (C-E-G) the subjects are trained to press C, and so on such as in Figure 4.a. The left hand and the corresponding notes of the chord are shown in Figure 4.b. The right hand of the subject is trained to play the notes from C to G. Using this arrangement, this software is able to instruct the user to perform simple music. The user can play with only the right hand (melody only) or with both hands (melody with simple chord).

This software also has the capability to produce sound of note. This voice corresponds with the finger as defined in Figure 3. The output of this information is transmitted to the computer sound system. Moreover, the haptic feedback augments the information for the user. Electric signal is transmitted to the nerve using electrodes patch. Fig 5 shows the illustration of the back of human palms. All the human fingers of both hands are placed the electrode. The software has facility to send electrical sensations appropriate to the visual and audio information. The next section will provide the detailed explanation of the electro-tactile feedback system.

Figure 4: The corresponding fingers with the notes (a) left hand (b) right hand.

Figure 5: The electrodes in hands of the subject

3.3. Haptic system

The electro-tactile feedback system, which we call TENS (Transcutaneous Electrical Nerve Stimulation) system, comprises of three main components, namely transmitter, receiver and electrodes. The first component is the transmitter; the function of which is to transmit the signal from the computer to the receiver wirelessly. The main element of this component is a Bluetooth transmitter. The receiver receives the information from the transmitter and transformed it into electrical pulses. The five channels receivers with one ground are used. For the experiments, two transmitters are used for both hands. This structure enables the subject to receive ten signal frequencies. The last component is the electrode, placed in the knuckles, and a ground is placed in the middle of each hand. This arrangement allows the nerves of the subjects to obtain the information. The electric signal has frequencies that vary in the range of 10Hz to 120Hz. The width of the pulse varied between 10μs to 100μs. Moreover, the width of the pulse is a factor in controlling the intensity of the signal. The signal has an amplitude range between 40V until 80V. Those features are able to be adjusted. This adjustable function is to attain the subjects’ comfort. Every subject has their own preferences. In the following section, the experiment with the results will be shown.

4. Experiment Methods and Results

This learning system is to train the subjects with passive and active mode at improving the basic skills of playing the piano. These trials were directed mainly to confirm the potential of the proposed system using audio and visual information and augmented with electrotactile feedback system. This objective can be done by comparing the experiment with a training system using only audio and visual feedback.

To do the experiment, nine subjects are divided into three groups. All the subjects are never trained to play the piano. They also have never used the electrotactile feedback. The subject’s ages vary between 15 until 21 years old. The subjects are comprised of five males and four females. The first group trains using the active mode with audio, visual and haptic information. The second group practices with the active mode, only using audio and visual feedback. The passive mode, with audio and haptic information, is given to the third group. The subjects doing the active mode have to tap their fingers as appropriate as the given information.

Before the trials, all the subjects are given basic music theory, focusing on the use of appropriate fingers and hands for specific note or chords. For the first five minutes, they are trained  to use the system. The aim of this training is to enable the subjects play with the virtual piano. Subjects were trained to tap their fingers according to the notes that appear on the screen until they become used to use their fingers under the sensors. This is followed with an initial test. In this state all the subjects tried to play a simple song. The system only shows the letter of the notes that they have to tap, and the errors are counted. Errors include the use of wrong fingers or missed notes.

After the initial test, for the groups where haptic feedback is used, the intensity of the electric signal is adjusted for each subject. This is for the comfort of the subject, also to adapt with the signal. This is then followed with five minutes training of the subjects with the method in their groups. In this time all the subjects are given information about which fingers must be bent in accordance with the letters of the notes shown in the screen. Then, in the last five minutes, all the subjects tried to play music assisted with the feedback and the letters of the notes in the screen. After a short break, the final test is conducted to prove the effectiveness of the haptic feedback. This task is the same with the initial task. The errors are calculated and the percentage of improvement of each subject can be counted.

Based on the experiment, on average, the first group, who learned to bend the fingers utilizing visual and audio information, and enriched with electrical sensation, on average achieved an improvement of around 40%. On the other hand, the second group, who learned through the active method using visual and audio feedback, but without haptic feedback, achieved 20% improvement. While the last group, which used the passive method, had about 35% improvements. Table 1 shows the number of errors for each subject before and after being trained in the method they use.

 Analysis of the outcome of the experiment shows that all the training methods offers positive results. All the methods are able to improve the musical skills of the subjects. However, the methods which employ the haptic feedback show a more significant performance improvement compared to the method not involving haptic feedback. This can be concluded that the systems that involve haptic information are more effective. Moreover, subjects who use electric stimulation feel that the feedback is not painful and easy to use.

Table 1: Error before and after training

5. Conclusion

This paper introduces a novel system to train fingers to play the piano using multimedia information. The audio, video and haptic feedback is used to teach. A computer’s speaker is used to provide audio information. The screen of the computer is used to show visual feedback, while haptic information is transmitted utilizing electrical sensation. This proposed system employs a gesture sensor to become a virtual piano, and a custom software is built to accommodate all the inputs and outputs of the system.

The results of the tests indicate that the system involving electro-tactile feedback using active and passive training methods have positive results. The electro-tactile feedback system enables to train the fingers of the users to move in association with the information reflexively. Moreover, this system augments to train users the new finger skill better than the method only using audio and visual. The improvement of the subjects who use haptic feedback is superior to the subjects who utilize the conventional one. Electro-tactile feedback is inexpensive, convenient to use, simple to wear, and does not hinder the user from moving their fingers.

This system has shown its potential to enhance finger training devices. It is desirable to conduct further research with more subjects, as well as a comparison with existing training systems to measure the performance of this proposed system.

Acknowledgment

This work is financially supported by Penelitian Terapan Unggulan Perguruan tinggi research fund at Indonesian Ministry of Education. Also acknowledgment to Politeknik Negeri Batam for the laboratory facilities.

1. T. Markow, N. Ramakrishnan, K. Huang, T. Starner, M. Eicholtz, S. Garrett, H. Profita, A. Scarlata, C. Schooler, A. Tarun, D. Backus, “Mobile music touch: Vibration stimulus in hand rehabilitation,” in 2010 4th International Conference on Pervasive Computing Technologies for Healthcare, Pervasive Health 2010, IEEE Computer Society, 2010, doi:10.4108/pervasivehealth.2010.8.
2. D. Kohlsdorf, T. Starner, “Mobile music touch: The effect of primary tasks on passively learning piano sequences,” in Proceedings – International Symposium on Wearable Computers, ISWC, 2010, doi:10.1109/ISWC.2010.5665877.
3. M.A. Eid, M. Mansour, A.H. El Saddik, R. Iglesias, “A haptic multimedia handwriting learning system,” in Proceedings of the ACM International Multimedia Conference and Exhibition, ACM Press, New York, New York, USA: 103–108, 2007, doi:10.1145/1290144.1290161.
4. J. Bluteau, S. Coquillart, Y. Payan, E. Gentaz, “Haptic Guidance Improves the Visuo-Manual Tracking of Trajectories,” PLoS ONE, 3(3), e1775, 2008, doi:10.1371/journal.pone.0001775.
5. C. Seim, D. Quigley, T. Starner, “Passive haptic learning of typing skills facilitated by wearable computers,” in Conference on Human Factors in Computing Systems – Proceedings, Association for Computing Machinery, New York, New York, USA: 2203–2208, 2014, doi:10.1145/2559206.2581329.
6. D.S. Pamungkas, K. Ward, “Electro-tactile feedback system for achieving embodiment in a tele-operated robot,” in 2014 13th International Conference on Control Automation Robotics and Vision, ICARCV 2014, 2014, doi:10.1109/ICARCV.2014.7064573.
7. D.S. Pamungkas, K. Ward, “Tele-operation of a robot arm with electro tactile feedback,” in 2013 IEEE/ASME International Conference on Advanced Intelligent Mechatronics: Mechatronics for Human Wellbeing, AIM 2013, 2013, doi:10.1109/AIM.2013.6584175.
8. D. Pamungkas, K. Ward, “Electro-tactile feedback system for a prosthetic hand,” in 22nd Annual International Conference on Mechatronics and Machine Vision in Practice, M2VIP 2015, 2015.
9. D. Pamungkas, K. Ward, “Tactile sensing system using electro-tactile feedback,” in ICARA 2015 – Proceedings of the 2015 6th International Conference on Automation, Robotics and Applications, 2015, doi:10.1109/ICARA.2015.7081163.
10. D.S. Pamungkas, K. Ward, “Electro-Tactile Feedback System to Enhance Virtual Reality Experience,” International Journal of Computer Theory and Engineering, 8(6), 465–470, 2016, doi:10.7763/ijcte.2016.v8.1090.
11. D.S. Pamungkas, A. Turnip, “Electro-Tactile Cues for a Haptic Multimedia Finger Motoric Learning System,” in ICSECC 2019 – International Conference on Sustainable Engineering and Creative Computing: New Idea, New Innovation, Proceedings, Institute of Electrical and Electronics Engineers Inc.: 127–132, 2019, doi:10.1109/ICSECC.2019.8906989.
12. Principles of Neural Science, Fifth Edition (Principles of Neural Science (Kandel)): 9780071390118: Medicine & Health Science Books @ Amazon.com.
13. I.P. Pavlov, “Conditioned reflexes: An investigation of the physiological activity of the cerebral cortex,” Annals of Neurosciences, 17(3), 2010, doi:10.5214/ans.0972-7531.1017309.
14. M. Mihelj, D. Novak, S. Beguš, Introduction to Virtual Reality, Kluwer Academic Publishers, 2014, doi:10.1007/978-94-007-6910-6_1.
15. Y. Yoon, D. Moon, S. Chin, “Fine Tactile Representation of Materials for Virtual Reality,” Journal of Sensors, 2020, 7296204, 2020, doi:10.1155/2020/7296204.
16. R.L. Peiris, Y.L. Feng, L. Chan, K. Minamizawa, “Thermalbracelet: Exploring thermal haptic feedback around the wrist,” in Conference on Human Factors in Computing Systems – Proceedings, Association for Computing Machinery, New York, New York, USA: 1–11, 2019, doi:10.1145/3290605.3300400.
17. K. Rangarajan, H. Davis, P.H. Pucher, “Systematic Review of Virtual Haptics in Surgical Simulation: A Valid Educational Tool?,” Journal of Surgical Education, 77(2), 337–347, 2020, doi:10.1016/j.jsurg.2019.09.006.
18. A. Abiri, J. Pensa, A. Tao, J. Ma, Y.Y. Juo, S.J. Askari, J. Bisley, J. Rosen, E.P. Dutson, W.S. Grundfest, “Multi-Modal Haptic Feedback for Grip Force Reduction in Robotic Surgery,” Scientific Reports, 9(1), 1–10, 2019, doi:10.1038/s41598-019-40821-1.

(Click Here)

(Click Here)