Examination of a Skill Sampling Method of an Athlete Using the Athlete's Movement and Eye Movement for the Development of an AI Coach

Article history: Received: 27 August, 2020 Accepted: 17 September, 2020 Online: 24 October, 2020 From amateur players who enjoy sports throughout their lives to top athletes who participate in international competitions, interest in improving sports skills is growing. Coaches and their coaching are indispensable for improving sports skills, but it is difficult for many athletes, especially amateur athletes, to secure coaching. However, we thought that anyone could easily receive coaching through the use of an artificial intelligence (AI) coach. In order to bring about AI coaching, learning is important. The set of learning data must include data such as players’ skills as they correspond to their gaze and performance. In particular, it is thought that analyzing gaze movement during sports may provide insight into exceptional athletic skills. In this study, we propose a skill sampling method of collecting learning data for the specific purpose of creating an AI coach, using a wireless eye movement measurement device and 4K imaging.


Introduction
There is growing interest in sports, including the promotion of lifelong sports with an eye on the aging population, as well as the improvement of the skills of athletes aiming to participate in the Olympic Games and other international competitions. Many people from professionals to amateurs wish to improve their athletic ability and enjoy sports as a regular activity. Having a good coach is essential to improve sports skills, however, not everyone has access to a superior coach. We believe that by extracting the skills of both athletes and coaches, and applying learning data, it is possible to develop an AI robot that can coach anytime and anywhere.
An important task in realizing such AI is to include skill learning. It is necessary to collect learning data as a set of both players' and coaches' performances, gazes, and ideal skills. However, to the best of our knowledge, no such sets of learning data yet exist. In the present paper, we examine a skill sampling method of collecting learning data based on the performance and gaze of an athlete.
We focused on eye movement because we believe that it is essential for players during sports to watch the ball, to watch the opponent, and to observe the situation of the game. In fact, in an experiment that analyzed the eye movements of athletes performing on a gymnastics pommel horse, it was confirmed that posture fluctuations can be predicted from the stability of eye movements [1]. In martial arts such as kendo and fencing, it is said that it is important to maintaim a wide field of view by concentrating on a single point without moving one's eyes (as the Japanese proverb says, enzan no metsuke: "eyes on the far mountain" in a rough English translation) [2]. In a study by Natsuhara et al. [3], experts and semi-experts in football clarify whether there is a difference in eye movements during play depending on skill.
It is thought that an examination of eye movements during competition in various sports can reveal the points of attention and important gaze points for each sport. [4][5][6][7][8] For example, in relation to eye movements, experiments that block vision while a subject shoots a basketball have demonstrated that visual information at the final stage of the shot is important [9].
However, simply measuring eye movements without further context leaves it unclear what game situation brought about those eye movements. Therefore, combining athlete images taken from different directions with eye movement data can help match the movement of the athlete at any given point to the specific playing situation and eye movement. By connecting an actual athlete's action output with data obtained in this way, a set of learning data can be obtained. The required learning data vary by sport. In table tennis, for example, useful data include the players' gaze, posture and movements, how to hit the ball, the score rate, etc. In dance, on the other hand, in addition to data on the dancer's gaze, posture and movements, it is also useful to know his or her velocity of rotation, etc. It is necessary to construct a database that associates the characteristic play of skilled athletes with results such as score rate and rotation speed.
In this paper, we examine a skill sampling method based on an athlete's performance and gaze in order to acquire a data set to be used in creating an AI coach. First, we consider the configuration and accuracy evaluation of a wireless eye movement measurement device developed to easily measure the eye movements of athletes who exercise intensely. Next, we present the results of experiments conducted during table tennis and dance rotation as examples of data acquisition.

Configuration of a wireless eye movement measurement device
Many sports science experiments using eye movements have been conducted [10]. In the work [11], experiments have been conducted with the aim of elucidating the role of head, eye and arm movements during table tennis competitions, using eye movement measuring devices and other equipment. However, the eye movement measurement device used in this experiment requires a wired connection to an interface PC and control unit etc, which is difficult to measure in competitions with a lot of distance to travel or competitions involving rotation. In addition, the study [12] describes the eye movements and shooting strategy in soccer penalty kicks. The eye movement measurement device used in this study was linked to a PC via a 10m fire wire cable from a recording device in a pouch wrapped around the waist, so it was not possible to measure in a completely free state. One study by [4] measured the eye movements of athletes watching competition videos. As in these examples, few studies have measured eye movements in a state where athletes are free to compete.
There are several reasons why it is difficult to measure the eye movements of athletes during competition. First, wired devices can hinder the athlete's performance. Also, calibrated sensors may be displaced by the intense movement of the athlete and, when a sensor shifts, the experimenter may not notice the shift unless measurement status can be confirmed in real time. We solved these problems using wireless measurement devices and by enabling real-time monitoring and recalibration of the measurement situation. Figure 1 shows the configuration of the developed eye movement measurement device, and a person wearing the device is shown in Figure 2. A video transmitter and external control system with a mouse were added to a TalkEye Lite system (Takei Science Instruments Co., Ltd., Niigata, Japan), which uses a corneal reflection method. The experimenter can control recording start/stop, offset of the line of sight, etc. with a Bluetooth mouse by watching a display transmitted wirelessly. Two devices can be used at the same time. Because experimental data is recorded in the TalkEye Lite, there are no direct effects of the wireless delay. The device can be fixed to the head with a hook and loop fastener so that the eye movement sensors do not slip off during exercise. Figure 3 shows the operation screens shown on the external display: (a) is the eye movement calibration screen, (b) is an offset button for correcting the displacement of gaze movement when the subject gazes at the center of the field of view before the experiment, and (c) is the experimental screen. Basic operation is possible wirelessly when the athlete carries the eye movement measuring equipment in a backpack. It is possible to measure the gaze movement of two athletes simultaneously by radio.
We evaluated the accuracy of this eye movement recording system using a dancer who agreed to serve as a subject for the experiments discussed in Section 4. We had the dancer perform two high-speed turns and compared eye movement accuracy before and after the turns. Five-point calibration was applied, as shown in Figure 4. The distributions of the gaze points when the subject gazed at the calibration points before and after two highspeed turns are shown in Figure 5. The subject's accuracy before turning was x: -0.82±0.64 deg., y:-0.72±0.69 deg., and that after turning was x: 0.92±0.87 deg., y:-1.71±1.75 deg. Considering that the diameter of the fovea, which has the highest resolution on the retina, is about 1 degree, it was confirmed that eye movement accuracy barely changes at all with high-speed turns. However, it was confirmed that the dancer's gaze shifted upward when viewing the lower index point after rotation.
The maximum transmission distance of video in the room is 19.7m, and the maximum distance at which the TalkEye Lite can be controlled is 43m. The device weighs 1.3 kg. The size of the goggles with the eye movement sensors and a field of view camera is 19 cm in length, 21 cm in width, and 9 cm in height.

Measurement of gaze movement during table tennis play
We first conducted experiments on table tennis as an example to demonstrate the wireless function of our device. Our subjects were two players on the table tennis team at Tokai University Takanawadai Junior & Senior High School. A table tennis table was brought to our 4K studio, each athlete wore a backpack device, and the eye movement of each was acquired. In order to record the actions of the athletes, video was taken with a 4K camera from the front and the side of the table tennis table and were recorded synchronously with recording by the eye movement measurement device. Figure 6 shows the layout of the equipment during the experiment, and Figure 7 shows an image taken during play.
An example of an image edited after measurement is shown in Fig8. "Cam 1" in Figure 8 shows the performance of Player A, "Cam 2" shows the movement of the ball during the game, and "Cam 3" shows the performance of Player B. The images marked "Player A" and "Player B" in the figure show their respective lines of sight. In addition to the images of the two athletes' gaze movement superimposed on the image from each camera, the expression and movement of the athletes during the game are displayed. Our 4K system makes it possible for individual images to maintain a resolution quality nearly as good as that of highdefinition television even when we divided an image, as in this figure; high-precision analysis was thus possible.
The purpose of this experiment was to sample the athlete's characteristic eye movements and performance during table tennis by using the above analysis method. The specific data to be collected include information on the athlete's gaze and performance during rallies, for example. These data will be useful in the creation of an AI coach. An example of analysis using this apparatus is shown in Figure 9, which shows the gaze movement of both players at the moment Player A hits the ball. The left and right lines of sight are indicated by the yellow and green numbers, respectively, where their number indicates their order.
Furthermore, we propose a method to compare the body and eye movements of Players A and B in the time series shown in Figure 10. Figure 10 compares the actions and lines of sight of Players A and B. Here, for greater clarity, the ball is circled in white and the line of sight is circled with yellow dots. The area over which the gazing points were distributed is expressed by the size of the circle.
In These results indicate that Player A's head movement supplemented his eye movement since his eye movement area was small, while Player B did not need to supplement his eye movement with head movement. In this regard, the coach of the high school table tennis team pointed out that small head movement during a game, like Player B's, is better.
Time (g) in Figure 10 also shows another characteristic result. When the ball hit by Player A reaches the top of its trajectory after bouncing off the table, Player B looks at the highest point of the ball. It is known that the rotation direction of the ball is easy to see when the vertical velocity of the ball is nearly at the top of its trajectory. In other words, we were able to capture the moment when Player B looked at the top of the trajectory to determine the rotation direction of the ball.
Two important features of our system are that two wireless eye movement analysis devices can be employed simultaneously and that it uses high-definition 4K imaging, making it possible to extract such moments from rallies between two players. By using two devices, it is possible to know the athletes' tactics using gaze. We believe that this can be useful as a sports video analysis method or as a new training method. By using this method, the athlete's body movement, situation, and eye movement can be obtained simultaneously at any point during competition. At present, coaching emphasizes movement but, by using gaze data, it is possible to provide coaching based on the information skilled athletes obtain from their own gaze and from their own plays as well as those of their opponents. Furthermore, a database can be created in which these measured data are combined with the score rate, smash success rate, etc. By then using this information as learning data, an AI coach that can teach expert skills can be created.

Measurement of gaze movement during dance
Next, let us consider an experimental example of dance. Considerable research on figure skating is currently being conducted. Resistance to vestibular system input is typically weak in novices who have just started figure skating, and it has been noted that they are remarkably susceptible to physiological and mental disorders. Rotation experiments using a Barany chair have shown heartbeat irregularities and malfunction of the coordination of movements of the hands and feet [13].
It is difficult to measure eye movements while a subject is rotating during competition. Therefore, experiments have been conducted in which circulation vection is generated using virtual reality in order to examine eye movements closely related to the vestibulo-ocular reflex (VOR). In a comparison of professional skaters and beginners in which the angle of the view to be displayed was changed, significant differences were observed all components of eye movement (decrease in fixation time, decrease in saccade amplitude, increase in blinking, etc.) in results corresponding to a visual angle exceeding 180° [14].
These studies show that training of the vestibulo is an important aspect of figure skating training, and eye movement is useful for evaluation. From the point of view of training athletes for sports that require high speed rotation, such as figure skating and dancing, the athlete should set the gaze position at the time of rotation at a single point and suppress the VOR, which is the function of the semicircular canal at the back of the ear. This is said to prevent eye movement in the direction opposite to the rotation of the head, which occurs to compensate for head movement during head rotation [15][16][17].    Previous studies have used rotating chairs and images such as VR environments but to the best of our knowledge, none have actually measured eye movements during performance. Since the athlete rotates, a wired system is impossible and a wireless system is indispensable. Furthermore, since the system undergoes heavy centrifugal force during rotation, it is necessary to ensure that the goggles for eye movement measurement do not shift, and robustness is also required for the eye movement measurement device itself. Additionally, due to the intense motion during performance, it is possible that the calibration of the eye movement device may shift. Thus, it is necessary for the experimenter to be able to recalibrate the eye movement measuring apparatus during performance of subject.
We believe that the device introduced in the present study meets the requirements for eye movement measurement during dance. In the present dance experiment, a dance expert with a dance history of 17 years beginning at the age of 4 wore a wireless eye movement measurement device and rotated in place. To the best of our knowledge, there have been no reported examples of eye movement measurement during dance using a wireless-type eye movement measurement device. All experiments were conducted at the 4K studio of our university, and videos were taken with GoPro from the top, and with a 4K camera from the front, right and left; all videos were recorded synchronously with the wireless eye movement measurement device. The sampling rate was 120 Hz for the GoPro device, 60 Hz for the 4K camera, and 30 Hz for the wireless eye movement measurement device. The experimental set-up is shown in Figure 11 and 12.
After the experiment, we edited the video from the three recording devices, and found the rotation angles of the head and body. An example of the edited video is shown in Figure 13. Images of the dancer's eye movement and of the dancer from various directions were recorded in high definition using a 4K production system. The purpose of this experiment was to sample a dancer's characteristic eye movements and body posture and movement during a dance competition. As mentioned above, to the best of our knowledge, there has been no actual measurement of gaze during a dance turn. Our proposed skill sampling method can clarify not only the gaze of the spinning dancer, but also his or her movement. A sequence of images from a dance turn is shown in Figs.14-17. Figure 14 shows the series of images taken from overhead, and the same sequence is then shown from the front (Figure 15), from the right (Figure 16), and from the left ( Figure  17). A picture was taken every 45°, and the numbers correspond across the different views. The present method makes it possible to acquire the relationships among the body, head, and eye movements of a dancer during a dance.
Figs.18 and 19 are graphs of the subject's lines of sight at the time of rotation. The solid line shows the rotation angle of the head, the broken line shows that of the body, and the dotted line shows that of the right eye (gaze). Where there is no dotted line, the subject blinked and eye movement could not be acquired. The color of each line corresponds to the number of the trial. The left vertical axis in Figure 18 shows the angle between the head and the body, the right vertical axis shows the angle of the line of sight, and the horizontal axis shows the corresponding numbers of the frames on the wireless eye movement measuring device. In Figure  19, the vertical axis shows the velocity of the corresponding rotation angle, and the horizontal axis shows the numbers of the frames on the wireless eye movement measuring device. First, let us focus on head movement. The head does not move significantly until Frames 7-10 in Figure 18. This is because the athlete is trying to look at his chosen front fixation point. After that, the head rotates rapidly from Frame 8 onward, overtaking the body, which rotated previously. The velocity of the head rotation angle reaches its maximum in Frames 10-14 in Figure 19, and the average of the velocity of the head rotation angle of 11 trials is 1712±112 deg/sec. Next, let us examine the motion of the body. The velocity of the body rotation angle (Figure 19) reached its maximum with the same timing as the head, at Frames 8-12. However, it can be seen that there is no large change in speed compared to the head; the body rotates at an approximately constant speed. The dancer's eye moves to the left until Frame 11, at which point the body and the head are both rotating to the right, indicating that the dancer is gazing at his chosen front fixation point on the left. Some data were lost due to blinking but, after 18 frames, the body and head passed right behind and turned to the right, and the dancer was gazing at his chosen front fixation point on the right. Eventually, the body and head rotate 360 degrees and again face the front and the eye movement is also 0 degrees, but before that time, a number of sawtooth-like eye movements like OKN (optokinetic nystagmus) that pass through the predetermined frontal fixation point and back again have occurred.
Next, Figs Figure 20 shows that the body rotates first, followed by the head, which turns rapidly around a body rotation angle of about 90° (③ in Figs. 14-17, 20 and 21) until the body rotation angle reaches about 140 deg (④ in Figure 14-17, 20 and 21), when the head begins to lead. Figure 21 confirms that the maximum velocity reached 1920 deg/sec when the head was sharply turned. The velocity of the body rotation angle did not change as rapidly as that of the head.
Generally, the eye rotates in the direction opposite to that of the rotation of the head due to the VOR, but when the head is rotating at a high speed, the VOR is prevented, and we confirmed that the eye did not move in Figure 21 (④ and ⑤). Furthermore, the video from the field-of-view camera of the wireless eye movement measuring device confirmed that the subject continued to gaze at the front camera between 0° and 90° (① -③ in Figure 14-17, 20 and 21) and between 270° and 360° (⑦ -⑨ in Figure 14-17, 20 and 21). From Figure 20, it was confirmed that the eye rotated to the left from the front as it rotated (① -④ ), and from the right to the front after turning (⑦ -⑨ ), in order for the dancer to gaze at a point that he had chosen in advance (the front camera, in this example) even as the body rotated. After that, eye movement showed a large reverse in direction at ⑦. Regarding gaze position, it was confirmed that the dancer gazed at a point that he had chosen in advance by asking "Where do you look when you are turning?" after the experiment. Figure 11: Experimental set-up from the top Generally speaking, when one sits on a rotating chair and rotates, the VOR causes dizziness. However, a dancer keeps his or her posture steady without dizziness by watching a certain point while rotating [11][12][13]. To the best of our knowledge, the present study is the first time that this could be confirmed through the use of an eye movement measuring device. The present method makes it possible to acquire the relationships among the body, head, and eye movements of a dancer while dancing. By using this skill sampling method to create a database of the head, body, and eye movement data obtained from many skilled athletes, an AI coach that can teach beginners the optimal rotation method can be created.

Conclusion
There is a big difference in the level of sports ability to aim for, from the promotion of lifelong sports aimed at healthy longevity to the training of athletes who are able to compete in the Olympic Games and other international competitions. For this reason, coaches for many different levels of athletes are required. However, it cannot be said that there are enough coaches, except for athletes who are aiming for professional sports and the Olympics. If an individual learns tips for improving and enjoying various sports, he or she will be able to start sports at an early age and continue throughout life; such tips may also help the handicapped and the elderly. Therefore, if we can develop an AI coach that can teach these tips, we believe that the shortage of coaches can be solved at once.
However, learning data are required in order to develop AI robots. For example, if the goal is a successful shot in soccer, the performance of the athlete up to that point and during the immediate situation becomes learning data. The situation during a game changes every second, and the exact same game situation is rarely, if ever, reproduced. Additionally, the athlete does not experience all possible game situations and does not calculate the next play based on all past experiences during games. It is possible to develop an AI coach that can make judgments similar to those of the skilled athlete by applying as much learning data and as many outcomes as possible. To the best of our knowledge, no analysis of the behavior of athletes during a match or performance has yet been carried out for this purpose. Therefore, given the fact that the eye movements of advanced and beginner athletes differ significantly in various skills, and taking advantage of the superb resolution of 4K imaging, which allows images taken during play to be synthesized with high resolution, we worked on building databases that make it possible to learn from the eye movements and performance of athletes during a game or performance. In this paper, we present measurement examples of athletes both playing table tennis and performing dance rotation, and analysis examples using 4K video; these data can be used to create a database.
In table tennis, the analysis included data collected when two players were competing, specifically, data regarding eye movement, the position of the racket, and the position of the ball.
In the case of dance rotation, eye movement, head movement, and body movement during rotation were examined. The data introduced here are those of non-professional athletes and are not necessarily sufficient as input for learning data. However, by applying our skill sampling method with cooperation from top athletes, it will be possible to collect the necessary data. Using our method, we would like to develop AI coaching through the use of learning data that match eye movements, body movements, game situations and outcomes during play.

Acknowledgment
This work was supported in part by the Hoso Bunka Foundation (2018/2019 Grant Cycle). We would like to express our sincere gratitude to the table tennis team at Tokai University Takanawadai Junior & Senior High School for kindly allowing us to take movies of them playing. We also thank Mr. Hiroki Kudo in our laboratory who danced for our dance experiment.