A Proposal of Exercise and Performance Learning Assistant Sys- tem for Self-Practice at Home

Article history: Received: 11 June, 2020 Accepted: 15 October, 2020 Online: 24 October, 2020


Introduction
Nowadays, the population of old people has rapidly increased around the world. The rapid increase of seniors may cause large negative impacts to the societies in many countries, if they have unhealthy lives. World Health Organization (WHO) has announced that the physical health determines the Quality of Life (QoL) of a human [2]. To achieve the stability and prosperity of the country, it is one of the most important policies for many countries to advance the QoL and health of the people including seniors.
However, a lot of seniors are in unhealthy conditions, may stay at home all the time, and do never have exercises. Before this study, we had interviews 20 seniors in Indonesia, and found that about 50% of them are suffering from pains in the bodies. More specifically, five seniors feel pains on their legs, three seniors on the backs, and two seniors on the shoulders. Nevertheless, our survey found that in one week on average, three seniors have exercises at more than three times, one has three times, two have two times, and six have one time. The remaining seniors have no exercise at all. Only 15% of them have regular exercises outside their homes more than three times a week to alleviate pains.
Besides, pandemic spreads of COVID-19 force people to stay at home in order to avoid infections among them [3]. Then, many people have lost opportunities of having regular exercises in daily lives, such as walking to offices, schools, or shopping centers, and enjoying sports or trainings in outdoors, fields, or gyms. The similar situations to seniors may also cause health problems to a lot of people around the world.
In this paper, we propose an Exercise and Performance Learning Assistant System (EPLAS) to assist people practicing exercises or learning performances by themselves at home. EPLAS offers a video content showing the model movement of an instructor for some exercise or performance such as Yoga, Tai Chi, or dance. The user should follow the actions suggested by the instructor as similarly as possible. In EPLAS, the rating function is implemented to rate the user reaction to each action of the instructor. It calculates the difference between the coordinates of the feature points in the image of the instructor and in that of the user. If the difference is small, the function gives the higher rate, since both actions are similar.
The feature points are actually extracted by applying OpenPose [4] to the corresponding images of the instructor and the user to each action. OpenPose is a popular opensource program for real-time human pose estimation, and has been developed by researchers at Carnegie Mellon University. It can estimate the poses of multiple persons in one image at the same time. First, OpenPose detects the feature points called keypoints of every person in the image. Then, it allocates them to distinct individuals in the image. The source codes and documents can be accessed at GitHub.
For evaluations of the proposal, we prepared videos of five simple Yoga poses, and asked 41 persons in Indonesia, Japan, and Taiwan to practice the Yoga poses using EPLAS. Then, we asked 20 persons including the Yoga instructor to evaluate each pose of every participant subjectively. The results show the strong correlation exists between the subjective evaluation results and the rating function outputs in EPLAS. The rating function also points out the feature points to be improved in each pose. Thus, the effectiveness of the proposal is confirmed.
The rest of this paper is organized as follows: Section 2 explores relevant studies in literature. Section 3 presents the overview of EPLAS Section 4 presents the rating function in EPLAS. Section 5 shows evaluation results of EPLAS. Finally, Section 6 concludes this paper with future works.

Related Works in Literature
In this section, we discuss related works to this paper in literature. Some of the papers focus on evaluating physical exercises for seniors using robot interactions or electrical muscles, and on helping seniors to record the personal health with the physical health record such as Fuzzy System, Exergames.
In [5], the author proposed a unique fuzzy system design for adjusting the cycle ergometer workload to the physical work capacity of the individual, and observed the physical work capacity by measuring the heart rate and the muscularfatigue-related index. A set of fuzzy membership functions were adopted for three different phases during a trial exercise with a progressively increasing workload.
In [6], the author presented the design, implementation, and user study evaluation of a socially assistive robot (SAR) system, which has been designed to engage elderly users in physical exercises aimed at achieving health benefits and improving the quality of life (QoL).
In [7], the author showed a moderate therapeutic effect associated with the self-application of the neuromuscular electrical stimulation (NMES) application protocol that targets improvements in the muscle strength and the cardiovascular exercise capacity of an elderly person. This work is the promising first step towards the development of an NMES application protocol that could eventually be widely used by seniors to counteract the detrimental effects of aging.
In [8], the author proposed a study for understanding why seniors still perceive the usability of Personal Health Records as low in spite of the publicly available guidelines. Personal Health Records focuses on user generated contents where health websites provide information for users to consume.
In [9], the author proposed the design, implementation, wide deployment, and evaluation of the low-cost physical exercise and gaming (exergaming) FitForAll (FFA) platform. The system usability, the user adherence to exercise, and the efficacy are explored. The design of FFA is tailored to elderly people, distilling literature guidelines and recommendations. The FFA architecture introduces standard physical exercise protocols in exergaming software engineering, as well as, standard physical assessment tests for the augmented adaptability through the adjustable exercise intensity.
In [10], the author developed an exercise-management platform with an end-user application software and a connective sport device to collect and store the elderly individual's exercise data as a reference for diagnosis and treatments. Compared to traditional methods for encouraging sports, this platform uses Near Field Communication (NFC) for automatically collecting data.
In [11], the author studied the correlation between the sense of presence and the attitude towards physical exercises in an active game-based exercise training program for older adults. The hypothesis in this study is that the more positive attitude towards physical exercises would lead to a higher sense of presence. There is the significant positive correlation between them in this program.
In [12], the author supported the positive effects of exergaming playing on the elderly's attitude towards exergames, psychosocial well-being (sociability and loneliness), and inter-generational perceptions. Exergames could be used as a good intervention to improve social interactions among the elderly and promote the healthy and active ageing. Senior Activity Centres (SAC) may introduce exergames and organize it as a weekly activity. Policymakers can use exergames to create an environment for elderly to take physical exercises, maintain social relationships, and in turn, improve the elderly's psychosocial well-being.
In [13], the author developed a sensor resistance-band for objective exercise measurements and preliminary trials of activities' classifications by using the artificial intelligence. The preliminary experiments show that the sensor resistanceband can be used as the replacement equipment of the traditional resistance-band in handling for data collections. A future study will be based on the sensed resistance-band to quantify the exercises objectively in elderly people.
In [14], the author presented a design of Embedded based Assistive System for Yoga (EASY) and its implementation for Yoga postures analysis, using Kinect and LabVIEW. The authors claimed that the novelty of the proposal is the integration of the skeleton method and the golden template techniques to identify the amount of deviation while performing the asana postures. However, there is no detail on how to extract the skeletons accurately, and also no detail on what is the golden template method.
In [15], the author presented a Yoga pose evaluation method using OpenPose. This method calculates the angle of the human pose at the important feature point, including the elbow and the knee, using the to-and-fro feature points together, and compares the angle of the instructor pose and that of the user pose. www.astesj.com In this section, we propose the Exercise and Performance Learning Assistant System (EPLAS) to help people to practice exercises and learn performances by themselves at home.

Required Devices
In EPLAS, a personal computer (PC) displays a video on the monitor and rates the user reaction. The PC camera should be installed to capture the motion of the user. For more realistic instructions, a flat wall in the room can be used as the monitor by adopting a projector. The wireless mouse will be used to control the PC by the user, because the user needs to stay at a distance place from the PC so that the PC camera can detect the whole body of the user. Depending on the exercise, the user may sit on a chair during the exercise.
Table shows the specifications of the adopted PC. A high-performance computer that has a lot of GPUs will be necessary to evaluate dynamic movements of poses using OpenPose, which will be in future studies.

Implemented Functions
For efficient self-practices by users, EPLAS provides the following functions:

Content selection function
A user can select and start the video of the exercise or performance to be practiced.

Mirror function
A user can view the movement or reaction of himself/herself on the monitor to each action of the instructor during the practice.

Rating function
A user can know the accuracy of his/her reaction to each action of the instructor.

Utilization Procedure
The user can use EPLAS for the self-practice by the following procedure: 1. Set up the necessary devices for EPLAS at home properly. Put it in front of the user. Here, it is necessary to make sure that the whole body of the user is detected by the camera.
2. Start running the EPLAS software.
3. Sit on a chair if necessary.
4. Select and run one video content at EPLAS.
5. Practice the exercise or performance by following the movements of the instructor in the video.
6. Reflect the practice by the result of the rating function.

EPLAS User Interface
For the trial use of EPLAS, we implemented simple user interface functions that run only on Chrome [17] browser. Chrome offers rich functions and is now very popular around the world. When the interface is opened, the EPLAS menu in Figure   When one of the yoga pose is chosen, the corresponding exercise interface in Figure 2 appears. This interface has three views.
The left top view displays the current movement or reaction of the user for the mirror function. This camera display function is realized using the built-in camera function in Chrome. Thus, the request for the camera access permission pops up automatically. By permitting it, the moving image of the camera appears there. The moving image can be recorded on the browser, if necessary.
The right top view displays the video of the exercise or performance by an instructor. By imitating the movements of the instructor in this view, the user can practice or enjoy the exercise or performance learning.
The left bottom view displays the recorded moving image of the user. By clicking the button, the user can see his/her movement or reaction for the instructor video. The recorded moving image will be used in the rating function in the next section. www.astesj.com

Rating Function
In this section, we present the rating function using Open-Pose to give a numerical feedback on the quality of movements or poses of the user in an exercise or a performance.

Idea in Rating Function
From a photo or a video frame containing a human body, OpenPose can extract the coordinates of the 18 feature points of the body on the coordinate system of the photo/frame, called keypoints. Figure 3 illustrates the feature points for determining a human pose. Here, we note that OpenPose can handle multiple human bodies in the same photo/frame at the same time. Then, we consider that the difference of the coordinates between the instructor pose and the user pose for the same action is a proper index to evaluate the accuracy of the user reaction.

Coordinate System Adjustment
However, the coordinate systems of two photos/frames can be different by the camera architecture or the distance between the subject and the camera. Since the instructor photo/video should be taken beforehand at a different place using a different camera, the coordinate system for the user pose must be adjusted to be coincident with that for the instructor pose as much as possible.
In this paper, we adjust the x-coordinate and the ycoordinate of the feature points through the liner functions, x = a x x + b x , y = a y y + b y , independently. The values of the coefficients, a x , b x , a y , and b y , are obtained by applying the least-squares method to the x-coordinates or the y-coordinates of the 18 feature points to minimize the difference between the instructor pose and the user pose, respectively.

Feature Point Pickup for Improvements
On a feature point of the user pose, if the corresponding coordinate is much different from that of the instructor pose, the user should care the point intensively to improve the whole pose or reaction. In this paper, we calculate the Euclid distance between the user/instructor coordinates for every feature point, and take the average and the standard deviation of the Euclid distances for all the points. Then, we choose the summation of the average and the standard deviation for the threshold, and pick up the feature points whose Euclid distance is larger than this threshold. In other words, we regard the 16% of the instances as abnormal, assuming they follow the normal distribution.

Procedure
Now, we present the procedure of the rating function for one pose. When multiple poses are rated, this procedure should be repeated for them.
1. Select the photo or capture the video frame that contains the pose to be rated. Here, the final pose to be rated is often longer that the other transition poses.

Run
OpenPose and extract the coordinates of the 18 feature points.
3. Repeat these steps for both the instructor photo/video and the user photo/video. 4. Apply the least squares method to the x-coordinates and the y-coordinates of the feature points to minimize the difference between the instructor photo/frame and the user one.

5.
Adjust the x-and y-coordinates of the feature points for the user pose by using the linear functions.
6. Calculate the Euclid distance between the two coordinates corresponding to each feature point after the adjustments.
7. Calculate the average and the standard deviation of the Euclid distances to the 18 feature points.
8. Take the summation of the average and the standard deviation as the threshold.
9. Pick up any feature point of the user pose if the Euclid distance is larger than both this threshold and the minimum threshold, and notice it to the user to be improved. Here, the minimum threshold should be selected properly. www.astesj.com To avoid pointing too many feature points to be improved, we select the points whose Euclid distance is larger than both the calculated threshold and the minimum threshold. If no feature point is selected for a pose, we select the point whose Euclid distance is the largest among the points in the pose, so that the user can know which part should be improved. For the minimum threshold, we choose the average threshold among the 41 persons in this paper. In future studies, the minimum threshold will be tuned into the proper value. In both figures, we mark the selected feature points.

Advantages of Proposal
The rating function of the proposal is applied to the important static postures of each Yoga pose where the user keeps the same body state for a while. The Euclid distances of the coordinates of the feature points found by OpenPose between the instructor and the user are calculated. Then, the correctness of the Yoga pose is evaluated at every feature point by comparing the distance with the threshold that is given by the summation of the average Euclid distance among all the feature points and its standard deviation. If the distance is larger than the threshold, the function feedbacks that the corresponding point should be improved. Thus, the proposal points out the individual feature points to be improved for the user. Besides, the proposal gives the overall evaluation of the whole posture of the user. The larger threshold suggests the worse pose, since it becomes larger when the difference between the user's pose and the instructor's pose is large. By comparing the thresholds between the users, the best/worst users and their Yoga poses can be known. Furthermore, by reviewing the past thresholds, the user can know his/her improvements, which will be in future works.

Evaluations
In this section, we evaluate the proposed EPLAS using five Yoga poses and the rating function through application to 41 persons in Indonesia, Japan, Myanmar, and Taiwan. We evaluate the user pose by objective evaluation using OpenPose by comparing the result of keypoint and JSON fie between the instructor Pose and the User Pose. Then we made a form subjective evalulation by scoring the instructor and 19 corespondence.

Evaluation Setup
For evaluations, we adopted the conventional personal computer (PC) in Table , because only the static postures are evaluated in the current system. It is noted that high-performance computer that has a lot of GPUs will be necessary to evaluate dynamic movements of poses using OpenPose, which will be in future studies.
Then, we prepared the videos for five Yoga poses in Figure 4, and asked 41 persons in Indonesia, Japan, Myanmar and Taiwan with various ages, to practice them by following the movements of the instructor as the EPLAS users. Then, we manually selected the frames for the final poses by the instructor and the users from the recorded videos, where the final pose stopped for several seconds. By following the procedure in Section 4, we calculated the threshold for each Yoga pose of every user. The correctness of the static posture of the Yoga pose is evaluated at every feature point in our proposal. For the static posture, we calculate the threshold by taking the summation of the average Euclid distance among all the feature points between the instructor and the user and its standard deviation, and compare the Euclid distance at every feature point with this threshold. If the distance is larger than the threshold, our system feedbacks to the user that the corresponding feature point should especially be improved for the better static posture, among the feature points. The larger threshold suggests the worse pose, since the threshold becomes larger when the difference between the user's pose and the instructor's pose is large. By comparing the thresholds between users, a user can know who shows the good performance and who does the bad one. Also, by reviewing the recorded thresholds, the user can know the improvements of the performance www.astesj.com Then, for the subjective evaluation, we requested the Yoga instructor in the videos and 19 users among the 41 users, to rate each pose of every user in the experiments with three points by showing the instructor's photo and the user's photo. The goodness of a Yoga pose can be subjective. In this rating, we asked them to rate 1 if they feel the pose in the photo is good, 2 if they feel it is neutral, and 3 if they feel it is bad. Then, we calculated the subjective results and comparing the subjective result with our system results.

Results for Individual Users
First, we discuss the evaluation results for individual users. Table 2 shows the rating results by the proposal and the subjective evaluation results for individual users. For each one of the 41 users in the experiments, this table presents the gender, the age, the average of the average Euclid distances among the 18 feature points for the five Yoga poses, the average of the standard deviations (SD) of the Euclid distances for the five poses, the average of the thresholds (TH), and the average of the total subjective rating results for the five poses among 41 users.
Then, to confirm the validity of the proposed rating function, we calculated the correlation coefficient between the average thresholds and the average rating results among the 41 users. The value is 0.746, which suggests the strong correlation exists between them. Actually, the two graphs in Figure 5 suggest the similarity between the average thresholds and the subjective rating results of the 41 users. When the average thresholds are compared among the users, younger users have smaller thresholds than older users, in general. As mentioned before, the threshold is given by the summation of the average and the standard deviation of the Euclid distances. The correlation coefficient between the age and the average threshold is 0.660, and that between the age and the average rating result is 0.628. Both suggest the moderate correlations.

Results for Five Poses
Next, we discuss the evaluation results for individual poses. Table 3 compares the average thresholds between the five poses. It indicates that Seated 2 exhibits the largest threshold (31.85), while Side-bend does the smallest threshold (12.41). As illustrated in Figure 4 (e), to stretch the left arm and bend it deeply at Seated 2 may be difficult for seniors, which makes the threshold large.
www.astesj.com In these figures, the feature points that should be improved are pointed out by the marked crosses for further improvements of the poses. They are extracted in the rating function, where the Euclid distances are larger than the thresholds. Here, the average threshold value among the 41 users in Table 3 is used for the minimum threshold.
In Figure 6 (a), the right hand is pointed, because it is not stretched sufficiently, if compared with the instructor in Figure 4 (b). In Figure 6 (b), the three points in the both hands are pointed out, where the both arms should be stretched. It is noted that due to the pain, this lady cannot stretch the arms. Besides, the left leg is also pointed out, because the both legs are not parallel. In Figure 7 (a), the right leg is pointed, because the both legs are not parallel. In Figure 7 (b), the left hand is pointed out, where the left arm should be stretched and bended to the right side. Besides, the left leg is pointed out, because the both legs are not parallel. It is expected that the users will improve their poses by caring these feature points.

Conclusion
This paper proposed the Exercise and Performance Learning Assistant System (EPLAS) to assist people practicing exercises or learning performances by themselves at home. Using an open-source software OpenPose, the rating function was implemented to evaluate the user pose and point out the feature points to be improved. For evaluations, the instructor videos of five Yoga poses were prepared, and EPLAS was applied to 41 persons with various ages and genders in Indonesia, Japan, and Taiwan. The quality of the poses was also subjectively evaluated by 20 persons, where the strong correlation was observed between the subjective results and the rating function outputs. Besides, EPLAS pointed out the feature points to be improved in each pose. Thus, the effectiveness of the proposal was confirmed. In future works, we will improve user interfaces for easy operations by seniors, collect a variety of exercise/performance video contents, evolve the rating function for evaluating dynamic motions, and apply EPLAS to more various people for evaluations.