1. Introduction

Soft robots are specialized sets of actuators that possess infinite degrees of freedom and, therefore, able to perform complex motions and manipulations, which would be difficult for rigid manipulators to perform. The control of such infinite degrees of freedom is, therefore, an integral part of the study and design of soft actuators. This paper aims to control a soft robotic artificial muscle using gesture-based control. This paper is an addition to the study initially presented in International Conference on Soft Computing & Machine Intelligence (ISCMI) [1], which employs hand gestures to control a soft rotary actuator. In this work, the same principle is employed to contract and expand a Soft Robotic Muscle using gesture signals.

The pneumatic air within pneumatically powered soft actuators can be controlled using manual or electronics control. Manual control involves the operation of the control hardware such as piston and cylinder via manual operation [2] while electronics operation makes use of electronic devices such as air pump and ON/OFF solenoid valves for the control hardware interfaces [3][4]. While electronics control has the superior advantage of automatic control compared with manual operation of pistons and cylinders, an electronics-free control system is sought after. This is because soft robots are inherently compliant, and as such, can be used in environments where Electromagnetic Interferences (EMI) needs to be minimized. An electronics-free control system is required because the air pump and solenoid valves used for electronics control generate EMI waves and, as such will limit the use of soft robots in such controlled environments. Electronics-free control makes use of fluidic valves, logic circuits, and fluidic processors. The switching valves and pumps are composed entirely of silicone rubber [5].

Soft robot control can be over a long-distance communication channel [6]. The transfer of control commands from the master device to a slave device via a remote communication channel results in delays. For instance, a teleoperated soft actuator controlled on the master side using hand gestures has been studied for the measured delays. With the Message Queuing Telemetry Transport (MQTT) server utilized to convey packets of data over the internet to the soft actuator, delays, which include communication delays, elastic saturation delays occurring as a result of non-linear effects of pneumatic action, delays due to noise and hardware imperfections were observable. A Kalman filter [7] has been used to effectively minimize the delay due to elastic saturation.

Unlike silicone-based soft roots that move via material stretching, fabric-based eversion robots are made from non-extensible nylon fabric. Therefore, more complex control such as extension, steering, tip position, and orientation, and variable stiffness control is possible using air pressure control [8].

As a result of the multiple degrees of freedom exhibited by soft robots, it implies that multiple sensors are required to measure these degrees of freedom for feedback control. Embedding a large number of sensors within a soft robot is impractical because embedded sensors can limit movement [9] and can give rise to increased material and computational cost. Therefore, an observer or estimator is necessary to estimate the states of the soft robot based on the available measurements from the sensors available. Before observer design can be done, a model of the soft robot is first required. A soft robot has been modeled using a network of wavelet and a Kalman filter used to predict output measurements [10].

In this paper, a soft robotic muscle described in [11] is controlled for contraction and expansion using a gesture device. The choice of an artificial muscle based soft device compared to other forms of the soft actuator is due to its ability to contract and expand in such a manner that can follow natural motions of the hands. Soft robotic muscles made of silicone have the advantage of high contractibility [12] and high force compared to conventional McKibben muscles [13]. Soft robotic muscle actuators make use of vacuum pressure to contract via buckling of air channels [14] and make use of air pressure to expand via the opening of air channels [11]. This is In contrast to McKibben’s artificial type muscles that shorten when pressurized air enters into the inner air cavities [15]. The novelty of the proposed solution is the control of artificial muscles for expansion and contraction purposes via hand gestures. Possible areas of application include the use of the soft muscles for rehabilitation purposes and the combined system for game purposes – a physiotherapy gaming device can be developed to exercise the hands and fingers of individuals that need to strengthen the muscles of the hands and fingers.

Hand gesture recognition refers to the combination of specialized sensors with mathematical algorithms to recognize the movement of the hands with the intention of controlling devices without touch. The applications of hand-gesture systems are in the areas of robotics, home automation, automatic sign language recognition, phone, and computer security and gaming devices. The sensors used for gesture recognition include magnetic sensors, inertial measurement units, single cameras, depth cameras, stereo cameras, and gesture-based gaming controllers. Hand gesture sensors can make use of 3D model-based algorithms, Skeletal-based algorithms (such as the Kinect and Leap Motion sensors), Appearance-based models (such as cameras) or Electromyography-based models (such as EMG electrodes) to track hand movements and recognize the gestures.

 An accelerometer system for drawing English letters and Arabic numerals as well as recognizing directional movement for gaming, was implemented [16] as a hand recognition device for recognizing the handwritten gestures. Hand movement is captured with the use of an accelerometer while the DTW algorithm processes it and sent to a mobile device using the BlueTooth interface. The gesture recognition and motion detection algorithm were validated by collecting hand gesture and directional motion trace for three participants. Being an accelerometer based system, the device performed best with directional movement detection compared to shape and digits recognition.

In order to miniaturize hand gesture devices, a microsystem consisting of ultrasonic traducers was proposed [17]. The hand gestures implemented include hand tap, hand click, hand rotate, hand press, and hand rub, which are precisely the way people use the fingers in actual scenarios. Ultrasonic sensors were used to distinguish the movement of the hands, and RF signal processing algorithms were used to excerpt features for the machine learning classifiers. Accuracies of conventional classification methods with  Radio Frequency (RF) and end-to-end architecture for 9 subjects were analyzed with accuracy greater than 80% for all the classifiers.

A segmentation method that uses both depth and RGB color data from a scene using a Kinect sensor to obtain the hand position and convex hull detection algorithm for identification of the fingertips position was presented [18]. The hand gesture image is first extracted, after which contour information is removed to give a sketch of the hand with palm and finger location highlighted. From the presented results, the proposed method was found to accurately determine hand gestures in realtime with distances of 0.6m, 1.0m, 1.5m, and 2.0m from the sensor. The gesture recognition rate was between 96.67% – 98.88% for the six gestures.

A multimodal hand gesture recognition system comprising of the combined use of a single Inertial Measurement Unit (IMU) and pressure-based transducers placed on the wrist has been evaluated [19]. The multimodal approach eliminates the problem of using a single IMU as a hand gesture detection system due to arm movements. The system comprised of an IMU unit (3-axis accelerometer, and 3-axis gyroscope) and acoustic signals (10 microphones) placed on the wrists of 10 subjects in order to detect and track thirteen common gestures for hand signaling.

A method for detecting gestures of the hand [20] using soft skin tactile sensors to measure wrist movements instead of using the more conventional methods involving IMUs, EMG, and FSR has been implemented. Eight specific gestures comprising the movement of the wrist could be distinguishable using the novel sensor. Filtering techniques were used to filter out pulse rate noise, which could be misinterpreted as a movement of the wrist. The verification of the performance of the proposed sensor involved comparison tests between the sensor with an EMG and FSR sensor was conducted. 97–99% accuracy of the stain sensor was obtained compared to the EMG and FSR sensors suggesting the superiority of the novel gesture recognition method with the tested conventional methods. Other conventional methods that have been used to control soft actuators other than hand gestures [21] include eye movement and tracking [22].

This section provided an introduction with background material on soft robotics control. It also describes hand gesture sensors and systems. Section two provides a description of the hand gesture control strategy employed to control the contraction and expansion of the soft robotic artificial muscle. Results and discussion are offered in Section three. The last section, Section four, provides a conclusion for the work.

2. Hand Gesture Control Design of Soft Robotic Muscle

In this work, a hand gesture system that controls a soft robotic muscle for contraction and expansion actuation is implemented. The system design comprises of the hardware and software design. The hardware system consists of a leap motion sensor to detect the hands and the soft robotic muscle. The software system is made up of the control algorithms that detect the gestures responsible for contracting and expanding the soft muscles.

2.1. Leap Motion Sensor Interface

The Leap Motion device continually detects hand, finger, or tool-like objects inside its field of view when powered ON via connection to a computer. Three infrared LEDs and two infrared cameras within the device facilitate hand motion tracking. The device is able to track any object located with its 150⁰ viewing area with a distance of 0.50m from it in the upward direction and 0.25m from it in other directions [23]. Hand and fingers can be detected by the Leap using the Software Development Kit (SDK) libraries. The leap sensor is displayed in Figure 1.

The leap motion sensor tracks the movement of the hands within its FOV by measuring distances in three directions, called the X, Y, and Z directions. The X distance is changed when a positioned hand moves left or right, the Y distance changes with a hand moving up and down from the sensor, while the Z distance changes with a positioned hand moving forward or backward.

Figure 1: The Leap Motion sensor powered via USB connection to a computer; the green status LED is ON and the three infrared LEDs come ON as shown.

2.2. Soft Robotic Muscle Interface

The Soft Robotic Muscle (SRM) used is the same described in [24]. The detailed fabrication process, together with mathematical equations describing its model, is presented in [11]. The SRM muscle is comprised of soft modular units arranged in series to improve the rate of shortening. The SRM muscle contracts upon the application of vacuum pressure and expands to its original length when positive pressure is applied to its inner chambers. The muscle is also able to bend bi-directionally by controlled variation of air and vacuum pressure within its inner walls. The SRM muscle is shown in Figure 2. The parameters of muscle include the wedge angle, θ = 60⁰, wedge length, l₁=20mm, number of modular units, n=9, air channel height, h_c=40mm, wall thickness, t=2.5mm, and muscle length, l=160mm. The soft robotic muscle can contract and expand partially. The zero position of the SRM is at its normal length of 160mm; its contacted length is 104mm while its expanded length is 170mm.

2.3. Integrated Hand Gesture and Soft Robotic Interface

The software aspect of the system involves designing the hand gesture system responsible for contracting and expanding the SRM muscle. This process first begins with detecting the two hands and then implementing the software that measures the distance between both hands. Two gestures which we term the “contracting” and “expanding” gestures are implemented. The two gestures are described as follows:

Figure 2: SRM Muscle having a total length of 160mm and defining angle of wedge as 60⁰, wedge length is 20mm, number of wedges is 9, the height of air channel equals 40mm and wall thickness is 2.5mm

• Contraction gesture: This gesture is represented by bringing both hands together, i.e., decreasing the distance between the left and right hands.
• Expansion gesture: This gesture is represented by bringing both hands farther apart, i.e., increasing the distance between the left and right hands.

Figure 3: From (a), when the contraction gesture is executed, the SRM muscle contracts; From (b), when the expansion gesture is executed, the SRM muscle expands to its original length.

The two gestures were developed because they follow the natural motions of how a human will contract and expand the muscle as if holding it physically without an actuating source. Therefore, when the contraction gesture is executed, control commands are sent to the SRM muscle to contract it. Conversely, when the expansion gesture is performed, a control command from the leap sensor interface is sent to the electronics hardware [4] to cause the expansion of the soft robotic muscle. The rationale behind using a leap sensor is so that expansion and contraction control of the SRM muscle can be implemented over a remote distance. This gesture interface allows for a master-slave control where the master is the movement of the two hands, and the slave is the soft robotic muscle that follows the motions of the master. The pictorial representation of the system design is seen in Figure 3.

The software flowchart for implementing the integrated system comprising hand gesture detection and soft robotic muscle control is shown in Figure 4. As seen from the Figure, the distance between the two hands is measured; if this distance is decreasing, i.e., its value is reducing, that means that a contraction gesture is being executed and a command is sent to the SRM muscle to contract it. If the measured distance is increasing, i.e., its value is getting larger from the previous timestamp, which means that an expansion gesture is being executed, and a command is sent to the SRM muscle to expand it to its normal resting length. If there is no change in the distance measured between the consecutive timestamps, then no gesture is being executed, and the SRM muscle does not change in length.

Figure 4: Flow chart for the control of soft robotic artificial muscle with hand gesture signals using the leap motion sensor

3. Results and Discussion

The controller hardware setup consisting of the electronic circuit, leap motion sensor, SRM Muscle, supply pumps for negative and positive pressurized air is shown in Figure 5. The detailed explanation of the experiment and description of the used circuit board and controller can be found in [3][4]. Through the USB connection of the Leap device to a computer, interpreted gestures commands are sent to the controller hardware with the microcontroller and leap connected to the same computer interface for the transfer control commands.

Figure 5: The hardware controller for the hand gesture system consisting of the controller circuitry, vacuum pump for contracting the muscle and air pump for expanding the SRM Muscle

Figure 6 shows absolute vacuum pressure and contraction ratio as a function of during contraction. A maximum absolute vacuum pressure of -20kPa is observed for a maximum contraction ratio of 0.35. The measured contraction time is 4 seconds

Figure 6: Contraction of the SRM Muscle, the contraction ratio moves from 0 to 0.35 of the muscle length in 4s, the vacuum pressure is shown to move from 0kPa to -20kPa during contraction

When the SRM Muscle is undergoing contraction, contraction ratio, and absolute vacuum pressure move from zero to the contraction ratio of 0.35 and vacuum pressure of -20kPa, respectively. Figure 7 shows absolute vacuum pressure and contraction ratio as a function of during expansion. The contraction ratio moves from 0.35, which is the contracted ratio to 0, which is the original muscle resting position in 6 seconds. The vacuum pressure is shown to move from -20kPa to -0kPa during expansion.

Figure 7: Expansion of the SRM Muscle, the contraction ratio moves from 0.35 which is the contracted form to 0 which is the original muscle resting position in 6s, the vacuum pressure is shown to move from -20kPa to -0kPa during expansion

When the SRM Muscle is undergoing expansion, contraction ratio, and absolute vacuum pressure move a contraction ratio of 0.35 and vacuum pressure of -20kPa to zero as shown. Through the use of the leap sensor to control the soft muscle, closed-loop control operation of the SRM muscle opens the possibility to obtain a specified muscle contraction. To achieve the desired operation, visual information from the users’ eyes measures the appropriate contraction or expansion length while control signals are sent to contract and expand the SRM muscle.

The novelty of the proposed solution is the control of artificial muscles for expansion and contraction purposes via hand gestures. Possible areas of application include the use of the soft muscles for rehabilitation purposes and the combined system for game purposes – a physiotherapy gaming device can be developed to exercise the hands and fingers of individuals that need to strengthen the muscles of the hands and fingers.

The demonstration of the control of the Soft Robotic Artificial Muscle with a hand gesture using the leap motion sensor is shown in Figure 8. A user performs the contraction and expansion gesture using the Leap motion sensor as a hand gesture interface to create contraction and contraction movements of the Soft Robotic Muscle. As shown, both hands must be within the 150⁰ field of view of the leap sensor in order that the accurate distance between the hands is measured. Also, both hands should be within 0.50m from the sensor in the upward direction and 0.25m from the sensor in the x and y direction.

Figure 8: A user performs the contraction and expansion gestures using the Leap motion sensor as hand gesture interface to create contraction and contraction movements of the Soft Robotic Muscle

4. Conclusions

A control design strategy for the control of artificial muscles using hand gesture signals have been proposed and implemented through the regulation of air pressure within the inner chambers. The artificial muscle can be made to expand and contract as a result of interpreted gesture commands from the leap motion device. The advantage of the employed hand gesture control compared is that it provides a more natural interface for the regulation of air pressure within the artificial muscle through the use of electronic and automatic control. Applications of this work include teleoperated control of a soft robotic muscle using the natural motions of how the muscle will be contracted and expanded if it is physically held.

Conflict of Interest

The authors declare no conflict of interest.

Acknowledgment

The authors acknowledge the management of Covenant University, Ota, Nigeria, for the publication support fund.

1. V. Oguntosin, A. Akindele and E. Oladimeji, “Gesture-Based Control of Rotary Pneumatic Soft Robot Using Leap Motion Controller” in 6th International Conference on Soft Computing & Machine Intelligence(ISCMI), 2019, pp. 169-174, 2019, DOI: 10.1109/ISCMI47871.2019.9004295
2. F. Ilievski, A.D. Mazzeo, R.F. Shepherd, X. Chen and G.M. Whitesides, “Soft Robotics for Chemists” Angewandte Chemie,, 123(8): 1930-1935. 2011, DOI: 10.1002/anie.201006464
3. V. Oguntosin, S. Nasuto, and Y. Hayashi, “Embedded Fuzzy Logic Controller for Positive and Negative Pressure Control in Pneumatic Soft Robots” in UKSim-AMSS 19th International Conference on Computer Modelling & Simulation, 2017, pp. 63–68. 2017, DOI: 10.1109/UKSim.2017.41
4. V. Oguntosin, S. J. Nasuto and Y. Hayashi, “A Compact Low-Cost Electronic Hardware Design for Actuating Soft Robots”, International Journal of Simulation Systems, Science & Technology (IJSSST), 16(3), pp. 1–11, 2016, DOI: 10.5013/IJSSST.a.16.03.06
5. S. T. Mahon, A. Buchoux, M. E. Sayed, L. Teng and A. A. Stokes, “Soft Robots for Extreme Environments: Removing Electronic Control”, IEEE International Conference on Soft Robotics, pp. 782-787, 2019. DOI: 10.1109/ROBOSOFT.2019.8722755
6. P.D.S.H. Gunawardane, R..E..A. Pallewela and Nimali T. Medagedara, “Tele-Operable Controlling System for Hand Gesture Controlled Soft Robot Actuator” in 2nd IEEE International Conference on Soft Robotics (RoboSoft), 2019, DOI: 10.1109/ROBOSOFT.2019.8722756
7. P.D.S.H. Gunawardane, R.E.A Pallewela, A.P.T.D. Pathirana and N. T. Medagedara, “Kalman Filter Based Approach to Overcome Elastic Saturation and Pneumatic Disturbances in Tele-Operated Soft Actuators” in Advances in Manufacturing Technology XXXIII, 2019, DOI:10.3233/ATDE190023
8. A. Ataka, T. Abrar, F. Putzu, H. Godaba, and K. Althoefer, “Model-based Pose Control of Inflatable Eversion Robot with Variable Stiffness”, IEEE Robotics and Automation Letters, 2020, DOI: 10.1109/LRA.2020.2976326
9. V. Oguntosin, A. Akindele,, O. Alashiri,. “Vision Algorithms for Sensing Soft Robots”, Journal of Physics: Conference Series, 1378 (3), 2019, DOI: 10.1088/1742-6596/1378/3/032102
10. J. Y. Loo, K. C. Kong, C. P. Tan1 and S. G. Nurzaman “Non-linear System Identification and State Estimation in a Pneumatic Based Soft Continuum Robot”, IEEE Conference on Control Technology Applications, 2019, DOI: 10.1109/CCTA.2019.8920693
11. V. Oguntosin, A. Akindele, “Design and Characterization of Artificial Muscles from Wedge-Like Pneumatic Soft Modules.” Sensors and Actuators A: Physical, 2019, DOI: 10.1016/j.sna.2019.07.047
12. K. Han, N. Kim, and D. Shin, “A Novel Soft Pneumatic Artificial Muscle with High-Contraction Ratio”, Soft Robotics, 00 (00), 2018, DOI: 10.1089/soro.2017.0114
13. S. M. Mirvakili and I. W. Hunter, “Artificial Muscles: Mechanisms, Applications, and Challenges” Advanced Materials, 1704407, 1-28, 2017, DOI: 10.1002/adma.201704407
14. D. Yang, M. S. Verma, J. So, B. Mosadegh, C. Keplinger, B. Lee, F. Khashai, E. Lossner, Z. Suo, and G. M. Whitesides, “Buckling Pneumatic Linear Actuators Inspired by Muscle”, Advanced Materials Technologies, 1600055, 1-6, 2016, DOI: 10.1002/admt.201600055
15. T. Abrar, F. Putzu, J. Konstantinova and K. Althoefer, “EPAM: Eversive Pneumatic Artificial Muscle” in 2nd IEEE International Conference on Soft Robotics (RoboSoft), 2019, DOI: 10.1109/ROBOSOFT.2019.8722787
16. A. Ahmed et al., “Modeling and Simulation of Office Desk Illumination Using ZEMAX,” in 2019 International Conference on Electrical, Communication, and Computer Engineering (ICECCE), 1–6, 2019. DOI: 10.1109/ICECCE47252.2019.8940756
17. Y. Sang, L. Shi, and Y. Liu, “Micro Hand Gesture Recognition System Using Ultrasonic Active Sensing”, IEEE Access, 6, pp. 49339 – 49347, 2018, DOI: 10.1109/ACCESS.2018.2868268
18. X. Ma and J. Peng, “Kinect Sensor-Based Long-Distance Hand Gesture Recognition and Fingertip Detection with Depth Information”, Hindawi Journal of Sensors, 5809769, 2017, pp. 1-9, DOI: 10.1155/2018/5809769
19. N. Siddiqui, R. H. M. Chan, “Multimodal hand gesture recognition using single IMU and acoustic measurements at wrist” PLoS ONE 15(1): e0227039. 2020, pp. 1-12. DOI: 10.1371/journal.pone.0227039
20. B. Sung-Woo and L. Seok-Pil, “Implementation of Hand Gesture Recognition Device Applicable to Smart Watch Based on Flexible Epidermal Tactile Sensor Array”, MDPI micromachines, 10, 692, 2019, pp. 1-15. DOI:10.3390/mi10100692
21. V. Oguntosin and A. Abdulkareem, “Design of a pneumatic soft actuator controlled via eye tracking and detection”, Heliyon. 6(7):e04388, 2020, DOI: 10.1016/j.heliyon.2020.e04388
22. V. Oguntosin and A. Abdulkareem, “Hand gesture control and design of a rotary pneumatic soft actuator using leap motion sensor”. Int J Intell Robot Appl, 2020, DOI: 10.1007/s41315-020-00140-5
23. B. Sanders, Mastering Leap Motion. Packt Publishing, 2014
24. V. Oguntosin, “Development of Soft Modular Robotics,” Ph.D Thesis, University of Reading, 2017. ISNI: 0000 0004 6499 6177

(Click Here)

(Click Here)