Learning How to Use a Supernumerary Thumb

Ali Seçkin Kaplan

, Emre Akın Ödemiş

, Emre Doğan

, Mehmet Orhun Yıldırım

, Youness Lahdili

Amr Okasha

and Kutluk Bilge Arıkan

1,3 a

Department of Mechanical Engineering, TED University, Ankara, Turkey

Department of Mechanical Engineering, Middle East Technical University, Ankara, Turkey

Neuroscience and Neurotechnology Center of Excellence, NÖROM, Ankara, Turkey

okasha.amr@metu.edu.tr

Keywords: Motor Learning, Supernumerary Limb, Mirror Paradigm, Leader-Follower Modality.

Abstract: This study presents a novel system consisting of a supernumerary robotic thumb and a virtual reality-based

mirror paradigm in a leader-follower mode. As the extra thumb skeleton, a planar robotic mechanism with

two degrees of freedom is utilized. The experimental setup poses the task of acquiring proficiency in

controlling the supernumerary second thumb throughout a five-day duration of engaging in the leader-

follower game. There is evidence that after five days of practice, a subject's motor performance improves and

motor variability decreases.

1 INTRODUCTION

Supernumerary Robotic Limbs (SRLs) are at the

forefront of human-robot cooperation and integration

research. They are designed to increase job efficiency

and safety, augment different human body functions,

and restore certain capabilities for persons with

disabilities (Yang et al., 2021; Tong & Liu, 2021).

Supernumerary limbs include robotic arms, robotic

legs, and supernumerary robotic fingers (SRFs).

Supernumerary robotic fingers are often intended to

serve two functions. The aforementioned goals are to

supplement a healthy person's hand (Ariyanto et al.,

2017; Hussain et al., 2017) or to compensate for lost

functions caused by severe disorders like strokes

(Salvietti et al., 2021; Lee et al., 2021). The majority

of supernumerary robotic fingers are meant to aid in

the grasping of daily objects, while some are aimed to

better a human-performed task, such as playing the

piano (Cunningham et al., 2018). The potential

applications of a supernumerary robotic finger design

may be constrained in comparison to other forms of

supernumerary robotic limbs, primarily due to

limitations imposed by size and weight

considerations. Most supernumerary robotic fingers

are designed with specialization in mind, focusing on

https://orcid.org/ 0000-0003-2093-1577

the accomplishment of a singular task, such as

grasping.

The objective of this study is to establish a

framework for examining the enhancement of manual

dexterity in healthy individuals through the utilization

of a robotic supplementary thumb as an additional

appendage. We considered a scenario in which,

instead of using the original thumb to push or

compress an elastic object, a supernumerary robotic

thumb is employed to execute the same action. In the

meanwhile, the original thumb is utilized to

manipulate a joystick. The proposed novel system

combines SRF with a virtual reality-based leader-

follower game. The interaction force due to the

compression of the spring is utilized in the virtual

reality environment to make the follower’s avatar

track the leader. While following the leader, the

subject is instructed to shoot a laser gun at a target on

the leader in order to score points by manipulating the

joystick. The game was played by a participant on a

daily basis, completing 15 rounds each day for a

consecutive period of 5 days. The performance was

analyzed within the context of motor learning. The

score attained during the phases of habituation,

learning, and retention are used to characterize motor

learning. Additionally, motor variability is used to

assess motor learning.

Kaplan, A., Ödemi¸s, E., Do

gan, E., Yıldırım, M., Lahdili, Y., Okasha, A. and Arıkan, K.

Learning How to Use a Supernumerary Thumb.

DOI: 10.5220/0012240000003543

In Proceedings of the 20th International Conference on Informatics in Control, Automation and Robotics (ICINCO 2023) - Volume 1, pages 489-494

ISBN: 978-989-758-670-5; ISSN: 2184-2809

489

2 EXPERIMENTAL SYSTEM

The experimental system is developed to study how

we learn to utilize the supernumerary thumb while

playing a virtual reality-based leader-follower game.

Figure 1 shows the components of the system and

Figure 2 illustrates typical data monitoring during the

rounds. For each round, the pedal and button force

inputs, joystick firing inputs, leader and follower

positions, and scores are all recorded. The system's

components are discussed in the following sections.

Figure 1: Overall system architecture.

Figure 2: Screen shots of a typical round.

2.1 Two Degrees of Freedom (DOF)

Robotic Thumb

The chosen topology for the supernumerary thumb is

a 2 degree-of-freedom planar robot arm, as depicted

in Figure 3. The mechanism offers alternative

learning opportunities by presenting challenges in

both task space and joint space mapping of pedal

inputs. The selection of working space is undertaken

to uncover potential avenues for exploration during

the training process, Figure 4. Determining link

lengths is achieved through simulations that aim to

reach the desired exploration space while considering

feasible transmission angles. Transmission angles

are necessary for maintaining the desired interaction

forces, such as when exerting pressure on the red

button.

Figure 3: Two degrees of freedom planar mechanism as the

second thumb mechanism.

Figure 4: Screen shot from the kinematic animation of the

mechanism tracing a circular path in workspace. Tip of the

mechanism is point D.

Figure 5: Solid model of the synthesized mechanism with

the actuators.

-100 -50 0 50

[m]

-20

100

120

[m]

[mm]

ICINCO 2023 - 20th International Conference on Informatics in Control, Automation and Robotics

490

2.2 Hand Interface and Pedals

The objective of this study is to acquire the skill of

pressing the red buttons depicted in Figures 2 and 6

using the supernumerary thumb. This action will

result in the generation of an interaction force, which

is produced by compressing a linear spring. The force

sensor detects the generated interaction force, which

is then processed by the microcontroller board. In the

game, the follower avatar utilizes it as a means of

generating upward propulsion. Furthermore, the input

is transformed into a linear mapping that is then

converted into a pulse width modulated (PWM)

signal. This signal is subsequently utilized to drive a

vibrating motor, which in turn delivers haptic

feedback to the player's wrist.

Figure 6: Pinching interface (in yellow circle) and pedals

(in green rectangle).

The experimental setup that has been developed

consists of two pedals, one for the left foot and

another for the right foot, as depicted in Figure 6. A

force sensor, specifically a force-sensing resistor, is

installed on each pedal. To address the issue of

parasitic noise originating from the lengthy 1.5-meter

wires connecting the sensors to the pedals, as well as

for signal conditioning purposes, an operational

amplifier (op-amp) is employed in a voltage-follower

configuration. This configuration is placed between

each sensor and its corresponding analog-to-digital

converter (ADC) input. The microcontroller employs

a linear mapping technique to convert pedal forces

into two pulse width modulation (PWM) signals.

These signals serve as reference position inputs for

the two mini servomotors that drive the two degrees

of freedom (2-DOF) thumb mechanism, as shown in

Figure 5. The position control systems that are

inherently present within the RC-servo motors are

exclusively employed.

2.3 Real-Time Control System

In this study, i.MX RT1024-EVK microcontroller

board is used as the real-time control hardware,

Figure 7.

Figure 7: Real-time control hardware.

Figure 8: Real-time control system built in Simulink and

runnig on MXP board – part 1 on the left, part 2 on the right.

The software architecture for this project has been

developed using MATLAB/Simulink. The system

can be conceptually separated into two primary

components: (1) the module responsible for acquiring

data and transmitting actuation signals to and from

the external environment, and (2) the module that

executes game control algorithms and manages the

physics of the system, as depicted in Figure 8.

The first part utilizes the MBDT toolbox,

providing access to the peripherals of the i.MX

RT1024-EVK, a robust microcontroller board

operating at a frequency of 500MHz. This board is

capable of supporting real-time scheduling. The ADC

Learning How to Use a Supernumerary Thumb

491

(Analog-Digital Converter) blocks and PWM (Pulse

Width Modulation) blocks were utilized to

successfully interface with the sensors and control the

actuation process. The connection between the two

stages of blocks is facilitated by a pipeline consisting

of commonly used Simulink gain, bias, numerical

filter, and saturation blocks. It is important to

acknowledge that the determination of the values in

these blocks was primarily based on empirical

methods, taking into consideration the mechanical

limitations of the servo motors. Significantly, every

MBDT block offers a means of accessing

MCUXpresso Config, an independent software tool

designed for the purpose of advanced configuration

of the peripherals of i.MX RT1024-EVK. It is at this

stage that we can configure the ADC resolutions,

PWM frequency, initial duty cycle, pin assignments,

and various other parameters that pertain to the low-

level layer of the microcontroller.

The subsequent component of our architectural

design is responsible for managing the underlying

principles that dictate the dynamics of the "leader-

follower" game. This includes the application of

Newtonian mechanics, the establishment of scoring

mechanisms, the implementation of a "leader" signal,

the establishment of boundaries, and the enforcement

of a time limit of 90 seconds for each iteration. The

aforementioned values are encoded and parsed into

packets, which are subsequently transmitted over the

User Datagram Protocol (UDP) to an IP address. This

IP address corresponds to the location where the

"leader-follower" game is actively listening. It is

important to note that this game, developed using the

Unity engine, operates concurrently on a distinct

instance that is external to MATLAB. The initial

component of the architecture, which is solely

concerned with sensing and actuating, operates at a

sampling frequency of 100 Hz. However, the

subsequent component responsible for executing

intricate algorithms operates at a higher frequency of

1 KHz.

The initial segment is executed on the

microcontroller board depicted in Figure 7, while the

subsequent segment is executed on the primary

computer that hosts MATLAB. The utilization of the

PIL (Processor-in-the-Loop) framework of the i.MX

RT1024-EVK board enables the achievement of this

capability. It is considered the recommended

approach for co-simulation of this nature, wherein

each processing platform assumes responsibility for

its respective native blocks.

2.4 Leader-Follower Game

The mirror paradigm's leader-follower modality is

implemented within a game format, utilizing the

Unity engine. The leader, equipped with a target pane,

is executing a uniaxial motion in both the upward and

downward directions. The objective of the follower

positioned on the left is to consistently track the

movements of the leader and skilfully engage the

target by discharging the laser weapon, thereby

acquiring points. Figure 10 presents a screenshot

from the implementation.

The leader's motion is made up of the sum of three

sine waves, which provides a complex reference for

the follower. However, such references have been

proven in the literature to be implicitly learned (Pew,

1974; Polat, 2022).

The force resulting from the interaction between

the second thumb and the button in the hand interface

is utilized as the vertical force for propelling the

avatar of the follower in an upward direction. In order

to induce downward movement of the avatar, it is

necessary to decrease the interaction force to a

magnitude that is lower than the weight of the avatar.

This reduction in force will generate a net force in the

negative y direction.

Figure 9: Leader-follower game in Unity.

2.5 Experimental Protocol

The participant, a 23-year-old male, was instructed to

complete the game challenge in the conventional

manner, excluding the use of the sixth finger, and

instead utilizing their own thumb. Then, the

individual puts on the robotic extra thumb on their

right hand and engages in the game as a daily regimen

consisting of 15 rounds, each lasting 90 seconds, over

a span of 5 consecutive days.

On a daily basis, the experimental procedure

consisted of a series of rounds. For each day, the

initial 2 rounds were designated as habituation runs,

ICINCO 2023 - 20th International Conference on Informatics in Control, Automation and Robotics

492

followed by 11 rounds of learning trials, and

concluding with 2 rounds specifically designed as

retention runs. The leader's pattern is the same

throughout the habituation and retention rounds.

During the training sessions, the leader moved in the

habituation pattern's mirror symmetry.

3 RESULTS AND DISCUSSION

The score of the subject is presented in Figure 11. The

black bar shows the performance without extra

robotic limb.

Figure 10: Average scores.

Daily, the subject's performance improves when

the scores of the habituation and retention trials are

compared. When the habituation scores of all sessions

are compared, there is a positive trend. However,

when learning and retention performance for the

whole experimental time is examined, we do not

detect a comparable tendency. As a result, it is

difficult to claim that a skill can be learned in 5 days

of training.

Additionally, we examined the motor variability

of the supernumerary thumb control. During motor

learning, it is known that the exploration phase must

be diminished, and the exploitation phase must be

prolonged. This reduces the motor variability of the

subject’s control action (Dhawale et al., 2017). We

discussed the trajectory of the tip point D of the

robotic thumb mechanism to evaluate the motor

variability. The trajectories observed during the final

round of training on the first and fifth days are shown

in Figure 12. When the trajectories are compared, the

reduced motor variability is interpreted as a sign of

motor learning progress.

Figure 11: Trajectories of point D – Top: Last round of

day 1, Bottom: Last round of day 5.

4 CONCLUSIONS

A novel system has been developed to investigate the

process of motor learning during practice involving

the control of a supernumerary second thumb. The

setup allows for the investigation of learning in either

the joint or task space of the supernumerary limb.

Further research will investigate the variations in

retention and transfer of motor learning that

correspond to the learning spaces.

In this study, the motor learning task can be

categorized into various regimens. The initial subtask

involves positioning the robotic finger's tip in

proximity to the red button. Subsequently, the finger

Sco

Learning How to Use a Supernumerary Thumb

493

must acquire the capability to apply force to the

button to propel the avatar in the game. Lastly, the

finger must possess the ability to anticipate the

movements of the leader in order to effectively pursue

the leader avatar. In addition, it is imperative to

acquire the skill of accumulating points via the act of

firing the laser beam by utilizing the organic thumb

in conjunction with the robotic thumb. Hence, the

experimental system can be utilized to investigate

diverse characteristics of motor learning. It is

important to assess the motor performance in each

phase separately as well.

For the first regime, where it is learned to navigate

the robotic finger's tip from its initial position to the

button location, motor performance increases over

repeated rounds. Motor variability in this phase is

reduced as shown in Figure 12. Nevertheless, there is

criticism regarding the necessity for additional rounds

and extended periods of time to acquire proficiency

to accumulate scores as following the leader avatar.

This inquiry serves as the central focus of our ongoing

research, namely, the investigation into the methods

by which motor learning can be facilitated or

enhanced. This study demonstrates the fundamental

integration of a robotic finger and a virtual reality

system using the mirror paradigm. The subsequent

stage involves the development of shared control

architectures with the aim of facilitating motor

learning. Furthermore, the utilization of force fields,

haptic interaction, and disturbances will be employed

to augment the process of motor learning (Özen et al.,

2021; Brookes et al., 2020). In addition to the

kinematic and kinetic data, neuroplasticity will be

evaluated by processing the EEG data. To evaluate

motor learning concretely, nonlinear measures will be

used.

REFERENCES

Ariyanto, M., et al. "Development of low cost

supernumerary robotic fingers as an assistive device,"

2017 4th International Conference on Electrical

Engineering, Computer Science and Informatics

(EECSI), Yogyakarta, Indonesia, 2017, pp. 1-6

Brookes J, Mushtaq F, Jamieson E, Fath AJ, Bingham G,

Culmer P, et al. (2020) Exploring disturbance as a force

for good in motor learning. PLoS ONE 15(5): e0224055

Cunningham, J., Hapsari, A., Guilleminot, P., Shafti, A.,

& Faisal, A. A. (2018). The supernumerary robotic

3rd thumb for skilled music tasks. 2018 7th IEEE

International Conference on Biomedical Robotics and

Biomechatronics (Biorob).

Dhawale, A. K., et al. (2017). The Role of Variability in

Motor Learning, Annu. Rev. Neurosci. 40:479–98

Hussain, I., Salvietti, G., Spagnoletti, G., Malvezzi, M.,

Cioncoloni, D., Rossi, S., & Prattichizzo, D.

(2017). A soft supernumerary robotic finger and mobile

arm support for grasping compensation and hemiparetic

upper limb rehabilitation. Robotics and Autonomous

Systems, 93, 1–12.

Lee J., et al. "Assistive supernumerary grasping with the

back of the hand," 2021 IEEE International Conference

on Robotics and Automation (ICRA), Xi'an, China,

2021, pp. 6154-6160.

Özen Ö, Buetler KA and Marchal-Crespo L (2021)

Promoting Motor Variability During Robotic

Assistance Enhances Motor Learning of Dynamic

Tasks. Front. Neurosci. 14:600059.

Pew, R. W. (1974). Levels of analysis in motor control,

Brain Research, vol. 71, no. 2-3, pp. 393–400.

Polat, Y. E. (2022), Development of a human-animation

interaction system to evaluate synchronization,

prediction and implicit learning, M.Sc. Thesis, TED

Universtiy.

Salvietti, G., Franco, L., Tschiersky, M., Wolterink, G.,

Bianchi, M., Bicchi, A., Barontini, F., Catalano, M.,

Grioli, G., Poggiani, M., Rossi, S., & Prattichizzo, D.

(2021). Integration of a passive exoskeleton and a

robotic supernumerary finger for grasping

compensation in chronic stroke patients: The softpro

wearable system. Frontiers in Robotics and AI, 8, 1–9.

Tong Y. and Liu J., "Review of Research and Development

of Supernumerary Robotic Limbs," in IEEE/CAA

Journal of Automatica Sinica, vol. 8, no. 5, pp. 929-952,

May 2021.

Yang, B., et al. "Supernumerary Robotic Limbs: A Review

and Future Outlook," in IEEE Transactions on Medical

Robotics and Bionics, vol. 3, no. 3, pp. 623-639, Aug.

2021.

ICINCO 2023 - 20th International Conference on Informatics in Control, Automation and Robotics

494