A Digital Hand to Mimic Human Hand in Real Time Operation

Making of Digital Finger with Partial Soft Skin and Rigid Bone

Hiroshi Hashimoto

, Sho Yokota

, Daisuke Chugo

and Kaoru Mitsuhashi

Industrial Technology Graduate Course, Advanced Institute of Industrial Technology, Tokyo, Japan

Department of Mechanical Engineering, Toyo University, Saitama, Japan

Department of Human System Interaction, Kwansei Gakuin University, Hyogo, Japan

Department of Mechanical Engineering, Tokyo University of Technology, Tokyo, Japan

Keywords: Digital Hand, Hand Anatomy, Mimic, Real Time Operation, Soft Skin.

Abstract: This paper presents a digital hand which is a type of model to mimic human hand operation in real time

operation. Human hand performs various difficult tasks in daily life and shows dexterous operation to use

tools or equipment, because it has numerous degree of freedom (DoFs) of finger joints and soft skin. To

realize the mimic of human hand operation in real time operation to overcome the problems such as high

DoFs, soft skin. We have developed the digital hand whose input to control hand posture is obtained from a

hand posture sensor and soft skin is designed as mesh structure. Here, the way to define parameters of mesh

structure is discussed. We demonstrate the simulation of the digital hand model and examine how the model

is able to mimic the motion of human hand.

1 INTRODUCTION

This paper presents a digital hand with soft skin

which is a type of model to mimic human hands in

real time operation.

Human hand performs various difficult tasks in

daily life and shows dexterous operation to use tools

or equipment, because it has numerous degree of

freedom (DoFs) of finger joints more than 22 DoFs

(Chao et al.,1989), (Kapandj, 2005). There are many

types of grasp such as power grasps, precision

grasps and miscellaneous grasps, and each types is

also divided into many various hand postures

(Edwards and Buckland,2002). These hand postures

can be made by the hand’s DoFs, basically the

posture of holding and arch ensure the various hand

posture. However, a study on dynamical operation of

hand using objects has not been made in the field of

anatomy, but only on grasping which shows static

situation to fixe objects.

On the other hand, the previous studies on digital

hand for robotics or CG (computer graphics) have

been developed very well. In the early period of

robot hand researches, its objective is to realize

stable grasping objects based on the theoretical

aspects (Nguyen,1986), (Yoshikawa,1996). Hence,

these consideration merely focused on the stable

grasping geometrically, not consideration of the

human like grasping/operations. Up to date,

dexterous grasping of robot hands have been

developed (Mouri et al., 2005), (Ishihara et al.,

2006), (Inoue and Hirai, 2009), it remains difficult to

realize dexterous manipulations as seen in actual

human hand operations.

In researches of CG, considering muscular,

freedom of joints and tendons, a precise digital hand

to mimic human hand has been tried to be made

(Lee and Kunii, 1995), (Sueda et al., 2008), (Endo

et al., 2008), (Mulatto et al., 2013). Its objective is to

evaluate product designs when it grasp an object. So,

the discussions were made about parameter

identifications of the digital hand, and grasping

situation on contact points between the digital hand

and object. These considerations focus on the static

states while grasping statically, not on dynamical

states such as pen spinning.

Here, the human hand is able to manipulate

objects dexterously described below. For example,

the human hand can pass a tool from appropriate

fingers to the other with only finger, this can be seen

such that a skilled engineer operates a driver with

one hand or a surgeon shows the neat exactness of

the surgeon's knife. These operations lead to a rapid

use of the instrument or create much valuable things.

Hashimoto, H., Yokota, S., Chugo, D. and Mitsuhashi, K.

A Digital Hand to Mimic Human Hand in Real Time Operation - Making of Digital Finger with Partial Soft Skin and Rigid Bone.

DOI: 10.5220/0005749900970102

In Proceedings of the 11th Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications (VISIGRAPP 2016) - Volume 1: GRAPP, pages 99-104

ISBN: 978-989-758-175-5

The dynamical operation of the digital hand has

been slightly considered in (Hashimoto et al., 2013),

(Hashimoto et al., 2014), not seen in the other

studies. In the researches, the body of the digital

hand was made from rigid body. Human hand is

covered by soft skin, which is deformable while

operating an object. Therefore, contact region

touched with the object is area not point for rigid

skin, so the dynamic relationship on the contact

region also becomes complex. This means the real

time operation of the digital hand requires numerous

computational load. Because the way of moving to

operate a thing dynamically is very enormous, the

programming to simulate all patterns of hand

postures is very troublesome.

To overcome the problems such as the

computational load with soft skin and real time

operation for various operation cases, first, we

propose a digital hand structure based on anatomy,

here, the reduced DoFs of joints is introduced to

decrease the computational load. Second, the

operation system with hand-posture sensor

LeapMotion (LeapMotion, 2015) and virtual

physical space which is realized with Bullet Physics

(BulletPhysics, 2015). To confirm the effectiveness

of the digita hand system, some operations are

examined.

2 SKELETON MODEL BASED ON

ANATOMY

The hand skeleton model is shown in Figure 1 based

on anatomical and medical hand investigation

(Kapandj, 2008).

Figure 1: Hand skeleton structure.

In Figure 1, abbreviated label for joints have

following meanings (arranged in order from

proximal to distal extremity). CMC stands for the

carpometacarpal joint, MCP for the metacarpo-

phlangeal joint, PIP for the proximal interphalangeal

joint, and DIP for the distal interphalangeal joint.

Other joint labels of thumb are: TMC for the

trapeziometacarpal joint, MCP for the meta-

carpophlangeal joint, and IP for the interphalangeal

joint.

Degrees of freedom of each joint is

approximately equivalent to those of the actual

human hand, except the TMC joints because of

complexity of joint structure of the actual human

thumb.

The skeleton has five fingers, i.e., the thumb,

index finger, middle finger, ring finger, and little

finger. The base of these fingers in the hand

structure is the carpus underneath the metacarpal

bones, which lies between the palm and wrist. The

carpus consists of 8 bones in the actual human hand

but is approximated as two bones in the model: one

corresponding to trapezium at bottom of thumb, and

the other corresponding to other carpal bones except

trapezium bone (assembly of other carpal bones,

namely scaphoid, lunate, capitate, triquetrum,

pisiform, trapezoid, and hamate bones).

Although metacarpal bones are all in the palm in

an actual hand, they are all separated to allow

motion relative to each other, and connected to a

corresponding phalangeal bone of each finger.

Each finger (not including the thumb) is

composed of three bone links, called phalangeal

bones. Each neighbouring pair of bone links are

connected with a joint, i.e. a constraint that restricts

relative translational motion of bone links in

dynamics simulation. The DIP, PIP and IP has one

DoF, the MCP has two DoFs, the CMC has two

DoFs and the TMC has three DoFs. So, the total

DoFs of human hand is 30.

Here, the dexterous pose and motion of hand

should be kept in good condition such as the arches

(Kamakura et al., 1980), (Edwards and Buckland,

2002). So, the DoFs of thumb needs to consider very

carefully. The range of deviation of MCP joint of the

thumb is so small for abduction/adduction that it can

be usually neglected, and its DoFs can be

approximated as one.

Not to force the action of bending the fingers, but

to act for flexion/extension, there is an angular

constraint condition between the angle of DIP and of

PIP for each finger such as refs. (Chao et al., 1989);

(Ying et al., 2005).

() ()

angle DIP angle PIP

(1)

GRAPP 2016 - International Conference on Computer Graphics Theory and Applications

100

where the function angle( joint ) means the angle of

joint and

≃

(2)

Thus, the DoFs of DIP is able to be eliminated.

From the fact, the angles of DIP and PIP are

linearly independent, so one DoF for each finger can

be reduced. Figure 2 shows the structures of the

skeleton model under the consideration described

above, and the total DoFs is reduced to 16 from the

original DoFs. In this figure, the dependent joints

means that DIP is dependent joint to PIP as shown

Equations (1) and (2), so the DoFs of DIP is able to

be eliminated.

Figure 2: Structure model of joints and bones.

The reduced DoFs will be used in stable control

of the digital hand described later.

3 STRUCTURE OF DIGITAL

HAND

Human hands consist of rigid bone and soft skin

which forms deformable surface when hand

grasping objects. The rigid bone support to pick at a

small object and the soft skin is to prevent to drop an

object with friction on the contact area between

deformable skin and the object. Therefore, a

complex operation of human hand is realized. First,

the design of the soft skin is described, then its

connection with rigid bone is shown.

3.1 Design of Soft Skin

It is difficult to join soft skins to rigid bones in

making the digital hand by using Bullet Physics

which is one of physics engines. This is the reason

why each schemes of collision detections is different.

Now, we think about only skin of the size that

only comes in contact with the object, the

connection between the soft skin and the rigid bone

uses an anchor combination provided of Bullet

Physics, not direct combination. From this, the skin

is designed to be able to be installed onto the tip of

the finger and the middle of each bone. The shape of

the skin of fingertip is made by Blender (Blender,

2015) shown in Figure 3.

Figure 3: Making of soft skin of fingertip in Blender

window.

The number of mesh that makes up part of the

fingertip and the finger pulp hemisphere will

become too large, then the calculation time required

for collision detection will be enormous, thus it is

difficult to achieve real time operation. Based on the

trade-off of computational load and feasibility of the

dexterous hand operation, the selection of the

number is determined by trial and error.

Next, the figure of the soft skin is introduced into

soft body of Bullet Physics, and some parameters

(Table 1) of soft body should be defined to set up it.

Table 1: Parameters of soft skin in Bullet Physics.

kDP

Damping coefficient; damps forces acting on

soft body nodes to reduce their oscillation

over time. Imagine a mass hanging on a

spring. Range [0,1]

kDG and kLF

Drag and Lift coefficient; relating to

aerodynamics (Wikipedia_Lift,2015,

NASA,2015), Range

[0, ]+∞

kDF

Dynamic friction coefficient; just friction of

nodes against surfaces, as with rigid bodies.

Range [0,1]

kMT

Pose matching coefficient; be used with

setPose(bool, bool). Range [0,1]

kCHR, kKHR

and kSHR

Rigid, kinectic and Soft contacts hardness;

controling how strict any overlap between

the soft body and other types is treated.

Range[0,1]

However, the effective way to identify them have

not shown yet, so we investigated that human hand

played the bar spinning as a manipulation with the

high-speed camera (1000 fps ) as shown in Figure 4.

Observing the situation of the deformable skin by

A Digital Hand to Mimic Human Hand in Real Time Operation - Making of Digital Finger with Partial Soft Skin and Rigid Bone

101

investigating the figure, the parameters are adjusted

to show the similar situation of the deformable soft

skin.

(a) t = 0.0 sec

(b) t = 0.1 sec

Figure 4: Scene of bar spinning (1000 fps).

3.2 Connection between Soft Skin and

Rigid Bone

The bone is made from a cylinder rigid body and the

DoFs is in accordance with the joint of the hand

described the previous section. When the skin is

connected with the bone, the gap of the joint is

sufficient distance movable range of each joint to be

achieved.

We use Panda3D (Panda3D, 2015) to develop

the digital hand, which is a development platform

with Bullet Physics, described in Python language.

Figure5 shows one finger conducted by the design

described above by using Panda3D. In Figure 5, the

rigid bone and the partial soft skin are connected

with anchors. The reason why the partial soft skin is

adopted is to reduce the computational effort.

For through hand operation in real time, and is

focused on seeing the mechanical interaction of the

hand and the object, this paper will not be rendered.

Because we focus on operating the digital hand in

real time and investigating the dynamical interaction

with the object, the rendering of CG is not

introduced.

Figure 5: Digital finger with partial soft skin and rigid

bone.

Figure 6 shows the extending this configuration

to the five fingers.

Figure 6: Digital hand with partial soft skin and rigid

bone.

3.3 Virtual Physical Space

Our digital hand is able to grasp and manipulate

objects in the Virtual Physical Space. In the

development with Bullet Physics, the space would

be not well defined yet. So, we define it such that the

Virtual Physical Space is the three-dimensional

extent shown in the computer simulation, in which

an approximate simulation of certain physical

systems, such as rigid body dynamics (including

collision detection), soft body dynamics is provided

by a proper physics engine.

The digital hand and appropriate objects are set

in the Virtual Physical Space, then gravity, collision

detection and rotation calculations for them are

calculated. So, in the space the digital hand is able to

grasp or manipulate the object.

GRAPP 2016 - International Conference on Computer Graphics Theory and Applications

102

4 REAL TIME OPERATION

SYSTEM

4.1 Hand Posture Sensing

To realize the digital hand to mimic human hand

operation in real time, a sensor which is able to

sense the hand posture and also the position of hand

is required, then the Leap Motion Controller (LMC)

is suitable for the requirement. The LMC observes a

roughly hemispherical area, to a distance of about 1

meter, and can get 3D position data of all joints of

fingers and palm within sampling rate 150-295 fps

(USB 3.0 connection), this is made possible by the

skeleton model of hand of the LMC

(LeapMotionSDK, 2015). Then, the position data is

sent through a USB cable to the host computer.

The position data sometime is disturbed caused

by light illumination or characteristics of human

hand such as skin color and condition. To get stable

data, the constraint condition shown in chapter 2 is

imposed upon the data.

4.2 Implementation

A demonstrative application has been developed to

evaluate the digital hand in operation by the

postures. The goal is to set up the digital hand in real

time operation. The software application is

executable on the CPU(Core i7-4900MQ, 2.8GHz)

and the GPU(Nvidia Quadro K4100M, 1152 Cuda

processors). In our goal, the roles of CPU and GPU

are assigned separately as following

x Finger Callback : CPU

x Graphics Thread : GPU

x Physics Simulation : GPU

These processing assigned to CPU and GPU is

enable to use PyCUDA (PyCUDA, 2015), because

Panda3D is built in Python and the assigned has

been developing in the present circumstances.

4.3 Experiment

The subject operates the digital hand to mimic the

human hand in real time processing, using the Leap

Motion as the input device of the human hand

posture is shown in Figure 7.

We have succeeded in the real time operation for

a Digital hand only with rigid bones, not soft skin as

shown in Figure 8. The digital hand grasps and

operates the bar dextrously. A scene of real time

operation of the digital hand with rigid bones and

soft skin is shown in Figure 9. According to the

movement of human figures and palm of the hand,

the digital hand change its posture to mimic the hand.

And when the digital hand grasp an object in the

virtual physic space, the collision detection between

the digital hand and the object is transmitted to the

physics engine, and the digital hand can grasp it

according to the varying hand posture in real time.

However, those computational load becomes

tremendous, so the real time operation be fit to use is

not sufficient.

Figure 7: Digital hand system with LMC to get hand

posture in real time.

Figure 8: Digital hand with no soft skin operating the bar.

Figure 9: Digital hand with partial soft skin and rigid

bone.

A Digital Hand to Mimic Human Hand in Real Time Operation - Making of Digital Finger with Partial Soft Skin and Rigid Bone

103

5 CONCLUSIONS

This paper proposed a novel design procedure of the

digital hand, which is in reduced DoFs, the design of

soft skin, rigid body and those connection approach,

and real time operation system.

The reduced DoFs of the digital hand is proposed

by considering anatomy, which is to be operated in

real time. The total number of reduced DoFs is 16,

which is less than actual DoFs.

The design of soft skin and rigid body is regular

way in CG creation, but the connection approach is

devised because the collision detection of each body

shows different phases. This approach relates on the

shape of the soft skin.

The real time operation is considered about the

digital hand with reduced DoFs and the usage of the

LMC. The applicable demonstration in real time

operation is able to be realized by tuning PyCUDA,

and it will be shown in the conference stage.

ACKNOWLEDGEMENTS

I would like to thank Dr. Akinori Sasaki who had

contributed the development of this study. This work

was supported by JSPS KAKENHI Gran Numbers

25280125, 25560009 and in part supported by JST

RISTEX Service Science, Solutions and Foundation

Integrated Research Program.

REFERENCES

Blender, 2015, http://www.blender.org/

BulletPhysics,2015, http://bulletphysics.org/wordpress/

Chao, E.Y. , Cooney, K.N.An, W.P. and Linscheid, R.L.,

1989, Biomechanics of the Hand, World Scientific

Publishing.

Edwards, S.I. and Buckland, D.J., 2002, Development and

Functional Hand Grasps, SLACK Incorporation.

Endo, Y., Kanai, S., Miyata, N., Kouichi, M., Mochimaru,

M., Konno, J., Ogasawara, M. and Shimokawa, M.,

2008, Optimization-Based Grasp Posture Generation

Method of Digital Hand for Virtual Ergonomic

Assessment, SAE Intl J. of passenger cars-electronic

and electrical systems, vol.1, issue1, pp.590-598.

Hashimoto, H., Sasaki, A., et.al., 2013, Bar Spinning as

Dexterous Manipulation of Digital Hand Based on

Human Hand, IASTED Intl Conf. on Modelling and

Simulation, pp.413-418.

Hashimoto, H., Sasaki, A., et.al., 2014, A Structure and

Soft Finger Model of Digital Hand for Real Time

Dexterous Manipulation , IASTED Intl Conf. on

Modelling, Identification and Control, pp.265-270.

Inoue, T. and Hirai, S., 2009, Mechanics and Control of

Soft-fingered Manipulation, Springer.

Ishihara, T., Namiki, A., Ishikawa, M. and Shimojo, 2006,

M., Dynamic pen spinning using a high-speed

multifingered hand with high-speed tactile sensor,

IEEE RAS Intl Conf. on Humanoid Robots, pp.258-

263.

Kamakura, N., Matuo, M., Ishii, H., Mitsuboshi F. and

Miura, Y., 1980, Patterns of Static Prehension in

Normal Hands, American Journal of Occupation

Therapy, vol.34, pp.437-445.

Kapandj, A.I., 2008, The Physiology of the Joints Vol.1-3,

Churchill Livingstone.

LeapMotion, 2015, https://www.leapmotion.com/

LeapMotionSDK, 2015, https://developer.leapmotion.com /

Lee, J. and Kunii, T., 1995, Model-Based analysis of Hand

Posture, IEEE Computer Graphics and Applications,

vol.15, pp.77-86.

Mulatto, S, Formaglio, A. and Prattichizzo, D., 2013,

Using Posture Synergies to Animate a Low-

Dimensional Hand Avatar in Haptic Simulation, IEEE

Transactions on Haptics, vol.6, pp.106-116.

Mouri, T., Kawasaki, H. and Umebayashi, K., 2005,

Developments of New Anthropomorphic Robot Hand

and Its Master Slave System, Proc. of International

Conference on Intelligent Robots and Systems,

pp.3225-3230.

Nguyen,V, 1986, Constructing stable force-closure grasps,

In Proc. of ACM Fall Joint Computer Conference,

pp.129-137.

NASA, 2015, The Drag Coefficient,

https://www.grc.nasa.gov/www/k-12/ airplane/dragco.

html.

Panda3D, 2015, https://www.panda3d.org/

PyCUDA, 2015, http://mathema.tician.de/software/

pycuda/

Sueda, S., Kaufman, A. and Pai, D.K. 2008,

Musculotendon Simulation for Hand Animation, Proc.

of ACM SIGGRAPH2008, vol.27, issue3, pp.1-8.

Ying, W., Lin, J. and Huang, T.S., 2005, Analyzing and

capturing articulated hand motion in image sequences,

IEEE Trans. on Pattern Analysis and Machine

Intelligence, vol.27, issue12, pp.1910-1922.

Yoshikawa, T, 1996, Passive and active closures by

constraining mechanisms, In Proc. of IEEE Intl Conf.

on Robotics and Automation, vol.2, pp.1477-1484.

Wikipedia_Lift, 2015, Lift coefficient, https://en.wikipedia

.org/wiki/Lift_coefficient.

Wolf, K, 2015, Grasp Interaction with Tablets (T-Labs

Series in Telecommunication Services), Springer.

Moore, R., Lopes, J., 1999. Paper templates. In

TEMPLATE’06, 1st Intl Conf. on Template

Production. SCITEPRESS.

Smith, J., 1998. The book, The publishing company.

London, 2nd edition.

GRAPP 2016 - International Conference on Computer Graphics Theory and Applications

104