The Curved Surface Visualization of the Expert Behavior for Skill

Transfer Using Microsoft Kinect

Kaoru Mitsuhashi

, Hiroshi Hashimoto

and Yasuhiro Ohyama

School of Computer Science, Tokyo University of Technology, Hachioji, Tokyo, Japan

Master Program of Innovation for Design and Engineering, Advanced Institute of Industrial Technology, Tokyo, Japan

Keywords: Skill Transfer, Microsoft Kinect, B-spline Curve Surface, Visualization, Gradient Curvature Distribution,

Tracking Motion, Experts and Beginner.

Abstract: Method of teaching and inheriting for skill is almost oral. It is not quantitative but qualitative. Quantitative

inheriting of skill is difficult. In this paper, after tracking of a subject's skill motion using Microsoft Kinect,

a subject's motion is visualized as the curved surface. A curved surface is fitted in the positions of a subject's

joint, or the direction of trajectories. Expert and beginner perform swimming and karate motion. After the

motions are tracked, the trajectories of joints are transformed to a curved surface. The difference of an

action between an expert and a beginner is extracted by investigating curvatures and form on the visualized

curved surface. Therefore, we expected that technical skill is transferred easily.

1 INTRODUCTION

The actions at the time of dances, sports, and

engineering are different greatly to an expert and a

beginner. However, methods of teaching and

inheriting for skill is almost oral. It is not

quantitative but qualitative. Quantitative inheriting

of skill is difficult. In the case of sports, the experts

express in abstract languages, such as onomatopoeia,

or metaphor of an object image. However, they can’t

teach or inherit exactly and quantitatively. In the

case of engineering (Takeo and Natsu, 2011), the

experts can’t express a motion of fingertips and arms

orally in technical parts, such as machine tool

operation. Then, after seeing an expert's operation,

the beginner trains by performing imitated the

operation. In addition, the inheritance is impossible

when experts leave suddenly. Moreover, since

quantitative evaluation cannot be performed, the

same motion is not always repeated. Then, an

expert's motion is captured by video camera

photography, and the motions are analysed in

research or software (Cheung, Baker and Kanade,

2003), (Sigal and Black, 2006). The method is the

motion capture by one or more camera sets, with the

background subtraction technique, extracts a

human's outline and displays only a human's motion.

The motion can be preserved, and the reproducibility

is high. However the extraction of human position is

difficult, and quantitative evaluation is limited or no

meaning. Furthermore, in order that motion capture

may require large scale equipment, the possible

capture place is restricted in many cases. By forcing

marker wearing on a subject, we can hardly expect

to track the usual motion.

Then, we focus Microsoft Kinect, which is a

reasonable and easy operation, and capture the

motion using it. Kinect can recognize pictures and

depth positions, which is a useful tool function and

expected the application to three-dimensional

measurement. Kinect can extract a human's outline,

and the position of the human skeletons and joints.

Therefore, a human motion can be extracted easily

on a small scale. In the conventional research, angles

of the skeleton and joint positions are measured

(Murao, Hirao and Hashimoto, 2011). However,

there is no research that the whole body motion is

evaluated. Moreover, the quantitative evaluation of

joint angle and extracting position may be no

meaning. Namely, joint angle evaluation is not

transferred easily, and exact joint angles is not

necessary in many cases. In this paper, our purpose

is that a human joint position of motion is visualized

to a curved surface, and we extract the difference

between beginners and experts from the form or

curvature of the curved surface. We focus the human

upper half body, investigate the trajectories to the

both hands, elbows, shoulders, and the neck.

550

Mitsuhashi K., Hashimoto H. and Ohyama Y..

The Curved Surface Visualization of the Expert Behavior for Skill Transfer Using Microsoft Kinect.

DOI: 10.5220/0005101305500555

In Proceedings of the 11th International Conference on Informatics in Control, Automation and Robotics (ICINCO-2014), pages 550-555

ISBN: 978-989-758-040-6

 2014 SCITEPRESS (Science and Technology Publications, Lda.)

Afterwards, B-spline curved surface are fitted to the

joint trajectories in post processing. The form and

curvature of a visualized curved surface are

displayed visually and quantitatively, and the

difference of motion between an expert and a

beginner is extracted. Therefore, we expected that

technical skill is transferred easily.

2 EXPERIMENT METHOD

2.1 Motion Tracking Method

In this paper, we track the motion of human’s joints

in drawing gesture expression using Kinect. A user

expresses object shape by moving the right hand, left

hand, or both hands with depth sensing and image

recognition. Figure 1 (a) shows the tracking

situation. Kinect is placed the height position of

1.0m and the distance between Kinect and a user is

2.0m. Figure 1 (b) shows an image recognition of

the user. In this paper, we measure the position of a

right hand, right elbow, right shoulder, left hand, left

elbow, left shoulder, and neck. Line segments by

gesture are displayed with measuring the position of

the hand (right or left hand) using OpenCV open

source. A user’s motion is tracked in every 0.03

second, and the measured position is placed with the

time series.

Figure 1: Motion tracking system.

2.2 Curved Surface Visualization

The subject's motion captured using Kinect is

visualized to a curved surface in the preceding

section. In order to visualize a curved surface, the

data of a subject's joint position of point cloud based

on a time series is preserved, and B-spline curved

surface is fitted to the point cloud. The curved

surface makes a subject's trajectory the direction of

u, and makes joint positions the direction of v.

Figure 2 shows motion and curved surface when the

subject opens the arms and squats down. The

generated curved surface calculates the size of a

curved surface, normal vectors, tangent vectors, and

curvatures using 3D-CAD software Rhinoceros in

Figure 2. Furthermore, the gradation display of

curvature and the zebra mapping display are also

performed. Zebra mapping is an analytical technique

to visualize continuities of the curvature.

Figure 2: Visualized curved surface.

Fitting method to a curved surface is an

approximation. The lines are only continuous

segments because the trace of the drawing is a

Kinect

Human

2.0m

(a) Situation

Stage

Motion

Tracking

1.0m

(b) Kinect view

Color image

Depth recognition

Hand

Elbow

Shoulder

Trajectory

Start

Stop

(a) Motion

(b) Gradient curvature

Joint direction

Trajectory direction

TheCurvedSurfaceVisualizationoftheExpertBehaviorforSkillTransferUsingMicrosoftKinect

551

discrete point cloud; that is, the drawing lines are not

enough to create curved lines. Then, the point cloud

is converted to fitting curve lines. Approximation is

the method for smoothly passing a curved line or

surface through only the neighbourhood of the point

cloud, not through all the points. It enables the

operator to control the occurrence of the gap and

swing of the drawing position fuzzily. Therefore, we

adopt the approximation method. In this study, the

curved line or surface is a uniform cubic B-spline. It

allows for drawing a singular point and maintaining

the curvature continuity. The expressions of the

uniform cubic B-spline curved line L(t) are as

follows.

QNL )()( tt 

(1)

Here, N(t) is the matrix of the B-spline function,

t is a parameter, Q is the matrix of control points Q

(i=0, …, nq-1). We must perform fitting, although

currently, the control points and the parameter are

unknown. Then the control points and the parameter

t can be obtained using the matrix P of the drawing

point P

(i=0, …, np-1)

PNQ )(





(2)

(3)

The expressions of the surface S(u, v) are as

follows, similar to that of the line.

vuu,v

)()()( QNNS 

(4)

PNNQ )()(





(5)

Here, the expressions of the parameters u, and v

are omitted because they are equivalent to t.

However, if all the motions are transferred the

curved surface display, a curved surface will be

twisted or overlapped. Then, tangent and normal

vectors are calculated, and the first standard normal

and tangent vectors are decided like Figure 3. And a

curved surface is divided if the angle between the

standard vector and the other is larger than 180

degrees. Furthermore, a curved surface is divided

also if the self-intersection on a curved surface or

edge is occurring. Then, we are able to prevent a

twist and overlap of a curved surface.

Figure 3: Condition of divided curved surface.

3 CURVED SURFACE

EVALUATION OF EXPERT

AND BEGINNER

3.1 Swimming Crawl Motion

We investigate the difference of crawl motion in

swimming between 10-year-experience expert and

beginner. Subjects repeat the crawl motion in the

front of Kinect. There are 15 pieces of curved

surface. Subjects rotate in the yaw direction (z axis)

of 45 degrees from Kinect front so that Kinect can

track the crawl motion easily. The situation of crawl

motion in swimming is shown in Figure 4. Figure 4

(a) shows expert’s motion, and Figure 4 (b) shows

beginner’s motion. The visualized curved surface of

the expert’s motion is shown in Figure 5. Figure 5

(a) shows the curved surface with the gradient

curvature distribution when a right arm is flung up,

and Figure 5 (b) shows the curved surface with zebra

mapping. The visualized curved surface of the

beginner’s motion is shown in Figure 6. Figure 6 (a)

shows the curved surface with gradient curvature

distribution when a right arm is flung up, and Figure

6 (b) shows the curved surface with zebra mapping.

From Figure 5 and Figure 6, curved surface of

expert’s motion had more flat parts than the

beginner’s motion on the whole. This result is the

same in zebra mapping. The striped zebra pattern of

the beginner’s motion is heterogeneous. On the other

hand, the expert’s surface of change of curvature is

focally larger than the beginner’s surface. According

to an expert’s opinion, the motion of scratching

water should be reduced as less as possible. In

addition, the size of an expert’s curved surface is

smaller than the beginner’s surface.

As mentioned above, the measuring result of the

maximum curvature and the curved surface area is

shown in Table 1 (a). From Table 1 (a), the expert’s

maximum curvature are larger than beginner’s

curvature, and the expert’s area is smaller than the

beginner’s area. Therefore, the curved surface



























)1(1

)2,1(

)0(0

npi

npit



ICINCO2014-11thInternationalConferenceonInformaticsinControl,AutomationandRobotics

552

change of curvature and area of the crawl should be

focused in order to crawl with expert’s motion.

(a) Expert (b) Beginner

Figure 4: Crawl motion in swimming.

(a) Gradient curvature distribution

(b) Zebra mapping

Figure 5: Curved surface of expert motion.

(a) Gradient curvature distribution

(b) Zebra mapping

Figure 6: Curved surface of expert motion.

3.2 Karate Thrust Motion

Like swimming, we investigate the difference of

thrust motion in karate between 5-year-experience

expert and beginner. Subjects repeat the thrust

motion in the front of Kinect. There are 10 pieces of

curved surface. Subjects rotate in the yaw direction

(z axis) of 45 degrees from Kinect front so that

Kinect can track the thrust motion easily. The

situation of thrust motion in karate is shown in

Figure 7. Figure 7 (a) shows expert’s motion, and

Figure 7 (b) shows beginner’s motion. The

visualized curved surface of the expert’s motion is

shown in Figure 8. Figure 8 (a) shows the curved

surface with the gradient curvature distribution when

a subject hit with a right arm and fist, and Figure 8

(b) shows the curved surface with zebra mapping.

The visualized curved surface of the beginner’s

motion is shown in Figure 9. Figure 9 (a) shows the

curved surface with the gradient curvature

distribution when a subject hit with a right arm and

fist, and Figure 9 (b) shows the curved surface with

zebra mapping. From Figure 8 and Figure 9, curved

surface of expert’s motion had more flat parts than

the beginner’s motion on the whole. The striped

pattern of the zebra is heterogeneous like swimming.

On the other hand, the expert’s surface of change of

curvature is focally larger than the beginner’s

surface. According to an expert’s opinion, the thrust

trajectory should be straight. In addition, the size of

an expert’s curved surface is smaller than the

beginner’s surface.

(a) Expert

(b) Beginner

Figure 7: Thrust motion in karate.

3.3 Curvature Evaluation

As mentioned above, the measuring result of the

Right hand

Right shoulder

Right elbow

Right hand

Right shoulder

Right elbo

TheCurvedSurfaceVisualizationoftheExpertBehaviorforSkillTransferUsingMicrosoftKinect

553

maximum curvature and the curved surface area is

shown in Table 1 (b). From Table 1 (b), the expert’s

maximum curvature are larger than beginner’s

curvature, and the expert’s area is smaller than the

beginner’s area. Therefore, the curved surface

change of curvature and the area should be focused

in order to thrust with expert’s motion in the same of

crawl motion.

(a) Gradient curvature distribution

(b) Zebra mapping

Figure 8: Curved surface of expert motion.

4 CONCLUSIONS

In this paper, a human joint position of motion is

visualized to a B-spline curved surface, and we

investigate the difference between beginners and

experts from the form or curvature of the curved

surface. The form and curvature of a visualized

curved surface are displayed visually and

quantitatively, and the difference of motion between

an expert and a beginner is extracted. In result, the

expert’s maximum curvature are larger than

beginner’s curvature, and the expert’s area is smaller

than the beginner ’ s area. The curved surface

change of curvature and the area should be focused

in order to act with expert’s motion. In future, the

effectiveness of this technique is established by

acquiring a large amount of the expert’s motion

database, and we track various expert’s motion to

transfer skill.

(a) Gradient curvature distribution

(b) Zebra mapping

Figure 9: Curved surface of expert motion.

Table 1: Curvature and area.

(a) Crawl in swimming

(b) Thrust in karate

ACKNOWLEDGEMENTS

This work was in part supported by JST RISTEX

Service Science, Solutions and Foundation

Integrated Research Program.

Maximum

curvature

[rad/mm]

Area

]

beginner 0.8 0.43

expert 5.6 0.19

Maximum

curvature

[rad/mm]

Area

]

beginner 1.7 0.66

expert 10.6 0.21

Right shoulder

Right hand

left hand

Neck

Left shoulder

Right shoulder

Right hand

Neck

Left hand

Left shoulder

ICINCO2014-11thInternationalConferenceonInformaticsinControl,AutomationandRobotics

554

REFERENCES

Yasushi Takeo ， Wataru Natsu, 2011. Development of

valuation Method for Measurement Skill Training,

Proceedings of International Symposium on

Standardization Education and Research 2011 Tokyo

Japan, pp.130-145.

Cheung, K. M. G., Simon Baker, and Takeo Kanade.,

2003. Shape-from-silhouette of articulated objects and

its use for human body kinematics estimation and

motion capture, Computer Vision and Pattern

Recognition, 2003. Proceedings. 2003 IEEE

Computer Society Conference on. Vol. 1. IEEE, pp.77-

84.

Sigal, Leonid, and Michael J. Black., 2006. Humaneva:

Synchronized video and motion capture dataset for

evaluation of articulated human motion. Brown

Univertsity TR 120.

Toshiyuki Murao, Yasuyuki Hirao and Hiroshi

Hashimoto, 2011. Skill Level Evaluation for Taijiquan

based on Curve Fitting and Logarithmic Distribution

Diagram of Curvature, SICE Journal of Control,

Measurement, and System Integration, 4, 1, pp.001–

005.

TheCurvedSurfaceVisualizationoftheExpertBehaviorforSkillTransferUsingMicrosoftKinect

555