Motion Curved Surface Analysis and Composite for Skill Succession
using RGBD Camera
Kaoru Mitsuhashi
1
, Hiroshi Hashimoto
2
and Yasuhiro Ohyama
1
1
Department of Mechanical engineering, School of Engineering, Tokyo University of Technology, Hachioji, Tokyo, Japan
2
Master Program of Innovation for Design and Engineering, Advanced Institute of Industrial Technology, Tokyo, Japan
Keywords: Skill Succession, Microsoft Kinect, B-spline Curve Surface, Visualization, Gradient Curvature Distribution,
Experts and Beginners, Composite Surface, Skeleton and Curved Surface Succeeding Method.
Abstract: The skill succession method is almost oral. It is not quantitative but qualitative. Quantitative succession is
difficult. In this research, after tracking of a subject's motion using RGBD camera, a subject's motion is
visualized as the motion curved surface. Expert and beginner perform the sports and entertainment motion,
and the character of the surface is analyzed. The character is the maximum curvature and surface area. In
addition, we suggest the composite surface, because one RGBD camera is not all tracking motion by occluding
the obstacle or subject’ body parts. Finally, we confirm the validity of skill succession by watching skeleton
motion movie and curved surface.
1 INTRODUCTION
Beginner trains to watch and imitate the expert
behavior, in the entertainment of the Noh play and
Kabuki play, and sports play, engineering of
operating a machine (Hashimoto et al., 2011).
However, their skill succession/teaching are
qualitative, not quantitative. They are expressions in
abstract languages, such as onomatopoeia, or
metaphor of an object image, and the quantitative
evaluation is difficult to express and perform
(Hasegawa and Fukumura, 1996). Therefore, the skill
succession/teaching cannot be confirmed, the same
behavior is not always repeated.
Then, an expert's motion is captured by video
camera photography, and the motions are analyzed in
research or software (Takeo and Natsu, 2011),
(Cheung et al., 2003; Sigal and Black, 2006). The
method is the motion capture by one or more camera
sets, with the background subtraction technique,
extracts a human's outline/marker joints and displays
only a human's motion. The motion can be preserved,
and the reproducibility is high. However the
extraction of human position is difficult, and
quantitative evaluation is limited or no meaning.
Furthermore, human joints are needed to capture
equipped markers, by forcing marker wearing on a
subject. Therefore, we can hardly expect to track the
usual motion.
We focus Microsoft Kinect, which is a reasonable
and easy operation, and capture the motion using it.
Kinect can recognize pictures and depth positions,
which is a useful tool function and expected the
application to three-dimensional measurement.
Kinect can extract a human's outline, and the position
of the human skeletons and joints. In the conventional
research, angles of the skeleton and joint positions are
measured (Murao et al., 2011; Hashimoto et al.,
2014). In addition, we visualize a human joint
trajectory of motion into a curved surface (it is called
a motion curved surface), and we extract the
difference between beginners and experts from the
form or curvature of the motion curved surface in
previous research (Mitsuhashi et al., 2014; Suneya et
al., 2014). Therefore, we can evaluate technical skill
quantitatively, and except the skill
succession/teaching for expert’s skill easily.
However, some joints are not tracked by occluding an
obstacle or body parts (joints, body, arm, leg, head...),
(Deutscher et al., 2000) because tracking view is one
direction (one Kinect). Therefore, the motion curved
surface is lacked by the occlusion. On the other hand,
we have never confirmed the validity of skill
succession using a motion curved surface.
In this research, we compose the motion curved
surfaces made from the multiple Kinect view, so as to
track the whole joint motion in more detail. Moreover
406
Mitsuhashi K., Hashimoto H. and Ohyama Y..
Motion Curved Surface Analysis and Composite for Skill Succession using RGBD Camera.
DOI: 10.5220/0005569904060413
In Proceedings of the 12th International Conference on Informatics in Control, Automation and Robotics (ICINCO-2015), pages 406-413
ISBN: 978-989-758-123-6
Copyright
c
2015 SCITEPRESS (Science and Technology Publications, Lda.)
we investigate the effectiveness of the composite
motion curved surface. In addition, we confirm the
validity of skill succession by watching skeleton
motion movie and curved surface.
2 EXPERIMENT METHOD
2.1 Motion Tracking Method
In this paper, we track the motion of human’s joints
in drawing gesture expression using Kinect. A user
expresses object shape by moving the right hand, left
hand, or both hands with depth sensing and image
recognition. Figure 1 shows the tracking situation.
Kinect is placed the height position of 1.75m and the
distance between Kinect and a user is 2.0m. Figure 2
shows an image recognition of the user. Figure 2(a)
shows an image recognition of the user. Figure 2(b)
shows a depth image recognition with human joints
and skeleton model. Position of joints and skeletons
is estimated by Kinect driver. In this paper, we
measure the position of a right hand, right elbow,
right shoulder, left hand, left elbow, left shoulder, and
neck. Line segments by gesture are displayed with
measuring the position of the hand (right or left hand)
using the OpenCV library. Line segments by gesture
are displayed with measuring the position of the hand
(right or left hand) using OpenCV library. Kinect
programing language is C/C++ and using openNI2,
Figure 1: Motion tracking situation.
(a) RGB color image (b) Depth image
Figure 2: Kinect view.
NiTE2 library. A user’s motion is tracked in every
0.02 second, and the measured position is placed with
the time series.
2.2 Curved Surface Visualization
The subject's motion is visualized to a curved surface
in the preceding section. In order to visualize a curved
surface, the data of a subject's joint position of point
cloud based on a time series is preserved, and B-
spline curved surface is fitted to the point cloud by the
approximation. The curved surface makes a subject's
trajectory the direction of u, and makes joint positions
the direction of v. Figure 3 shows motion, trajectory
and curved surface when the subject opens the arms
and squats down. The generated curved surface
calculates the size of a curved surface, normal vectors,
tangent vectors, and curvatures using 3D-CAD
software Rhinoceros in Figure 3. Figure 3(b) shows
the trajectory of upper joints. Figure 3(c) shows the
visualized curved surface. Figure 3(d) shows the
gradation display of curvature and Figure 3(e) the
zebra mapping display. Zebra mapping is an
analytical technique to visualize continuities of the
curvature.
(a) RGB color image
(b) Joints trajectory (c) Visualized surface
(d) Zebra mapping (e) Gradient curvature
Figure 3: Visualized motion curved surface.
However, if all the motions are transferred the
curved surface display, a curved surface will be
Start
End
MotionCurvedSurfaceAnalysisandCompositeforSkillSuccessionusingRGBDCamera
407
twisted or overlapped. Then, tangent and normal
vectors are calculated, and the first standard normal
and tangent vectors are decided. And a curved surface
is divided if the angle between the standard vector and
the other is larger than 180 degrees. Furthermore, a
curved surface is divided also if the self-intersection
on a curved surface or edge is occurring. Then, we are
able to prevent a twist and overlap of a curved surface.
3 ANALYSIS OF SKILL MOTION
3.1 Throw Motion in Darts
We investigate the difference of throwing motion in
Darts between 5-year-experience expert and
beginner. Subjects perform the darts motion in the
front of Kinect, their upper half of the body (hand,
elbow, and shoulder) is tracked. The situation of
throwing motion in darts is shown in Figure 4. Figure
4 (a) shows expert’s motion of RGB color image and
depth image with joints and skeletons, and Figure 4
(b) shows beginner’s motion. The visualized curved
surface of the expert’s motion is shown in Figure 5.
Figure 5 (a) shows the curved surface with the
gradient curvature distribution when a right arm is
throwing and Figure 5 (b) shows the curved surface
with zebra mapping. The visualized curved surface of
the beginner’s motion is shown in Figure 6. The
curved surface is divided hand-elbow and elbow-
shoulder, because there is the self-intersection in
Figure 6. Figure 6 (a) shows the curved surface with
gradient curvature distribution when a right arm is
throwing and Figure 6 (b) shows the curved surface
with zebra mapping. In this result, the expert’s motion
(a) Expert (mirror image)
(b) Beginner (mirror image)
Figure 4: Throwing motion in darts.
(a) Zebra mapping (b) Gradient curvature
Figure 5: Motion curved surface of expert in darts.
(a) Zebra mapping (b) Gradient curvature
Figure 6: Motion curved surface of beginner in darts.
Table 1: Maximum curvature, area, and shape in darts.
surface is fan shaped, the elbow is fixed. Namely, the
expert’s hand trajectory is curved. Expert turns his
hand toward the target in the finish. According to an
expert’s opinion, fixing the elbow is most important.
On the other hand, the beginner’s surface is triangle
shape. Namely, the beginner’s hand trajectory is
straight. From Figure 5 and Figure 6, curved surface
of expert’s motion had more flat parts than the
beginner’s motion on the whole. This result is the
same in zebra mapping. The striped zebra pattern of
the beginner’s motion is heterogeneous. As
mentioned above, the measuring result of the
maximum curvature and the curved surface area is
shown in Table 1. From Table 1, the expert’s surface
of change of curvature is focally larger than the
beginner’s surface. Beginner’s surface at start is
largest curvature. On the other hand, the expert’s
surface area is similar to the beginner’s area.
3.2 Leg Raising Motion in Stretch
We investigate the difference in leg raising motion
(side plank) of Stretch 3-year-experience expert and
beginner. Subjects perform the darts motion in the
front of Kinect, their lower half of the body (foot,
Maximum
curvature
[rad/mm]
Area
[m
2
]
Shape
Expert 0.91 0.18
Fan shape
Beginner 0.24 0.12
Triangle
ICINCO2015-12thInternationalConferenceonInformaticsinControl,AutomationandRobotics
408
knee, and hip) is tracked. Side plank is a motion that
the whole body is supported using the arm and leg,
and opposite leg is raising up and down. The situation
of throwing motion in side plank is shown in Figure
7. Figure 7 (a) shows expert’s motion of RGB color
image and depth image with joints and skeletons, and
Figure 7 (b) shows beginner’s motion. Figure 8 shows
the multiple curved surfaces of whole body in leg
raising motion. In Figure 8, we need to focus the
lower half of the body. Figure 9 (a) shows the curved
surface with the gradient curvature distribution when
a right leg of expert is raising and Figure 9 (b) shows
the curved surface with zebra mapping. Figure 10 (a)
shows the curved surface with the gradient curvature
distribution when a right leg of beginner is raising and
Figure 10 (b) shows the curved surface with zebra
mapping. From figures, the stripped zebra pattern of
expert is along the curved lines, but beginner’s pattern
is not. As mentioned above, the measuring result of
the maximum curvature and the curved surface area
is shown in Table 2. From Table 2, the expert’s
surface of change of curvature is focally larger than
the beginner’s surface. On the other hand, the expert’s
surface area is larger than beginner’s area.
3.3 Evaluation of Curved Surface
Table 3 shows the maximum curvature, area, and
shape of motion curved surface in swimming the
crawl, karate thrusts (Mitsuhashi, Ohyama, and
Hashimoto, 2014), darts throw, side plank, and
calligraphy. From Table 3, we find the expert’s
maximum curvature and area is larger than
(a) Expert
(b) Beginner
Figure 7: Leg raising motion in side plank.
Figure 8: Motion curved surfaces of body in side plank.
(a) Zebra mapping (b) Gradient curvature
Figure 9: Motion curved surface of expert in side plank.
(a) Zebra mapping (b) Gradient curvature
Figure 10: Motion curved surface of beginner in side plank.
Table 2: Maximum curvature, area, and shape in side plank.
Table 3: Maximum curvature, area, and shape
of the motion curved surfaces.
beginner’s. Additionally the expert’s shape is
complicated, the beginner’s shape is flat. Therefore,
we need to focus the maximum curvature, area, and
shape, and their outputs or results are also important.
Maximum
curvature
[rad/mm]
Area
[m
2
]
Shape
Expert 0.87 0.29
Triangle
Beginner 0.21 0.13
Rectangle
Maximum
curvature
[rad/mm]
Area
[m
2
]
Shape
Expert 5.60 0.19 Concave
Beginner 0.82 0.43 Convex
Expert 10.63 0.21 Thin
Beginner 1.74 0.66 Thick
Expert 0.91 0.18 Fan shape
Beginner 0.24 0.12 Triangle
Expert 0.87 0.29 Triangle
Beginner 0.21 0.13 Rectangle
Expert 0.46 0.15 Projection
Beginner 0.24 0.10 Flat
Swimming
crawl
Karate
thrust
Darts
throw
Side plank
leg raise
Caligraphy
MotionCurvedSurfaceAnalysisandCompositeforSkillSuccessionusingRGBDCamera
409
4 COMPOSITE CURVED
SURFACE
4.1 Limitations of One Kinect View
Body joints are tracked by one Kinect until preceding
chapter, for that reason, some joints are not tracked
by occluding an obstacle or body parts oneself. In
addition, the motion curved surfaces are different
shape from the front view and back view. Because the
motion surface exist both the correct tracking parts
and incorrect tracking parts. Therefore the tracking all
motion in large or rotating motion is difficult for one
Kinect. Then, we suggest to track the motion from the
multiple direction using the multiple Kinect, the
multiple curved surfaces are composited. Figure 11
shows the surface composition method. One correct
surface and the other correct surface are composited,
and the incorrect surfaces are ignored from Figure 11.
Figure 12 shows the tracking situation of multiple
Kinect. The composite curved surface is made with 2
Kinect in this paper.
Figure 11: Schematic of composite surface.
Figure 12: Tracking situation of multiple Kinect.
4.2 Validity of Composite Curved
Surface
Throw in darts in preceding chapter is tracked using
2 Kinect, one Kinect is tracking from front view
position to 1.50m distance, and the other is from a
side view position to 1.50m distance. Because we
confirm the validity of the composite curved surface.
Figure 13 (a) shows the motion curved surface from
side view, Figure 13 (b) shows from front view. The
curved surface of Figure 13 (a) exists the incorrect
parts at the start and end and the correct parts in hand
trajectory, because the depth sensor is less exact than
image processing. On the other hand, the motion
curved surface of Figure 13 (b) exists the correct parts
in all motion. Then, the Edge 2 and Edge 3 are
changed to the Edge 5 and Edge 6, watching the RGB
image view. In this result, the composite surface is
similar to the motion surface from front view. Figure
13 (c) shows the composite curved surface.
Therefore, we prove the composite surface from
multiple view valid.
(a) Left view (b) Front view
(c) Composite curved surface
Figure 13: Composite motion curved surface of expert in
darts throwing motion.
4.3 Swing Motion in Tennis
Large and rotating motion is tracked using the
composite curved surface method. It is a swing
motion in tennis. Some joints are occluded by the
arms of oneself at start and end. Figure 14 shows the
situation of multiple Kinect for rotating motion in
tennis. One Kinect is placed in the front, the other is
placed in the back. Figure 15 shows the motion
curved surface from top view using multiple Kinect.
Figure 15 (a) shows from back view, Figure 15 (b)
shows from front view. The curved surface of Figure
15 (a) exists the correct parts in Edge 1 and Edge 2 (at
the start) and the incorrect parts at the end by
occluding the arm. On the other hand, the motion
curved surface of Figure 15 (b) exists the correct parts
in Edge 3 (at the end) and the incorrect parts at the
ICINCO2015-12thInternationalConferenceonInformaticsinControl,AutomationandRobotics
410
(a) Perspective view
(b) Back view (c) Front view
Figure 14: Situation of swing motion in tennis.
(a) Back view (b) Front view
(c) Composite curved surface
Figure 15: Composite motion curved surface of expert in
darts throwing motion.
start. Then, the composite surface is made from
selected the Edge 1, Edge 2, and Edge 5 watching the
RGB image view. Figure 15 (c) shows the composite
curved surface. From Figure 15(c), expert’s motion
surface from top view is more smooth and concave
than the beginner’s surface. Then, the measuring
result of the maximum curvature and the curved
surface area is shown in Table 4. From Table 4, the
expert’s surface of change of curvature is focally
larger than the beginner’s surface. On the other hand,
the expert’s surface area is larger than the beginner’s
area. Therefore, we can make the motion curved
surface and the difference of the expert and beginner
from large and rotating motion also.
Table 4: Maximum curvature, area, and shape of
composite surface in tennis.
5 SKILL SUCCESSION USING
MOTION CUVED SURFACE
We have never succeeded using a motion curved
surface, only confirmed the difference of the expert
and beginner in a motion curved surface character.
The skill succession method is watching a skeleton
motion (animation) and curved surface. The
animation is 3D viewer. It can rotate and translate the
view position and focal position. The animation
program is written by C++ and openGL library, and
it read the joint data. Figure 16 shows the skeleton
movie and motion curved surface of right upper half
of the body in swimming the crawl. Then, we confirm
the validity of skill succession using a motion curved
surface. First, two beginner subjects are tracked the
throw motion in darts. Next, one subject watch only
the motion movie, and the movie are expert’s motion.
The other watch the skeletons motion (animation) and
curved surface, the skeleton motion (animation) is the
expert’s or subject oneself. And both subjects train to
watch or move in 10 minutes. Figure 17 (a) shows the
watching only movie and moving situation, Figure 17
(b) shows the skeleton motion and curved surface of
right upper half of body (hand, elbow, and shoulder).
After training, two beginner subjects are tracked the
throw motion in darts again
Figure 16: Skeleton animation and motion curved surface
in swimming the crawl.
Maximum
curvature
[rad/mm]
Area
[m
2
]
Shape
Expert 2.40 1.84
Concave
Beginner 1.30 1.16
Flat
MotionCurvedSurfaceAnalysisandCompositeforSkillSuccessionusingRGBDCamera
411
(a) Watching video (b) Motion with surface
Figure 17: Training the throwing motion in darts.
Figure 18 shows the before training curved
surface for two beginner. They are triangle shape or
exist straight trajectory of hand, and two subjects
don’t fix the elbow. Figure 19 shows the after training
curved surface. Figure 19 (a) shows the only watch
motion movie, Figure 19 (b) shows the skeleton
motion and curved surface. From figures, the surface
of Figure 19 (a) remain the same shape. On the other
hand, the surface of Figure 19 (b) is similar to expert’s
fan-shaped surface, and his trajectory is a curved line
(segment). Table 5 shows the maximum of curvature
and area for expert, only movie training, and skeleton
and curved surface training. The skeleton and curved
surface trainer’s motion are similar to expert’s motion
from Table 5. In this result, we prove the skill
succession using the motion curved surface valid.
(a) Watching video (b) Motion with surface
Figure 18: Motion curved surface of before training.
(a) Watching video (b) Motion with surface
Figure 19: Motion curved surface of after training.
6 CONCLUSIONS
In this research, expert and beginner perform the
sports and entertainment motion for skill succession,
Table 5: Maximum curvature, area, and shape
throwing motion training in darts.
and the character of the surface is analyzed.
Additionally we combine the motion curved surfaces
made from the multiple Kinect view, so as to track the
whole joint motion in more detail. We investigate the
effectiveness of the composite motion curved surface.
In addition, we confirm the validity of skill
succession by watching skeleton motion movie and
curved surface. In this result, we find the expert’s
maximum curvature and area is larger than
beginner’s. Moreover the expert’s shape is
complicated, the beginner’s shape is flat. In addition,
we can make the composite motion curved surface
from multiple Kinect, and the difference of the expert
and beginner from large and rotating motion also.
Finally, we prove the skill succession using the
skeleton animation and motion curved surface valid.
In future work, the database is constructed the other
motion curved surface, and the motion velocity or
acceleration curved surface is suggested. In addition,
the new method of skill succession using motion
curved surface.
ACKNOWLEDGEMENTS
This work was in part supported by JST RISTEX
Service Science, Solutions and Foundation Integrated
Research Program and JSPS KAKENHI Grant
Number 60208460.
REFERENCES
Hashimoto, H., Yoshida, I., Teramoto, Y., Tabata, H., and
Han, C., 2011. Extraction of Tacit Knowledge as Expert
Engineer's Skill based on Mixed Human Sensing,
Proceedings of the 20th IEEE International Symposium
on Robot and Human Interactive Communication,
pp.413-418, 2011.
Taki, T., Hasegawa, J. I., & Fukumura, T. (1996,
September). Development of motion analysis system for
quantitative evaluation of teamwork in soccer games.
In Image Processing, 1996. Proceedings, International
Conference on Vol. 3, pp. 815-818. IEEE.
Maximum
curvature
[rad/mm]
Area
[m
2
]
Shape
0.91 0.18 Fan shape
Before 0.26 0.15 Triangle
After 0.47 0.16 Triangle
Before 0.53 0.13 Triangle
After 0.82 0.17 Fan shape
Expert
Only movie
Skeleton and
Surface
Beginner
ICINCO2015-12thInternationalConferenceonInformaticsinControl,AutomationandRobotics
412
Yasushi Takeo Wataru Natsu, 2011. Development of
valuation Method for Measurement Skill Training,
Proceedings of International Symposium on
Standardization Education and Research 2011 Tokyo
Japan, pp.130-145.
Cheung, K. M. G., Simon Baker, and Takeo Kanade., 2003.
Shape-from-silhouette of articulated objects and its use
for human body kinematics estimation and motion
capture, Computer Vision and Pattern Recognition,
2003. Proceedings. 2003 IEEE Computer Society
Conference on. Vol. 1. IEEE, pp.77-84.
Sigal, Leonid, and Michael J. Black., 2006. Humaneva:
Synchronized video and motion capture dataset for
evaluation of articulated human motion. Brown
Univertsity TR 120.
Toshiyuki Murao, Yasuyuki Hirao and Hiroshi Hashimoto,
2011. Skill Level Evaluation for Taijiquan based on
Curve Fitting and Logarithmic Distribution Diagram of
Curvature, SICE Journal of Control, Measurement, and
System Integration, 4, 1, pp.001–005.
Hiroshi Hashimoto, Mizuki Nakajima, Seiichi Kawata,
Sang-Gyu Shin, Toshiyuki Murao, 2014. Skill Level
Evaluation of Taijiquan based on 3D Body Motion
Analysis, Proc. Of IEEE International Conference on
Industrial Technology (ICIT2014), CD-ROM.
Kaoru Mitsuhashi, Hiroshi Hashimoto, and Yasuhiro
Ohyama, 2014. The Curved Surface Visualization of the
Expert Behavior for Skill Transfer Using Microsoft
Kinect, 11th International Conference on Informatics in
Control, Automation and Robotics (ICINCO 2014),
September1-3, Wien, Austria, pp.550-555.
Mitsuki Suneya, Masaki Kutsuna, Kaoru Mitsuhashi,
Yasuhiro Ohyama, and Hiroshi Hashimoto, 2014. The
Curved Surface Visualization of Behaviors for Skill
Inheriting using RGBD camera. SICE System
Integration 2014 (SI2014), December 14-17, Tokyo,
Japan. Pp.67-71, (in Japanese).
Deutscher, Jonathan, Blake, Andrew, Reid, Ian, 2000.
Articulated body motion capture by annealed particle
filtering. In: Computer Vision and Pattern Recognition,
2000. Proceedings. IEEE Conference on. IEEE, pp.126-
133.
MotionCurvedSurfaceAnalysisandCompositeforSkillSuccessionusingRGBDCamera
413