3D Reference-Based Skeletal Movement Evaluation

Lars Lehmann

, Roman Seidel

and Gangolf Hirtz

Digital Signal Processing and Circuit Technology, Chemnitz University, Reichenhainer Str. 70, Chemnitz, Germany

Keywords:

Segment Reference, Visualisation, Feedback Control, Assisted Motion Control, Motion Error Detection, 3D

Realtime Animation.

Abstract:

In a medical therapy, the exact execution of the training exercises developed by the therapist is a crucial task for

the success of the therapy. Currently, a therapist has to treat up to 15 patients at the same time on an outpatient

basis. To compensate this deﬁcit, an automated assistance system needs to be created. Previous approaches

have focused on a parameterised segment angle-based assessment for training exercise feedback. This work

focuses on a reference-based approach. This reference is created by the therapist and thus corresponds to

the ideal movement model and can be individually adapted to the patient. It is necessary to compare this

reference with the patients’ real movement in real time, to detect deviations and to output them as errors. For

this purpose, the reference can be adapted to the body size of the patient and the patients’ current position

and orientation can be taken into account, or it can be described by reference segments, i.e. an angle-based

comparison of the reference. Our work highlights the the segment and reference-based assessment approach

and compares them to each other.

1 INTRODUCTION

After a hip operation, the persons treated must com-

plete a rehabilitation phase of several weeks in a spe-

cial stationary therapy centre. During the therapy,

one therapist looks after up to 15 patients (Nitzsche,

2018). It is not possible for the therapist to carry out

the therapy in the required quality because they lack

the necessary overview. This lack of specialist control

leads to errors in the execution of the patients’ move-

ment. These systematic incorrect movements due to

unavailable control aggravate their hip problems and

delay or even prevent recovery (L

osch, 2019).

Demographic change will increase the number

of patients and decrease the number of therapists

(Budliger, 2021). In order to prevent a collapse of

therapeutic medicine, computer-supported assistance

in the implementation of therapy is indispensable.

Therefore, a prototype was developed, which uses a

depth sensor to realise a skeletal extraction and de-

tects and evaluates the patients’ movements directly

at runtime. The markerless assistance system works

in real time and supports the therapists in this way.

For visualisation, the assistance system uses a three-

https://orcid.org/0000-0001-9778-8479

https://orcid.org/0000-0002-3144-1488

https://orcid.org/0000-0002-4393-5354

dimensional avatar, modelled on a human body, to

represent a patient and to detect and map his move-

ments. The avatar is formed by a three-dimensional

mesh model whose surface is described by triangles

and is connected to the extracted skeleton so that the

patients’ movements directly affect the model. The

mesh model is divided into segments according to

the coupled skeleton, which are used for direct vi-

sualisation(Lehmann, 2020). If an error is detected

during the movement, the corresponding segment in-

volved in the error is coloured directly on the mesh

model(Wolff, 2018). Figure 1 shows an error and the

colouring of the involved segments with the help of

known trafﬁc light colours.

Figure 1: Representation of direct error mapping.

When coloured red, this extraction segment is directly

involved in the error. Yellow stands for a warning, the

150

Lehmann, L., Seidel, R. and Hirtz, G.

3D Reference-Based Skeletal Movement Evaluation.

DOI: 10.5220/0011969200003497

In Proceedings of the 3rd International Conference on Image Processing and Vision Engineering (IMPROVE 2023), pages 150-155

ISBN: 978-989-758-642-2; ISSN: 2795-4943

 2023 by SCITEPRESS – Science and Technology Publications, Lda. Under CC license (CC BY-NC-ND 4.0)

segment is in a critical movement area. If the segment

is coloured green or according to the mapped texture

(Capizzi, 2002), no error has been detected and the

movement is correct.

An angle-based comparison of the skeleton’s bone

joints for motion immobilisation is relatively easy to

realise, but it also has some disadvantages. Creating

the comparative movements is quite time-consuming,

as each individual aspect of the movement must be

speciﬁed and agreed separately. It would be sim-

pler to have a movement recording that can be gen-

erated quickly and used for comparison with the cur-

rent movement. These recorded references are indi-

vidualised ideal movements generated by the thera-

pist. Two of these reference implementations are pre-

sented and compared in this paper.

2 STANDARD SEGMENT-BASED

METHOD

The general segment-based movement assessment

(SBE) concentrates on the observation of the skele-

ton points and their position in relation to each other.

The individual points of the extraction skeleton are

known and also their connection to each other. These

are used to read angles between segments and match

them. In this way it can be determined whether a

movement within a predeﬁned frame or is ending out-

side of it(Zhao et al., 2014). This method is a stan-

dard method for a motion assessment(R

oder et al.,

2023)(Ren et al., 2005).

At the beginning of the movement evaluation, it is

necessary to deﬁne the individual 3D movements and

their corresponding error patterns(Ant

on D, 2015).

The movement deﬁnitions are needed to realise the

repetition counts. The error patterns help to deter-

mine deviations in the intended movement and to give

hints for the real execution of the movement. In the

assistance system, various SBE scenarios have been

created. There are abduction, ﬂexion and extension

movements for both the lower and upper extremities

that can be detected and evaluated. In this section an

example creation of the right leg abduction and its er-

ror patterns are shown.

Abduction is deﬁned by the movement of a body part

away from the body axis (Menche, 2003). In the ﬁg-

ure 2 the whole skeleton is shown during an abduction

of the right leg. The movement is visualised mirror-

inverted because the sensor extraction is done in this

way.

The joint points A, B, C and D necessary for cal-

culating the abduction angle are marked in green. The

angle results from observing the position of two inter-

Figure 2: 2D representation of an abduction of the right leg.

The angle α between the two through the straight lines AB

and CD is determined and used as a motion measurement.

section lines. The intersection line results from two

joint points belonging to each bone. The actual an-

gle α is obtained by substituting the intersection lines

into equation 1.

cos(α) =

u · v

|u| ·|v|

(1)

The angle α is the desired goal of this movement

where u = S − A and v = S − C. This angle must be

achieved during the real-time movement in order to

realise a correct repetition of the movement.

Additional parameters for the movement guidance

must be checked in order to achieve an optimal train-

ing result. For example, it must be ensured that the

movement takes place primarily in the xy-plane if the

patient is standing frontally in front of the extraction

sensor. This plane should not be left during the abduc-

tion movement. In addition, the knee, i.e. the angle

between the vectors CE and DE, can be bent too much

during the movement. This simpliﬁed movement re-

sults in a compensatory movement that is contrary to

recovery. A bent upper body, i.e. a tilting of the entire

upper area of the extraction points, also has an un-

favourable effect on the target movement. Only when

all these factors are taken into account can a success-

ful movement be counted and processed as a repeti-

tion. Otherwise, the additional parameters create an

error image that is immediately presented to the pa-

tient visually in order to correct his or her movement

execution during the next repetition.

If this SBE-based movement is to be changed or

even extended, application-related problems arise. If

the angle of the abduction is changed, the associated

variable must be changed either directly in the source

code or with the help of a GUI connection, which is

still quite easy to do with this simple abduction move-

ment. Although in this case the user has to switch

through the four parameter defaults and parameterise

the appropriate one, this can be done. If, however, in

addition to this abduction, a ﬂexion of the left arm and

a slightly offset extension of the right arm with a pen-

3D Reference-Based Skeletal Movement Evaluation

151

dulum movement of the upper body are to be realised,

a GUI-based arrangement of the training movement

quickly reaches its limits, becomes extremely time-

consuming to handle and results in confusing layouts.

Temporally different movement sequences are difﬁ-

cult to implement with the SBE.

3 REFERENCE-BASED

METHODS

In order to avoid the time-consuming manual individ-

ual parameterisation of an SBE training, reference-

based methods of training planning can be used. The

creation and use of the individualized patient refer-

ence is carried out by the therapists with the assis-

tance system (Lehmann et al., 2021). In the assis-

tance system’s reference recording mode, the ther-

apist’s movements are recorded and stored as a se-

quence in the system. These movement sequences are

available as digitized 3D skeletal movements. A ref-

erence is equivalent to a repetitive motion where the

start and end points can be identiﬁed. The advantage

of the reference is that it can be created by an expe-

rienced therapist, making it an ideal therapy and indi-

vidually adapted to the patient.

Figure 3: A short movement reference that shows the rais-

ing and lowering of both arms.

The ﬁgure 3 shows a reference image, where the ref-

erence consists of 100 individual steps and serves as

the basis for the patients’ assisted movement rehabil-

itation. The movement in the ﬁrst position of the ref-

erence and the last one are similar in their orientation

and position.

The original skeleton extraction does not deter-

mine the exact and real extraction points in ev-

ery frame due to varying lighting conditions, sensor

deﬁcits and model extraction inaccuracies. As a re-

sult, when motion vectors of the individual frames are

recorded, shifts and deviating, geometrically jumping

points occur. This has to be smoothed in order to de-

sign the subsequent processing of the reference in the

best possible way and to reduce input errors as much

as possible.

The comparison of a current movement with a move-

ment reference is explained using two methods inte-

grated in the assistance system.

Figure 4: The original reference on the left is imprecise due

to errors in the extraction of the individual points and re-

quires smoothing. The mean reference in the middle was

smoothed with the SMA(Dreszer, 1975) and ten neighbor-

ing points in each direction of the gradient. The right refer-

ence was smoothed ten times consecutively, involving two

adjacent points.

3.1 Reference Based Evaluation (RBE)

Therapists and patients differ in their physical size.

The reference created by medical professionals is

therefore not immediately applicable to the anatom-

ically deviating patients. An adjustment of the sizes

of skeletal extractions must be made beforehand. The

current skeletal information of the patient is used for

the 3D visualisation of the avatar. Since the current

extraction points are used for the visualisation, the

size must be adapted to the reference (ﬁgure 5).

Figure 5: The original reference (blue) has been adjusted to

the bone length of the current real sequence (orange). The

adapted reference (magenta) is the target of the adjustment.

For the comparison of the movements between the

reference and the current skeleton some calculations

and parameter declarations are needed. Blue sections

in ﬁgure 6 are reference sections, orange stands for

the current movement and the sections in the orange

box are calculated at runtime.

At the beginning, the length of the patient’s bones

must be determined in order to use this for the ad-

justment of the reference skeleton. For this purpose,

the assistant system tells the patient to stand relaxed

and calmly frontally in front of the extraction sensor

in order to keep his bone length extractions as stable

as possible. The initialisation process must provide at

least 60 frames of almost identical length values in or-

der to use these as the basis for the adaptation. These

bone lengths are applied to the smoothed reference

and this is adapted once to the corresponding size.

IMPROVE 2023 - 3rd International Conference on Image Processing and Vision Engineering

152

Figure 6: The stages of RBE based movement assessment

to be carried out.

The scaled reference S must be adjusted to the

patients’ position. For this, the difference vector v,

which is deﬁned between the root node of the ref-

erence skeleton S

and the root node of the patient

skeleton P

is calculated. The root vector of the skele-

ton is the vector from which all extraction points of

the skeleton originate and is usually equated with the

middle hip.This difference vector v shows the location

difference between reference S and the current extrac-

tion P and comes about when the standing positions

of the patient and the reference are not exactly on top

of each other. To compensate for this, every extrac-

tion point and every sequence of the scaled reference

S is now subtracted with this displacement vector v.

Another adjustment to be made for each extraction

run is the rotational shift of the current motion skele-

ton to the associated reference frame. This difference

can arise when there is a deviation in the orientation

rotation of the reference to the current motion. The ro-

tation of the entire skeleton can best be understood at

the hip, because this is anatomically ﬁxed. It is there-

fore not possible to move the left and right hip extrac-

tion point independently of each other, since these two

are connected by the hip bone(Schwegler and Lucius,

2016).

Figure 7: When comparing a non-rotated (left) and rotated

hip (right), the larger distance between the skeletal points is

noticeable in the former.

The problem of the resulting angle between the hip

vectors to be compared is shown in ﬁgure 7. In both

cases the reference has been adjusted to the current

movement. The comparison was made on an identi-

cal frame index of the animation. Hip rotation was

ignored in the left ﬁgure. The distances between the

extraction points of the blue reference and the ma-

genta adaptation skeleton are much larger than on the

rotated side.

After the size, the position and the rotation differ-

ence have been applied to the reference S, the compar-

ison of the extraction vectors can be generated. For

this, all adapted reference sequences R

are compared

with the current extraction points P. Since the move-

ment executions are not completely identical, a 3D

vector ∆ is used to calculate the valid tolerance range

for the current extraction point comparison.

∑

j=0

∑

m=0

(S[ j]

− ∆) < P

∑

j=0

∑

m=0

(S[ j]

+ ∆) (2)

The above inequality 2 is checked for each dimen-

sion. If one of the three-dimensional extraction points

is outside of this tolerance range, it is detected as a

motion error and is colored.

3.2 Segment Reference Based

Evaluation (SRBE)

The RBE considered in the previous section is a rel-

atively complex procedure to carry out the desired

movement evaluation. To overcome this complex

preparatory steps such as stable bone length calcula-

tion and its adapted reference adjustment as well as

hip transformation adjustments per updated extraction

frame must be carried out. These disadvantages can

be avoided with SRBE.

Figure 8: The stages of SRBE based movement assessment

to be carried out.

Blue sections in ﬁgure 8 are reference sections, or-

ange stands for the current movement, the sections in

the orange box are calculated at runtime. The angles

between the bones are independent from the length

of the bones. Starting from a root extraction point,

which is found in the middle hip extraction point, the

known bone connections of neighboring extractions

and their connection angles can be used to determine

the position of the extraction points under consider-

ation. In contrast to the reference-based comparison,

3D Reference-Based Skeletal Movement Evaluation

153

the one-off calculation of the bone lengths is saved, as

is the adjustment of the reference skeleton sizes, the

permanent shift of the reference to the origin and the

rotation of the hip, since the angular considerations of

the skeletal extraction points already include them.

The calculation of the angles between the indi-

vidual bone segments of the reference must also be

carried out once, at the beginning of the movement

tracking. The already smoothed reference animation

is required for this. The calculation of the individual

angles is done segment by segment for each recorded

movement sequence by iterating step by step over the

total number of reference movement images. To do

this, it is necessary to know how the skeletal extrac-

tion points are assigned to the segments, i.e. which

bones are connected to each other. Then the re-

quired angles between these joined bones can be cal-

culated. The connected bone segments of the skeleton

are taken as straight lines and each is transferred into

the parametric form. It is known that these straight

lines intersect in space as they converge at the associ-

ated extraction point. The equation of a straight line

is as follows.

x = a + λu (3)

The vector a is the extraction point of a bone from

the bone segments table that corresponds to the ﬁrst

index. The parameter u is the direction vector of this

bone line. That is the difference between the end vec-

tor b of the straight line and its start vector a. Each

value of λ corresponds to exactly one point on the

straight line. The associated indices of the bones can

be read from the given composite bone segment table

of the skeleton extraction.

Once the parametric forms have been determined

for both lines that belong together, the angle between

the two can then be calculated. For this, the direction

vector u of the ﬁrst bone in the segment is processed

with the direction vector v of the second bone in the

formula 1.

The arccosine of the angle α returns a value be-

tween 0° and 180°. These angles are recorded for

all segments of the frame and across all frames of

the animation and saved in the assistance system as

a variable rsa for later processing. After the angles

have been determined for all segments of the ref-

erence frame and across all possible frames of the

recorded animation, they can now be used for a move-

ment comparison with the current angle segments.

However, it is not enough just to determine the angle

of the bone segment. The position of the composite

bones in relation to the connection vector S must also

be evaluated. For this, the position of the points A and

B must be evaluated. Therefore, the angles between

Figure 9: Calculation of the connection angle of the bone

segment and the positional relationships of the composite

bones.

AS and SB to the standard planes are calculated. By

looking at the x, y and z values of the two points A and

B and comparing them with the corresponding values

of S, the direction of execution can be determined ex-

actly. Positive or negative values are obtained, which

are multiplied by the angles to the standard planes to

determine the ﬁnal direction angles. For each angle α

there are three positive or negative angles for A and B.

The current angles are determined from the pa-

tient’s current movement extraction. The calculation

is analogous to the calculation of the reference angles,

with the difference that the collection is not based on

the total reference, but only on the current extraction.

It is then checked whether a frame that meets the crite-

ria is found in the set of reference segment angles R

and the alignment angles of R

and R

of the entire

reference animation. If a matching angular segment

frame is available, the patient’s current movement is

judged to be correctly executed based on the segment

angles P. Otherwise, a movement error is detected in

the deviating extraction segment.

∑

j=0

∑

m=0

(R[ j]

− ∆) < C

∑

j=0

∑

m=0

(R[ j]

+ ∆) (4)

The deviation tolerances ∆ of the current patient an-

gle C

are shown in the formula 4 and can be set in

the system. The tolerance allow a controllable devi-

ating movement execution with respect to the refer-

ence. This ensures that all patients, according to their

ﬁtness level, can perform these reference exercises.

To increase the exercise intensity, only the tolerance

has to be reduced to bring the patients more and more

into the desired optimal starting reference during their

exercises. For the comparisons of R

and C

and of

and C

, there are similar tolerence frames. Only

when all seven conditions are evaluated as correct is

the executed patient movement also correct.

4 SUMMARY

The SBE is the standard method for performing the

evaluation. Only the current angles between the de-

ﬁned and selected bone segments of the individual

IMPROVE 2023 - 3rd International Conference on Image Processing and Vision Engineering

154

extraction frames are taken into account. If this an-

gle corresponds to the previously created parameters

and does not touch any of the necessary error patterns,

the movement is evaluated as correct, otherwise it is

marked as a non-compliant movement.

The RBE is a reference-based method and is there-

fore not parameterised like the SBE, but receives its

comparison values from a previously created refer-

ence movement. This reference movement is created

individually for the respective patient and serves as

the basis for the targeted movement comparison.

With the RBE, the reference is adjusted to the size

of the current patient and their speciﬁc spatial coordi-

nates. During the execution of the patient movement,

the patient is guided through the repetition with the

help of bounding boxes that reﬂect the tolerance level

of the movement deviation. If the patient moves out-

side the permissible range, an error is detected and a

corresponding visual message is given to the patient.

The second reference-based method considered is

the SRBE. This is not based on length comparisons

but on the angles between the bone segments of the

reference. These angles are also evaluated in terms of

their orientation to determine the correct direction of

execution of the given joint angle. Due to the angle

evaluation, the SRBE is not limited to static length

information and also works with changing reference

skeletal sizes, for example when a patient moves to-

wards or away from the extraction sensor.

Table 1: Advantages and disadvantages of the motion eval-

uation methods.

Description SBE RBE SRBE

clearly error patterns ✓ (✗) (✗)

ﬁxed error patterns ✓ ✗ ✗

high creation time ✓ ✗ ✗

idealized reference ✗ ✓ ✓

speed evaluation ✗ ✓ ✓

fast execution speed ✓ (✓) (✓)

single-joint overview ✗ ✓ ✗

real-time ✓ (✓) ✓

change position to cam ✓ ✗ ✓

easy GUI usage ✗ ✓ ✓

5 CONCLUSION AND FUTURE

WORK

The generation and use of an individual reference

has the advantage of subject-speciﬁc correct execu-

tion. A patient can then follow this reference to per-

form the best possible therapy. The two reference-

based methods presented here can realise movement

assessment. The RBE shows exactly those local 3D

positions of the skeletal points that have to be ap-

proached by the patient, the SRBE enables this pre-

cise movement tracking on the basis of the evaluated

angle sequences within the reference. The beneﬁts of

reference-based movement capture do not only extend

to medical movement therapy. Rapid reference gen-

eration and reference evaluation could also be applied

in other areas where skeleton-based motion detection

would be helpful. The reference could be used for

training steps in the assembly industry as well as for

learning a musical instrument. As long as the move-

ment patterns can be recorded and extracted, they can

be evaluated for all movement-speciﬁc tasks and ap-

plied to the situation.

REFERENCES

Ant

on D, Go

ni A, I. A. (2015). Exercise recognition for

kinect-based telerehabilitation. In Methods Inf Med.

2015, pages 145–155.

Budliger, H. (2021). Demograﬁscher Wandel und

Wirtschaft. Springer Gabler, Berlin, 1st edition.

Capizzi, T. (2002). Inspired 3D Modeling and Texture Map-

ping. Cengage Learning PTR.

Dreszer, J. (1975). Mathematik Handbuch. Verlag Harri

Deutsch.

Lehmann, L. (2020). Visigrapp 2020. In Individual Avatar

Skeletal based Animation Feedback for Assisted Mo-

tion Control, pages 206–213. Scitepress.

Lehmann, L., Seidel, R., and Hirtz, G. (2021). Supervised

skeleton-based reference movement adjustment. Cur-

rent Directions in Biomedical Engineering, 7(2):1–4.

osch, C. (2019). Paper templates. In Validation of an as-

sistance system for motion analysis. German Journal

of Sports Medicine.

Menche, N. (2003). Biologie Anatomie Physiologie, vol-

ume 5. Urban und Fischer Verlag.

Nitzsche, N. (2018). Assistierte Bewegungskontrolle in der

Rehabilitation durch intelligente Sensortechnologie.

readbox unipress, M

unster, 1st edition.

Ren, L., Patrick, A., Efros, A., Hodgins, J., and Rehg, J.

(2005). A data-driven approach to quantifying natural

human motion. ACM Trans. Graph., 24:1090–1097.

oder, T., M

uller, M., and Clausen, M. (2023). Ge-

ometrische relationen f

ur die bewegungsanalyse.

Schwegler, J. and Lucius, R. (2016). Der Mensch -

Anatomie und Physiologie. CNE Bibliothek. Thieme.

Wolff, D. (2018). OpenGL 4 Shading Language Codebook.

Packt Publishing, Birmingham, 3nd edition.

Zhao, W., Feng, H., Lun, R., Espy, D. D., and Reinthal,

M. A. (2014). A kinect-based rehabilitation exer-

cise monitoring and guidance system. In 2014 IEEE

5th International Conference on Software Engineer-

ing and Service Science, pages 762–765.

3D Reference-Based Skeletal Movement Evaluation

155