Virtual Reality Environment for the Validation of Bone Fracture

Reduction Processes

J. J. Jim

enez-Delgado

, A. Calzado-Mart

ınez

, F. D. P

erez-Cano

and A. Luque-Luque

Graphics and Geomatics Group, University of Ja

en, Ja

en, Spain

Keywords:

Fracture Reduction, Virtual Reality, Reduction, Validation, Usability, Interaction.

Abstract:

This work presents a virtual environment for the validation by experts of computer-assisted bone fracture

reduction. This environment is composed of VR glasses and 3D controllers (HTC Vive) that allow interaction

and immersion in the scene in a realistic way. The virtual environment developed allows loading fractured

bone models (fragments) so that the specialist performs a virtual fracture reduction and its results can be used

for the validation of algorithms and assisted reduction techniques. Once the fragments are loaded, the user can

perform an interactive reduction of the fracture, visualizing the fragments in 3d from different views, moving

the fragments in 3d and placing the fragments in space to observe the reduction in detail. Once completed,

it allows the reduction to be exported so that it can be compared with other fracture reduction systems. The

system has been tested by specialists in traumatology and a usability study has been carried out. Finally,

the system has been empirically validated and used to compare the performance of other computer-assisted

reduction systems.

1 INTRODUCTION

Fracture reduction is a surgical procedure to repair a

fracture or dislocation in the correct alignment. This

sense of the term reduction does not imply any type

of elimination or quantitative decrease, but rather im-

plies a restoration. It must be taken under consider-

ation that multiple fragments can be generated in a

fracture, some of them with microscopic size. The re-

duction of a fracture requires the union of the larger

fragments, ignoring the smaller ones. Therefore, the

reduction of a fracture does not imply that the frag-

ments must be completely joined at all points.

When a bone is fractured, the fragments lose their

alignment in the form of displacement or angulation.

For the fractured bone to heal without any deformity,

the bone fragments must be realigned to their normal

anatomical position. Orthopedic surgery attempts to

recreate the normal anatomy of the fractured bone by

reducing displacement.

The goal of virtual reality is to create an immer-

sive environment that is as realistic as possible. The

https://orcid.org/0000-0003-3014-0496

https://orcid.org/0000-0001-8690-6409

https://orcid.org/0000-0002-8065-8173

https://orcid.org/0000-0001-5869-5683

use of virtual reality to work with bone fragments, im-

proves the visualization, the perception of the fracture

zone, to achieve a better reduction, and the interaction

of the user with the fragments is more realistic. These

features allow the use of the tool to perform precise

fracture reductions and their subsequent comparison

with reduced fractures using fracture reduction algo-

rithms allowing validation of the results. In addition,

it could be extended as a training tool for fracture re-

duction, although the focus of the tool should be the

inverse, with the help of experts, datasets of fractures

can be manually reduced to provide a reference to val-

idate the results of algorithm.

The summary of this article is as follows. In the

next section we analyze the previous work on reduc-

tion of bone fractures, analyzing the methods used for

the validation of the obtained models. It is comple-

mented with information on other fracture reduction

simulation environments. After the background the

design of the tool for the validation of fracture re-

duction processes is presented. The following section

presents and discusses the results obtained in the pro-

cess of using the simulator for validation. Finally, the

conclusions summarize the objectives of this tool and

propose the work to be developed in the future.

Jiménez-Delgado, J., Calzado-Martínez, A., Pérez-Cano, F. and Luque-Luque, A.

Virtual Reality Environment for the Validation of Bone Fracture Reduction Processes.

DOI: 10.5220/0009175003990405

In Proceedings of the 15th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications (VISIGRAPP 2020) - Volume 1: GRAPP, pages

399-405

ISBN: 978-989-758-402-2; ISSN: 2184-4321

399

2 BACKGROUND

At the moment, data banks of bone fractures and

healthy bones are still inaccessible and a challenge

for the future. P

erez and Jim

enez (P

erez-Cano and

Jim

enez-Delgado, 2019a) have developed a tool that

allows the fracturing of bones and the obtaining of the

different fragments, ﬁgure 1. The problem with this

approach is that it is necessary the use of bone mod-

els and the results are based on the use of 2D frac-

ture patterns so that information about the fracture is

lost and is not entirely realistic. Therefore, the easi-

est, and most accurate method of obtaining 3D bone

models, is the segmentation of medical images of real

fracture cases. There are other approaches to obtain

geometric models of bones such as 3D scanning that

yield similar results, but the main problem they have

is time and loss of accuracy as identiﬁed by P

erez and

Jim

enez (P

erez-Cano and Jim

enez-Delgado, 2019b)

in evaluating the different approaches to obtaining

a geometric bone model. Paulano (Paulano et al.,

2014) concluded that traditional segmentation meth-

ods work correctly with healthy bones, but that in

fragmented bones, it was not possible to identify the

different fragments that make up a fracture. Paulano

also proposes a method based on 2D region growing

(Justice et al., 1997; Fan et al., 2005). This method is

based on the establishment of several seeds along the

bone so that the regions grow and allow the identiﬁ-

cation of the different fragments.

Figure 1: Extraction of the fracture pattern through the frac-

ture of a femur after a process of mechanical experimenta-

tion. Figure a represents the femur at the time the fracture

occurs, ﬁgure b indicates the fracture zone with red colour

and ﬁgure c the fracture pattern obtained. Extracted from

erez-Cano and Jim

enez-Delgado, 2019a).

Studies such as the directed by Citak (Citak et al.,

2008), emphasizes the importance of using new tech-

nologies to achieve a more realistic visualization. As

demonstrated in this study, improved visualization

and interaction allows for better planning of fracture

reduction and therefore obtaining more precise and

efﬁcient reductions as a result. The advantages of vir-

tual reality in the ﬁeld of bone fracturing have been

known for quite some time. Tsai (Tsai et al., 2001)

also conclude that simulations in a virtual environ-

ment allow better planning, mainly by improving the

visualization of the fracture, and obtaining better re-

sults. In terms of interaction in these environments,

Gusai (Gusai et al., 2017) analyzed the interaction of

a natural user in a realistic environment. The controls

of virtual reality systems have 6 degrees of freedom,

as in the case of HTC Vive. The use of these devices

are more intuitive in positioning tasks in immersive

environments compared to the use of other types of

options such as the use of gestural interfaces.

There are currently many works that attempt to

increase the degree of automation of bone fracture

reduction, but there is no standardized way to vali-

date the results obtained. The most direct and evident

method is a subjective visual assessment, but this does

not provide quantiﬁable results although it serves to

validate results from quantitative methods. One of

the forms used to objectively validate the reduction

of bone fractures is the comparison with the symmet-

rical bone (F

urnstahl et al., 2012; Vlachopoulos et al.,

2018) of the same patient. This is not always possi-

ble since in real clinical cases only the study of the

fracture area is normally done, mainly because of the

cost in money and time. There are occasions in which

it is not possible to obtain the scan of the symmetri-

cal bone, bones that do not present symmetry, medical

anomalies or traumatisms with fracture in both bones.

When studies are performed on donated human bones

or on animal bones, which are mechanically frag-

mented to simulate different types of lesions, there

is the alternative of scanning the bone before and af-

ter the experiment, which permits comparison of the

reduced fragments with the intact model (Liu et al.,

2019). In the research by Paulano (Paulano-Godino

and Jim

enez-Delgado, 2017) a set of tools developed

by the same team (Paulano et al., 2012) is used for

the manual reduction carried out by experts, then the

measurements are extracted and compared with the

reduction obtained by its algorithm to extract the dif-

ference in translation and rotation between manual

and automatic reduction.

Lately a multitude of Virtual Reality, Augmented

or Mixed devices have appeared, which have been

tested in hospital environments and facilitate the re-

duction of a fracture. This work aims to address

the problem of validating the reduction of bone frac-

tures based on the work of Paulano (Paulano-Godino

and Jim

enez-Delgado, 2017), taking advantage of ad-

vances in computer person interaction and visualiza-

tion using Virtual Reality techniques.

GRAPP 2020 - 15th International Conference on Computer Graphics Theory and Applications

400

3 VIRTUAL ENVIRONMENT

The proposed virtual reality environment has been

implemented using the Unity graphics engine, as seen

in ﬁgure 2. This tool recreate a virtual world familiar

to future users of the system, an operating room, with

the aim of improving immersion and focusing users

in a surgical environment for fracture reduction. The

hardware incorporated in the HTC Vive bundle pro-

vides full immersion, the kit is formed by a VR helmet

with multiple sensors to determine its spatial position,

proximity sensors and gyroscope. For the control of

the environment, it incorporates controllers that allow

manipulating the fragments in 3D space and provide

feedback through vibration.

The system allows to select a fragment, place it in

the space and proceed to perform the reduction with

the remaining fragment, ﬁgure 3. The ﬁrst time that

a fragment is selected, by clicking the right trigger

while the beam touch it, it jumps in front of the right

controller and stays connected to to motion of the con-

troller, once the user click the right trigger again the

fragment changes to a non selected status and starts

to follow the left controller. Non selected fragments

stay attached to the left controller, so the user can

move and rotate independently the inactive fragments

with the left hand and the selected fragment with the

right hand, this allows users to observe the reduction

process from different angles to improve results. The

fragments on the left hand can be selected as many

times as necessary, the second and subsequent times

when the fragments are selected they do not move to

the right controller, they only get attached to it so the

user can make small corrections.

Throughout the process the user receives contin-

uous feedback through vibration and visualization of

speciﬁc elements located on the fracture zone when

the fragments collide, to perceive the interaction be-

tween fragments like a real collision. These compo-

nents are called ”colliders” in the Unity system, in

ﬁgure 4 shows the active colliders as small yellow

spheres.

Once a satisfactory result is reached the reduction

can be ﬁnished by clicking the left trigger, ﬁgure 5

presents a reduced fracture with Windows Mixed Re-

ality controllers, the use of Unity allows the applica-

tion to be multiplatform and compatible with a wide

range of RV helmets. Finally, the relative positions

of both fragments, translation and rotation, are stored

and used as ground truth to compare the results from

other automatic or semi-automatic fracture reduction

systems.

4 RESULTS AND DISCUSSION

As has been seen in previous sections, virtual reality

provides better visualization and greater control over

daily actions. In the ﬁeld of traumatology, an immer-

sive environment allows the users to have better con-

ditions when conducting studies. In this section, the

advantages of the tool for reducing bone fractures are

analysed, comparing the results achieved with the ob-

tained through studies of a similar scope. A research

has also been carried out about the experience of the

use of the tool by experts in fracture reduction pro-

cesses.

4.1 Automatic versus Manual

Reduction

A software has been developed that allows the valida-

tion of automatic algorithms for the reduction of bone

fractures using the bone reductions conducted by ex-

perts in a virtual environment as the ground truth.

Thus, the error is then calculated as the absolute value

of the difference between the result of the automatic

and manual reduction in translation and rotation, the

closer each value is to zero, the more accurate it is.

The ﬁrst parameter is the distance error between

the two fragments measured from the center of mass

of each model. The second one is the rotation error

that was calculated in two ways, as the average differ-

ence around the fragments three 2nd moment vectors,

proposed by Paulano (Paulano-Godino and Jim

enez-

Delgado, 2017), and as α and β errors, used in the

work of F

urnstahl (F

urnstahl et al., 2012). α is the

difference around the two largest 2nd moment vec-

tors and β represents the rotational difference around

the smallest 2nd moment vectors.

In experiments, some cases has been tested and

compared with the results obtained by the algorithm

of Paulano (Paulano-Godino and Jim

enez-Delgado,

2017). In complex cases, when the fracture consist of

more than two fragments the reduction of the fracture

is applied in pairs, that means, ﬁrst two fragments are

reduced, one to each other, and then the obtained frag-

ment is reduced with the remaining fragment. Figure

6 illustrates the automatic reduction of the ﬁbula frac-

ture, the same fracture used in section 3 to demon-

strate the functioning of the virtual environment. The

fracture of the tibia is composed by three fragments,

the second reduction was performed using the previ-

ous reduction of fragment 1 and 3 and fragment 2.

At ﬁnal stages of development, the system was

used to tune the parameters of a automatic frac-

ture reduction process based on a modiﬁcation of

the ICP algorithm, which is being currently devel-

Virtual Reality Environment for the Validation of Bone Fracture Reduction Processes

401

Figure 2: Scene designed in Unity of the virtual environment.

Figure 3: Bone fragment selection system, the beam emit-

ted from the controller improves accurate picking.

oped. The table 1 shows the results of the new al-

gorithm, which are very promising, improving those

obtained in the works of Paulano (Paulano-Godino

and Jim

enez-Delgado, 2017) and F

urnstahl (F

urnstahl

et al., 2012), table 2.The average translational error

has been reduced from 1.80 and 1.11, Paulano and

urnstahl results respectively, to 0.58, which repre-

sents an average increase in accuracy of 100%. The

rotation error in the work of Paulano is 3.25, F

urnstahl

Figure 4: The colliders are shown as small yellow spheres

that generate haptic feedback in the controllers.

obtains an error value in alpha and beta of 3.1 and

3.51, the algorithm in development has achieved val-

ues of 2.41, 3.30 and 0.92 in such parameters, only

the alpha error is slightly worse but the beta value has

been really improved.

The error of the fracture reduction algorithm could

be minimized thanks to this framework, once reached

a certain level of precision in the reduction of frac-

tures is difﬁcult to differentiate visually if one result

is better than another. This is why it is important to

create an environment and a standardized process in

which the results in fractures of any bone can be com-

pared and improved.

One of the most important advantages of virtual

reality is the simplicity of use, replacing the interac-

tion with keyboard and mouse by a more immersive

GRAPP 2020 - 15th International Conference on Computer Graphics Theory and Applications

402

Table 1: Results of an improved experimental reduction algorithm achieved with the aid of the developed system.

Fracture Fragments Translation error (mm)

Rotation error (mm)

Average Alpha Beta

Humerus 1 → 2 1,3202 1,3902 1,8582 0,6494

Fibula 1 → 2 0,2945 3,2845 4,5854 1,1135

Tibia 1 → 3 0,3586 3,9431 5,3503 1,6489

Tibia 1 3 → 2 0,3283 1,0213 1,4163 0,2622

Figure 5: Successful reduction of a ﬁbula fracture with a

Windows Mixed Reality device.

Figure 6: Improved automatic reduction of a ﬁbula fracture

thanks to the use of the developed framework.

experience with elements that track in real time the

head and hands of the user, which achieves a com-

pletely natural interface to the 3D world. For this

reason, the learning time is practically non existent

compared to the use of a 2D environment. This soft-

ware has been tested by three different user proﬁles,

all users have been able to successfully reduce several

bone fractures without the need to spend a great deal

of time reducing a bone fracture.

The equipment needed to run the virtual reality

framework is noticeably more powerful, we have used

Table 2: Average results of different algorithms when eval-

uating errors.

Study Traslation Rotation Alpha Beta

Furnstahl 1,11 — 3,10 3,51

Paulano 1,80 3,25 — —

Original 0,58 2,41 3,30 0,92

a PC equipped with a ﬁrst generation i7 microproces-

sor with 8GB of RAM, an NVidia 1060 graphics and

the virtual reality glasses, HTC Vive.

4.2 Usability Tests

The system has passed through a validation process in

which a sample of specialists has been selected. This

user group formed by two experts with experience in

fracture reduction, two users with experience in the

use of fracture reduction applications and a radiolo-

gist expert in the analysis of traumatology images. In

addition to the difﬁculty of ﬁnding real cases of bone

fractures due to conﬁdentiality reasons, the collabo-

ration with the medical community has proved more

costly than expected, mainly due to the lack of time

of the professionals and the rejection that such recent

technologies produce.

Table 3: Results of the user experience survey.

Feature Rating

Image quality 4.83±0.37

Immersive sensation 4.83±0.37

Realism 3.83±0.69

Graphics ﬂuidity 4.50±0.50

Intuitive control 3.66±0.94

Learning curve 4.33±0.47

To conduct the usability study, the virtual reality

device was ﬁrst shown to users, equipped with it, and

asked to observe and interact with the environment

without explanation of the application workﬂow. In

order to evaluate the ﬁrst impression of the system, a

questionnaire had been designed with general ques-

tions about the quality of the images, immersive sen-

sation, realism, ﬂuidity of the graphics, intuitive con-

trol and learning curve. The questionnaire has been

Virtual Reality Environment for the Validation of Bone Fracture Reduction Processes

403

Table 4: Results of the questionnaire about the reduction process.

Fracture

Ease of selection

and movement

of fragments

Accuracy and feedback

of the collisions

Ease of the reduction

process

Accuracy

Fibula 4.66±0.62 3.83±0.99 4.16±0.69 4.16±0.69

Femur 1 4.75±0.43 3.83±0.80 3.66±0.75 4.00±0.58

Femur 2 4.66±0.47 3.75±0.43 3.33±1.11 3.50±0.76

Femur 3 4.83±0.37 3.58±0.49 3.16±0.69 3.33±0.47

Table 5: Score of the reductions granted by the experts.

Fracture Accuracy

Fibula 4.50±0.60

Femur 1 4.66±0.82

Femur 2 4.83±0.47

Femur 3 3.83±0.80

designed using ﬁve level Likert responses.

Table 3, shows that the experience of use is re-

ally satisfactory, only two values, ”realism” and ”in-

tuitive control” obtain a score slightly lower than 4

points. According to users, the low level of realism

is mainly due to the resolution of the models, which

have been extracted from CT images without apply-

ing any smoothing ﬁlter that could reduce accuracy,

and to the scale applied to easily handle the smaller

fragments. The score in this aspect could be improved

in the future by applying an algorithm to smooth the

model obtained after segmentation, but preserve as

much detail as possible.

The lowest value was ”intuitive control”, which is

to be expected considering that the use of the software

has not been previously indicated. Subsequently, brief

instructions for the reduction procedure have been

provided to each user, after which they have been

asked to perform four reductions of bone fractures

with two fragments each.

After completing the reductions, they were asked

to answer a questionnaire about the ease of selection

and movement of the fragments, the accuracy and

feedback of the collisions, the ease of the reduction

process, and ﬁnally each user was asked to rate the

accuracy of the result obtained (table 4). The values

obtained in relation to ease of movement and selec-

tion validate the user interface implemented using the

VR controllers, while the score associated with feed-

back indicates a need to adjust the vibration response

curve.

The ease of reduction and accuracy are markers

that are very interrelated, if users have a perception

that they do not get an optimal result they will nega-

tively evaluate both parameters. Considering the con-

clusions obtained in section 4.3, it could be assumed

that both aspects of the process have been subjectively

evaluated by the sensation of low performance of the

users.

4.3 Validation by Experts

In section 4.2, users have been asked about the ex-

perience using the system. As a result, it has been

obtained low values in the perception of accuracy in

the reduction. To conﬁrm whether the value obtained

is objective or users rated their own result negatively,

users repeated each reduction three times and then the

result of their peers was presented to each expert to be

scored, to increase reliability the whole process was

conducted hiding the author of each reduction.

In table 5, it can be observed that the evaluation of

the reduction of fractures by the rest of the experts is

greater. The fragments of the fracture femur 3 show

loss of osseous material and some degree of defor-

mation, which explains the lower level of precision

appreciated.

5 CONCLUSIONS

This article proposes a system for reducing fractures

using a virtual environment that provides fast and re-

alistic results for the validation of bone fractures, as

an alternative to rapid prototyping reduction of bone

fragments and subsequent capture of fragment posi-

tions.

The main objective that has been sought and

achieved is the validation of automatic algorithms for

the reduction of fractures. For this purpose, the re-

sults generated by expert users have been compared

with the results obtained from fracture reduction al-

gorithms using different metrics deﬁned in the results

section.These results have demonstrated that the pro-

posed process allows to compare and evaluate pre-

cisely the reductions of fractures improving those

present in the literature in different aspects.

Moreover, during the creation of the system sev-

eral tests have been carried out with different types

of users in order to study in depth how the immer-

sion of users in a virtual environment facilitated the

GRAPP 2020 - 15th International Conference on Computer Graphics Theory and Applications

404

reduction of fractures.These studies have proved that

the improvement in visualization and interaction have

been essential to the improvement of the results ob-

tained in the reduction of fractures.

In view of the ease of use and the natural way in

which bone fragments are manipulated, a future mod-

iﬁcation has been studied for use as training software

for traumatologists. Although this work has been used

exclusively for medical use, it could be extended to

other ﬁelds such as forensic medicine or archaeology.

ACKNOWLEDGEMENTS

This work has been supported by the Ministerio

de Econom

ıa y Competitividad and the European

Union (via ERDF funds) through the research project

DPI2015-65123-R.

REFERENCES

Citak, M., Gardner, M. J., Kendoff, D., Tarte, S., Krettek,

C., Nolte, L.-P., and H

ufner, T. (2008). Virtual 3d

planning of acetabular fracture reduction. Journal of

Orthopaedic Research, 26(4):547–552.

Fan, J., Zeng, G., Body, M., and Hacid, M.-S. (2005).

Seeded region growing: an extensive and comparative

study. Pattern Recognition Letters, 26(8):1139–1156.

urnstahl, P., Sz

ekely, G., Gerber, C., Hodler, J., Snedeker,

J. G., and Harders, M. (2012). Computer assisted re-

construction of complex proximal humerus fractures

for preoperative planning. Medical Image Analysis,

16(3):704–720.

Gusai, E., Bassano, C., Solari, F., and Chessa, M. (2017).

Interaction in an immersive collaborative virtual real-

ity environment: A comparison between leap motion

and HTC controllers. In New Trends in Image Anal-

ysis and Processing – ICIAP 2017, pages 290–300.

Springer International Publishing.

Justice, R. K., Stokely, E. M., Strobel, J. S., Ideker, R. E.,

and Smith, W. M. (1997). Medical image segmen-

tation using 3d seeded region growing. In Hanson,

K. M., editor, Medical Imaging 1997: Image Process-

ing. SPIE.

Liu, B., Zhang, S., Zhang, J., Xu, Z., Chen, Y., Liu, S.,

Qi, W., and Yang, L. (2019). A personalized pre-

operative modeling system for internal ﬁxation plates

in long bone fracture surgery—a straightforward way

from CT images to plate model. The International

Journal of Medical Robotics and Computer Assisted

Surgery, 15(5).

Paulano, F., Jim

enez, J. J., and Pulido, R. (2014). 3d seg-

mentation and labeling of fractured bone from CT im-

ages. The Visual Computer, 30(6-8):939–948.

Paulano, F., Pulido, R., and Jim

enez, J. J. (2012). An Appli-

cation to Interact with 3D Models Reconstructed from

Medical Images. In Navazo, I. and Patow, G., editors,

Spanish Computer Graphics Conference. The Euro-

graphics Association.

Paulano-Godino, F. and Jim

enez-Delgado, J. J. (2017).

Identiﬁcation of fracture zones and its application in

automatic bone fracture reduction. Computer Meth-

ods and Programs in Biomedicine, 141:93–104.

erez-Cano, F. D. and Jim

enez-Delgado, J. J. (2019a).

Macroescale fracturation of osseous models. In Vip-

IMAGE 2019, pages 492–500. Springer International

Publishing.

erez-Cano, F. D. and Jim

enez-Delgado, J. J. (2019b). To-

wards the modelling of osseous tissue. In Proceedings

of the 14th International Joint Conference on Com-

puter Vision, Imaging and Computer Graphics Theory

and Applications. SCITEPRESS - Science and Tech-

nology Publications.

Tsai, M.-D., Hsieh, M.-S., and Jou, S.-B. (2001). Virtual

reality orthopedic surgery simulator. Computers in Bi-

ology and Medicine, 31(5):333–351.

Vlachopoulos, L., Sz

ekely, G., Gerber, C., and F

urnstahl, P.

(2018). A scale-space curvature matching algorithm

for the reconstruction of complex proximal humeral

fractures. Medical Image Analysis, 43:142–156.

Virtual Reality Environment for the Validation of Bone Fracture Reduction Processes

405