A Multi-modal Brain Image Registration Framework for US-guided
Neuronavigation Systems
Integrating MR and US for Minimally Invasive Neuroimaging
Francesco Ponzio, Enrico Macii, Elisa Ficarra and Santa Di Cataldo
Department of Control and Computer Engineering, Politecnico di Torino,
Corso Duca degli Abruzzi 24, 10129 Torino, Italy
Keywords:
Neuroimaging, US-based Neuronavigation, Multi-modal Image Registration, Image Processing, MR-US
Image Integration.
Abstract:
US-guided neuronavigation exploits the simplicity of use and minimal invasiveness of Ultrasound (US) imag-
ing and the high tissue resolution and signal-to-noise ratio of Magnetic Resonance Imaging (MRI) to guide
brain surgeries. More specifically, the intra-operative 3D US images are combined with pre-operative MR im-
ages to accurately localise the course of instruments in the operative field with minimal invasiveness. Multi-
modal image registration of 3D US and MR images is an essential part of such system. In this paper, we
present a complete software framework that enables the registration US and MR brain scans based on a multi
resolution deformable transform, tackling elastic deformations (i.e. brain shifts) possibly occurring during the
surgical procedure. The framework supports also simpler and faster registration techniques, based on rigid
or affine transforms, and enables the interactive visualisation and rendering of the overlaid US and MRI vol-
umes. The registration was experimentally validated on a public dataset of realistic brain phantom images, at
different levels of artificially induced deformations.
1 INTRODUCTION
Neuronavigation consists in a comprehensive set
of computer-assisted and neuroimaging technologies
that help the neurosurgeons “navigating” within the
brain to guide complex surgical interventions, such as
brain biopsies or tumor resections.
Among the brain imaging modalities, Magnetic
Resonance Imaging (MRI) provides the best discrim-
ination between soft tissues inside the brain, and it is
able to image tumors’ borders with a good level of de-
tail. Furthermore, differently from Computer Tomog-
raphy (CT), it does not expose the tissues to danger-
ous ionising radiations, hence it is minimally invasive
for the patient. This makes it the preferred imaging
modality for planning brain surgical procedures.
On the other hand, planning a brain surgery based
on pre-operative MR scans, which might have been
acquired few days or even weeks before the surgery,
raises many issues, especially in the practice of neu-
rosurgical oncology. For example, in case of rapidly
evolving neoplasms, the navigation might be ex-
tremely error prone because of the changing geometry
and size of the lesion.
As a solution to this issue, modern neuronaviga-
tion systems integrate the information coming from
pre-operative MRI with intra-operative MRI scans,
possibly acquired several times after critical stages
of the surgical procedure. Besides avoiding macro-
scopic errors due to the evolution of the lesion after
the first set of scans, this setting allows to address
brain shifts naturally occurring during the surgery.
Brain shifts might alter the original position of crit-
ical cerebral structures, and include elastic deforma-
tions due to craniotomy, tissue resections, cyst de-
compressions, removal of cerebrospinal fluids or even
haemorrhage or use of diuretics. Leveraging on the
real-time anatomical information provided by intra-
operative MRI, the surgical plan can be altered to ac-
count for such shifts.
In spite of its many advantages, using intra-
operative MRI has also a number of critical logistic
drawbacks. In fact, it requires specialised operating
suites, costly and bulky instrumentations, as well as
longer anaesthesia and operating room time.
To overcome these drawbacks, in the last few
years ultrasound (US)-based techniques are becom-
ing more and more attractive for neuronavigation pur-
114
Ponzio F., Macii E., Ficarra E. and Di Cataldo S.
A Multi-modal Brain Image Registration Framework for US-guided Neuronavigation Systems - Integrating MR and US for Minimally Invasive Neuroimaging.
DOI: 10.5220/0006239201140121
In Proceedings of the 10th International Joint Conference on Biomedical Engineering Systems and Technologies (BIOSTEC 2017), pages 114-121
ISBN: 978-989-758-215-8
Copyright
c
2017 by SCITEPRESS Science and Technology Publications, Lda. All rights reserved
Figure 1: Scheme of principle of US-guided neurosurgery. A moving set of US volumes acquired during the surgical procedure
are combined with a fixed set of pre-operative MR scans.
poses (C. Nikas et al., 2003).
Indeed, US-based neuronavigation has major ad-
vantages compared to traditional MRI-based systems.
These advantages include its relatively low cost, sim-
plicity of use and minimal invasiveness, both in terms
of volume and complexity of equipment and of im-
pact on the patient. On the other hand, US images
have well-known limitations, such as low signal-to-
noise ratio and penetration depth. Moreover, they
have much lower soft tissue discrimination capabili-
ties compared to MR technology.
Hence, the latest trend in neuronavigation is the
use of hybrid surgical planning techniques, integrat-
ing pre-operative MRI scans with intra-operative US.
In this case, the procedure is guided by a fixed set
of pre-operative MRI scans and a moving set of US
volumes acquired during the surgery (see the scheme
of Figure 1). The pre-operative MR scans are used
to construct a structural model of the patient’s head,
and provide a detailed anatomical 3D map of the brain
and of the targeted lesions. The position of the US
probe with respect to the patient’s coordinate system
is obtained real-time using a probe tracking system.
Then, the pre-operative MR scans are registered and
overlaid on the US images acquired in the Operating
Room, updating the structural model of the brain’s pa-
tient based on the new anatomical information pro-
vided by US.
In such US-guided hybrid system, the accurate
automated registration of MR and US scans plays a
fundamental role. Nonetheless, while multi-modal
brain image registration has a very consolidated tra-
dition in other imaging technologies such as MR and
CT (Sarkar et al., 2005), the registration of US and
MR images is still a research topic in development,
with a number of challenges that need to be tackled:
(i) low signal-to-noise conditions typical of US
imaging. (ii) absence of highly contrasted anatomical
structures (e.g. bones, high-density tissues) driving
the registration algorithm. (iii) possible presence of
brain shifts induced by the surgical procedure.
Most of the currently available solutions do not
explicitly deal with elastic brain shifts (Coup´e et al.,
2012), or are user-dependent, in that they rely ei-
ther on the interactive delineation of markers or sur-
faces (Lunn et al., 2001; Liu et al., 2014), which is not
feasible in the context of real-time neuronavigation.
In this paper, we present an automated framework
tackling these issues. It takes as input MR and US
scans and probe positioning information, as provided
by a tracking system, and allows a fully-automated
registration and overlay of the two volumes, without
requiring any interaction from the user.
Our methodology is based on a non-rigid regis-
tration algorithm, in order to tackle possible non-
linear deformations, with a self-adjusting parameters
search. Nonetheless, the tool supports also other sim-
pler registration techniques, which can be selected
when to tackle stages of the procedure not implying
elastic warping between US and MR scans.
The registration accuracy is experimentally vali-
dated using a publicly available set of MR and US
scans from an anatomically realistic human brain
phantom, even in presence of extensive elastic defor-
mations.
The rest of the paper is organised as follows. In
Section 2, we describe the main modules of the pro-
posed framework. In Section 3, we discuss the multi-
modal image registration technique. In Section 4, we
provide few details on the algorithm’s set-up. In Sec-
tion 5, we present and discuss the experimental re-
sults. In Section 6 we conclude the paper.
2 SOFTWARE FRAMEWORK
The software was implemented in python and C++,
making use of ITK and VTK libraries (Yoo et al.,
2002; Schroeder et al., 2003). In the following, we
briefly describe the main modules, as shown in Fig-
ure 2.
A Multi-modal Brain Image Registration Framework for US-guided Neuronavigation Systems - Integrating MR and US for Minimally
Invasive Neuroimaging
115
Figure 2: Diagram of software architecture, with main modules.
2.1 Volumes Acquisition
Through this module, the software receives as input:
(i) the pre-operative 3D MRI scans of the patient.
(ii) the intra-operative US scans (reconstructed vol-
umes).
The two volumes are in their own reference space
and Field of View (FOV). Hence, the 2D slices com-
posing the two different input volumes do not match.
The module supports either DICOM, NIfTI or vtk im-
age formats.
2.2 Realignment and Reslicing
Through this module, the coordinate spaces of the
pre-operative MR images and the intra-operative US
volumes remapped to a common patient’s reference
space and FOV and resliced for voxels’ coherency.
This way, the 2D orthogonal slices of the two image
volumes are coherently reconstructed for both image
modalities.
To match the reference systems of US and MRI,
the software communicates with an external optical
motion tracking system, that is able to record the po-
sition and orientation of the US probe during image
acquisition. The tracking system processes this infor-
mation and provides a transformation matrix to our
software, in the form of a 4X4 matrix T:
x
= T · x (1)
which in matricial form is:
x
1
x
2
x
3
1
=
q
1,1
q
1,2
q
1,3
q
1,4
q
2,1
q
2,2
q
2,3
q
2,4
q
3,1
q
3,2
q
3,3
q
3,4
0 0 0 1
·
x
1
x
2
x
3
1
(2)
2.3 Multi-modal Image Registration
Through this module, the two realigned volumes are
co-registered.
More in detail, the MR volume containing the pre-
operative structural information of the patient’s brain
is deformed in order to match the intra-operative US
scans. Hence, brain shift deformations occurring dur-
ing the surgery can be compensated.
As mentioned in Section 1, this molule supports
three different registration methods, which will be de-
scribed in more details later.
These methods are:
(i) Multi-resolution rigid transform. Simplest
solution consisting in rotation and translation, which
ensures best computational time.
(ii) Multi-resolution affine transform. Con-
sisting in rotation, translation, scaling and shearing,
which was found to be the best option when elastic
deformations are negligible (e.g. at stages not requir-
ing brain resections).
(iii) Multi-resolution non-rigid transform. Con-
sisting in a non-linear deformable transform, where
BIOIMAGING 2017 - 4th International Conference on Bioimaging
116
local deformations at the voxel-level, including lo-
cal warping, are defined by a Free Form Deformation
(FFD) model as a mesh of control points. This ap-
proach was found to be the best for the correction of
extensive elastic brain-shift deformations.
2.4 Rendering/Visualization
Through this module, the realigned and co-registered
volumes can be visualised and overlaid. Two different
options of visualisation are supported:
(i) Volume rendering, with clipping box. When
this option is selected, the overlaid MR and US im-
ages are visualised using 3D volume rendering based
on a ray casting and 3D texture mapping technique.
Two different sets of colours are applied to render
the two imaging modalities. The user can interac-
tively rotate the volumes to obtain a better view of the
anatomical details in each modality. The visualisa-
tion supports cropping, clipping and blending modes.
Thanks to these features, the user can clip away the
volume at any point, allowing a better view of the in-
ternal structures.
(ii) Interactive 3D slider. When this option is
selected, the user can interactively switch from vol-
ume rendering to a 2D visualisation of single slices.
The user can navigate along either coronal, sagittal
or transverse directions using a sliding bar or simply
scrolling the mouse, select a plane of interest on the
rendered volume and then visualise the 2D image (ei-
ther MR, US, or a superposition of the two) obtained
cutting the volume through the selected plane.
3 REGISTRATION TECHNIQUE
The US-MR registration technique is implemented as
an iterative optimisation problem, where the search
for the optimal transform matching US and MR vol-
umes is driven by the maximisation of a fitness value
quantifying the similarity between the two input im-
ages. This can be schematically represented by the
components reported in Figure 3.
3.1 Similarity Metric
Of the large number of image registration similar-
ity measures that have been proposed over the years
(among the most relevant: cross-correlation, sum of
squared intensity differences, ratio image uniformity,
etc.), Mutual Information (MI) is widely acknowl-
edged as the best choice for the registration of multi-
modality images. Indeed, this metric is best suited to
measure the matching degree of images having differ-
ent characteristics in terms of range of image intensi-
ties, because it does not assume a linear relationship
among the two sets of intensity values. On top of that,
MI is an automatic intensity-based metric that does
not require the preventivesegmentation of anatomical
parts, surfaces or landmarks on the images, nor the
definition of specific features to drive the registration
process.
MI applies the joint probability distribution of pix-
els from the two images to measure the certainty that
the values of the first image pixels map to similar val-
ues in the other image, which is a reliable quantitative
measure of how similar the images are. Higher val-
ues of MI imply a large reduction in the uncertainty
(i.e. entropy) between the two distributions of pixels’
values, which is a clear indication that the two images
are better aligned.
More specifically, if U and V are the two image
volumes, their Mutual Information is defined as:
MI = H(U) + H(V) H(U,V), (3)
where H(U) and H(V) are the individual marginal en-
tropies of U and V, respectively, and H(U,V) is the
joint entropy.
On the other hand, the individual and joint en-
tropies are defined as:
H(U) =
i
P
U
(u
i
) · logP
U
(u
i
), (4)
H(V) =
j
P
V
(v
j
) · logP
V
(v
j
), (5)
H(U,V) =
i, j
P
UV
(u
i
,v
j
) · logP
UV
(u
i
,v
j
), (6)
where P is the marginal probability density distri-
bution of the intensities in the image, estimated based
on the method by (Mattes et al., 2001) using Parzen
histograms (Xu et al., 2008).
3.2 Optimizer
In order to reduce the computational cost of the al-
gorithm, the optimization problem is implemented as
Figure 3: Diagram of the multi-modal registration module.
A Multi-modal Brain Image Registration Framework for US-guided Neuronavigation Systems - Integrating MR and US for Minimally
Invasive Neuroimaging
117
Figure 4: Scheme of the multi-resolution optimization ap-
proach.
multi-resolution strategy, schematically represented
in Figure 4. The rationale of this approach is the fol-
lowing:
- First, the optimal transform parameters are
searched applying at a rough scale in terms of image
resolution (i.e. the input images are resampled so that
only a fraction of the available voxels are used for the
optimization)
- The transform determined at the starting level is
used to initialize image registration at the next stage,
at a finer scale (i.e. a higher number of voxels is used
for the optimization).
- This process is repeated until the finest scale pos-
sible (full image resolution) is reached.
3.3 Transform
As brain shifts might easily imply elastic deforma-
tions of the brain, the proposed method is a non-
rigid registration technique, that leverages on a de-
formable transform of points from one N-dimensional
space to another N-dimensional space. This deforma-
tion is defined at the voxel-level in terms of a Free
Form Deformation (FFD) model, where a set of de-
formation vectors~g is applied to a sparse regular grid
of control points, one deformation vector per control
point (see Figure 5).
As mentioned in Section 2, in order to provide the
user with less computationally intensive solutions in
stages of the procedure not involving elastic brain de-
formations, the framework supports also the use of
affine transforms (applying a linear combination of
translation, rotation, scaling and shearing, i.e. non-
uniform scaling in some directions), or of even sim-
pler rigid transforms (applying only translation and
rotation). In this case, the transforms can be expressed
geometrically in terms of transform matrices, as in
Equation 1.
3.4 Interpolator
The interpolator is applied to compute the voxel-wise
intensities of the transformed image at non-grid posi-
tions. The interpolation method affects the smooth-
ness of the optimisation search space as well as the
overall computation time. As a compromise between
accuracy and computational costs, our system imple-
ments B-spline interpolation for non-rigid transform
and linear interpolation for either rigid body or affine
transform models.
4 ALGORITHM SET-UP
Summarising, the registration algorithm works to-
wards the maximisation of MI similarity between US
and MR scans. This metric is embedded into an objec-
tive function and optimized numerically. As reported
in Equation 3, this requires an estimation of the proba-
bility density functions (PDFs) associated with the US
and MR images, as well as their joint intensities. Such
PDFs are not known a priori, hence they have to be
estimated from intensity samples, treated as indepen-
dent and identically distributed random measures. In
our implementation, a set of intensity samples is ran-
domly selected on the input image. Then, the PDF is
estimated at discrete bins uniformly spread within the
dynamic range. Entropy values are then computed by
summing over such bins. Hence, an accurate choice
of the number of bins is crucial for registration accu-
racy.
Besides sample bins, the main parameters that
need a careful set-up are the number of samples used
for the calculation (the higher this number, the higher
the significance of the estimation), as well as the size
of the mesh associated with the deformation field (the
higher the size, the higher the variety of deformations
that can be modelled by the system).
Figure 5: Non-rigid registration.
BIOIMAGING 2017 - 4th International Conference on Bioimaging
118
Figure 6: PVA brain phantom (http://pvabrain.inria.fr/).
As all these parameters can dramatically impact
on the computational time, the set-up needs to find
a reasonable trade-off between number of iterations
of the optimisation problem and registration accuracy.
In our system, this trade-off can be automatically ad-
justed based on a training pair of US and MR scans. A
simple heuristic algorithm is implemented, that starts
with a very coarse grid of parameters. Then, increas-
ingly finer search grids are applied around the optimal
parameters determined at each stage of the heuristics.
5 EXPERIMENTAL RESULTS
To experimentally validate our framework, we used
the PVA Brain 3D image dataset, a publicly avail-
able dataset of multi-modal images acquired from a
human brain phantom (Chen et al., 2010). The phan-
tom, made of polyvinyl alcohol cryogel (PVA-C), was
obtained using a shape mold 3D printed from ex-
vivo human brain MR images, hence it is anatomi-
cally very accurate and realistic (see Figure 6). The
phantom contains two inflatable catheters, allowing
to artificially induce brain shift deformations of a pri-
ori known entity. The dataset contains corresponding
MR and US scans at different stages of deformation.
Per each stage, the dataset contains also the transfor-
mation matrices provided by an external neuronavi-
gation probe tracking system, which can be used to
remap US and MR scans into a common reference
system, obtaining an initial rough alignment. Hence,
the dataset can be used to test image registration and
brain shift compensation techniques.
In order to obtain an objective measure of regis-
tration accuracy, we implemented the following vali-
dation procedure:
(i) With the help of a skilled neurologist, we inter-
actively extracted a set of nine anatomical landmarks
(i.e. points of interest) on the MR volume (see Fig-
ure 7).
(ii) We extracted the corresponding anatomical
landmarks on the US volume.
(iii) We run our registration pipeline on the US and
MR volumes.
We repeated steps (i) and (ii), independently, on
the US and MR volumes obtained after registration.
Then, we computed the Euclidean distance between
homologous MR-US markers before and after regis-
tration. The same validation procedure was run on
corresponding US and MR scans at three different de-
formation configurations, respectively with the two
catheters inflated by 0 ml - 0 ml (i.e. absence of de-
formation), 0 ml - 5 ml and 5 ml - 10 ml.
The ideal target of a US-guided neurosurgical sys-
tem is that MR and US volumes should be perfectly
superimposed (see an example of US-MR overlay be-
fore and after registration in Figure 8), and that the
homologous markers should coincide. Hence, we can
establish a validation procedure based on the follow-
ing:
- the Euclidean distance between homologous MR
and US markers provides a measure of the initial spa-
tial mismatch between the two sets of scans before
registration (due to either imperfect probe tracking or
to non-linear brain shift deformations occurring dur-
ing the surgery).
- The same distance computed after registration
provides a measure of the residual spatial mismatch
(where some small mismatch, reasonably in the order
of a mm, is possibly due to manual selection of the
markers).
- The comparison of distances computed before
and after registration provides a measure of registra-
tion quality, in terms of compensation of spatial mis-
match between US and MR scans.
Figure 7: Anatomical landmarks on the PVA brain phantom
images.
A Multi-modal Brain Image Registration Framework for US-guided Neuronavigation Systems - Integrating MR and US for Minimally
Invasive Neuroimaging
119
Figure 8: Example of registration outcome on PVA brain
images. Overlay of MR and US scans before (left) and after
(right) registration.
The obtained results are reported in the boxplots
of Figure 9, respectively for registration based on
rigid transform, affine transform and non-rigid de-
formable transform.
As it can be easily gathered from the boxplots,
even after the initial realignment based on US probe
tracking the US and MR scans have a consistent spa-
tial mismatch (up-to 8 mm). This mismatch is hardly
corrected by a simple rigid-body registration, because
it is mainly due to elastic deformations induced by the
inflatable catheters. Indeed, the most successful re-
sults are obtained by the non-rigid registration option
(see third graph of Figure 9). This technique allows
to obtain the best values in terms of residual distance
between the homologous landmarks, of about 2.5 mm
on average. On top of that, the obtained registration
accuracy is reasonably stable at different stages of de-
formation. This confirms that a deformable transform
is the preferable registration strategy.
As an additional analysis, we performed a paired-
sample t-test on the obtained results, which revealed
a significant difference between landmarks distance
before and after the registration process (p = 7.091e-
14). This confirms the qualitative observations drawn
on the basis of Figure 9.
6 CONCLUSIONS
In this paper, we presented a software framework
based on ITK and VTK that allows the automated reg-
istration and overlay of US and MR brain images, in
the context of US-based neuronavigation.
The framework is a modular system that takes
as input the pre-operative MR image and the intra-
operative US image as well as the positioning in-
formation provided by an external probe tracking
system, and provides as output the two volumes
remapped in a common reference system and regis-
tered. Experimental results on a publicly available
dataset of US and MR images of an anatomically re-
alistic brain phantom demonstrate that the software
is able to compensate non-linear deformations of the
Figure 9: Distances of homologous landmarks in US and
MR scans, before and after registration, for three different
deformation stages (respectively, with the two catheters in-
flated by 0 ml - 0 ml, 0 ml - 5 ml, 5 ml - 10 ml).
same order of surgical brain shifts with a good level
of accuracy.
As future work, we plan to integrate our software
BIOIMAGING 2017 - 4th International Conference on Bioimaging
120
with a custom US probe tracking system, in order to
provide a complete US-based neuronavigation frame-
work.
ACKNOWLEDGEMENTS
This research was pursued within the project
DeNeCor (Devices for NeuroControl and NeuroRe-
habilitation), co-funded by the ENIAC Joint Under-
taking under the sub-programme Health and the Ag-
ing Society: Home and Hospital Healthcare, ENIAC
JU Grant Agreement number 324257.
REFERENCES
C. Nikas, D., Hartov, A., Lunn, K., Rick, K., Paulsen, K.,
and W. Roberts, D. (2003). Coregistered intraopera-
tive ultrasonography in resection of malignant glioma.
Neurosurgical Focus, 14(2):1–5.
Chen, S. J.-S., Hellier, P., Gauvrit, J.-Y., Marchal, M.,
Morandi, X., and Collins, D. L. (2010). An Anthro-
pomorphic Polyvinyl Alcohol Triple-Modality Brain
Phantom Based on Colin27, pages 92–100. Springer
Berlin Heidelberg, Berlin, Heidelberg.
Coup´e, P., Hellier, P., Morandi, X., and Barillot, C.
(2012). 3d rigid registration of intraoperative ultra-
sound and preoperative mr brain images based on hy-
perechogenic structures. Journal of Biomedical Imag-
ing, 2012:1:1–1:1.
Liu, Y., Kot, A., Drakopoulos, F., Yao, C., Fedorov, A.,
Enquobahrie, A., Clatz, O., and Chrisochoides, N.
(2014). An itk implementation of a physics-based
non-rigid registration method for brain deformation in
image-guided neurosurgery. Frontiers in Neuroinfor-
matics, 8:33.
Lunn, K. E., Hartov, A., Hansen, E. W., Sun, H., Roberts,
D. W., and Paulsen, K. D. (2001). A quantitative com-
parison of edges in 3d intraoperative ultrasound and
preoperative mr images of the brain. In Proceedings
of the 4th International Conference on Medical Im-
age Computing and Computer-Assisted Intervention,
MICCAI ’01, pages 1081–1090, London, UK, UK.
Springer-Verlag.
Mattes, D., Haynor, D. R., Vesselle, H., Lewellyn, T. K.,
and Eubank, W. (2001). Nonrigid multimodality im-
age registration.
Sarkar, A., Santiago, R. J., Smith, R., and Kassaee, A.
(2005). Comparison of manual vs. automated multi-
modality (ct-mri) image registration for brain tumors.
Medical Dosimetry, 30(1):20 – 24.
Schroeder, W. J., Martin, K., and Lorensen, W. (2003). The
visualization toolkit: An object-oriented approach to
3d graphics, third edition.
Xu, R., Chen, Y.-W., Tang, S.-Y., Morikawa, S., and Ku-
rumi, Y. (2008). Parzen-window based normalized
mutual information for medical image registration.
IEICE - Trans. Inf. Syst., E91-D(1):132–144.
Yoo, T. S., Ackerman, M. J., and Lorensen, W. E. (2002).
Engineering and algorithm design for an image pro-
cessing API: A technical report on itk-the insight
toolkit. Proc. of Medicine Meets Virtual Reality, pages
586–592.
A Multi-modal Brain Image Registration Framework for US-guided Neuronavigation Systems - Integrating MR and US for Minimally
Invasive Neuroimaging
121