A Multi-modal Brain Image Registration Framework for US-guided

Neuronavigation Systems

Integrating MR and US for Minimally Invasive Neuroimaging

Francesco Ponzio, Enrico Macii, Elisa Ficarra and Santa Di Cataldo

Department of Control and Computer Engineering, Politecnico di Torino,

Corso Duca degli Abruzzi 24, 10129 Torino, Italy

Keywords:

Neuroimaging, US-based Neuronavigation, Multi-modal Image Registration, Image Processing, MR-US

Image Integration.

Abstract:

US-guided neuronavigation exploits the simplicity of use and minimal invasiveness of Ultrasound (US) imag-

ing and the high tissue resolution and signal-to-noise ratio of Magnetic Resonance Imaging (MRI) to guide

brain surgeries. More speciﬁcally, the intra-operative 3D US images are combined with pre-operative MR im-

ages to accurately localise the course of instruments in the operative ﬁeld with minimal invasiveness. Multi-

modal image registration of 3D US and MR images is an essential part of such system. In this paper, we

present a complete software framework that enables the registration US and MR brain scans based on a multi

resolution deformable transform, tackling elastic deformations (i.e. brain shifts) possibly occurring during the

surgical procedure. The framework supports also simpler and faster registration techniques, based on rigid

or afﬁne transforms, and enables the interactive visualisation and rendering of the overlaid US and MRI vol-

umes. The registration was experimentally validated on a public dataset of realistic brain phantom images, at

different levels of artiﬁcially induced deformations.

1 INTRODUCTION

Neuronavigation consists in a comprehensive set

of computer-assisted and neuroimaging technologies

that help the neurosurgeons “navigating” within the

brain to guide complex surgical interventions, such as

brain biopsies or tumor resections.

Among the brain imaging modalities, Magnetic

Resonance Imaging (MRI) provides the best discrim-

ination between soft tissues inside the brain, and it is

able to image tumors’ borders with a good level of de-

tail. Furthermore, differently from Computer Tomog-

raphy (CT), it does not expose the tissues to danger-

ous ionising radiations, hence it is minimally invasive

for the patient. This makes it the preferred imaging

modality for planning brain surgical procedures.

On the other hand, planning a brain surgery based

on pre-operative MR scans, which might have been

acquired few days or even weeks before the surgery,

raises many issues, especially in the practice of neu-

rosurgical oncology. For example, in case of rapidly

evolving neoplasms, the navigation might be ex-

tremely error prone because of the changing geometry

and size of the lesion.

As a solution to this issue, modern neuronaviga-

tion systems integrate the information coming from

pre-operative MRI with intra-operative MRI scans,

possibly acquired several times after critical stages

of the surgical procedure. Besides avoiding macro-

scopic errors due to the evolution of the lesion after

the ﬁrst set of scans, this setting allows to address

brain shifts naturally occurring during the surgery.

Brain shifts might alter the original position of crit-

ical cerebral structures, and include elastic deforma-

tions due to craniotomy, tissue resections, cyst de-

compressions, removal of cerebrospinal ﬂuids or even

haemorrhage or use of diuretics. Leveraging on the

real-time anatomical information provided by intra-

operative MRI, the surgical plan can be altered to ac-

count for such shifts.

In spite of its many advantages, using intra-

operative MRI has also a number of critical logistic

drawbacks. In fact, it requires specialised operating

suites, costly and bulky instrumentations, as well as

longer anaesthesia and operating room time.

To overcome these drawbacks, in the last few

years ultrasound (US)-based techniques are becom-

ing more and more attractive for neuronavigation pur-

114

Ponzio F., Macii E., Ficarra E. and Di Cataldo S.

A Multi-modal Brain Image Registration Framework for US-guided Neuronavigation Systems - Integrating MR and US for Minimally Invasive Neuroimaging.

DOI: 10.5220/0006239201140121

In Proceedings of the 10th International Joint Conference on Biomedical Engineering Systems and Technologies (BIOSTEC 2017), pages 114-121

ISBN: 978-989-758-215-8

Figure 1: Scheme of principle of US-guided neurosurgery. A moving set of US volumes acquired during the surgical procedure

are combined with a ﬁxed set of pre-operative MR scans.

poses (C. Nikas et al., 2003).

Indeed, US-based neuronavigation has major ad-

vantages compared to traditional MRI-based systems.

These advantages include its relatively low cost, sim-

plicity of use and minimal invasiveness, both in terms

of volume and complexity of equipment and of im-

pact on the patient. On the other hand, US images

have well-known limitations, such as low signal-to-

noise ratio and penetration depth. Moreover, they

have much lower soft tissue discrimination capabili-

ties compared to MR technology.

Hence, the latest trend in neuronavigation is the

use of hybrid surgical planning techniques, integrat-

ing pre-operative MRI scans with intra-operative US.

In this case, the procedure is guided by a ﬁxed set

of pre-operative MRI scans and a moving set of US

volumes acquired during the surgery (see the scheme

of Figure 1). The pre-operative MR scans are used

to construct a structural model of the patient’s head,

and provide a detailed anatomical 3D map of the brain

and of the targeted lesions. The position of the US

probe with respect to the patient’s coordinate system

is obtained real-time using a probe tracking system.

Then, the pre-operative MR scans are registered and

overlaid on the US images acquired in the Operating

Room, updating the structural model of the brain’s pa-

tient based on the new anatomical information pro-

vided by US.

In such US-guided hybrid system, the accurate

automated registration of MR and US scans plays a

fundamental role. Nonetheless, while multi-modal

brain image registration has a very consolidated tra-

dition in other imaging technologies such as MR and

CT (Sarkar et al., 2005), the registration of US and

MR images is still a research topic in development,

with a number of challenges that need to be tackled:

(i) low signal-to-noise conditions typical of US

imaging. (ii) absence of highly contrasted anatomical

structures (e.g. bones, high-density tissues) driving

the registration algorithm. (iii) possible presence of

brain shifts induced by the surgical procedure.

Most of the currently available solutions do not

explicitly deal with elastic brain shifts (Coup´e et al.,

2012), or are user-dependent, in that they rely ei-

ther on the interactive delineation of markers or sur-

faces (Lunn et al., 2001; Liu et al., 2014), which is not

feasible in the context of real-time neuronavigation.

In this paper, we present an automated framework

tackling these issues. It takes as input MR and US

scans and probe positioning information, as provided

by a tracking system, and allows a fully-automated

registration and overlay of the two volumes, without

requiring any interaction from the user.

Our methodology is based on a non-rigid regis-

tration algorithm, in order to tackle possible non-

linear deformations, with a self-adjusting parameters

search. Nonetheless, the tool supports also other sim-

pler registration techniques, which can be selected

when to tackle stages of the procedure not implying

elastic warping between US and MR scans.

The registration accuracy is experimentally vali-

dated using a publicly available set of MR and US

scans from an anatomically realistic human brain

phantom, even in presence of extensive elastic defor-

mations.

The rest of the paper is organised as follows. In

Section 2, we describe the main modules of the pro-

posed framework. In Section 3, we discuss the multi-

modal image registration technique. In Section 4, we

provide few details on the algorithm’s set-up. In Sec-

tion 5, we present and discuss the experimental re-

sults. In Section 6 we conclude the paper.

2 SOFTWARE FRAMEWORK

The software was implemented in python and C++,

making use of ITK and VTK libraries (Yoo et al.,

2002; Schroeder et al., 2003). In the following, we

brieﬂy describe the main modules, as shown in Fig-

ure 2.

A Multi-modal Brain Image Registration Framework for US-guided Neuronavigation Systems - Integrating MR and US for Minimally

Invasive Neuroimaging

115

Figure 2: Diagram of software architecture, with main modules.

2.1 Volumes Acquisition

Through this module, the software receives as input:

(i) the pre-operative 3D MRI scans of the patient.

(ii) the intra-operative US scans (reconstructed vol-

umes).

The two volumes are in their own reference space

and Field of View (FOV). Hence, the 2D slices com-

posing the two different input volumes do not match.

The module supports either DICOM, NIfTI or vtk im-

age formats.

2.2 Realignment and Reslicing

Through this module, the coordinate spaces of the

pre-operative MR images and the intra-operative US

volumes remapped to a common patient’s reference

space and FOV and resliced for voxels’ coherency.

This way, the 2D orthogonal slices of the two image

volumes are coherently reconstructed for both image

modalities.

To match the reference systems of US and MRI,

the software communicates with an external optical

motion tracking system, that is able to record the po-

sition and orientation of the US probe during image

acquisition. The tracking system processes this infor-

mation and provides a transformation matrix to our

software, in the form of a 4X4 matrix T:

′

= T · x (1)

which in matricial form is:







′













1,1

1,2

1,3

1,4

2,1

2,2

2,3

2,4

3,1

3,2

3,3

3,4

0 0 0 1



















(2)

2.3 Multi-modal Image Registration

Through this module, the two realigned volumes are

co-registered.

More in detail, the MR volume containing the pre-

operative structural information of the patient’s brain

is deformed in order to match the intra-operative US

scans. Hence, brain shift deformations occurring dur-

ing the surgery can be compensated.

As mentioned in Section 1, this molule supports

three different registration methods, which will be de-

scribed in more details later.

These methods are:

(i) Multi-resolution rigid transform. Simplest

solution consisting in rotation and translation, which

ensures best computational time.

(ii) Multi-resolution afﬁne transform. Con-

sisting in rotation, translation, scaling and shearing,

which was found to be the best option when elastic

deformations are negligible (e.g. at stages not requir-

ing brain resections).

(iii) Multi-resolution non-rigid transform. Con-

sisting in a non-linear deformable transform, where

BIOIMAGING 2017 - 4th International Conference on Bioimaging

116

local deformations at the voxel-level, including lo-

cal warping, are deﬁned by a Free Form Deformation

(FFD) model as a mesh of control points. This ap-

proach was found to be the best for the correction of

extensive elastic brain-shift deformations.

2.4 Rendering/Visualization

Through this module, the realigned and co-registered

volumes can be visualised and overlaid. Two different

options of visualisation are supported:

(i) Volume rendering, with clipping box. When

this option is selected, the overlaid MR and US im-

ages are visualised using 3D volume rendering based

on a ray casting and 3D texture mapping technique.

Two different sets of colours are applied to render

the two imaging modalities. The user can interac-

tively rotate the volumes to obtain a better view of the

anatomical details in each modality. The visualisa-

tion supports cropping, clipping and blending modes.

Thanks to these features, the user can clip away the

volume at any point, allowing a better view of the in-

ternal structures.

(ii) Interactive 3D slider. When this option is

selected, the user can interactively switch from vol-

ume rendering to a 2D visualisation of single slices.

The user can navigate along either coronal, sagittal

or transverse directions using a sliding bar or simply

scrolling the mouse, select a plane of interest on the

rendered volume and then visualise the 2D image (ei-

ther MR, US, or a superposition of the two) obtained

cutting the volume through the selected plane.

3 REGISTRATION TECHNIQUE

The US-MR registration technique is implemented as

an iterative optimisation problem, where the search

for the optimal transform matching US and MR vol-

umes is driven by the maximisation of a ﬁtness value

quantifying the similarity between the two input im-

ages. This can be schematically represented by the

components reported in Figure 3.

3.1 Similarity Metric

Of the large number of image registration similar-

ity measures that have been proposed over the years

(among the most relevant: cross-correlation, sum of

squared intensity differences, ratio image uniformity,

etc.), Mutual Information (MI) is widely acknowl-

edged as the best choice for the registration of multi-

modality images. Indeed, this metric is best suited to

measure the matching degree of images having differ-

ent characteristics in terms of range of image intensi-

ties, because it does not assume a linear relationship

among the two sets of intensity values. On top of that,

MI is an automatic intensity-based metric that does

not require the preventivesegmentation of anatomical

parts, surfaces or landmarks on the images, nor the

deﬁnition of speciﬁc features to drive the registration

process.

MI applies the joint probability distribution of pix-

els from the two images to measure the certainty that

the values of the ﬁrst image pixels map to similar val-

ues in the other image, which is a reliable quantitative

measure of how similar the images are. Higher val-

ues of MI imply a large reduction in the uncertainty

(i.e. entropy) between the two distributions of pixels’

values, which is a clear indication that the two images

are better aligned.

More speciﬁcally, if U and V are the two image

volumes, their Mutual Information is deﬁned as:

MI = H(U) + H(V) − H(U,V), (3)

where H(U) and H(V) are the individual marginal en-

tropies of U and V, respectively, and H(U,V) is the

joint entropy.

On the other hand, the individual and joint en-

tropies are deﬁned as:

H(U) = −

∑

) · logP

), (4)

H(V) = −

∑

) · logP

), (5)

H(U,V) = −

∑

i, j

) · logP

), (6)

where P is the marginal probability density distri-

bution of the intensities in the image, estimated based

on the method by (Mattes et al., 2001) using Parzen

histograms (Xu et al., 2008).

3.2 Optimizer

In order to reduce the computational cost of the al-

gorithm, the optimization problem is implemented as

Figure 3: Diagram of the multi-modal registration module.

A Multi-modal Brain Image Registration Framework for US-guided Neuronavigation Systems - Integrating MR and US for Minimally

Invasive Neuroimaging

117

Figure 4: Scheme of the multi-resolution optimization ap-

proach.

multi-resolution strategy, schematically represented

in Figure 4. The rationale of this approach is the fol-

lowing:

- First, the optimal transform parameters are

searched applying at a rough scale in terms of image

resolution (i.e. the input images are resampled so that

only a fraction of the available voxels are used for the

optimization)

- The transform determined at the starting level is

used to initialize image registration at the next stage,

at a ﬁner scale (i.e. a higher number of voxels is used

for the optimization).

- This process is repeated until the ﬁnest scale pos-

sible (full image resolution) is reached.

3.3 Transform

As brain shifts might easily imply elastic deforma-

tions of the brain, the proposed method is a non-

rigid registration technique, that leverages on a de-

formable transform of points from one N-dimensional

space to another N-dimensional space. This deforma-

tion is deﬁned at the voxel-level in terms of a Free

Form Deformation (FFD) model, where a set of de-

formation vectors~g is applied to a sparse regular grid

of control points, one deformation vector per control

point (see Figure 5).

As mentioned in Section 2, in order to provide the

user with less computationally intensive solutions in

stages of the procedure not involving elastic brain de-

formations, the framework supports also the use of

afﬁne transforms (applying a linear combination of

translation, rotation, scaling and shearing, i.e. non-

uniform scaling in some directions), or of even sim-

pler rigid transforms (applying only translation and

rotation). In this case, the transforms can be expressed

geometrically in terms of transform matrices, as in

Equation 1.

3.4 Interpolator

The interpolator is applied to compute the voxel-wise

intensities of the transformed image at non-grid posi-

tions. The interpolation method affects the smooth-

ness of the optimisation search space as well as the

overall computation time. As a compromise between

accuracy and computational costs, our system imple-

ments B-spline interpolation for non-rigid transform

and linear interpolation for either rigid body or afﬁne

transform models.

4 ALGORITHM SET-UP

Summarising, the registration algorithm works to-

wards the maximisation of MI similarity between US

and MR scans. This metric is embedded into an objec-

tive function and optimized numerically. As reported

in Equation 3, this requires an estimation of the proba-

bility density functions (PDFs) associated with the US

and MR images, as well as their joint intensities. Such

PDFs are not known a priori, hence they have to be

estimated from intensity samples, treated as indepen-

dent and identically distributed random measures. In

our implementation, a set of intensity samples is ran-

domly selected on the input image. Then, the PDF is

estimated at discrete bins uniformly spread within the

dynamic range. Entropy values are then computed by

summing over such bins. Hence, an accurate choice

of the number of bins is crucial for registration accu-

racy.

Besides sample bins, the main parameters that

need a careful set-up are the number of samples used

for the calculation (the higher this number, the higher

the signiﬁcance of the estimation), as well as the size

of the mesh associated with the deformation ﬁeld (the

higher the size, the higher the variety of deformations

that can be modelled by the system).

Figure 5: Non-rigid registration.

BIOIMAGING 2017 - 4th International Conference on Bioimaging

118

Figure 6: PVA brain phantom (http://pvabrain.inria.fr/).

As all these parameters can dramatically impact

on the computational time, the set-up needs to ﬁnd

a reasonable trade-off between number of iterations

of the optimisation problem and registration accuracy.

In our system, this trade-off can be automatically ad-

justed based on a training pair of US and MR scans. A

simple heuristic algorithm is implemented, that starts

with a very coarse grid of parameters. Then, increas-

ingly ﬁner search grids are applied around the optimal

parameters determined at each stage of the heuristics.

5 EXPERIMENTAL RESULTS

To experimentally validate our framework, we used

the PVA Brain 3D image dataset, a publicly avail-

able dataset of multi-modal images acquired from a

human brain phantom (Chen et al., 2010). The phan-

tom, made of polyvinyl alcohol cryogel (PVA-C), was

obtained using a shape mold 3D printed from ex-

vivo human brain MR images, hence it is anatomi-

cally very accurate and realistic (see Figure 6). The

phantom contains two inﬂatable catheters, allowing

to artiﬁcially induce brain shift deformations of a pri-

ori known entity. The dataset contains corresponding

MR and US scans at different stages of deformation.

Per each stage, the dataset contains also the transfor-

mation matrices provided by an external neuronavi-

gation probe tracking system, which can be used to

remap US and MR scans into a common reference

system, obtaining an initial rough alignment. Hence,

the dataset can be used to test image registration and

brain shift compensation techniques.

In order to obtain an objective measure of regis-

tration accuracy, we implemented the following vali-

dation procedure:

(i) With the help of a skilled neurologist, we inter-

actively extracted a set of nine anatomical landmarks

(i.e. points of interest) on the MR volume (see Fig-

ure 7).

(ii) We extracted the corresponding anatomical

landmarks on the US volume.

(iii) We run our registration pipeline on the US and

MR volumes.

We repeated steps (i) and (ii), independently, on

the US and MR volumes obtained after registration.

Then, we computed the Euclidean distance between

homologous MR-US markers before and after regis-

tration. The same validation procedure was run on

corresponding US and MR scans at three different de-

formation conﬁgurations, respectively with the two

catheters inﬂated by 0 ml - 0 ml (i.e. absence of de-

formation), 0 ml - 5 ml and 5 ml - 10 ml.

The ideal target of a US-guided neurosurgical sys-

tem is that MR and US volumes should be perfectly

superimposed (see an example of US-MR overlay be-

fore and after registration in Figure 8), and that the

homologous markers should coincide. Hence, we can

establish a validation procedure based on the follow-

ing:

- the Euclidean distance between homologous MR

and US markers provides a measure of the initial spa-

tial mismatch between the two sets of scans before

registration (due to either imperfect probe tracking or

to non-linear brain shift deformations occurring dur-

ing the surgery).

- The same distance computed after registration

provides a measure of the residual spatial mismatch

(where some small mismatch, reasonably in the order

of a mm, is possibly due to manual selection of the

markers).

- The comparison of distances computed before

and after registration provides a measure of registra-

tion quality, in terms of compensation of spatial mis-

match between US and MR scans.

Figure 7: Anatomical landmarks on the PVA brain phantom

images.

A Multi-modal Brain Image Registration Framework for US-guided Neuronavigation Systems - Integrating MR and US for Minimally

Invasive Neuroimaging

119

Figure 8: Example of registration outcome on PVA brain

images. Overlay of MR and US scans before (left) and after

(right) registration.

The obtained results are reported in the boxplots

of Figure 9, respectively for registration based on

rigid transform, afﬁne transform and non-rigid de-

formable transform.

As it can be easily gathered from the boxplots,

even after the initial realignment based on US probe

tracking the US and MR scans have a consistent spa-

tial mismatch (up-to 8 mm). This mismatch is hardly

corrected by a simple rigid-body registration, because

it is mainly due to elastic deformations induced by the

inﬂatable catheters. Indeed, the most successful re-

sults are obtained by the non-rigid registration option

(see third graph of Figure 9). This technique allows

to obtain the best values in terms of residual distance

between the homologous landmarks, of about 2.5 mm

on average. On top of that, the obtained registration

accuracy is reasonably stable at different stages of de-

formation. This conﬁrms that a deformable transform

is the preferable registration strategy.

As an additional analysis, we performed a paired-

sample t-test on the obtained results, which revealed

a signiﬁcant difference between landmarks distance

before and after the registration process (p = 7.091e-

14). This conﬁrms the qualitative observations drawn

on the basis of Figure 9.

6 CONCLUSIONS

In this paper, we presented a software framework

based on ITK and VTK that allows the automated reg-

istration and overlay of US and MR brain images, in

the context of US-based neuronavigation.

The framework is a modular system that takes

as input the pre-operative MR image and the intra-

operative US image as well as the positioning in-

formation provided by an external probe tracking

system, and provides as output the two volumes

remapped in a common reference system and regis-

tered. Experimental results on a publicly available

dataset of US and MR images of an anatomically re-

alistic brain phantom demonstrate that the software

is able to compensate non-linear deformations of the

Figure 9: Distances of homologous landmarks in US and

MR scans, before and after registration, for three different

deformation stages (respectively, with the two catheters in-

ﬂated by 0 ml - 0 ml, 0 ml - 5 ml, 5 ml - 10 ml).

same order of surgical brain shifts with a good level

of accuracy.

As future work, we plan to integrate our software

BIOIMAGING 2017 - 4th International Conference on Bioimaging

120

with a custom US probe tracking system, in order to

provide a complete US-based neuronavigation frame-

work.

ACKNOWLEDGEMENTS

This research was pursued within the project

DeNeCor (Devices for NeuroControl and NeuroRe-

habilitation), co-funded by the ENIAC Joint Under-

taking under the sub-programme Health and the Ag-

ing Society: Home and Hospital Healthcare, ENIAC

JU Grant Agreement number 324257.

REFERENCES

C. Nikas, D., Hartov, A., Lunn, K., Rick, K., Paulsen, K.,

and W. Roberts, D. (2003). Coregistered intraopera-

tive ultrasonography in resection of malignant glioma.

Neurosurgical Focus, 14(2):1–5.

Chen, S. J.-S., Hellier, P., Gauvrit, J.-Y., Marchal, M.,

Morandi, X., and Collins, D. L. (2010). An Anthro-

pomorphic Polyvinyl Alcohol Triple-Modality Brain

Phantom Based on Colin27, pages 92–100. Springer

Berlin Heidelberg, Berlin, Heidelberg.

Coup´e, P., Hellier, P., Morandi, X., and Barillot, C.

(2012). 3d rigid registration of intraoperative ultra-

sound and preoperative mr brain images based on hy-

perechogenic structures. Journal of Biomedical Imag-

ing, 2012:1:1–1:1.

Liu, Y., Kot, A., Drakopoulos, F., Yao, C., Fedorov, A.,

Enquobahrie, A., Clatz, O., and Chrisochoides, N.

(2014). An itk implementation of a physics-based

non-rigid registration method for brain deformation in

image-guided neurosurgery. Frontiers in Neuroinfor-

matics, 8:33.

Lunn, K. E., Hartov, A., Hansen, E. W., Sun, H., Roberts,

D. W., and Paulsen, K. D. (2001). A quantitative com-

parison of edges in 3d intraoperative ultrasound and

preoperative mr images of the brain. In Proceedings

of the 4th International Conference on Medical Im-

age Computing and Computer-Assisted Intervention,

MICCAI ’01, pages 1081–1090, London, UK, UK.

Springer-Verlag.

Mattes, D., Haynor, D. R., Vesselle, H., Lewellyn, T. K.,

and Eubank, W. (2001). Nonrigid multimodality im-

age registration.

Sarkar, A., Santiago, R. J., Smith, R., and Kassaee, A.

(2005). Comparison of manual vs. automated multi-

modality (ct-mri) image registration for brain tumors.

Medical Dosimetry, 30(1):20 – 24.

Schroeder, W. J., Martin, K., and Lorensen, W. (2003). The

visualization toolkit: An object-oriented approach to

3d graphics, third edition.

Xu, R., Chen, Y.-W., Tang, S.-Y., Morikawa, S., and Ku-

rumi, Y. (2008). Parzen-window based normalized

mutual information for medical image registration.

IEICE - Trans. Inf. Syst., E91-D(1):132–144.

Yoo, T. S., Ackerman, M. J., and Lorensen, W. E. (2002).

Engineering and algorithm design for an image pro-

cessing API: A technical report on itk-the insight

toolkit. Proc. of Medicine Meets Virtual Reality, pages

586–592.

A Multi-modal Brain Image Registration Framework for US-guided Neuronavigation Systems - Integrating MR and US for Minimally

Invasive Neuroimaging

121