DIAGNOSIS

A Global Alignment and Fusion Medical System

E. Faliagka, V. N. Syrimpeis, A.Tsakalidis

Computer Engineering and Informatics Department, University of Patras, Greece

G. K. Matsopoulos

School of Electrical and Computer Engineering, National Technical University of Athens, Greece

J. Tsaknakis, G. Tzimas

Department of Applied Informatics in Administration and Economy, National Technical University of Messolonghi, Greece

Keywords: Medical systems, Image processing, Registration, Fusion.

Abstract: In this paper, a global registration-fusion system of medical data is presented in detail. The system is

comprised by the following basic subsystems: (1) the multimodal medical image archiving and

communication subsystem, (2) the image processing subsystem, and (3) the multimodal registration and

fusion subsystem. The system offers various capabilities such as storage, retrieval, distribution and

presentation of images from different medical modalities in DICOM format, supports multiple examinations

of a patient and uses parallel processing threads to perform the processing of the acquired three-dimensional

(3D) data in almost real time. The paper discusses the basic features of the proposed system, analyzes the

proposed algorithms for image preprocessing, registration and fusion and presents the results of an

experimental study that was carried out for evaluating its performance. The innovation of the proposed work

is multilayered. It provides automatic matching based on both segmented surfaces and on different levels of

gray and it allows comparison of registration accuracy for the different techniques based on specific criteria

to quantify registration. Finally, it improves the registration when there is movement and / or distortion in

the data collection of the patient from different imaging systems.

1 INTRODUCTION

Medical imaging is a vital component of diagnostic

medicine, and it also has a significant role in the

areas of surgical planning and radiotherapy (Maintz,

1998). Often, medical images acquired in the clinical

track are using different imaging technologies.

Integrating these images, which are often

complementary in nature, is a challenging problem.

The first step in the integration process is bringing

the tomographic images into spatial registration, so

that the same anatomical regions coincide, a

procedure referred to as registration (Hajnal, 2001).

After registration, a fusion step is required in order

to combine information from different modalities, or

from the same modalities at different examination

periods (

Hawks, 1992).

A prominent example where the fusion of

registered images maximizes the available

diagnostic information is tumor diagnosis and

radiotherapy treatment. The Magnetic Resonance

(MR) imaging system, the SPECT medical imaging

and the Positron Emission Tomography (PET)

provide functional information even at very early

stages of cancerous tumors, but they do not reliably

depict the anatomical characteristics of the tested

organs. On the other hand, tomographic imaging

techniques such as Computer Tomography (CT) and

magnetic (MR) scanners, the ultrasound and X-rays

provide anatomical information, but usually

determine the existence of a cancer tumor only when

it is in a later stage compared to the functional

techniques. Thus, the combined use of different

modalities that offers complementary clinical

information is much more effective, allowing early

Faliagka E., Syrimpeis V., Tsakalidis A., Matsopoulos G., Tsaknakis J. and Tzimas G. (2010).

DIAGNOSIS - A Global Alignment and Fusion Medical System.

In Proceedings of the Third International Conference on Health Informatics, pages 21-28

DOI: 10.5220/0002695500210028

 SciTePress

diagnosis and accurate identification of a cancer

tumor and hence the effective planning of the

radiotherapy treatment.

It is often necessary to align medical data to

illustrate the changes between the data retrieved at

different times so as to assess the progress of a

disease, or to assess the effectiveness of the

treatment. In this case the fusion of data is

implemented to illustrate the changes, as in the

measurement of bone support for implants using

dental radiographs. Moreover, the data registration

applies to cases where data from anatomical atlases

in conjunction with real clinical data and studies on

patient populations are used.

In this work, a global alignment-fusion system of

medical data was developed, which was named

«dIaGnosis». Comparable software systems for

processing and visualization of medical data are also

implemented by Philips Medical Systems Inc.,

Siemens Medical Systems Inc and others. Medical

data in commercial systems are represented in

DICOM format, which is the prominent medical

data protocol. Most commercial software provide

semi-automatic and automatic registration options,

as well as possibilities for data fusion after

registration alignment, either on sections base (2D

problem) or on surfaces base (3D problem). The

proposed system overbalances the existing

registration techniques. Specifically, it provides

automatic matching based on both segmented

surfaces and on different levels of gray, while

algorithms are applied directly to three-dimensional

(3D) data. In addition, it allows the application of

different geometric transformations, including an

elastic transformation to improve the registration

when there is movement and / or distortion in the

data collection of the patient from different imaging

systems. Finally, it allows comparison of registration

accuracy for the different techniques based on

specific criteria to quantify registration.

This paper is organized as follows. Section 2

outlines the architecture of the system proposed and

presents the algorithms used for image

preprocessing, registration and fusion. Section 3

describes the working environment of the

implemented system. Section 4 presents the baseline

scenario where most of the procedures supported by

the system are shown. The efficiency of registration

techniques was tested during the pilot study on skull

patient data collected from CT and MR scanners.

2 SYSTEM ARCHITECTURE

In Figure 1 the overall system architecture is

depicted. The system consists of five complementary

layers-subsystems, managing the registration and

fusion of the medical data, as well as the interaction

with the final user.

Figure 1: The system architecture.

According to the proposed architecture, the

system consists of the following five subsystems.

2.1 Data Collection and Data

Management Subsystem

The subsystem of data collection and management

allows the storage, retrieval, distribution and

presentation of medical images:

 Using Magneto-optical instrument, and

 Data transfer via network from the diagnostic

consoles of the CT and MR scanners, or a

workstation where digital medical data are

acquired.

The medical data collected are a series of sections

from the same patient from different imaging

modalities (CT and MR scanners) and correspond to

a specific region of the human body. In the pilot

version of the system the data correspond to the

region of the skull as acquired from both CT and

MR scanners.

As mentioned earlier, DICOM format is the

prominent international protocol for medical data.

Thus, the medical data acquired by the scanners are

compatible with this format. The entry, management

and export data are in DICOM format too. (NEMA,

2006)

In this subsystem we implemented a function

that reads the header of the DICOM file (DICOM

header) and includes automatically the following

technical characteristics of the system: the number

of sections, the number of pixels per section, a data

analysis per section (mm/pixel), the interval sections

(mm), the number of bits/pixel and the patient data

(patient code, DATE examination, etc.), if available.

The basic technical capabilities of the subsystem

HEALTHINF 2010 - International Conference on Health Informatics

include:

 Patient (code) correspondence with the initial

data of his/her examination.

 Multiple examinations per patient (through

appropriate code).

 Data display with multiple horizontal sections

in icon size.

 Data storage after their process (in DICOM or

other format) in the hard disk of the computer

system.

 Determination of reference data from the user.

 Ability to support multiple data to align

common reference data.

2.2 Medical Data Preprocessing

Subsystem

The data preprocessing is an optional step. It applies

to data which are characterized by high levels of

noise and the containment is achieved by using the

appropriate filters. So, it is usual that before the

registration a re-sampling of one or both data sets

that have the same discretionary analysis is needed.

Thus in the subsystem an appropriate technique for

re-sampling is incorporated (

Unser, 1993). The data

pre-processing subsystem includes the segmentation

technique as developed. In this case, anatomical

information is extracted from the two data sets (for

example the external surfaces of the skull from both

CT and MR scanners), which is then used to perform

the registration.

2.2.1 Pre-processing Techniques

The acquired 3D data may include noise and/or

characterized by heterogeneous background. This

noise is undesirable and should be removed, without

the loss of significant anatomical information

contained in images. For noise reduction, suitable

filters are implemented to improve the quality of

images, which are applied on section based (two-

dimensional problem - 2D) (Gonzalez, 1993).

Specifically, within the subsystem the following

filters have been implemented to improve image

quality:

 Mean filter: It is a low-pass filter which reduces

high-frequency noise in an image.

 Median filter: it is another filter for noise

containment.

 Gamma correction: The factor γ determines the

function which distributes the values of pixels,

according to the intensity of brightness of the

screen. The factor γ is equal to one when there

is a linear relationship between pixel values

and intensity of brightness. Images that appear

darker usually require the factor γ have values

larger than one, while those which appear

bright usually require the factor γ have values

smaller than one.

 Histogram Equalization filter: it is a commonly

used technique for better visualization of the

diagnostic information of an image. In cases

where the biological tissue of interest shows

rates (different levels of gray), which vary

between certain limits in the digital image, the

visualization of the tissue is significantly

enhanced if the function which corresponds

the values of pixels in the image with

brightness in screen changes.

 Adjust brightness and contrast: It is one of the

most basic functions for image editing. The

implementation of this subsystem provides the

opportunity to change the brightness and

contrast of images by the simple linear

transformation:

(

)

(

)

byxaIyxI +

,,'

(1)

Where I(x,y) is the pixel of the initial image with

coordinates (x,y) and I’(x,y) is the pixel of the

adjusted image.

2.3 Medical Data Alignment Subsystem

In many cases in the current clinical practice it is

desirable to combine information provided by two or

more imaging modalities or to monitor the

development of a treatment based on data collected

at different times by the same modality. In

particular, when monitoring the development of a

treatment, it is very often the imaging anatomical

structures displayed in two sets of data that have

been collected at different times to be characterized

by geometrical movements, revolutions, etc. It is

necessary to find an appropriate geometric

transformation, which achieves the spatial

coincidence of anatomical structures of the two

images. This process of finding the transformation is

called registration.

The medical data alignment subsystem consists

of a set of techniques for 3D registration of brain

data on surface based or using the levels of gray

(gray-based). Particular attention has been given to

the design of the automatic registration techniques.

Alternatively, there is the option of manual

registration using appropriate surface driving points

as selected by the expert.

Within the design of this subsystem three

registration techniques were implemented:

 Automatic registration based on surfaces,

 Automatic registration based on gray levels and

DIAGNOSIS - A Global Alignment and Fusion Medical System

 Manual registration.

2.3.1 Surface-based Registration Technique

This technique is automatic and based on the spatial

matching of segmented anatomical structures of data

from different imaging modalities (Matsopoulos,

2003).

The basic stages of the automatic method for

surfaces registration include:

 Surface Pre-alignment. The stage of pre-

alignment includes the spatial displacement of

two triangulated surfaces, so that the centres

of mass coincide. Also, a transformation of

scale in each axis is done separately, based on

the voxel sizes of the two images

(Matsopoulos, 2000).

 Geometric transformation application. The

second phase implements an overall geometric

transformation. Its parameters are calculated

by optimizing a function that quantifies the

spatial matching of a triangulated surface of

the reference image (computer tomography -

CT) and the modified image (MRI - MRI).

Four models of geometric transformation in

three dimensions, are explored and evaluated

based on the final results of the registration

(Van den Elsen, 1993).

 Matching function definition. The registration

can be seen as the optimization of a Measure

of Match - MOM according to the variables of

the selected transformation. At the case of

surfaces matching an appropriate matching

function is the average of the geometric

distance between the transformed points of the

magnetic scanner data and the corresponding

closest points of computer scanner data.

2.3.2 Registration Based on Gray Levels

This data registration technique is based on the

automatic spatial identification of data from

different imaging systems and is applied on image

values directly, without the prior requirement for

segmenting common anatomical structures (Kagadis,

2002).

2.3.3 Manual Registration

In the case of the manual registration method, the

expert selects points in the respective sections of the

two imaging modalities and the registration of the

data is based on the selection of a particular

geometric transformation (Maurer, 1997). This

method has been developed so that its performance

can be compared to the performance of the proposed

automatic registration methods.

2.4 Medical Data Fusion Subsystem

Medical data fusions scope is to combine

information from different modalities, after the

application of the medical image registration

process. The fusion subsystem is designed

appropriately to allow the composition of anatomical

information from the aligned medical data using

techniques such as the pseudo-colour scale, logic

functions for the diverse overlay of image parts on

another image and change the degree of

transparency in the overlay of anatomical structures

(Matsopoulos, 2008).

2.4.1 Fusion Techniques

Within the proposed system, the following

techniques for medical data fusion, were developed

and applied after registration:

 Implementation of logical functions for the

diverse parts overlay of one image on the

other. Specifically, after the data registration,

the anatomical information derived from data

of the CT Scanner overlays on the respective

aligned sections of the MR scanner in order to

fuse the information from the two imaging

systems. This process is mathematically

standardized with the logical operator

Exclusive Or (XOR), which is implemented as

follows:

(

)

(

)

(

)

(

)()()

yxMyxIyxMyxIyxI

,,,1,, +−=

(2)

where Ι

and Ι

the reference image and the

image to align respectively and Μ is the mask

that has value 1 at the pixels that overlay from

both the Ι

to the Ι

. The mask Μ may be the

segmented structure of interest of the Ι

image

that has to be visualized from the reference

system of Ι

, or repeated normalized

geometric shapes, where the aim is to visually

confirm the accuracy of the registration of Ι

relatively to Ι

In the pilot study, information from the CT

scanner was isolated and was inserted in the

aligned data of the magnetic scanner using

logic functions (XOR). This fusion method

allows the expert – a doctor to assess the

accuracy of the registration method, while it

gives information on the position of the bones

from the computer tomography in comparison

with other soft tissues or tumors, as shown in

MR.

HEALTHINF 2010 - International Conference on Health Informatics

 Ability to change the degree of transparency ‘a’

during the overlay of anatomical structures in

order to achieve a combination of information

and assess the quality of the registration result.

This technique was implemented on the basis

of the relationship:

() ()( ) ()

yxaIayxIyxI

,1,, +

−

(3)

 Fusion of data in 3D is a particularly difficult

problem because the extra dimension makes

the data display difficult even without the

extra complexity of the data fusion. In this

work, a simultaneous demonstration of

common anatomical structures - surfaces

before and after the registration is achieved

using the proposed representation techniques

in the form of VRML.

 The results are visualized using pseudo-

coloring according to the medical system used

(e.g. red for the visualization of the anatomical

structure of the axial scanner and blue for the

magnet scanner), to make clear to the expert

the relative position of the two surfaces and

the change before and after the registration

(

Gomes, 1998).

2.5 Graphical Interface Subsystem

The graphical interface subsystem is an important

part of the developed system, as it allows the final

user use the necessary functions of the registration

software. The subsystem was developed having in

mind the following requirements:

 Ease of use and user friendliness,

 Speed enforcement functions and

 Reliable performance of the software’s

individual applications

(a)

(b)

Figure 2: (a) Selection of registration parameters (b)

Selection of the fusion parameters and implementation of

the fusion settings.

The key features of the graphical interface

subsystem include:

 Creation of an appropriate graphical

environment: This feature concerns the design

and development of an appropriate interface

that offers: a) easy navigation to the

software’s menus, b) easy access to medical

data and c) a clear definition of the integrated

techniques – algorithms.

 Visualization of the medical information: An

important feature of the software is the ability

to visualize data and present the results of the

applications and techniques applied in an

comprehensible manner. It provides: a)

visualization of the original medical data, b)

visualization of the data processing results,

particularly the results of automatic

registration methods and c) visualization of

the fusion of information from the registration.

In particular it allows simultaneous display of

relevant medical data (e.g. CT and MR

sections) before and after the registration and

presentation of the fusion results.

Quantification of the registration results:

Beyond the visualization of the registration

and fusion results another important feature is

the presentation of quantitative data. The data

are related to registration results based on

a)specific success criteria and b) on

geometrical differences (displacements and

rotations) of the data to align from the

reference data.

3 WORKING ENVIRONMENT

The user can use the basic components and navigate

to the input and output data using a tree structure

(Figure 3). The tree structure starts from the node of

the project. The project is the main component of the

system. A project consists of source images,

processing settings and output images and can be

saved and retrieved at will without losing the

settings of the user. It is the root of the tree that

represents, while the intermediate nodes and leaves

of the tree represent individual project data or

processing information.



Figure 3: System information data.

DIAGNOSIS - A Global Alignment and Fusion Medical System

Figure 4: Starting the application.

There are also collection nodes, which group

different snapshots of similar information existing in

each project. Each node, depending on the

information that represents may correspond to a

dynamic description, have associated notes, contain

interfaces presenting information and have

properties that are processed by the user and others.

New nodes can be created by the user and added to

the project while some existing may be removed.

In Figure 4 the general working environment of

the application is shown. The working environment

is dynamic. The user interfaces can be aggregated

into tabs, to activate the automatic concealment

within the window, to match them all together and

more. The user options are saved by closing the

application, and retrieved the next time booted. Also

some templates of the user interface are created and

the user can easily select the one he likes.

The images can be loaded either from an existing

list or with the process of surveying examination

(Figure 5).

As the recovery process of the examining image

from DICOM files may be slow, the system makes

the process to use parallel processing threads.

During the information retrieval the system notifies

the user about the status of recovery and does not

allow access to the node’s data.

Figure 5: Image loading with the process of surveying

examination.

4 PILOT SCENARIO

In order to have an exhaustive testing of the system

a testing scenario was defined. This scenario uses all

the processes supported.

Specifically, the system was installed and

evaluated by an expert-radiologist on the credibility

of the operation and performance of the registration

techniques.

Data sets from axial (CT) and magnetic

tomography (MRI) from 5 patients from

Strahlenklinik of the Stadtische Kliniken Offenbach

of Germany were used. The axial tomography data

were the «Reference data», while the magnetic

tomography data were the «Data to align».

After any registration technique an overlay -

fusion of the CT data on the corresponding sections

of the MRI data took place. In this way, the expert

assessed optically the performance of the

registration techniques.

Figure 6 shows characteristic results of the

registration-fusion techniques using real medical

data. Based on an analysis of these results we came

to the following conclusions:

 The performance of the automatic registration

techniques is much better compared to the

semi-automatic alignment technique.

 Among the automatic registration techniques

based on gray levels, the technique of mutual

information has better performance compared

to the technique using the correlation

coefficient.

 The technique of surface registration is worse

compared to the technique using mutual

information and is almost equivalent to the

technique using the correlation coefficient.

(a)

(b)

(c)

(d)

Figure 6: Registration results using (a) Automatic

registration using correlation coefficient. (b) Automatic

surfaces registration (c) Automatic registration using

mutual data coefficient. (d) Semi-automatic registration

technique.

HEALTHINF 2010 - International Conference on Health Informatics

4.1 Quantitative Analysis of

Registration Results

The four registration techniques implemented within

the proposed system were further quantitatively

evaluated in terms of accuracy. Towards this

direction, five patients were used forming five pairs

of sets, each set consisting of CT and MRI head data

of the same patient. The accuracy of each

registration technique was measured as the mean

distance of the centers of all the external markers for

each data set before and after registration (in pixels).

The centers of the external markers were obtained

manually by an experienced radiotherapist.

Comparisons on the performance of these

registration techniques based on this criterion are

shown in Table 1.

From the quantitative result in Table 1 it is shown

that all automatic techniques were performed better

than the semi-automatic technique. Furthermore, the

mutual information registration technique was

outperformed from the other two automatic

registration techniques. Finally, the surface and the

correlation coefficient registration techniques were

performed equivalently.

4.2 Three-dimensional Display of

Anatomic Structures

An important factor in the process of medical data

fusion is the ability of the system to visualize the

results of the registration. Specifically, the system

supports the display of the exported external.

Table 1: Registration Techniques comparison.

Data Sets

Registration Techniques

Automatic

Mutual

Information

Automatic

Correlation

Coefficient

Set 1 0.27 ± 0.01 0.59 ± 0.02

Set 2 0.28 ± 0.02 0.67 ± 0.03

Set 3

0.32 ± 0.01

0.80 ± 0.02

Set 4

0.31 ± 0.01

0.61 ± 0.05

Set 5

0.29 ± 0.01 0.46 ± 0.02

Data Sets

Registration Techniques

Automatic –

Surface

Registration

Semi-

automatic

Registration

Set 1 0.60 ± 0.03 1.89 ± 0.37

Set 2

0.63 ± 0.01 1.87 ± 1.21

Set 3

0.49 ± 0.07 1.33 ± 0.07

Set 4

0.50 ± 0.01 0.47 ± 0.00

Set 5 0.43 ± 0.00

0.53 ± 0.04

Figure 7: (a) Before registration (b) Registration using the

surface registration method.

anatomical structures - three-dimensional surfaces -

before and after the registration

Furthermore, an indicative visual result of the

overlay of the axial and magnetic scanner surfaces is

presented using VRML and a surface representation

algorithm.

It may be noted that the pre-aligned skin surfaces

are different and the area of the axial tomography is

external and above the area of the MRI. With the

method of surfaces registration the registration

between two surfaces is enhanced, as shown by the

alternation of the two colours of the surfaces.

In Figure 7 we can see the overlay of the skin

surface of the axial tomography (red colour) on the

corresponding surface of the magnetic tomography

(blue colour) for a specific couple using the

algorithm for surface representation in VRML

format.

5 CONCLUSIONS

In this paper we have illustrated our registration-

fusion system in detail, described the algorithms

used and shown the basic scenario of the

application’s usage. Diagnosis is an integrated

environment that facilitates the automatic matching

based on both segmented surfaces and on different

levels of gray and it allows comparison of

registration accuracy for the different techniques

based on specific criteria to quantify registration. It

also improves the registration in case of movement

and / or distortion in the data collection of the

patient from different imaging systems.

After the implementation of the system, a

number of tests were performed for evaluating the

developed registration techniques both qualitatively

and quantitatively in order to test the stability and

accuracy of the techniques. As for future work, we

plan to extend our system by developing further

fusion and registration techniques. Additionally,

more tests will be conducted to support the

efficiency of the implemented system.

DIAGNOSIS - A Global Alignment and Fusion Medical System

REFERENCES

J. B. A. Maintz and M. A. Viergever, 1998. A survey of

medical image registration, Medical Image Analysis,

vol. 2, no. 1, pp.1–36,

J.V. Hajnal, D.L.G. Hill, and D.J. Hawkes, 2001. Medical

image registration, CRC Press.

D.J. Hawks, D.L.G. Hill, and E.C.M.L. Bracey, 1992.

Multi-modal data fusion to combine anatomical and

physiological information in the head and the heart, in

J.H.C. Reiber and E.E. Van der Wall (Eds.):

Cardiovascular Nuclear Medicine and MRI, Kluwer

Academic Publishers, Dordrecht, The Netherlands, pp.

113-130.

2006, Digital Imaging and Communications in Medicine

(DICOM) Part 1: Introduction and Overview. National

Electrical Manufacturers Association.

Unser M, Aldroubi, Eden M. B-Spline, 1993. Signal

processing: Part II – Efficient design and applications.

IEEE Trans. Signal Proc.; 2: 834-83-1000.

Gonzalez RC, and Woods RE, 1993. Digital image

processing, Addison Wesley, Second Edition,

G.K. Matsopoulos, K.K. Delibasis, N.A. Mouravliansky,

P.A. Asvestas, K.S. Nikita, V.E. Kouloulias, N.K.

Uzunoglu, 2003. CT-MRI Automatic Surface-based

Registration Schemes Combining Global and Local

Optimization Techniques, Technology and Health

Care: Official Journal of the European Society for

Engineering and Medicine, vol. 11, no. 4, pp. 219-232.

Matsopoulos G.K., Delibasis K.K., Mouravliansky N.,

Nikita K.S, 2000. A Combination of Global and

Elastic Transformations for the Automatic Medical

Surface Registration. Scattering Theory and

Biomedical Technology: Modelling and Applications.

C.V. Massalas, G. Dassios, K. Kiriaki, D. Fotiades, A.

Payatakes (Eds.), pp. 250-264, World Scientific

Publishing Co., Inc., River Edge, NJ, USA (ISBN 981-

02-4391-X).

Van den Elsen PA, Pol EJD, Viergever MA. 1993.

Medical image matching—a review with

classification. IEEE Engng Med Biol; 12: 26-39.

G.C. Kagadis, K.K. Delibasis, G.K. Matsopoulos, N.A.

Mouravliansky, P.A. Asvestas, G.C. Nikiforidis, 2002.

A Comparative Study of Surface- and Volume-based

Techniques for the Automatic Registration between

CT and SPECT Brain Images, Medical Physics, vol.

29, no. 2, pp.201-213

Maurer C. R., Fitzpatrick J. M., Wang M. Y., Galloway R.

L., Maciunas R. J. and Allen G. G., 1997. Registration

of head volume images using implentable fiducial

markers, IEEE Trans. Med. Imag.; 16: 447-462.

G.K. Matsopoulos, 2008. Automatic Correspondence

Methods Towards Point-based Medical Image

Registration: An Evaluation Study, Chapter 26, in

“Handbook of Research on Advanced Techniques in

Diagnostic Imaging and Biomedical Applications”,

edited by T. Exarchos, IGI Global publication, pp.

407-425, (ISBN (ISBN 978-3-540-78870-6).

J. Gomes, L. Darsa, B. Costa, and L. Velho, 1998.

Warping and Morphing of Graphical Objects, in Computer

Graphics, The Morgan Kaufman Series.

HEALTHINF 2010 - International Conference on Health Informatics