A Multi-Modality Approach to Medical Case Retrieval for

Alzheimer's Disease

First Katarina Trojachanec Dineva

, Ivan Kitanovski, Ivica Dimitrovski, Suzana Loshkovska

and Alzheimer’s Disease Neuroimaging Initiative

Faculty of Computer Science and Engineering, Ss. Cyril and Methodius University,

Rugjer Boshkovik 16, PO Box 393, Skopje, N. Macedonia

Keywords: Alzheimer's Disease, Medical Case Retrieval, Multi-Modal, ADNI, TADPOLE.

Abstract: In this research, we evaluate medical case retrieval for AD on the bases of descriptors generated by combining

different modalities (Magnetic Resonance Imaging (MRI) markers, Fluorodeoxy-glucose Positron Emission

Tomography (FDG-PET) based measures, Cerebrospinal Fluid (CSF) protein levels, and Apolipoprotein-E

(APOE) genotype and age as risk factors). We investigated whether they would provide complementary

information aiming to improve medical case retrieval for AD. According to the obtained results, we concluded

that this approach outperformed the retrieval results in the current reported research by gaining MAP value

of 0.98 yet providing an efficient medical case retrieval for AD and keeping low dimensional feature vector.

1 INTRODUCTION

Alzheimer’s Disease (AD), as an irreversible,

progressive, neurodegenerative disorder, is one of the

most common forms of dementia. It causes neuronal

loss, spreading through different parts of the brain

(Alzheimer's Association, 2022; Porsteinsson, 2021).

A vast amount of data is continuously generated

as part of the medical cases related to AD in the

clinical and research centers, containing variety of

data types. Those include medical imaging markers

such as Magnetic Resonance Imaging (MRI),

Positron Emission Tomography (PET), and Diffusion

Tensor Imaging (DTI), biological markers, namely

Apolipoprotein E (APOE) status, cerebrospinal fluid

(CSF) measures, cognitive tests results etc., carrying

powerful information. The necessity for their efficient

organization, storage, and representation so to be able

to provide appropriate and easier access to the

medical cases, as well as, precise, efficient, and

clinically meaningful retrieval, analysis, knowledge

discovery, prediction and prognosis, is evident and

still, a challenge and an open question (Meyer, 2019).

https://orcid.org/0000-0002-7050-4947

* Data used in preparation of this article were obtained

from the Alzheimer’s Disease Neuroimaging Initiative

(ADNI) database (adni.loni.usc.edu). As such, the

investigators within the ADNI contributed to the design

The rapid development in machine learning (ML) is

having a profound impact in biomedical domain,

opening space for applying ML algorithms to provide

a solution to this challenge. This could support and

improve the diagnostic and therapeutic processes.

The aim of this research is to provide more

efficient and more comprehensive medical case

representation that will enable more precise and

clinically relevant medical case retrieval for AD.

Considering that multi-modality markers may capture

the variety of crucial aspects of the disease and its

progression (Marinescu, 2020; Moguilner, 2022), we

base the medical case representation on multiple

modalities, not the single one, to investigate whether

they can provide a complementary information

contribute to better retrieval results. Hence, we

combined information from MRI imaging markers,

Fluorodeoxyglucose (FDG) - PET based markers,

cognitive tests scores, CSF derived protein levels, as

well as risk factors provided by the Alzheimer's

Disease Neuroimaging Initiative (ADNI, 2022), to

represent each medical case.

and implementation of ADNI and/or provided data but did

not participate in analysis or writing of this report

(http://adni.loni.usc.edu/wp-content/uploads/how_to_apply/

ADNI_Acknowledgement_List.pdf)

554

Dineva, K., Kitanovski, I., Dimitrovski, I., Loshkovska, S. and Alzheimer’s Disease Neuroimaging Initiative, .

A Multi-Modality Approach to Medical Case Retrieval for Alzheimer’s Disease.

DOI: 10.5220/0011939800003414

In Proceedings of the 16th International Joint Conference on Biomedical Engineering Systems and Technologies (BIOSTEC 2023) - Volume 5: HEALTHINF, pages 554-561

ISBN: 978-989-758-631-6; ISSN: 2184-4305

 2023 by SCITEPRESS – Science and Technology Publications, Lda. Under CC license (CC BY-NC-ND 4.0)

The paper is organized as follows. Section 2

provides the related work. The materials and methods

are covered in Section 3. The experimental results and

discussion are presented in Section 4. Section 5

contains the concluding remarks.

2 RELATED WORK

During the past years, AD has been intensively

researched, in an attempt to ensure an early diagnosis,

prognosis, and ultimately finding an appropriate

therapy, including a cure for the disease (Marinescu,

2020b; Weber, 2021; Alzheimer's Association,

2022;). In some research, the traditional approach for

feature extraction from the MRI is used, meaning that

the descriptors are based directly on the visual image

content (Vinutha, 2019; Chethan, 2020; Sagayam,

2021). Other research is using the domain knowledge

to overcome the limitations of the traditional

approach. In this regard, baseline images were used

to estimate of the volumes of the brain structures, and

cortical thickness of the cerebral cortex regions are

calculated in (Trojacanec, 2015; Kruthika, 2019) or to

calculate Spatial Pattern of Abnormality for Early

detection of Alzheimer's Disease (SPARE-AD score)

in (Trojacanec, 2017). On the other hand, longitudinal

approach to estimate static features at multiple time

points is used in (Trojacanec, 2015), and to estimate

dynamic features for patients' descriptors was

proposed in (Trojachanec, 2017). Combination of

static, dynamic features, as well as the SPARE-AD

index (Dineva, 2022) was evaluated for image

retrieval for AD. With the aim to increase the early

detection performance for AD, authors in (Kruthika,

2019) used pre-trained 3D-autoencoder, 3D Capsule

Network, and 3D-Convolutional Neural Network.

Most of the current research is based on a single

modality, mainly MRI features. To induce

improvements in the current research, aiming to

provide more precise and relevant medical case

retrieval for AD, our research is focused on multi-

modal medical case retrieval. For that purpose, we

use multiple biomarkers, usually available as part of

the medical cases for AD acquired during the

examination period in the clinical centres. In fact, this

research aims to evaluate a representation based on a

combination of a variety of biomarkers, extracted

from MRI, FDG-PET, CSF protein levels, and risk

factors, such as APOE genotype and age.

Providing efficient and precise medical case

retrieval for AD might be very beneficial from

different aspects: (1) decision support by providing

clinicians with powerful and relevant information at

the right moment at the right place on the bases of the

existing medical cases of other examined patients; (2)

knowledge discovery from the large, continuously

growing, medical databases by retrieving the most

similar cases; (3) pattern discovery and understanding

of the disease, providing new insights, biomarkers

identification, and analysis of the disease progression;

(4) assessment and analysis of the response to a

possible therapy; (5) providing valuable knowledge

for students, clinicians and scientists.

3 MATERIALS AND METHODS

3.1 Medical Case Retrieval for

Alzheimer's Disease

The medical case retrieval process consists of

generating a representation of the query medical case

and all the medical cases previously stored in the

database using the same representation technique.

After that, the feature vector (descriptor) of the query

medical case is compared to the descriptors of all the

other medical cases. All the medical cases in the

database are then sorted by similarity to the query, so

that the most similar one is at the top. This sorted list

of the database medical cases is the result of the

retrieval. We used leave-one-out strategy because of

the small number of patients used in the evaluation.

To calculate the distance between the medical

case descriptors, we used Manhattan and Canberra

distance (Cha, 2007), chosen on an experimental

basis because they led to the best retrieval results

compared to several other distances such as

Euclidean, Chebyshev, Square Euclidean distance,

Bray-Curtis dissimilarity and Cosine similarity.

To evaluate the proposed strategy in this research,

we used the standard evaluation metric MAP (Mean

Average Precision) for quantitative measurement of

the retrieval performance.

3.2 Medical Case Representation

According to the recent research regarding AD, a

combination of different kinds of biomarkers may

provide complementary and powerful information

(Gupta, 2019; Marinescu, 2020b), possibly enriching

medical case representation. Thus, we combine

different biomarkers suggested by the TADPOLE

challenge (TADPOLE, 2022; Marinescu, 2018)

including MRI ROI derived measures, FDG-PET

based measures, CSF protein levels, APOE status,

and age, to represent the medical cases and we base

the medical case retrieval on multimodal descriptors.

A Multi-Modality Approach to Medical Case Retrieval for Alzheimer’s Disease

555

3.2.1 Magnetic Resonance Imaging

Regarding MRI measurements, we used dynamic and

static measures of the brain structures, as well as

SPARE-AD score, following the strategy used in

(Dineva, 2022) and the fully automated longitudinal

pipeline from the FreeSurfer version 5.3 (FreeSurfer,

2022; Reuter, 2012). According to the previous

research (Trojachanec, 2017), percent change with

respect to the value obtained from the linear fit at

baseline (PCfit) and symmetrized percent change

(SPC) of the volumetric measures of the cortical and

subcortical regions have proven to be most powerful

in longitudinal MRI retrieval for AD. Volumes and

cortical thickness (static) measurements form the

third and fourth time point (12- and 24- month follow-

up), led to the best retrieval precision (Trojacanec,

2015). We also used SPARE-AD score (Davatzikos,

2009; Trojacanec, 2017), available in ADNI database.

In this research, we used a combination of PCfit

or SPC of the volumetric measures (dynamic

features), static volumetric and cortical thickness

measurements at 12- and 24- month follow-ups, and

SPARE-A, because, they provided superior results in

longitudinal retrieval for AD (Dineva, 2022).

3.2.2 Fluoro-Deoxyglucose (FDG) PET

Molecular processes which are thought to be some of

the earliest to become abnormal due to AD

(Marinescu, 2020b), can be represented by PET.

(Jack Jr, 2018). FDG-PET measures of angular,

temporal, and posterior cingulate, are available on

ADNI website and used in this research.

We also use values for two global indices that

were recently added to ADNI: (1) hypometabolic

convergence index (HCI) (Chen, 2011), and (2)

statistical region of interest (sROI) (Chen, 2010; Van

Dyck, 2019). HCI is a cross-sectional measure,

indicating the extent to which the pattern and

magnitude of hypometabolism for a given patient

matches that of patients with AD, while, sROI

was

established longitudinally. representing the standard

uptake value ratio between those regions affected by

the disease and the regions spared by AD.

In contrast, measures of the amyloid-beta load in

the brain (AV45 PET) and tau load in the brain

(AV1451 PET), are not available for most of the

subjects, hence, not used in this research.

3.2.3 Main Cognitive Tests

To be able to measure cognitive decline in patients,

cognitive test is administered by medical expert as

one of the latest indicators of AD, (Jack, 2013).

Regarding the cognitive tests results, as

recommended by the TADPOLE challenge, we used

the following: Clinical Dementia Rating Sum of

Boxes (CDRsb), Alzheimer’s Disease Assessment

Scale (we used both ADAS11, and ADAS13 results),

Mini-Mental State Examination (MMSE), and Rey

Auditory Verbal Learning Test (RAVLT)

(Marinescu, 2020b). Regarding RAVLT test results,

we used the following features: RAVLT immediate,

RAVLT learning, RAVLT forgetting, and RAVLT

forgetting percent. Тhe results of all cognitive tests

are available through the ADNI database.

3.2.4 CSF Biomarkers

Representing the concentration of amyloid-beta and

tau abnormal proteins, not related to any particular

part of the brain, CSF based measurements are

powerful AD indicator, reflecting abnormalities

many years before symptom onset (Marinescu,

2020a; Marinescu, 2020b). CSF biomarkers are

recommended by the TADPOLE challenge to be used

as early AD indicators (Marinescu, 2018), and

therefore we also use them in our research as

amyloid-beta level in CSF (ABETA), tau level

(TAU), and phosphorylated tau level (PTAU).

3.2.5 Risk Factors

APOE ε4 is found to be very powerful genetic risk

factor lowering the age of onset AD (Marinescu,

2018; Gupta, 2019). Considering that the older

subjects are more likely to develop AD, another

important risk factor is the age (Marinescu, 2020b),

and following the recommendations from the

TADPOLE challenge, we used exactly these two.

3.3 Data

In this research, we used the data provided by ADNI,

acquired for the participants from the standardized list

from ADNI-1. The aim of ADNI is to enable research

that will provide an answer to the question whether

combining imaging markers and biological markers,

along with neuro-psychological and clinical

assessments, may indicate the presence and allow

assessment of the progression of MCI and AD.

Following the main goal of ADNI, we are

investigating whether multi-modal markers may lead

to a more precise and efficient medical case retrieval.

We also followed recommendations summarized

for the TADPOLE challenge, in terms of different

data modalities such as: (1) MRI, (2) main cognitive

tests, (3) PET, (4) DTI, (5) CSF biomarkers, and (6)

risk factors and demographic information.

HEALTHINF 2023 - 16th International Conference on Health Informatics

556

We applied several levels of selection to be able

to keep fully complete dataset, thus reducing the

influence of the missing data or the algorithm used to

cope with the missing data on the retrieval results.

Firstly, to provide the possibility of fair

comparison with other research about medical case

retrieval for AD, we selected the subjects that belong

to AD or normal control (NL) group and have

available MRI scans acquired at four time points, at

baseline, and the 6-, 12-, and 24-month follow-ups.

Thus, we obtained a total of 267 patients from the

standardized list, 168 AD, and 99 NL.

After the MRI processing, we selected only the

cases without global or regional failures in all time

points (153 patients in total, 41 AD and 112 NL), to

ensure a complete automatic processing and exclude

the necessity to involve a medical expert.

The cognitive tests results were not available for

three more patients, ending up with 150 patients in

total, 39 AD and 111 NL (subset 1). Significant

number of missing data were detected regarding FDG

PET measures, resulting in 68 patients, 17 AD, 47 NL

(subset 2). CSF biomarkers were available for 85

patients, 24 in AD group, 61 in NL group (subset 3).

3.4 Experimental Setup

To be able to obtain fully complete dataset for the

evaluation, each of the evaluations was conducted

using one of the subsets listed in subsection 3.3. The

goal is to provide a fair comparison between different

medical case representations, not influenced by the

missing data or the algorithm used to overcome the

problem of missing data. Hence, in this research, we

performed three separate evaluations, described in the

following subsections (3.4.1 – 3.4.3). To reduce the

dimension of the descriptor, select the most relevant

features, we also applied Correlation-based Feature

Selection (CFS) algorithm (Hall, 2003), chosen on an

experimental basis. The optimal feature subset was

determined separately in case of each query.

Because most of the current research addressing

the problem of medical case retrieval for AD is based

on MRI (Trojacanec, 2015; Trojachanec, 2017,

Trojacanec 2017; Kruthika, 2019a; Kruthika, 2019b,

Vinutha, 2019; Chethan, 2020), in each evaluation

scenario we used the case in which the descriptor

contains only MRI features for reference.

3.4.1 MRI Measures, Cognitive Scores, and

Risk Factors

The first evaluation was performed on using the

scenarios in Table 1, based on the following features:

 MRI (combination of dynamic (VolPCfit or

VolSPC), static measures (Vol34+CT34), and

SPARE-AD)

 Cognitive tests (combination of CDRsb,

ADAS11, ADAS13, MMSE, RAVLT

immediate, RAVLT learning, RAVLT

forgetting, and RAVLT forgetting percent)

 Risk factors (APOE ε4 and Age)

Table 1: Scenarios Based on MRI Measures, Cognitive

Tests and Risk Factors.

Scenario Descri

tor

S1.1 MRI_VolPCfit+Vol34+CT34+SPARE-AD

S1.2 MRI_VolSPC+Vol34+CT34+SPARE-AD

S1.3 S1.1+CognitiveT

S1.4 S1.2+Co

nitiveT

S1.5 S1.1+RiskF

S1.6 S1.2+RiskF

S1.7 S1.1+Cognitive + RiskF

S1.8 S1.2+Cognitive + RiskF

3.4.2 MRI, FDG PET Measures, Cognitive

Scores, and Risk Factors

The second evaluation was performed on the subset

2, following the scenarios given in Table 2 on the

bases of the following features:

 MRI (combination of dynamic (VolPCfit or

VolSPC), static measures (Vol34+CT34), and

SPARE-AD),

 Cognitive tests scores (CDRsb, ADAS11,

ADAS13, MMSE, RAVLT immediate,

RAVLT learning, RAVLT forgetting, and

RAVLT forgetting percent), and

 Risk factors (APOE ε4 and Age)

 FDG PET measurements (FDG PET, HCI,

sROI

, for each of the following visits,

baseline, 6

, 12

and 24

month follow-ups)

Table 2: Scenarios Based on MRI Measures, FDG-PET

measures, Cognitive Tests and Risk Factors.

Scenario Descri

tor

S2.1 MRI

VolPCfit+Vol34+CT34+SPARE-AD

S2.2 MRI_VolSPC+Vol34+CT34+SPARE-AD

S2.3 S2.1+FDG_PET+HCI+sROI

S2.4 S2.2+FDG_PET+HCI+sROI

S2.5 S2.1+FDG

PET+HCI+sROI

+RiskF

S2.6 S2.2+FDG

PET+HCI+sROI

+RiskF

S2.7

S2.1+FDG_PET+HCI+sROI

+Co

nitiveT

S2.8

S2.2+FDG_PET+HCI+sROI

+CognitiveT

S2.9 S2.3+Cognitive + RiskF

S2.10 S2.4+Co

nitive + RiskF

A Multi-Modality Approach to Medical Case Retrieval for Alzheimer’s Disease

557

3.4.3 MRI Measures, Cognitive Scores, Risk

Factors, and CSF Biomarkers

The influence of the CSF biomarkers ABETA, TAU,

and PTAU to the retrieval performance is mainly

evaluated through the evaluation scenarios listed in

Табле 3. The following features are used to generate

the feature vector in these scenarios:

 MRI (combination of dynamic (VolPCfit or

VolSPC), static measures (Vol34+CT34), and

SPARE-AD),

 Cognitive tests (combination of CDRsb,

ADAS11, ADAS13, MMSE, RAVLT

immediate, RAVLT learning, RAVLT

forgetting, and RAVLT forgetting percent)

 Risk factors (APOE ε4, Age)

 CSF markers (ABETA, TAU, and PTAU)

Table 3: Scenarios Based on MRI Measures, FDG-PET

measures, Cognitive Tests and Risk Factors.

Scenario Descriptor

S3.1 MRI_VolPCfit+Vol34+CT34+SPARE-AD

S3.2 MRI

VolSPC+Vol34+CT34+SPARE-AD

S3.3 S3.1+RiskF

S3.4 S3.2+RiskF

S3.5 S3.1+CognitiveT

S3.6 S3.2+CognitiveT

S3.7 S3.1+CFS

S3.8 S3.2+CFS

S3.9 S2.3+Co

nitiveT+CFS

S3.10 S2.4+CognitiveT+CFS

4 EXPERIMENTAL RESULTS

AND DISCUSSION

In this section, results of the medical case retrieval

evaluated through the three evaluation scenarios

described in subsection 3.4 are provided and

discussed.

4.1 Results Based on MRI, Cognitive

Scores, and Risk Factors

Results of the medical case retrieval based on a

combination of MRI imaging markers, cognitive tests

scores as well as risk factors, are given in Table 4.

This table lists the MAP values for each scenario in

case of two distances for calculation of the similarity

between the query patient's descriptor and all the

patients' descriptors stored in the database, Canberra

and Manhattan. For each subset, we include the

results for MRI based descriptors proposed in

(Dineva, 2022) for reference.

Table 4: Evaluation of the Medical Case Representation

Based on MRI ROI Measures and Cognitive Tests – value

of MAP.

Descriptor

MAP

(MD)

MAP

(CD)

S1.1:MRI_VolPCfit+Vol34+CT34

+SPARE-AD

0.88 0.86

S1.2:MRI_VolSPC+Vol34+CT34+

SPARE-AD

0.88 0.86

S1.3:S1.1+Co

nitiveT 0.95 0.97

S1.4:S1.2+Co

nitiveT 0.95 0.97

S1.5:S1.1+RiskF 0.87 0.86

S1.6:S1.2+RiskF 0.88 0.86

S1.7:S1.1+Cognitive + RiskF 0.95 0.97

S1.8:S1.2+Co

nitive + RiskF 0.95 0.97

Table 4 shows significant improvement of the

value of MAP when cognitive tests are included in the

descriptor (S1.3 i S1.4). The MAP value considering

these scenarios increased to 0.95 in case of Manhattan

distance, while in case of Canberra distance, the MAP

is 0.97, meaning that the inclusion of the cognitive

tests provides great improvement. Despite that

Canberra distance is a weighted version of the

Manhattan distance, we believe that its nature to be

more robust to outliers, but very sensible to values

around 0, which might happen in the case with

cognitive scores values, led to these results.

Additional benefit in the case of the combination

of MRI with the cognitive test scores is that the

dimension of the descriptor is lower than in the case

of MRI-based descriptor. In fact, in S1.3 and S1.4

scenarios, in most of the cases only 28-29 features

were selected, while in S1.1 and S1.2 scenarios, 34-

40 features were selected in most of the cases. From

the cognitive tests taken into consideration in this

research, CDR-SB, ADAS_13, MMSE, RAVLT

(5_sum), RAVLT (perc. forgetting), and FAQ were

selected in almost all cases as most relevant features.

On the other side, the combination of the MRI

imaging markers and risk factors such the Age and

APOE status (S1.5 and S1.6 scenarios), did not

induce any improvement. In only 19 cases the age

was selected to be a part of the descriptor, while the

APOE status was not selected at all.

Similarly, the influence of the risk factors in terms

of improvement of medical case retrieval for AD is

not the case also when all measures were

concatenated (S1.7 and S1.8 scenarios).

4.2 Results Based on MRI, FDG PET,

Cognitive Scores, and Risk Factors

Results of the evaluation for the descriptors S2.1-

S2.10 using the subset 2 are given in Table 5.

HEALTHINF 2023 - 16th International Conference on Health Informatics

558

Table 5: Evaluation of the Medical Case Representation

Based on MRI ROI Measures, Cognitive Tests, Risk

Factors, and FDG PET measures – value of MAP.

Descriptor

MAP

(MD)

MAP

(CD)

S2.1:MRI_VolPCfit+Vol34+CT34+

SPARE-AD

0.87 0.84

S2.2:MRI_VolSPC+Vol34+CT34+S

PARE-AD

0.87 0.84

S2.3:S2.1+FDG

PET+HCI+sROI

0.88 0.84

S2.4:S2.2+FDG

PET+HCI+sROI

0.88 0.84

S2.5:S2.1+FDG_PET+HCI+sROI

+RiskF

0.89 0.82

S2.6:S2.2+FDG_PET+HCI+sROI

+RiskF

0.89 0.82

S2.7:S2.1+FDG_PET+HCI+sROI

+Co

nitiveT

0.98 0.98

S2.8:S2.2+FDG_PET+HCI+sROI

+Co

nitiveT

0.98 0.98

S2.9:S2.3+Co

nitive + RiskF 0.98 0.98

S2.10:S2.4+Co

nitive + RiskF 0.98 0.98

According to the results, we came to the conclusion

that FDG-PET based measures provide a slight

increase of the MAP value (gaining a value of 0.88)

when Manhattan distance is used to calculate the

similarity. It should be emphasized that in this case,

from the used FDG-PET measures, HCI at 6- and 24-

month follow-up, as well as sROI

at 12- and 24-

month follow-ups were selected, and the descriptor

was long 36 features in most of the cases.

Moreover, the inclusion of the risk factors APOE

and age also led to an improvement when evaluation

was performed on the subset 2 with Manhattan

distance, by selecting the age as a relevant feature, but

not the APOE. The dimension of the feature vector in

most of the cases was 21 features.

Similarly like in the case of the evaluation based

on the subset 1, the cognitive tests scores provided

significant improvement of the retrieval precision in

the case on both similarity measures. But the

combination of all the features considered in this

evaluation, did not contribute to additional

improvement. In most of the cases, 23 features were

selected.

4.3 Results Based on MRI, Cognitive

Scores, Risk Factors, and CSF

Table 6 contains the results of the performed

evaluation for the descriptors S3.1-S3.10. Their

influence on the retrieval results is evaluated using

the subset 3. It is selected so that contains fully

competed data for all of the measures evaluated in this

scenario (MRI, cognitive tests, risk factors, and CSF

biomarkers: ABETA, TAU, and PTAU)

Table 6: Evaluation of the Medical Case Representation

Based on MRI ROI Measures, Cognitive Tests, Risk

Factors, and CSF biomarkers – value of MAP.

Descriptor

MAP

(MD)

MAP

(CD)

S3.1:MRI_VolPCfit+Vol34+CT34

+SPARE-AD

0.86 0.87

S3.2:MRI_VolSPC+Vol34+CT34+

SPARE-AD

0.86 0.87

S3.3:S3.1+RiskF 0.86 0.87

S3.4:S3.2+RiskF 0.86 0.87

S3.5:S3.1+Co

nitiveT 0.96 0.98

S3.6:S3.2+CognitiveT 0.96 0.98

S3.7:S3.1+CFS 0.86 0.88

S3.8:S3.2+CFS 0.86 0.88

S3.9:S2.3+Co

nitiveT+CFS 0.96 0.98

S3.10:S2.4+Co

nitiveT+CFS 0.96 0.98

The results showed that CFS biomarkers led to a

slight improvement of the medical case retrieval

precision. In almost all of the cases, ABETA level

was selected automatically by the algorithm, and

TAU and PTAU values were selected in fewer cases.

In most of the cases, the dimension of the descriptor

was 20 features.

Following the trend of the evaluation results

obtained on the subset 1 and 2, the combination with

the cognitive tests scores provided the best MAP.

4.4 Discussion

Instead of focusing on one modality, i.e. one type of

data, usually MRI, this research takes into

consideration multi-modal medical case

representation appropriate to AD, and evaluates the

medical case retrieval in terms of different

information contained in the medical case descriptor.

Following the TADPOLE recommendations, and the

considering that the reported research about medical

case retrieval for AD (mostly focused on information

extracted from the MRI images, cross-sectionally or

longitudinally) (Trojacanec, 2015; Trojachanec,

2017, Trojacanec 2017; Kruthika, 2019a; Kruthika,

2019b, Vinutha, 2019; Chethan, 2020; Dineva, 2022)

and less on other modalities (Gupta, 2019), we

performed a wide evaluation on different kinds of

medical case representations.

We performed the evaluation using fully complete

datasets, to be able to provide fair evaluation not

influenced by the missing data or the strategy used to

cope with the missing data. But, on the other hand,

trying to provide fully compete datasets, we ended up

with small subsets for some of the evaluation

scenarios and unbalanced classes (subset 2 and 3). In

this regards, FDG-PET measures were available for a

A Multi-Modality Approach to Medical Case Retrieval for Alzheimer’s Disease

559

very small number of patients (64), while CSF

measures for only 85 patients from ADNI 1.

According to the obtained results, with this

research we got significant improvement over the

current results in the reported research (Trojacanec,

2015; Trojachanec, 2017, Trojacanec 2017; Kruthika,

2019a; Kruthika, 2019b, Vinutha, 2019; Chethan,

2020, Gupta, 2019. In fact, cognitive tests scores

significantly increased the MAP in all three

evaluation subsets, leading to MAP value of 0.98.

Among the cognitive scores, CDR-SB, ADAS_13,

MMSE, RAVLT (5_sum), RAVLT (perc.

Forgetting), and FAQ were most frequently selected

features. Additionally, the FDG–PET based measures

(among which HCI at 6- and 24- month, and sROI

at 12- and 24- month follow-ups were automatically

selected in most of the cases) and the age as a risk

factor, provided a slightly better performance or no

improvement at all. Regarding the CSF, the most

frequently selected feature was ABETA level, but did

not significantly improved the retrieval results.

It should be emphasized that the features (of a

given modality) selected in most of the cases for a

given scenario, were similarly selected through all the

evaluations for which they were available. Despite

the fact that most of them are significant markers for

AD, this frequent automatic selection makes them

stable and appropriate to be used to represent medical

cases for AD.

The strategy for medical case representation used

in this research reflects the current condition of the

brain / atrophy, the degree of degeneration and the

progression of the disease, the nerve cells damage and

the brain metabolism, covering all the stages of the

disease in the medical case retrieval for AD, using the

entire cascade of disease indicators. Moreover, it can

be easily adopted for addressing specific aspects of

other neurological disease. Additionally, considering

the low number of features in the descriptor and high

precision at the same time, this strategy can be easily

adopted and integrated as an efficient and clinically

relevant decision support in the standard practice and

to assist medical experts. However, the challenging

part that needs to be addressed and involved in the

clinical workflow is rapid information processing

regarding different modalities for the a patient

currently examined (all other medical cases may be

processed off-line and stored in the database).

5 CONCLUSIONS

This research investigated the medical case retrieval

for AD based on different modalities of data. For this

purpose, we made a research in order to find a

relevant and efficient representation of the medical

case that will provide precise, efficient and clinically

relevant retrieval. Particularly, we focus our research

on multi-modal patient representation including the

following kinds of data: structural MRI, FDG-PET

based measures, CSF protein levels, and APOE

genotype and age as risk factors.

According to the performed evaluation, we

outperformed the results gained by the other research

on medical case retrieval for AD, which is mostly

based on single modality. We also provided a good

basis for further investigation and analysis in this

domain, by using a comprehensive approach. In the

future, we are going to extend the research to a wider

dataset, including more subjects and more phases of

the disease (early-MCI, late-MCI, …). We are also

are going to investigate deep learning methods in

terms of feature engineering.

ACKNOWLEDGEMENT

Data collection and sharing for the research in this

paper was funded by the Alzheimer's Disease

Neuroimaging Initiative (ADNI) (National Institutes

of Health Grant U01 AG024904) and DOD ADNI

(Department of Defense award number W81XWH-

12-2-0012).

REFERENCES

Alzheimer's Association, 2022. 2022 Alzheimer's disease

facts and figures. [https://www.alz.org/media/docu

ments/alzheimers-facts-and-figures.pdf] Аccessed:

30.10.2022.

Alzheimer's Disease Neuroimaging Initiative: ADNI, 2017.

[https://adni.loni.usc.edu/] Accessed: 5.11.2022.

Cha, S. H. (2007). Comprehensive survey on

distance/similarity measures between probability

density functions. City, 1(2), 1.

Chen, K., Langbaum, J. B., Fleisher, A. S., Ayutyanont, N.,

Reschke, C., Lee, W., ... & Alzheimer's Disease

Neuroimaging Initiative. (2010). Twelve-month

metabolic declines in probable Alzheimer's disease and

amnestic mild cognitive impairment assessed using an

empirically pre-defined statistical region-of-interest:

findings from the Alzheimer's Disease Neuroimaging

Initiative. Neuroimage, 51(2), 654-664.

Chen, K., Ayutyanont, N., Langbaum, J. B., Fleisher, A. S.,

Reschke, C., Lee, W., ... & Reiman, E. M. (2011).

Characterizing Alzheimer's disease using a

hypometabolic convergence index. Neuroimage, 56(1),

52-60.

HEALTHINF 2023 - 16th International Conference on Health Informatics

560

Chethan, K., & Bhandarkar, R. (2020). Hybrid Feature

Extraction Technique on Brain MRI Images for

Content-Based Image Retrieval of Alzheimer’s

Disease. In Advances in Communication, Signal

Processing, VLSI, and Embedded Systems (pp. 127-

141). Springer, Singapore.

Davatzikos, C., Xu, F., An, Y., Fan, Y. and Resnick, S.M

(2009). Longitudinal progression of Alzheimer's-like

patterns of atrophy in normal older adults: the SPARE-

AD index. Brain, 132(8), 2026-2035.

Dineva, K. T., Kitanovski, I., Dimitrovski, I., &

Loshkovska, S. (2022). Combining Static and Dynamic

Features to Improve Longitudinal Image Retrieval for

Alzheimer’s Disease. In International Conference on

ICT Innovations (pp. 107-120). Springer, Cham.

FreeSurfer. [https://surfer.nmr.mgh.harvard.edu/].

Accessed: 21.10.2022.

Gupta, Y., Lama, R. K., Kwon, G. R., & Alzheimer's

Disease Neuroimaging Initiative. (2019). Prediction

and classification of Alzheimer’s disease based on

combined features from apolipoprotein-E genotype,

cerebrospinal fluid, MR, and FDG-PET imaging

biomarkers. Frontiers in computational neuroscience,

13, 72.

Hall, M. A., & Holmes, G. (2003). Benchmarking attribute

selection techniques for discrete class data mining.

IEEE Transactions on Knowledge and Data

engineering, 15(6), 1437-1447.

Jack Jr, C. R., Bennett, D. A., Blennow, K., Carrillo, M. C.,

Dunn, B., Haeberlein, S. B., ... & Silverberg, N. (2018).

NIA-AA research framework: toward a biological

definition of Alzheimer's disease. Alzheimer's &

Dementia, 14(4), 535-562.

Kruthika, K. R., Maheshappa, H. D., & Alzheimer's Disease

Neuroimaging Initiative. (2019a). Multistage classifier-

based approach for Alzheimer's disease prediction and

retrieval. Informatics in Medicine Unlocked, 14, 34-42.

Kruthika, K. R., Maheshappa, H. D., & Alzheimer's Disease

Neuroimaging Initiative. (2019b). CBIR system using

Capsule Networks and 3D CNN for Alzheimer's disease

diagnosis. Informatics in Medicine Unlocked,14,59-68.

Marinescu, R. V., Oxtoby, N. P., Young, A. L., Bron, E. E.,

Toga, A. W., Weiner, M. W., ... & Alexander, D. C.

(2018). Tadpole challenge: Prediction of longitudinal

evolution in Alzheimer's disease. arXiv preprint

arXiv:1805.03909.

Marinescu, R. V., Oxtoby, N. P., Young, A. L., Bron, E. E.,

Toga, A. W., Weiner, M. W., ... & Alexander, D. C.

(2020a). The alzheimer's disease prediction of

longitudinal evolution (TADPOLE) challenge: Results

after 1 year follow-up. arXiv preprint

arXiv:2002.03419.

Marinescu, R. V. (2020b). Modelling the Neuroanatomical

Progression of Alzheimer's Disease and Posterior

Cortical Atrophy. arXiv preprint arXiv:2003.04805.

Meyer, P.F., Tremblay-Mercier, J., Leoutsakos, J., Madjar,

C., Lafaille-Maignan, M.É., Savard, M., Rosa-Neto, P.,

Poirier, J., Etienne, P., Breitner, J. and PREVENT-AD

research group (2019). INTREPAD: A randomized trial

of naproxen to slow progress of presymptomatic

Alzheimer disease. Neurology, 92(18), e2070-e2080.

Moguilner, S., Birba, A., Fittipaldi, S., Gonzalez-Campo,

C., Tagliazucchi, E., Reyes, P., ... & Ibáñez, A. (2022).

Multi-feature computational framework for combined

signatures of dementia in underrepresented settings.

Journal of Neural Engineering, 19(4), 046048.

Porsteinsson, A.P., Isaacson, R.S., Knox, S., Sabbagh,

M.N. and Rubino, I. (2021) Diagnosis of early

alzheimer’s disease: Clinical practice in 2021. The

Journal of Prevention of Alzheimer's Disease, 8(3),

371-386.

Reuter, M., Schmansky, N.J., Rosas, H.D. and Fischl, B

(2012). Within-subject template estimation for

unbiased longitudinal image analysis. Neuroimage,

61(4), 1402-1418.

The Alzheimer's Disease Prediction Of Longitudinal

Evolution (TADPOLE) Challenge

[https://tadpole.grand-challenge.org/] Accessed:

20.11.2022.

Trojacanec, K., Kitanovski, I., Dimitrovski, I. and

Loshkovska, S. October. Medical image retrieval for

Alzheimer’s disease using data from multiple time

points. In International Conference on ICT Innovations,

pp. 215-224. Springer, Cham (2015).

Trojacanec, K., Kalajdziski, S., Kitanovski, I., Dimitrovski,

I., Loshkovska, S. and Alzheimer’s Disease

Neuroimaging Initiative. Image Retrieval for

Alzheimer’s Disease Based on Brain Atrophy Pattern.

In International Conference on ICT Innovations, pp.

165-175. Springer, Cham (2017).

Trojachanec, K., Kitanovski, I., Dimitrovski, I. and

Loshkovska, S. Longitudinal brain MRI retrieval for

Alzheimer's disease using different temporal

information. IEEE Access, 6, 9703-9712 (2017).

Van Dyck, C. H., Nygaard, H. B., Chen, K., Donohue, M.

C., Raman, R., Rissman, R. A., ... & Strittmatter, S. M.

(2019). Effect of AZD0530 on cerebral metabolic

decline in Alzheimer disease: a randomized clinical

trial. JAMA neurology, 76(10), 1219-1229.

Vinutha, N., Sandeep, S., Kulkarni, A. N., Shenoy, P. D., &

Venugopal, K. R. (2019, March). A Texture based

Image Retrieval for Different Stages of Alzheimer’s

Disease. In 2019 IEEE 5th International Conference for

Convergence in Technology (I2CT) (pp. 1-5). IEEE.

Weber, C. J., Carrillo, M. C., Jagust, W., Jack Jr, C. R.,

Shaw, L. M., Trojanowski, J. Q., ... & Weiner, M. W.

(2021). The Worldwide Alzheimer's Disease

Neuroimaging Initiative: ADNI‐3 updates and global

perspectives. Alzheimer's & Dementia: Translational

Research & Clinical Interventions, 7(1), e12226.

Jack Jr, C. R., Knopman, D. S., Jagust, W. J., Petersen, R.

C., Weiner, M. W., Aisen, P. S., ... & Trojanowski, J.

Q. (2013). Update on hypothetical model of

Alzheimer’s disease biomarkers. Lancet neurology,

12(2), 207.

A Multi-Modality Approach to Medical Case Retrieval for Alzheimer’s Disease

561