Detecting 2D NMR Signals Using Mask RCNN

Hadeel Saad Alghamdi

, Alexei Lisitsa

, Igor Barsukov

and Rudi Grosman

Computer Science , University of Liverpool, U.K.

Biochemistry & Systems Biology, University of Liverpool, U.K.

Keywords:

NMR Spectre Analysis, Peak Detection, Mask R-CNN.

Abstract:

Picking peaks in two-dimensional Nuclear Magnetic Resonance (NMR) spectra has been a critical research

problem and a very time-consuming important step in further analyses of NMR biological molecular sys-

tems. Here, we implemented machine learning approach for peak detection and segmentation using machine

learning framework Mask R-CNN.The model was trained on a large number of synthetic spectra of known

conﬁgurations, and we show that our model demonstrates promising results up to 0.93 accuracy. We imple-

mented uniform scaling on the data matrix during training to further improve detection to achieve 10.17%

FPs and 1.7% FNs rate. We show the utility of Mask R-CNN on NMR spectra where the data range plays an

important role in peak detection.

1 INTRODUCTION

Nuclear Magnetic Resonance (NMR) spectroscopy is

one of most powerful technique for the conﬁrmation

of structural identity and for structure elucidation of

unknown compounds (Elyashberg, 2015).NMR spec-

troscopy is a widely popular approach (K., 1986) as

it provides information on the molecular structure of

complex chemical compounds and mixtures. It ﬁnds

its applications in various areas including medical di-

agnosis, drug discovery by monitoring inter molec-

ular interactions(Pellecchia et al., 2008),and identi-

fying components by screening complex mixtures in

various environmental science studies. One of the

critical steps in NMR structure determination is peak

picking (Pellecchia et al., 2008).A typical 2D spec-

trum (see Fig. 1) can contain hundreds to thousands

of peaks, and the identiﬁcation and quantitative char-

acterization of those peaks has a signiﬁcant impact on

all downstream analyses and further data interpreta-

tion. Each peak is distinguished by its centre position

(frequency coordinates corresponding to, so called

chemical shifts), peak shape along each dimension

and peak amplitude(Hesse et al., 2007).The follow-

ing steps are invariably entailed in the analysis of an

NMR spectrum: (i) identiﬁcation of the entire set of

peaks, known as peak picking; (ii) assignment of each

peak to the atoms it belongs to; and (iii) quantiﬁca-

tion of each peak by determining the peak amplitude

or volume(Li et al., 2021).These steps have been only

be partially automated despite years of progress.The

biggest difﬁculty in automation peak picking comes

from picking peaks in small molecular and distin-

guishing them from artifacts making this task tiring,

time-consuming and complicated without expert hu-

man assistance.

Figure 1: High resolution 2D NMR spectrum section of a

complex mixture. Typically NMR data range is between 0-

12 ppm on

H and 0-200 ppm on

C. Left panel shows the

typical contour visualisation of the data used during analy-

ses. Right panel shows the same data in a 3D visualisation.

Numerous approaches have been proposed to

solve this problem. These include picking signals

based on their intensities, volumes, or local signal-to-

noise ratio and are frequently combined with meth-

ods that improve spectrum quality,such as wavelet

transform(Liu et al., 2012),Bayesian methods(Cheng

et al., 2014) and spectra-decomposition-based meth-

ods(Tikole et al., 2014).Semi manual peak picking

940

Alghamdi, H., Lisitsa, A., Barsukov, I. and Grosman, R.

Detecting 2D NMR Signals Using Mask RCNN.

DOI: 10.5220/0011804700003393

In Proceedings of the 15th International Conference on Agents and Artiﬁcial Intelligence (ICAART 2023) - Volume 3, pages 940-947

ISBN: 978-989-758-623-1; ISSN: 2184-433X

 2023 by SCITEPRESS – Science and Technology Publications, Lda. Under CC license (CC BY-NC-ND 4.0)

usually provides more precise results with the as-

sistance of interactively picking features of com-

mon NMR software such as POKY(Goddard and

Kneller, 2007), NMRView (Johnson and Blevins,

1994), CCPN (SP et al., 2016) and some popular com-

mercial packages such as TopSpin and Mnova. The

applications of various machine learning techniques

for NMR peak picking have been studied over the

years. These include multi-layer back-propagation

artiﬁcial neural networks (Corne et al., 1992), sup-

port vector machines (Klukowski.P and al, 2015), and

variants of deep learning (Li et al., 2021) among oth-

ers. These approaches mainly targeted peak recogni-

tion in large protein molecules. Low intensity peaks

in small molecules is still a challenge up to date.

Here, we introduce a more generalized NMR

spectral peak picking method, using Mask R-

CNN(He et al., 2017) to automatically localize,

segment and classify peaks and their Full Width

Half Height area (FWHH) in both large and small

molecules.We report our preliminary results obtained

by using synthetic datasets of artiﬁcial spectra with

the small number of peaks for training Mask-R-CNN

models. We present tuning of default Mask-R-CNN

conﬁgurations and study the effects of scaling of data.

Overall performance appears to be promising (up

to 93% accuracy), but it requires further systematic

evaluation on more realistic spectra with the larger

amounts of peaks.

2 PEAKS IN NMR SPECTRA

Due to exponential decay of the NMR signals the

peaks have a theoretical Lorentzian line shape. How-

ever, the use of window functions in the processing

of 2D NMR data leads to the shape changes that

can be approximated by a mixture of Gaussian and

Lorentzian functions (Davis et al., 2020).Therefore,

we used both of the functions G(x, y) and L(x, y) to

generate the data for the training:

G(x, y) =

√

2π

exp



−

(x −x

)

2σ

(y −y

)

2σ



L(x, y) =

πγ

1 + (

x−x

y−y

)

Fig. 2 demonstrates peaks modelled by Gaussian

and Lorentzian functions. The height of the peaks is

represented by the color.

Figure 2: Filled contours of peaks calculated with A) Gaus-

sian functions on both dimensions and B) Lorentzian func-

tion on both dimensions.

3 METHOD

3.1 Mask R-CNN Framework

Mask R-CNN is a deep neural network aimed to solve

instance segmentation problem in machine learning

or computer vision (He et al., 2017). Mask R-CNN

model is commonly used for object detection and seg-

mentation. It has usefully been trained to detect and

segment objects in various areas including medicine,

(Chiao et al., 2019). planetary science (Davis et al.,

2020) and engineering (Zhao et al., 2018).The In-

put in most of those cases is originally an image in

RGB format with relatively large and well outlined

objects.In contrast, the NMR peaks are not conﬁned

to a well-deﬁned region of the spectrum. Their inten-

sities gradually decrease, but in the absence of noise,

are theoretically present in any part of the spectrum.

The noise in the experimental spectra makes the peak

contribution undetectable once it falls below the noise

level, however for each peak this boundary depends

on the peak amplitude, line-width and shape. This

presents a major challenge in assigning a mask to the

peak. If, similar to objects in photographs, the mask

is considered as the area outside of each no contribu-

tion from the peak can be detected, no mask can be

assigned in the absence of noise, while the different

degree of noise would change the mask of the same

peak object, making the mask dependent on the condi-

tions rather than on the peak properties. Additionally,

peaks in the NMR spectra have wide range of am-

plitudes, and the overall spectra are arbitrary scaled

during the measuring and data processing, while in

the images there is an objective image scaling, and

much more limited range of intensities.To address

these challenges in the practical NMR spectroscopy,

the cross-section at half-height is used to characterise

the peak, which is only dependent on the peak param-

eters, but not on the amplitude or noise level. This

area is straightforward to calculate theoretically, mak-

Detecting 2D NMR Signals Using Mask RCNN

941

ing it objective and suitable for generation of train-

ing data. And prediction of this mask would be much

more informative in the real data analysis as it will di-

rectly reﬂect the peak parameters. However, as illus-

trated in Fg.4, the half-height mask is much smaller

than the total area of the detectable peak intensities.

It is unclear whether Mask R-CNN developed for the

recognition of well-deﬁned shapes is suitable for the

analysis of ”diffused” objects, such as NMR peaks,

that do not have a well-deﬁned invariable mask that

covers the whole object. In this study we investigate

a general applicability of Mask R-CNN to the recog-

nition of peaks in the NMR spectra.

Figure 3: Mask R-CNN structure.

Fig. 3 outlines the architecture of Mask R-CNN.

It consists of a pre-trained CNN network on image

classiﬁcation tasks to generate feature maps of the in-

put. Region Proposal Network (RPN) slides a win-

dow over the feature maps to generate anchors for

each image. The anchors are used then by RPN to

generate Regions of Interest (RoI) which bounding

box that may or may not contain a peak.Mask R-CNN

handles RoIs without digitizing thus yielding feature

maps of ﬁxed sized. The feature maps are fed into

fully connected layers to make classiﬁcation, where

boundary box prediction is reﬁned using the regres-

sion. The warped features are also fed into Mask

branch classiﬁer in parallel in the existing branches

for classiﬁcation and localization.The mask branch

is a small fully connected network (FCN) applied to

each RoI predicting a pixel wise segmentation mask

of the object(He et al., 2017).

3.2 Mask Calculation

Training data for Mask R-CNN contains two forms of

input, the target objects and corresponding masks.The

masks in our dataset is calculated to cover the area

at full width half height (FWHH) values around each

peak.

Fig. 4 demonstrates the mask area which doesn’t

cover the whole object (peak) as it is the case in most

object detection projects.

Figure 4: Mask covers only FWHH area.

3.3 Dataset

The dataset is synthetically generated using NMRglue

(Helmus and Jaroniec, 2013),a python library pack-

age for NMR analysis. Each generated spectrum con-

tains a number of peaks with the following properties:

1. Spectra are generated as matrices of 64-bit ﬂoats.

2. Each matrix has a shape of 128x128 (minimum

size for Mask R-CNN(He et al., 2017).

3. Each matrix contains k peaks randomly located.

where k = [1..4] with uniform distribution.

4. The peaks are either Gaussian at both axis or

Gaussian at one axis and Lorentzian at the other.

5. Maximum amplitude = 1.

6. Masks are calculated to cover the area of full

width half height (FWHH) values around each

peak (local maximum) (see Fig. 5 for an illustra-

tion).

Figure 5: Three peaks and their masks.

7. No noise present during the training.

8. No overlapping masks present.

3.4 Pre-Processing

Mask R-CNN have been primarily applied to the

recognition of natural images, which feature perspec-

tive (distance-size dependence, e.g. objects placed

further appear smaller), ﬁxed aspect ratio and are reg-

ular grids of discrete-value pixels (0–255 in RGB

space). Mask R-CNN applies some standard prepos-

sessing procedures (He et al., 2017)to the input data

such as:

1. Non-square data is resized to square shape by Re-

size and pad with zeros to get a square shape.

2. Inputs must have three channels.

ICAART 2023 - 15th International Conference on Agents and Artiﬁcial Intelligence

942

3. Input data is converted to 32-bit ﬂoats followed

by subtraction of mean value. Post training, mean

value is added back and data is converted to 8-bit

integers.

4. Batch Normalization Layers are frozen by default.

The model was trained in two implementation scenar-

ios where the ﬁrst one as standard as possible and the

second one with sensible changes better suited to the

nature of our data.

3.5 Evaluation Criteria

For evaluation of the results of training we use mAP

measure and the rates of False Positive (FP) and False

Negative (FN) detections. mAP is a evaluation metric

used in Mask R-CNN for object localisation and clas-

siﬁcation(Everingham et al., 2010). The deﬁnitions of

FP and FN should take into account not only predicted

classes but also the locations of predicted bounding

boxes. Hence we deﬁne TP, FP, FN as follows. In

Mask-R-CNN the predicted detection {(b

, c

, p

)}

indexed by object j consists of:

• Bounding Box (BB) b

• predicted class (‘peak’ or ‘no peak’ in our case)

• conﬁdence score p

A predicted detection (b, c, p) is regarded as a True

Positive (TP) if:

• The predicted category c equals the ground truth

label c

• The overlap ratio IOU (Intersection Over

Union)(Everingham et al., 2010).

IOU(b, b

) =

area (b ∩b

)

area (b ∪b

)

between the predicted bounding box b and the

ground truth bounding box b

is not smaller than a

predeﬁned threshold ε. A predicted detection (b, c, p)

is regarded as a False Positive (FP) if:

• shared a ground truth with another detection (i.e

both predicted detection have IOU with the same

ground truth bounding box in excess of ε) which

has higher conﬁdence score p

, or

• has no ground truth bounding box with IOU

greater than ε.

A False negative (FN) is detected if:

• a ground truth peak is found without predicted

bounding box with IOU greater than ε

The occurrences False Positives and True Posi-

tives detections are illustrated in Fig. 6.

Figure 6: A, B, C and D display TP, FP and FN in different

scenarios.

4 STANDARD

IMPLEMENTATION

We use the implementation of Mask-R-CNN avail-

able at (He et al., 2017). The model hyper parameters

are listed in Table 1.

Table 1.

Hyper parameters Value Deﬁnition

BACKBONE resent50

The backbone networks

DETECTION MIN CONFIDENCE 0.9

Minimum conﬁdence threshold

to accept the ﬁnal detected ROI

CHANNEL NUMBER COUNT 3

number of channel is 3 for RGB

In our case, we kept it 3 by replicating

the same channel three times

IMAGE MAX DIM 128 x 128

Deﬁne input image

resizing (none)

TRAIN BN Freeze

Train or freeze batch

normalization layers

Number OF EPOCHS 20 - 90

deﬁne the number of epochs

4.1 Detection Results

Starting with 700 spectra data size, and the Testing

method used here is the traditional 20/80 split. The

model scored a mAP of 0.660 with 2.15 % of FPs

and FNs 19 % of FNs. More detailed analysis was

conducted of the detected peaks by extracting their

details such as height, width and data mean of a whole

spectra, they fall into three categories:

• Correctly Detected Peaks: peaks with maximum

value >= 3.4 are correctly detected and classiﬁed.

• Valid peaks with overlapped false peaks: valid

peaks with corresponding Ground Truth GT, those

peaks have also false overlapped peaks.

Detecting 2D NMR Signals Using Mask RCNN

943

• Not Detected Peaks: missed peaks where they

have GT, but the model failed to detect them.The

majority of the those peaks have maximum value

<= 3.4.

Figure 7: Detected peaks.

To better understand the detection results and im-

prove the accuracy we have applied magnitude scal-

ing for the data. Here we have relied also on the

fact that all essential information in NMR spectra is

magnitude-invariant, that is scaling up or down by a

constant factor does not change the information con-

tent of NMR spectra, but it may well affect the per-

formance of machine learning algorithms on the data.

In practice one can also should take into account the

scaling limitations caused by a particular implemen-

tation (e.g by the range of datatype values available).

In Table 2 we represent the detection results

for various scaling factors applied to testing phase

only.Then the range mean is calculated. The range is

deﬁned here as the difference between the maximum

and minimum values for each spectra then taking the

mean of those values for whole testing dataset.

Table 2.

Scaling factor Range mean mAP FPs % FNs %

No scaling 0.697 0.660 2.150 19.120

5 3.785 0.751 2.370 9.762

10 7.350 0.769 1.280 7.710

15 11.010 0.771 1.080 7.310

50 34.900 0.817 2.720 7.900

100 69.000 0.815 3.000 8.700

150 97.560 0.841 0.540 7.560

200 145.230 0.813 1.560 6.510

255 177.450 0.781 2.460 8.760

As seen from the table the model started preform-

ing better when the data mean between 34.90 and

177.23.Since the scaling was only applied during test-

ing and to form a complete conclusion,Table 3 shows

the outcomes of training the model with scaling dur-

ing: both training and testing, training only and test-

ing only.The scaling factor is determined by selecting

the best preformed model from Table 2 which is 150.

Table 3.

Scaling applied mAP FPs % FNs %

No scaling 0.660 2.150 19.120

Train & Test 0.898 6.480 6.940

Train only 0.579 8.5 13.9

Test only 0.781 2.460 8.760

Scaling during testing only or training and testing

combined show similar results.

4.2 Larger Dataset Size

To eliminate the possibility of inefﬁcient number

of training samples,we generated bigger dataset (5k

spectra).The dataset description and hyper parame-

ters are the same as 700 spectra data size. The model

scored a mAP of 0.616 with 12.15% of FPs and FNs

31% of FNs. Table 4 presents the scaling results dur-

ing the testing stage, and it displays decline in FP

and FN percentage when the data ranges from 2.52

to 7.104 and an increase in mAP score

Table 4.

Description Range mean mAP FPs % FNs %

No scaling 0.5002 0.616 12.19 31.55

5 2.5200 0.630 12.01 9.81

10 4.7690 0.684 10.61 8.90

15 7.1040 0.686 9.10 11.55

50 25.4770 0.557 8.36 21.25

100 48.8730 0.549 8.09 17.78

150 73.1000 0.489 12.32 20.24

200 100.7300 0.464 10.82 18.67

255 122.4400 0.492 11.59 16.60

Similar to the 700 dataset, scaling was also ap-

plied during the three stages. However, scaling during

training and testing combined outperformed the other

scenarios as shown in Table 5.

Table 5.

Scaling applied mAP FPs % FNs %

No scaling 0.616 12.19 31.55

Train & Test 0.784 7.27 2.59

Train only 0.781 3.19 16.66

Test only 0.492 11.59 16.60

Finally,The two best preformed models are se-

lected by scaling to the optimal data range and trained

with the optimal number of epochs see Table 6.

• Scaling of the range mean between 83.23 and

122.7.

ICAART 2023 - 15th International Conference on Agents and Artiﬁcial Intelligence

944

• Scaling in both the training and testing stages.

Table 6.

Data size mAP FPs % FNs %

700 0.788 0.400 1.200

5k 0.971 1.160 2.320

5 ADJUSTED IMPLEMENTATION

As mentioned in the previous section, most unde-

tected peaks that have maximum value <= 3.4. Those

peaks are relatively small and that might be a reason

for low performance of Mask R-CNN standard imple-

mentation. In order to improve performance we have

made several changes to the framework.

5.1 Modiﬁed Normalization

The default Mask R-CNN pre-processing in explained

in Section III-D, converts ﬂoats into integers directly

causing the values to be ﬂoored to 0.In order to pre-

serve the data the pre-processing was modiﬁed. Fig.8,

shows the effects of standard normalization (upper

part) and modiﬁed normalization (bottom part). The

bottom part of the ﬁgure shows the modiﬁed method

where the subtraction of zeros (instead of mean value)

and the conversion to original datatype (ﬂoat64) in-

stead to unint8 has been applied.

Standrad

Modified

Input loading

• Converting to float

• Substrating mean values

• Adding mean values

• Converting to unint8

Input loading

• Converting to float

• Substrating (zeros)

as mean values.

• Adding mean values

• Converting to oringial

datatype ( float64)

• Read original as Float64 matrix

• Convert to flaot32