SuperResolution-aided Recognition of Cytoskeletons in Scanning
Probe Microscopy Images
Sara Colantonio
1
, Mario D’Acunto
1,2
, Marco Righi
1
and Ovidio Salvetti
1
1
Institute of Information Science and Technologies, National Research Council of Italy,
Via G. Moruzzi 1, 56124, Pisa, Italy
2
Institute of Structure of Matter, National Research Council, ISM-CNR, Via Fosso del Cavaliere 100, 00133, Rome, Italy
Keywords: Super-resolution, Pattern Recognition, Scanning Probe Microscope, Cytoskeleton Recognition.
Abstract: In this paper, we discuss the possibility to adopt SuperResolution (SR) methods as an important preparatory
step to Pattern Recognition, so as to improve the accuracy of image content recognition and identification.
Actually, SR mainly deals with the task of deriving a high-resolution image from one or multiple low
resolution images of the same scene. The high-resolved image corresponds to a more precise image whose
content is enriched with information hidden among the pixels of the original low resolution image(s), and
corresponds to a more faithfully representation of the imaged scene. Such enriched content obviously
represents a better sample of the scene which can be profitably used by Pattern Recognition algorithms. A
real application scenario is discussed dealing with the recognition of cell skeletons in Scanning Probe
Microscopy (SPM) single image SR. Results show that the SR allows us to detect and recognize important
information barely visible in the original low-resolution image.
1 INTRODUCTION
Recent advances in SuperResolution (SR) methods
are fostering an increasing interest in the possibility
to apply SR processing to improve the accuracy of
image content recognition. The most frequent
applications in this direction are oriented to video
surveillance and intelligent traffic control (Shih-
Ming et al., 2011; Suresh et al., 2007; Aliyan S.,
Broumandnia, 2012), though, obviously, any image
based task can profitably benefit from such a
technique.
Actually, SR mainly deals with the task of
deriving a high-resolution image from one or
multiple low resolution images of the same scene
(the multiple images have usually very slight
difference from one another since corresponding to
following frames of a video). High resolution is
meant both as an improvement of content precision,
thanks to denoising and content enhancement, and as
spatial enlargement.
The result in both cases is a more precise image
whose content is enriched with information hidden
among the pixels of the original low resolution
image or multiple images, which correspond more
faithfully to the imaged scene. Such enriched
content obviously represents a better sample of the
scene which can be profitably used by Pattern
Recognition (PR) algorithms.
Starting from this statement, we argue that SR
and PR can be valuably combined in a
computational framework to recognize and
understand image content.
In this paper, we briefly introduce this
framework and then show an example of its
application to the recognition of cytoskeleton in
Scanning Probe Microscopy (SPM) images.
Indeed, in recent years, the study of
Mesenchimal Stem Cells (MSCs) has attracted a lot
of attention in tissue engineering and regenerative
medicine thanks to MSCs ability to be committed,
along several lineages, through chemical and
physical stimuli. MSCs are usually analyzed via
Atomic Force Microscopy (AFM), one of the often
preferred SPM imaging techniques used to obtain
mechanical information on cell surfaces and
deposited extra-cellular matrix molecules (Danti et
al., 2006).
The goal is to correlate morphological,
functional, and mechanical aspects of human MSCs
to obtain a deeper understanding of their effects on
cells functions, metabolism and finally shape. These
703
Colantonio S., D’Acunto M., Righi M. and Salvetti O..
SuperResolution-aided Recognition of Cytoskeletons in Scanning Probe Microscopy Images.
DOI: 10.5220/0004830407030709
In Proceedings of the 3rd International Conference on Pattern Recognition Applications and Methods (ICPRAM-2014), pages 703-709
ISBN: 978-989-758-018-5
Copyright
c
2014 SCITEPRESS (Science and Technology Publications, Lda.)
aspects can be revealed, from a microscopy point of
view, by identifying the cytoskeletal components
and organs (see Figure 1).
Figure 1: Cell cytoskeleton consists of microtubules
(approximately, 25 nm in diameter), actin filaments (5–7
nm in diameter), intermediate filaments (8–12 nm in
diameter), and other binding proteins.
With the support of biologists of the BioLab located
at CNR in Pisa, the cytoskeleton was prepared
according to a method (Hawkins et al., 2013) which
allowed us to work with images containing
stabilized microtubule filaments.
However, the identification of such constituent
microtubules is generally non trivial due to
physiological variations in fiber surface properties
and to AFM acquisition modality, which affect
image visual appearance, such as tip-cell contact. In
this frame, our solution, based on the use of SR to
improve the image definition, can be a viable
approach to semi-automatic identification and
recognition of cytoskeleton components in AFM
single image SR. The method improves spatial and
photometric resolution, thus allowing the effective
image recognition. In particular, the method
highlights the hidden underlying biological
structures.
The paper is organized as follows: Section 2
reports a brief overview of the computational
framework for the combination of SR and PR
techniques; hence, Section 3 focuses on the
recognition of cytoskeleton in AFM images, and,
finally, results and discussion are reported in Section
4.
2 SUPERRESOLUTION-AIDED
PATTERN RECOGNITION – AN
OVERVIEW OF THE
METHODOLOGY
Pattern Recognition (PR) applied to image content
can be roughly defined as the “art” of detecting and
identifying relevant structures and/or their
relationships present in an image, usually with the
final aim to (semi-)automatically perform an image-
based task.
PR techniques heavily rely on the quality of the
visual appearance of the image, i.e. on the definition
and precision of the structures imaged in it. In this
frame, PR can dramatically benefit from SR
processing aimed at enhancing the visual quality of
images as well as magnifying their spatial resolution
so as to enlarge and highlight relevant structures
barely visible and recognizable in the original low-
resolution images.
Indeed, a pre-processing step, usually intended to
image enhancement and restoration, is normally
included in PR processing chain. In this frame,
systematic SR is a viable solution, focused on image
content enrichment based on the recovery of missing
high-resolution details that are not explicitly found
in low-resolution images. This is what we are going
to illustrate in this paper.
In particular, we here concentrate on single
images; this means that both PR and SR techniques
are applied to a still image (in literature, this case is
also referred to as single-frame SR). Further work
will deal with PR in images from video or multiple
imagery data (i.e., multiple-frame SR). In this case,
the SR processing can benefit from the presence of
multiple images of the same scene and then exploit
the information hidden in such a pack of data.
In the following, we report an overview of the
framework already introduced in (D’Acunto et al.,
2013).
Formally, we assume the following image
acquisition model (Liu et al, 2008).
x,
y
x
,y′
∗
x
x,
y
y
x,
y
(1)
where Lx,y is the acquired image, x′,y′ is the
Point Spread Function (PSF), Hx'‐x,y'‐y is the
ideal image and Nx,y is the noise.
The PSF is strictly correlated to the image
acquisition instrument and the degree of spreading
(i.e., blurring) of a point object actually measures the
quality of the imaging system. In many cases, PSF is
a complex function depending on instruments
characteristics and limits as well as possible artefacts
ICPRAM2014-InternationalConferenceonPatternRecognitionApplicationsandMethods
704
introduced during the acquisition.
For instance, in SPM imaging, PSF results from
all the artefacts introduced by the AFM tip-sample
contact, the tip-ample convolution or finite tip
radius, and sample changing stiffness under tip
pressure. Another source of artefact during the scan
of biological sample is the temperature change,
which could introduce drifts due to piezo-tube with
subsequent sample structure deformation (D’Acunto
and Salvetti, 2011).
In the general framework we propose, the main
idea is to reconstruct the ideal image
x
x,y
y
by firstly de-noising the acquired image L, so as
to eliminate the noise component x,y; and then
by reducing the SPF in two ways: by (i) eliminating
the acquisition artefacts and (ii) super-resolving the
artefact- and noise-free image.
The latter step allows us to recover an image as
close as possible to the ideal image, including
scarcely visible details that are not explicitly found
in the original acquired low-resolution image L.
Once recovered such an image, PR techniques
can be applied to understand image content, and
hence solve the specific image-based task at hand.
2.1 SuperResolution Method
A SR method gets the original low-resolution still
image as input and creates the high-resolution image
by filling the new image grid with all the available
low-resolution image pixels. During this filling
process, the SR algorithm leaves some empty pixels,
whose values are then estimated by a filling
function.
According to the approach followed to define
this function, existing methods can be categorized in
(a) interpolation-based, (b) reconstruction-based,
and (c) example-based.
Interpolation-based SR methods assume that
images are spatially smooth and can be adequately
approximated by polynomials such as bilinear,
bicubic or level-set functions (Park et al., 2003;
Morse and Schwartzwald, 2001; Fattal, 2007). This
assumption is usually inaccurate for natural images
and thus over-smoothed edges as well as visual
artifacts often exist in the reconstructed high-
resolution images.
The reconstruction-based approach faces SR as
an inverse problem consisting in recovering the
original high-resolution image by fusing multiple
low-resolution images, based on certain assumed
prior knowledge of an observation model that maps
the high-resolution image to the low resolution
images (Irani and Peleg, 1991; Lin and Shum, 2004).
Each low-resolution image imposes a set of linear
constraints on the unknown high-resolution pixel
values. When a sufficient number of low-resolution
images are available, the inverse problem becomes
over-determined and can be solved to recover the
high-resolution image. However, it has been shown
that the reconstruction-based approaches are
numerically limited to a scaling factor of two (Lin
and Shum, 2004).
Example-based methods learn the mapping
between low-resolution and high-resolution image
patches from a representative set of image pairs, and
then the learned mapping is applied to super resolve
the image at hand. The underlying assumption is that
the missing high-resolution details can be learned
and inferred from the low-resolution image and a
representative training set. Numerous methods have
been proposed for learning the mapping between
low-resolution and high-resolution image pairs with
promising results (Freeman et al., 2002; Sun et al.,
2003; Chang et al., 2004; Sun et al., 2008; Yang et
al., 2008; Xiong et al., 2009).
With the initial intent to verify that our idea has
real potentialities, we have selected the most
promising SR method among a set of single-frame
state-of-the-art techniques. In particular, the SR
method proposed in (Kim and Kwon, 2010) is an
application-agnostic example-based SR method. It
works in the spatial domain and consists in a multi-
step procedure that merges interpolation and
learning. More precisely, after a first step of cubic
spline interpolation to obtain the image at the
desired scale, the method estimates the missing
values by generating a set of candidate high-
resolution images according to a local patch-based
regressive approach. This candidate images are then
combined to form a final high-resolution image.
More precisely, for each image location ,, the
pixel value is obtained as the convex combination of
the N candidates according to the following softmax
scheme:
,
,
,,
,…,
(2)
where
ω
x,
y
e
|
,
|
∑
e
,
,…,
(3)
and {d
x,y
}
i
=1..N is the estimation of distances
between the unknown considered pixel and each
candidate. This estimate is calculated using a set of
SuperResolution-aidedRecognitionofCytoskeletonsinScanningProbeMicroscopyImages
705
linear regressors:
,
|
,
|
1,…,
(4)
where , is a vector constructed using the
concatenation of all columns of a spatial patch of
centred at , and the parameters W
} are
optimized based on the patch-based regression
results L for a subset of training images.
A final post-processing step is included so as to
improve edge appearance.
3 CYTOSKELETON
RECOGNITION IN SPM
IMAGES
The study of Mesenchimal Stem Cells (MSCs) relies
on the identification of their skeletal components
and organs. These have usually a microtubule shape
with a particular distribution pattern, as shown in
Figure 1. Indeed, it is well-known that living cells
are in general very soft and mechanically
inhomogeneous; hence the corresponding
cytoskeleton forms a rigid network that controls and
supports both the cell shape and the cell movement.
AFM is usually the most used SPM technique to
investigate cell skeletons. The AFM works using a
probe to image the cell sample. Such tiny probe can
be considered as a paraboloid with a final sphere
(normally the radius of the sphere is 10-20nm) in
permanent or intermittent contact with the sample
generally considered flat (this corresponds to two
different modes of acquisition).
Based on the contact force between the probe
and the cell sample, the image recorded with an
AFM present a shot of the cell cytoskeleton. Being
the cytoskeleton composed by a complex network of
different cell components, such as actin filaments,
microtubules, proteins etc, it can be a rather complex
challenge to identify the different cyto-components
(see Figure 2).
Nevertheless, SR processing can significantly
improve the identification of such components (as
recently shown also by Chacko et al. 2013).
In this sense, we applied our framework to semi-
automatically identify the microtubule structures of
cytoskeletons depicted in AFM images.
As evident in Figure 2, besides clearly visible
filaments, many other structures are barely visible
and distinguishable in the microscopy image. SR
processing is a viable solution to face this issue.
According to the general framework introduced
above, a multi-step procedure is applied to identify
the different microtubules and filaments:
- image correction and denoising;
- image contrast improvement;
- image resolution improvement;
- microtubule recognition.
Figure 2: An AFM image depicting the microtubule
structures of an MSC skeleton.
More precisely, due to the characteristics of the
microscope imaging device, a tilt correction is
initially required. Then, contrast enhancement is
carried out according to Zuiderveld’s method
(Zuiderveld, K., 1994). As introduced above, SR
processing relies on the application of the Kim-
Kwon method.
Finally, the super-resolved image is processed
using a patch-wise semi-automatic pattern
recognition algorithm.
The aim is to identify a specific area
x
,y
⊆
x,y
of the super-resolved image H
corresponding to a microtubule.
Starting from a selected area of the image, the
PR algorithm selects a central pixel p
and applies a
kind of region growing method based on the
gradient value of pixel neighbourhood. More
precisely, the algorithm constructs a connected
region corresponding to a microtubule by adding,
neighbour by neighbour, a pixel connected with the
previous if the derivate between these two pixels is
lower a certain value (relative derivate ∆
) and if the
distance between the analysed pixel and the start
pixel is lower than a certain value (absolute delta
∆
).
Formally, starting from the selected pixel
p
∈
, a new pixel is inserted in
if and only if:
p
∈I
⟺p
∈I
,
|
p
p
|
∆
,
|
p
p
|
∆
(5)
ICPRAM2014-InternationalConferenceonPatternRecognitionApplicationsandMethods
706
4 RESULTS
The proposed multi-step procedure has been
implemented in Matlab and applied to AFM images
of MSC cytoskeletons, as the one shown in Figure 2.
Results show that, thanks to the SR methods,
also filaments barely visible in the original low-
resolution image have been identified.
Figure 3 shows an example of such result. A
patch of the original low-resolution image has been
selected, as shown in Figure 3.A and Figure 3.B
shows its rough enlargement. The SR method
allowed a 4X super-resolved image to be obtained,
i.e., the one reported in Figure 3.C. This way, a
“hidden” filament could be discovered and
characterized. Indeed, the PR method was able to
identify and delineate it as shown by the result in
Figure 3.D.
Figure 4 shows another example on a different
sample.
Measuring the dimension of the recognized
patterns provided a quantitative confirmation of the
results by consulting biologists of BioLab in Pisa. A
pixel in super resolved images corresponded to
about five nanometers. In the example of Figure 3,
we recognized fourteen microtubule structures,
Figure 3: Results of the SR-aided pattern recognition
method for the detection of microtubule cell structures. A:
The original image and the selected patch. B: roughly
enlargement of the original content of the selected patch.
C: the 4x super-resolved image of the selected patch. D:
the results of the pattern recognition method applied to the
super-resolved image.
Figure 4: Another example of application of the SR-aided
pattern recognition method. A and C are SR areas and B
and D are the respective recognized patterns.
considering different square sub-images. In all these
cases, both length and width of the recognized
pattern were in agreement with typical values
(Schaap et al., 2006) of microtubules.
These instances show how effectively SR
processing can improve the original image and then
facilitate the recognition of specific patterns.
We also tested our method by applying it to
synthetic images containing a set of cylindrical
shapes. We found that these shapes could be
recognized after both reduction of resolution and
addition of noise. We found that the percentage error
(number of pixels either wrongly assigned or non-
assigned to the pattern to identify) was 0.8% when
the signal-to-noise ratio was 11.6 dB and was 7.4 %
when the signal-to-noise ratio was 8.7 dB.
Figure 5.A gives the 3D representation of the
high-resolution area shown in Figure 4.C, while
Figure 5.B gives the 3D representation of the
original area corresponding to Figure 4.C.
5 CONCLUSIONS
The method proposed consisted in the application of
PR methods to single images enhanced by SR
algorithms. The application we carried out to the
recognition of cytoskeleton microtubules led to
biologically significant results as confirmed by a
SuperResolution-aidedRecognitionofCytoskeletonsinScanningProbeMicroscopyImages
707
Figure 5: On the left a 3D perspective of the SR image, on the right a 3D perspective of the original image.
group of biologists. This confirmed the vast range of
effectiveness of SR and allowed introducing a useful
specific tool in the field of the recognition of
biological structures.
Futures research will concern the following
points. Firstly, the method will be applied to a
greater number of experimental images. This will
allow improving it according to the properties of
new data and to better assess its validity.
Secondly, the stage of proper PR, following the
stage of image enhancement, will be further tested
and possibly improved.
Thirdly, more precise criteria will be given for
the selection of appropriate sub-images, with the aim
of possibly making this stage automatic.
Finally, other methods of image processing will
be taken into account with the purpose of
introducing modification and/or additions to our
method. For instance, methods of filaments
estimation should be considered (see, e.g., Genovese
et al., 2012).
ACKNOWLEDGEMENTS
The authors wish to thank in particular dr. Davide
Chiarugi, biologist, for the reliable support in data
validation.
REFERENCES
Aliyan, S., Broumandnia, A., 2012. A New Machine
Learning Approach to Deblurring License Plate Using
K-Means Clustering Method. In International Journal
of Advanced Research in Artificial Intelligence, Vol. 1,
No. 2, 2012.
Chacko, J.V., Cella Zanacchi, F., Diaspro, A., 2013.
Probing Cytoskeletal Structures by Coupling Optical
Superresolution and AFM Technqiues for a
Correlative Approach, in Cytoskeleton, Wiley,
doi:10.1002/cm.21139.
Chang, H., Yeung, D.Y., Xiong, Y., 2004. Super-
resolution through neighbor embedding. In
Proceedings of IEEE Conference on Computer Vision
and Pattern Recognition, 2004, 275-282.
D'Acunto, M., Salvetti, O., 2011. Pattern recognition
methods for thermal drift correction in atomic force
microscopy imaging. In Pattern Recognition and
Image Analysis - PRIA, vol. 21 (1) pp. 9.
D'Acunto, M., Pieri, G., Righi, M., Salvetti, O., 2013. A
Methodological Approach for combining super-
resolution and pattern recognition to image
identification. In Pattern Recognition and Image
Analysis – PRIA (in press).
Danti, S., D’Acunto, M., Trombi, L., Berrettini, S.,
Pietrabissa, A., 2006. A Micro/Nanoscale Surface
Mechanical study on Morpho-Functional Changes in
Multilineage-Differentiated Human Mesenchimal
Stem Cells, in Macromolecular Bioscence.
Fattal, R., 2007. Image upsampling via imposed edge
statistics. In Proc. of SIGGRAPH '07: ACM
SIGGRAPH 2007 papers, ACM, 2007.
Freeman, W.T., Jones, T.R., Pasztor, E.C., 2002.
Example-based super-resolution. In IEEE Computer
Graphics and Applications , 2002, pp. 56-65.
Genovese, C.R., Perone-Pacifico M., Verdinelli I.,
Wasserman L., 2012. The geometry of nonparametric
flament estmation. In Journal of the American
Statistical Association, 2012, pp. 788-799.
Irani, M., Peleg, S., 1991 Improving resolution by image
registration. In Computer Vision, Graphics and Image
Processing 53, 1991, 231-239.
ICPRAM2014-InternationalConferenceonPatternRecognitionApplicationsandMethods
708
Kim, K. I., Kwon, Y., 2010. Single-image super-resolution
using sparse regression and natural image prior. In
IEEE Trans. Pattern Analysis and Machine
Intelligence, vol. 32, no. 6, p. 1127-1133.
Lin, Z., Shum, H.Y., 2004 Fundamental limits of
reconstruction-based superresolution algorithms under
local translation. In IEEE Transactions on Pattern
Analysis and Machine Intelligence 26, 2004, 83-97.
Liu, H.Y., Zhang, Y., Song, J.I., 2008. Study on the
Methods of Super-resolution Image Reconstruction. In
International Society for Photogrammetry and Remote
Sensing, vol. XXXVII, pp. 461-466.
Morse, B., Schwartzwald, D., 2001. Image magnification
using level set reconstruction. In Proceedings of IEEE
Conference on Computer Vision and Pattern
Recognition, 2001, pp. 333-341.
Park, S.C., Park, M.K., Kang, M.G, 2003. Super-
resolution image reconstruction: A technical overview.
In IEEE Signal Processing Magazine, 2003, 21-36.
Schaap, I.A.T., Carrasco, C., J. de Pablo, P., MacKintosh,
F.C., Schmidt, C.F., Elastic Response, Buckling, and
Instability of Microtubules under Radial Indentation,
Biophysical Journal, Volume 91, Issue 4, 2006, pp.
1521-1531.
Schaap, I., Yang-Ting, C., Szu-Hua, W., Jar-Ferr, Y.,
2011. Multi-Resolution Local Probabilistic Approach
for Low Resolution Face Recognition, In Proc. of
International Conference on Intelligent Computation
and Bio-Medical Instrumentation (ICBMI), 2011, p.
220-223, 14-17 Dec. 2011.
Shih-Ming, H., Yang-Ting, C., Szu-Hua, W., Jar-Ferr, Y.,
2011. Multi-Resolution Local Probabilistic Approach
for Low Resolution Face Recognition, In Proc. of
International Conference on Intelligent Computation
and Bio-Medical Instrumentation (ICBMI), 2011, p.
220-223, 14-17 Dec. 2011.
Sun, J., Sun, J., Xu, Z., Shum, H.Y., 2008. Image super-
resolution using gradient profile prior. In Proceedings
of IEEE Conference on Computer Vision and Pattern
Recognition, 2008.
Sun, J., Zheng, N.N., Tao, H., Shum, H.Y., 2003. Image
hallucination with primal sketch priors. In Proceedings
of IEEE Conference on Computer Vision and Pattern
Recognition. Volume 2, 2003, 729-736.
Suresh, K.V., Mahesh Kumar, G., Rajagopalan, A.N,
2007. Superresolution of License Plates in Real Traffic
Videos. In IEEE Transactions on Intelligent
Transportation Systems, Vol. 8, No. 2, 2007.
Xiong, X., Sun, X., Wu, F., 2009. Image hallucination
with feature enhancement. In Proceedings of IEEE
Conference on Computer Vision and Pattern
Recognition, 2009.
Yang, J., Wright, J., Huang, T., Ma, Y., 2008. Image
super-resolution via sparse representation of raw
image patches. In Proceedings of IEEE Conference on
Computer Vision and Pattern Recognition, 2008.
Hawkins, T.L., Spet, D., Mogessie, B., Straube, A., Ross
J.L., 2013. Mechanical Properties of Doubly
Stabilized Microtubule Filaments. In Biophysical
Journal, Cell Press, pp. 1517–1528.
Zuiderveld, K., 1994. Contrast Limited Adaptive
Histogram Equalization. In Graphics Gems IV,
Academic Press, pp. 474–485.
SuperResolution-aidedRecognitionofCytoskeletonsinScanningProbeMicroscopyImages
709
