Potential Use of Smartphone as a Tool to Capture Embryo Digital

Images from Stereomicroscope and to Evaluate Them by an Artificial

Neural Network

Diego de Souza Ciniciato

1,*

, Maria Beatriz Takahashi

1,*

, Marcelo Fábio Gouveia Nogueira

and José Celso Rocha

Laboratório de Matemática Aplicada, School of Sciences and Languages, Universidade Estadual Paulista (Unesp),

Av. Dom Antonio 2100, Assis, Brazil

Laboratório de Micromanipulação Embrionária, School of Sciences and Languages, Unesp,

Av. Dom Antonio 2100, Assis, Brazil

Both authors contributed equally to the study

Keywords: Smartphone, Embryo Classification, Artificial Neural Network-based Software, Graphic User Interface.

Abstract: An online graphical user interface connected to a server was developed aiming to facilitate access to

professionals worldwide that face problems with bovine blastocysts classification. The blastocysts

assessment is carried on using images taken from an inverted microscope, which usually requires more

expensive devices such as digital camera and computer software. Smartphone camera quality and tasks

processing are getting better with technology advances. Therefore, a smartphone can be attached to the

eyepiece lens to provide Real-Time evaluation, and thus reducing costs when comparing to computers,

cameras, and software that are commonly used for this purpose.

1 INTRODUCTION

Brazilian cattle production has an important

contribution to the economy and social aspects of

this country. With approximately 215 million

livestock units and leader in meat exportations since

2004, Brazil is also the leader in in vitro production

of bovine embryos worldwide (Ministério do

Planejamento, Desenvolvimento e Gestão - Instituto

Brasileiro de Geografia e Estatística IBGE, 2016).

This production has utmost importance for

international and national improvement in cattle

genetics and productivity. The production of cattle

embryos for commercial purposes follows the steps:

they are produced in vitro and transferred to

synchronized receptors when they reach the

blastocyst stage (Hyttel et al., 2010). To help to

identify the quality embryos, which is associated

with the success of pregnancy, the International

Embryo Technology Society (IETS) recommends an

embryo classification system. This system is based

on morphological evaluation and establishes three

quality grades: excellent or good, “1”; fair, “2”; or

poor, “3” (Bó and Mapletoft, 2013).

However, this classification is directly affected

by the embryologist’s accuracy and experience to

evaluate the embryo variables related to the

development and pregnancy potential (Lindner and

Wright, 1983; Bó and Mapletoft, 2013). The reason

for this interference is that the morphological

analysis does not measure any objective variables to

determine the embryo classification. Moreover, the

analysis by the human vision is based on a

comparison between objects or images. In this

regard, human vision has difficulties at judging color

or brightness of shapes and features, which requires

measuring scales or relative size, angle and positions

of several objects to identify their characteristics

(Russ, 2016). Thus the analysis by an embryologist

is subjective and has low reproducibility (Bényei et

al., 2006). Indeed, the same embryo can be

classified with different degrees of quality by

different embryologists (inter-evaluator error) or

even by the same embryologist at different moments

(intra-evaluator error), especially in cases when the

quality grade is borderline (Farin et al., 1995).

Together with inexperience, the tiredness and the

mood of the evaluator could contribute to the major

Ciniciato D., Takahashi M., Gouveia Nogueira M. and Rocha J.

Potential Use of Smartphone as a Tool to Capture Embryo Digital Images from Stereomicroscope and to Evaluate Them by an Artiﬁcial Neural Network.

DOI: 10.5220/0006518501850189

In Proceedings of the International Conference on Computer-Human Interaction Research and Applications (CHIRA 2017), pages 185-189

ISBN: 978-989-758-267-7

causes of the subjective and low reproducibility of

this standard system of embryo assessment.

Therefore, several methods have been or are

being developed to provide an optional evaluation

for embryo classification that does not have external

effects. Some of them includes a semi-automatized

image segmentation process with the use of artificial

intelligence (AI) for human embryos (Gonzalez,

2004), an automatic segmentation procedure of

bovine embryos without AI (Melo et al., 2014), a

semi-automatized grading method of human

blastocyst using a support vector machine (Santos

Filho et al., 2012), embryo metabolism analysis,

cellular respiration measurements, the use of zona

pellucida birefringence, microRNA profile

determination, analysis based on logistic regression

and evaluation by time-lapse video (reviewed by

Rocha et al., 2016). However, none of these

methods are totally effective, and, despite being

subjective and old, the visual morphological analysis

is still widely used (Lindner and Wright, 1983; Farin

et al., 1995; Richardson et al., 2015).

Recently, there have been attempts at creating a

method based on digital image processing to

determine the viability of human embryos by

detecting blastomeres (Singh et al., 2014; Tian et al.,

2014) or trophectoderm (Singh et al., 2015).

Additionally, using processing and digital image

analysis in the quality evaluation of mouse

blastocysts, a previous study used an artificial neural

network technique with significant success (Matos,

Rocha and Nogueira, 2014). However, as far as we

can determine from the studied literature, a

classification method using digital image processing

has not been applied to bovine blastocysts.

In this context, a method based on artificial

neural network (ANN) combined with genetic

algorithm (GA) was developed to train an ANN to

classify bovine blastocyst images based on the IETS

standards (Rocha et al., 2017). In this study, a 482

bovine blastocysts images dataset were used to train

some ANNs, from which the best obtained 76.4% of

accuracy. The input set was the variables extracted

from image processing and the output was the mode

from grading of three experienced embryologists.

The use of three evaluations avoids the bias of using

a single evaluation as the standard for the ANN

training. The Kappa index of the inter-evaluator

agreement was 0.571 (482 images, P<0.001), and the

three ANNs obtained 0.616 for the same dataset

(482 images, P<0.001). This represents that the

ANN technique was more consistent than the

embryologists’ evaluation. Moreover, the intra-

evaluator agreement was 0.28, 0.41 and 0.47 (48

images, P<0.001), and when compared to the ANNs,

there were 100% agreement (Kappa index of 1.0),

which supports the robustness and low subjectivity

of an ANN.

The present position paper is a continuation in a

deeper way of the previous work (Rocha et al.,

2017), aiming the development of a Graphical User

Interface due to users that could not be familiar with

the programming environment and do not use/have

an inverted microscope. In addition, embryologists

from around the world can access the technique

online, without downloading or install the software.

Furthermore, we describe the application of

smartphone adapters for stereomicroscope ocular

lens to classify embryos in Real-Time.

2 METHODOLOGY

A server for image processing and classification of

bovine blastocysts was developed aiming to

democratize the technology available in our research

group. The access to the server is by the link below:

http://blasto3q.com. The image processing and

evaluation are carried out by the algorithm

Blasto3Q, which is described in (Matos, Nogueira

and Rocha, 2012, 2014). The users can access this

computational tool by a multiplatform application

available on the same server. The application has a

friendly and intuitive interface for users, and it has

additional functionalities comparing to the desktop

version, such as the evaluation of multiple images in

parallel. Due to the high processing cost for each

image, we choose to centralize this operation on the

server. If this action were carried on in the

smartphone, the execution time should increase

considerably, which is not desired by users.

Therefore, the smartphone just captures the

blastocyst images and receive the results from

classification.

On the server-side, there is a MATLAB

application (version R2017a) that works in service

mode, which executes several scanning of databases

to search non-processed requisitions. Each service

runs one process at a time, however, it can process

several instances, and thus the processing of

different requests will be performed in parallel and

simultaneously.

For a greater user experience, an intuitive user

interface was developed to general users, which runs

on the client-side. This interface communicates by

requests to the server. Each new processing request

is initialized by the desired image uploaded into the

server. This request is added to the database in the

end of the requests queue and it will be processed

according to its rank. The process is finished with

the output that the users want.

Therefore, there is an extremely light and fast

application that can perform in devices compatible

with HTML5 (more modern navigators). Nowadays

a large part of devices provides this markup

language, and an advantage is that it is possible to

use the application in different operating systems

(for example, both Android and iOS can execute the

application). The user can access this software

wherever they are, if they have an internet

connection. The results are processed in few

seconds.

Figure 1: Graphical User Interface evaluating bovine

blastocyst as grade 2 (“fair”) since the highest vector was

the yellow that is related to the fair degree.

In Figure 1, there is an example of the

application interface describing an in vitro produced

bovine embryo image taken from a smartphone

juxtaposed to an eyepiece of the stereomicroscope.

For this purpose, we used a Samsung S6 coupled to

a macro lens (Figure 2) to allow the proximity of the

smartphone lens with the eyepiece. With this

apparatus, the image of the embryo on the eyepiece

could be captured by the smartphone lens and using

the zoom of the phone to fill the screen with the

image. Moreover, we used the maximum of

magnification of the stereomicroscope (i.e., 60 x,

Leica M80).

3 DISCUSSIONS

The smartphone development allowed the creation

of new technologies and applications. Nowadays the

daily tasks made by a computer and a smartphone

are very similar, as we can access websites and

software on both devices. The smartphones have

advantages due to easy portability, and they are

lighter and cheaper than computers. Moreover, the

recent improvement of new generation smartphones’

cameras allowed taking images with higher quality

than previous generations.

Figure 2: Illustrative image of the macro lens attached to

the lens of the smartphone (left) and the macro lens alone

with its clipper (right).

In this context, the application of cheaper and

robust technology in research laboratories is

required to reduce costs. The image records using

microscopy in any laboratory usually requires

desktop computers and expensive digital cameras

and software to analyze them. Also, expensive

inverted microscopes are often required to obtain

those images with a high quality when recording

mammalian oocytes and embryos. The development

of an application that is functional in any device

(smartphone or other devices, such as desktop and

tablet) to evaluate images of blastocysts from

microscopy allows Real-time assessment, reduce

costs, and solve a subjective issue in blastocysts

classification.

Several adapters were developed to attach the

smartphones to the eyepieces of a microscope,

which provide better ergonomic, simple and fast

ways to take pictures of the sample, as in anatomic

pathology analysis using mobile devices (Lehman

and Gibson, 2013; Roy et al., 2014) and diagnostic

of diseases using deep learning (Quinn et al., 2016)

both attached to conventional light microscope. In

this way, besides the macro lens attached to the

smartphone lens, an adapter to a better stabilization

and focal plane quality could useful. Also, the

standardization of the blastocysts images by image

processing steps keeps the features from the

blastocysts, which allows the software interprets it

properly.

The improvement of this technology processing

can be achieved by cloud computing, which is a

model related to applications called ‘Software as a

service’. The basis for this model is to run on distant

computers linked by a cloud that is owned and

operated by others and that connect to users’

computers via a web browser. The advantages of this

method are the access to applications and data from

different computers that are connected to the cloud,

less risk of missing data and dynamically scalable

(Armbrust et al., 2010).

4 CONCLUSIONS

The bovine blastocyst classification by Artificial

Neural Network available as a graphical user

interface provides a robust method to embryologists

that are not familiar with programming languages.

Also, the smartphone adapters for microscope

eyepiece should provide better ergonomic and a

Real-Time assessment of bovine embryos.

ACKNOWLEDGEMENTS

The authors’ research is supported by grants #

2012/50533-2, 2013/05083-1, 2006/06491-2,

2011/06179-7, 2012/20110-2 and 2016/19004-4

from São Paulo Research Foundation (FAPESP).

We also thank Agência UNESP de Inovação (AUIN)

for processing the national and international patents

of the invention.

REFERENCES

Armbrust, M. et al. (2010) ‘A view of cloud computing’,

Communications of the ACM, 53(4), p. 50. doi:

10.1145/1721654.1721672.

Bényei, B. et al. (2006) ‘The effect of internal and

external factors on bovine embryo transfer results in a

tropical environment’, Animal reproduction science,

93(3–4), pp. 268–279.

Bó, G. and Mapletoft, R. (2013) ‘Evaluation and

classification of bovine embryos’, Animal

Reproduction, 54(935), pp. 344–348.

Farin, P. W. et al. (1995) ‘Agreement among evaluators of

bovine embryos produced in vivo or in vitro’,

Theriogenology, (95), pp. 339–349.

Gonzalez, R. (2004) Digital Image processing using

Matlab. Upper Saddle River New Jersey: Pearson

Prentice Hall.

Hyttel, P. et al. (2010) Essentials of domestic animal

embryology. Saunders/Elsevier.

Lehman, J. S. and Gibson, L. E. (2013) ‘Smart

teledermatopathology: a feasibility study of novel,

high-value, portable, widely accessible and intuitive

telepathology methods using handheld electronic

devices’, pp. 513–518. doi: 10.1111/cup.12108.

Lindner, G. and Wright, R. W. J. (1983) ‘Bovine embryo

morphology and evaluation’, Theriogenology,

20(313), pp. 407–416.

Matos, F. D., Nogueira, M. F. G. and Rocha, J. C. (2012)

‘Método para determinação da viabilidade e qualidade

de embriões’. Brazil.

Matos, F. D., Nogueira, M. F. G. and Rocha, J. C. (2014)

‘Method for determining embryo viability and

quality’. Brazil.

Matos, F. D., Rocha, J. C. and Nogueira, M. F. G. (2014)

‘A method using artificial neural networks to

morphologically assess mouse blastocyst quality’,

Journal of Animal Science and Technology, 56.

Melo, D. H. et al. (2014) ‘Algorithms for automatic

segmentation of bovine embryos produced in vitro’,

Journal of Physics: Conference Series 490, pp. 1–4.

Ministério do Planejamento, Desenvolvimento e Gestão -

Instituto Brasileiro de Geografia e Estatística IBGE.

Produção da pecuária municipal 2015. IBGE 43, 1-47

(2016). e-Book (bulletin) <http://biblioteca.ibge.

gov.br/visualizacao/periodicos/84/ppm_2015_v43_br.

pdf>.

Quinn, J. A. et al. (2016) ‘Deep Convolutional Neural

Networks for Microscopy-Based Point of Care

Diagnostics’, in Proceedings of International

Conference on Machine Learning for Health Care.

Richardson, A. et al. (2015) ‘A clinically useful simplified

blastocyst grading system’, Reproductive BioMedicine

Online, 31, pp. 523–530.

Rocha, J. C. et al. (2016) ‘Methods for assessing the

quality of mammalian embryos: How far we are from

the gold standard?’, JBRA Assisted Reproduction,

20(3), pp. 150–158.

Rocha, J. C. et al. (2017) ‘A method based on artificial

intelligence to fully automatize the evaluation of

bovine blastocyst images’, Scientific Reports,

7(Article number: 7659).

Roy, S. et al. (2014) ‘Smartphone adapters for digital

photomicrography’, J Pathol Inform, (October). doi:

10.4103/2153-3539.137728.

Russ, J. C. (2016) The Image Processing Handbook.

Seventh. CRC press.

Santos Filho, E.

et al. (2012) ‘A method for semi-

automatic grading of human blastocyst microscope

images’, Human Reproduction, 27, pp. 2641–2648.

Singh, A. et al. (2014) ‘Automatic blastomere detection in

day 1 to day 2 human embryo images using partitioned

graphs and ellipsoids’, IEEE International Conference

on Image Processing (ICIP), pp. 917–921.

Singh, A. et al. (2015) ‘Automatic segmentation of

trophectoderm in microscopic images of human

blastocysts’, IEEE Transactions on Biomedical

Engineering, 62(1), pp. 382–393.

Tian, Y. et al. (2014) ‘Automatic Blastomere Recognition

from a Single Embryo Image’, Computational and

Mathematical Methods in Medicine, 2014.