Animal Health Informatics: Towards a Generic Framework for

Automatic Behavior Analysis

Position Paper

Dmitry Kaplun

, Aleksandr Sinitca

, Anna Zamansky

, Stephane Bleuer-Elsner

, Michael Plazner

Asaf Fux

and Dirk van der Linden

Saint Petersburg Electrotechnical University “LETI”, Russia

University of Haifa, Israel

University of Bristol, U.K.

Keywords:

Animal Behavior, Animal Welfare, Veterinary Informatics, Computer Vision, Deep Learning.

Abstract:

The ﬁeld of veterinary healthcare informatics is still in its infancy, and state of the art solutions from human

healthcare are not easily adapted. IoT and wearable technologies may be bringing a wind of change, as large

amounts of health data of animals are now being produced. It makes this a timely moment to initiate a discus-

sion on the possibilities for cross-fertilization between the worlds of human and animal health informatics. In

this position paper we report on an ongoing project developing a framework for automatic video-based anal-

ysis of animal behavior and describe its concrete application for decision support of behavioral veterinarians.

The framework is generic, allowing for reuse across species and different analytical tasks. We further discuss

the possibilities for cross-fertilization between human and animal behavior analysis in the context of health

informatics.

1 INTRODUCTION

The notion of veterinary informatics was coined by

Talbot (Talbot, 1991), surveying the many applica-

tions of medical informatics in the veterinary pro-

fession. Such applications have explored messaging

standards, terminology lists, and standardized diag-

nosis lists (Santamaria and Zimmerman, 2011), as

well as applications of bioinformatics techniques to

the veterinarian domain (Sujatha et al., 2018). Yet

more than two decades after Talbot, the ﬁeld is still in

its infancy (Smith-Akin et al., 2007; Alpi, 2009) and

state of the art methodologies used in human health-

care informatics are neither easily adaptable, nor eas-

ily adopted by veterinarians.

Recent technological advances in IoT and big data

may bring the wind of change to the world of animal

health informatics. These technologies have started

to transform the living conditions of farming animals,

bringing to public attention also issues of food quality

and sustainability. The pet market is also catching up

on the hype of wearable devices, with a variety of ac-

tivity and ﬁtness trackers for pets which include sen-

The pet-wearables market is expected to grow at a

CAGR of 13.5% before 2025, with exponentially growing

demand from Asia following initial popularity in the West.

sors that can measure activity, sleep and vital signs of

pets. There are initiatives such as the PetCommunity

Blockchain, promoted by Petpace, aiming to create a

global network of interconnected health pet data, to be

produced and consumed by pets, owners, veterinari-

ans, and service providers. Thus animals have started

to produce signiﬁcant amounts of health data. This

holds a great promise for a cross-fertilization between

the worlds of human and animal health informatics.

Tools developed by informatics are useful not only

for veterinary science, but also for animal welfare –

especially for automatic assessment of behavior. An-

imal welfare science seeks to handle welfare issues

raised by the keeping and use of animals. It deﬁnes

different welfare parameters with regard to physio-

logical state, affective state, and natural living state

(Broom, 1996; Fraser, 2009). Behavior analysis is

an important tool in the assessment of animal welfare

used for the assessment of pain, injury and disease.

Behaviour is also of crucial importance in gauging

what animals want (Zamansky et al., 2017), most ob-

viously in the use of choice and preference tests, but

also through other methods that are particularly suit-

able for on-farm welfare assessment (Dawkins, 2004).

Thus behavior analysis and assessment is an in-

tegral part of animal health and welfare. There are

436

Kaplun, D., Sinitca, A., Zamansky, A., Blauer-Elsner, S., Plazner, M., Fux, A. and van der Linden, D.

Animal Health Informatics: Towards a Generic Framework for Automatic Behavior Analysis Position Paper.

DOI: 10.5220/0007566504360441

In Proceedings of the 12th International Joint Conference on Biomedical Engineering Systems and Technologies (BIOSTEC 2019), pages 436-441

ISBN: 978-989-758-353-7

voices calling for objectivization of methods for as-

sessing and analyzing behavior. As Karen Overall

noted in (Overall, 2014), “A review of behavioral data

over the past decade supports a serious shift to crisper

deﬁnitions of terms and quantiﬁable assessment of be-

haviors. Veterinary behavior and veterinary behav-

ioral medicine are coming later to this approach, but

the change is welcome.”

Informatics can have a great impact on such ob-

jectivization, supporting behavior analysis in a more

quantiﬁable and objective way, both in terms of pre-

cision and in terms of volumes of processable data.

Indeed, automatic tracking systems based on video

analysis are widely used for different species of an-

imals, such as wild animals (Burghardt and

Cali

2006), pigs (Ahrendt et al., 2011; Tillett et al., 1997),

poultry (Sergeant et al., 1998), insects (Noldus et al.,

2002), and many more. Well-developed systems for

rat and mice behavior recognition such as Ethovision

(Van de Weerd et al., 2001) and Laboras (Spink et al.,

2001) are widely employed in behavioral research.

While companion animals such as dogs are the

most frequently addressed species in animal health

research and practice, until now very few works ad-

dressed automated analysis of their behavior. One

approach, which has been gaining increasing inter-

est with the development of commercial smart collars

and trackers, is using wearable sensors such as ac-

celerometers and gyroscopes to recognize behaviors

and analyze motion. Several such systems have been

proposed (Gerencs

er et al., 2013; Brugarolas et al.,

2013), but currently only basic behaviors can be rec-

ognized with reasonable accuracy. Commercial track-

ers, such as FitBark and Petpace, may offer better ac-

curacy, but are yet to be scientiﬁcally validated to be

applied in health related settings.

In this paper we describe a video-based approach

to automatic analysis of animal behavior. Our project

started out focusing on companion animals, but ex-

panded into a generic approach, which can be used

across species, as well as across health related anal-

ysis tasks. Another feature of our approach its low

cost and simplicity, so that it can be easily applied

in different practical health related settings, and e.g.,

can be integrated into clinician’s workﬂow. It is low

cost and simple to use by working with data ob-

tained from the most basic affordable equipment, i.e.,

a cheap web camera that requires no special installa-

tion. Our approach consists of two layers: (i) a com-

puter vision layer, which performs automatic track-

ing of the animal by applying state-of-the-art machine

learning techniques, and thus is easily trained on dif-

ferent species of animals in different settings, and (ii)

a sense making layer, which interprets the informa-

tion extracted by the computer vision layer according

to the health related task at hand. This separation of

concerns facilitates the genericity of our approach.

In the rest of this paper we describe our approach

and its concrete applications in animal health infor-

matics. We further highlight the need for generic and

simple solutions for animal health informatics, and

call for cross-fertilization between animal and human

health informatics.

2 THE BLYZER FRAMEWORK

AND ITS APPLICATIONS

The current study is part of an ongoing research

project, which is a collaboration between the

Tech4Animals Lab

at the University of Haifa and the

AI lab at St. Petersburg Electrotechnical University

”LETI”. This project aims at developing automatic

tools for animal behavior analysis using state-of-the-

art machine learning techniques. We start by describ-

ing the Blyzer approach, and then present two con-

crete applications in different settings related to ani-

mal health.

The Blyzer (Behavior analyzer) approach pro-

vides a generic solution for automatic behavior analy-

sis. By ‘generic’ we mean that it can be easily reused

across species and environments, and across different

analysis tasks, as we explain below.

Figure 1 provides an overview of the approach.

The Blyzer architecture consists of two different mod-

ules. The computer vision module tracks the animal

in its chosen environment, producing spatio-temporal

data (location of animal in each frame, speed, orien-

tation, etc.). The sense making module uses the ex-

tracted spatio-temporal data for producing the cho-

sen analysis. To calibrate the computer vision mod-

ule, sufﬁcient amount of video footage of the cho-

sen species in the chosen environment needs to be

recorded. The frames are manually tagged, and ma-

chine learning techniques are applied to produce an

appropriate classiﬁer for the computer vision module.

To adapt the sense making module to the needs of the

user, the analysis task needs to be formulated in pre-

cise terms.

The results can then be shown to the user by cre-

ating an appropriate interface. We describe concrete

applications of the framework in the context of animal

health in the following section.

See: http://www.tech4animals.org

Animal Health Informatics: Towards a Generic Framework for Automatic Behavior Analysis Position Paper

437

ComputerVisionModule

{module}

SenseMaking

{module}

Spatio-temporal

{data}

Blyzer Framework

Visualization

{data}

Analysis

{data}

Video footage

{data}

User

Interface

{module}

…calculates…

…interpreted by…

InterfaceCalibration

Training data

{data}

Frames

Tags

…used by…

…generates…

…analyzes…

Behavioral indicators

{data}

…used for…

Model

{classiﬁer}

…trains…

Figure 1: Blyzer architecture.

3 DECISION SUPPORT FOR

BEHAVIORAL

VETERINARIANS

Behavioral disorders of domestic dogs (Canis Famil-

iaris) such as aggressive behavior, lack of self-control

and anxiety are a major threat for the well-being of

companion dogs and their owners, leading to dogs’

relinquishment to shelters, or preventing shelter dogs

from being adopted, eventually leading to their eu-

thanasia (Winslow et al., 2018; Scarlett et al., 2002;

Sherman and Serpell, 2008). One behavioral problem

frequently encountered in veterinary behavior prac-

tice is dogs’ hyperactive behavior. Such dogs are im-

pulsive, restless and inattentive, are difﬁcult to train

and may be dangerous to children and adults during

play due to lack of control of their bites. The increas-

ing interest in this disorder is reinforced by recent ev-

idence that the domestic dog is an adequate model

for the human Attention Deﬁcit Hyperactive Disorder

(ADHD) (Hoppe et al., 2017; Lit et al., 2010).

Diagnosis and treatment of behavioral problems

require special expertise in veterinary behavior. A

consultation with veterinary behaviorist lasts longer

than a regular checkup and is usually held according

to the following protocol. The vet starts with dis-

cussing the dog’s behavioral issues and main com-

plaints. He will then gather information from the

owner on adoption history, systematic behaviour work

up (eating, drinking, sleeping, playing, exploring, ag-

onistic, housetraining, somatosensory, phobias, sex-

ual), attachment quality, previous training history and

methods, living conditions (garden access, daily ex-

ercise: type and length). Depending on his diagno-

sis hypothesis and after ralizing the physical exam,

he might want to rule out some medical causes. At

this stage the dog may undergo complementary tests

(blood and urine tests, X-ray, MRI, etc). The direct

observation is done by the vet during this discussion

with the owner. That’s usually the time of the consul-

tation when the behavioral diagnosis is established.

The classiﬁcation of pathological disorders and

treatment decisions of a behavior specialist veterinar-

ian are usually done on the basis of (i) observation

of the dog during consultation, and (ii) owners’ self-

reporting. Concerning diagnoses, there is no current

consensus in the behavior community and several ap-

proaches exist. On one hand, most of them use a de-

scriptive approach, centered on describing the symp-

toms like ”storm phobia, food related aggression, anx-

iety” but no pathology name is established gathering

the symptoms (Overall et al., 1997). On the other

end, the French Veterinary Psychiatry based on the

nosography established by Pageat uses the same de-

scriptive symptoms but gathers them into pathologies

(e.g., deprivation syndrome) (Pageat, 1998). Further

discussion of these approaches is beyond the scope

of this paper. While in our evaluation we used cases

diagnosed using the French approach, our automated

approach is independent of the underlying behavioral

approach.

HEALTHINF 2019 - 12th International Conference on Health Informatics

438

The ﬁnal aim of a behavioral consultation is estab-

lishing a treatment plan, whose possibile options in-

clude e.g., environment modiﬁcation, behavior mod-

iﬁcation, and medication

, or some combination of

these options. To establish a treatment plan, behav-

ior assessment needs to be performed.

To quantify behavior assessment, several scales

and questionnaires have been developed and vali-

dated, such as the C-BARQ (Hsu and Serpell, 2003)

or the dog ADHD rating scale (Vas et al., 2007).

However, these scales are owner administered, and

thus are subject to non-objectivity and inaccuracy

(i.e., there is evidence that owners may misinterpret

the behavior of their pets (Mariti et al., 2012)). The

animal behavior research community has thus been

calling for further “objectivization” of behavioral as-

sessment (Overall, 2014).

In our ongoing project, we aim to support behav-

ioral veterinarian decision making, providing objec-

tive assessment tools for behavioral problems such as

ADHD-like behavior to be used during consultation.

For this particular disorder, for instance, we currently

aim to measure the degree of ‘erratic’ movement of

the dog around the consultation room, which indicates

the level of self-control exhibited by the dog, which

in many cases appears to be correlated with ADHD-

like behavior (at least in the pure cases without co-

morbidities).

The system based on the Blyzer framework is cur-

rently deployed in three behavioral clinics in Israel

and France. Using a simple camera attached to the

consultation room ceiling, the vet records the dog.

The recorded video, along with dog’s diagnostic data,

obtained from the owner report and additional dog’s

physical exam, is stored in a database. The veterinar-

ian can compare the different parameters which mea-

sure, e.g., how erratic the movement is (indicating the

level of self control), between different dogs, and be-

tween different visit of the same dog (e.g., before and

after receiving medical treatment for ADHD-like be-

havior).

In this particular case, the Blyzer framework is

instantiated as follows. The classiﬁer model for

dog detection was trained on video footage (see

Fig. 2) recoded in the clinics. The model was

developed using TensorFlow Object Detection API

faster rcnn resnet101, and was trained on around

6000 manually tagged frames. The computer vision

module now reaches a very high accuracy (above

90%), producing a spatio-temporal data in the form

of time-series (location of the dog per frame). The

sense making module currently supports visualizing

In some cases, surgical procedures, such as castration,

can be part of the management plan.

the dog’s movement, computing its different param-

eters and characteristics helpful for quantifying the

amount of “hyperactive” movement. Figures 3 and

4 demonstrate a visualization of the movement of two

dogs: normal and pathologically hyperactive during

the ﬁrst 3 mins of visiting the clinic.

Figure 2: Computer Vision Module tracking the dog in a

clinic.

Figure 3: movement of a normal dog (ﬁrst 3 mins).

Figure 4: movement of a hyperactive dog (ﬁrst 3 mins).

Animal Health Informatics: Towards a Generic Framework for Automatic Behavior Analysis Position Paper

439

A different example of an analysis task is provided

in our work (Zamansky et al., 2018), where similar

video analysis of the movement of dogs diagnosed

with anxiety when interacting with a moving toy was

analyzed, showing that anxious dogs were more in-

hibited and moved less when interacting with the toy.

4 DISCUSSION AND FUTURE

RESEARCH

We have described our approach aiming at genericity

and simplicity of use for behavior analysis for multi-

ple species and health settings. We hope that these

ideas can facilitate the growth of animal health in-

formatics and promote digitalization in veterinary sci-

ence and animal welfare. We envision many different

applications for the Blyzer framework, and are cur-

rently working in several its extensions, expanding

the scope of animal health informatics to the domain

of animal welfare.

Animal welfare is commonly centered around

three broad objectives: (1) to ensure good physical

health and functioning of animals, (2) to minimize

unpleasant “affective states” (pain, fear, etc.) and to

allow animals normal pleasures, and (3) to allow an-

imals to develop and live in ways that are natural for

the species (Fraser, 2009). According to Dawkins,

behaviour analysis plays a major role in these objec-

tives: “It is used in the clinical and pre-clinical as-

sessment of pain, injury and disease, and potentially

could have an even greater role, particularly if used

in conjunction with new technology. It is also of cru-

cial importance in gauging what animals want, most

obviously in the use of choice and preference tests,

but also through other methods that are particularly

suitable for on-farm welfare assessment.” (Dawkins,

2004).

Having simple and generic automatic tools for be-

havior analysis thus has the potential not only to sup-

port clinicians in their diagnostic decisions, but also

to impact welfare of companion, farm and zoo an-

imals. In his keynote talk at the Animal-Computer

Interaction Conference in 2007, Donald Broom, the

founding father of animal welfare science encouraged

informatics to offer technological solutions to press-

ing problems of welfare of animals, keeping which

for production, work and entertainment continues to

be a norm in modern society. We therefore suggest to

expand the notion of veterinary informatics, coining

the term ‘animal welfare informatics’. To give a con-

crete example, one extension of the Blyzer framework

we are currently working on involves analysis of the

sleep quality of sheltered animals and animals kept in

zoo environments to improve their physiological and

psychological wellbeing.

The development of such tools could also push the

boundaries of human health informatics. While anal-

ysis of video is one of the main research methods in

studying animal behavior (see, e.g., (Palestrini et al.,

2010; Cannas et al., 2014)), it has some parallels to

the use of video in human healthcare contexts (see,

e.g., (Stronach and Wetherby, 2014; Konofal et al.,

2001)), which may provide some hints towards the

possible generalizability of some results obtained in

veterinary informatics to human context.

Moreover, some behavioral disorders of animals

have strong connections to similar disorders in peo-

ple, and data analysis performed in animal health-

care informatics may inform and complement human

healthcare research. For instance, the domestic dog

has been suggested as a model to investigate human

behavioral disorders such as the Attention Deﬁcit Hy-

peractive Disorder (ADHD) (Hoppe et al., 2017; Lit

et al., 2010).

To summarize, state-of-the-art machine learning

techniques make it possible to easily develop auto-

matic tools for behavior analysis, which can greatly

impact animal health informatics. It is our hope that

this paper will also initiate a discussion on possible

cross-fertilization opportunities between animal and

human informatics with respect to automatic behav-

ior analysis.

REFERENCES

Ahrendt, P., Gregersen, T., and Karstoft, H. (2011). De-

velopment of a real-time computer vision system for

tracking loose-housed pigs. Computers and Electron-

ics in Agriculture, 76(2):169–174.

Alpi, K. M. (2009). Exploring the state of veterinary infor-

matics. In Positioning the profession: the Tenth Inter-

national Congress on Medical Librarianship, pages

1–15.

Broom, D. M. (1996). Attempts to cope with the environ-

ment. Acta Agric. Scand. Sec. A. Anim. Sci. Suppl,

27:22–28.

Brugarolas, R., Loftin, R. T., Yang, P., Roberts, D. L., Sher-

man, B., and Bozkurt, A. (2013). Behavior recogni-

tion based on machine learning algorithms for a wire-

less canine machine interface. In Body Sensor Net-

works (BSN), 2013 IEEE International Conference on,

pages 1–5. IEEE.

Burghardt, T. and

Cali

c, J. (2006). Analysing animal be-

haviour in wildlife videos using face detection and

tracking. IEE Proceedings-Vision, Image and Signal

Processing, 153(3):305–312.

Cannas, S., Frank, D., Minero, M., Aspesi, A., Benedetti,

R., and Palestrini, C. (2014). Video analysis of dogs

HEALTHINF 2019 - 12th International Conference on Health Informatics

440

suffering from anxiety when left home alone and

treated with clomipramine. Journal of Veterinary Be-

havior: Clinical Applications and Research, 9(2):50–

57.

Dawkins, M. (2004). Using behaviour to assess animal wel-

fare. Animal welfare, 13(1):3–7.

Fraser, D. (2009). Assessing animal welfare: different

philosophies, different scientiﬁc approaches. Zoo Bi-

ology: Published in afﬁliation with the American Zoo

and Aquarium Association, 28(6):507–518.

Gerencs

er, L., V

arhelyi, G., Nagy, M., Vicsek, T., and

Mikl

osi, A. (2013). Identiﬁcation of behaviour in

freely moving dogs (canis familiaris) using inertial

sensors. PloS one, 8(10):e77814.

Hoppe, N., Bininda-Emonds, O., and Gansloßer, U. (2017).

Correlates of attention deﬁcit hyperactivity disorder

(adhd)-like behavior in domestic dogs: First results

from a questionnaire-based study. Vet Med Open J,

2(3):95–131.

Hsu, Y. and Serpell, J. A. (2003). Development and vali-

dation of a questionnaire for measuring behavior and

temperament traits in pet dogs. Journal of the Amer-

ican Veterinary Medical Association, 223(9):1293–

1300.

Konofal, E., Lecendreux, M., Bouvard, M. P., and Mouren-

Simeoni, M.-C. (2001). High levels of nocturnal ac-

tivity in children with attention-deﬁcit hyperactivity

disorder: A video analysis. Psychiatry and Clinical

Neurosciences, 55(2):97–103.

Lit, L., Schweitzer, J. B., Iosif, A.-M., and Oberbauer,

A. M. (2010). Owner reports of attention, activity,

and impulsivity in dogs: a replication study. Behav-

ioral and Brain Functions, 6(1):1.

Mariti, C., Gazzano, A., Moore, J. L., Baragli, P., Chelli, L.,

and Sighieri, C. (2012). Perception of dogs’ stress by

their owners. Journal of Veterinary Behavior: Clinical

Applications and Research, 7(4):213–219.

Noldus, L. P., Spink, A. J., and Tegelenbosch, R. A. (2002).

Computerised video tracking, movement analysis and

behaviour recognition in insects. Computers and Elec-

tronics in agriculture, 35(2):201–227.

Overall, K. L. (2014). The ethogram project. Journal of

Veterinary Behavior: Clinical Applications and Re-

search, 9(1):1–5.

Overall, K. L. et al. (1997). Clinical behavioral medicine

for small animals. Mosby-Year Book, Inc.

Pageat, P. (1998). Pathologie du comportement du chien.

Ed. du Point v

erinaire.

Palestrini, C., Minero, M., Cannas, S., Rossi, E., and Frank,

D. (2010). Video analysis of dogs with separation-

related behaviors. Applied Animal Behaviour Science,

124(1):61–67.

Santamaria, S. L. and Zimmerman, K. L. (2011). Uses of

informatics to solve real world problems in veterinary

medicine. Journal of veterinary medical education,

38(2):103–109.

Scarlett, J. M., Salman, M., New, J. G., and Kass, P. H.

(2002). The role of veterinary practitioners in re-

ducing dog and cat relinquishments and euthanasias.

Journal of the American Veterinary Medical Associa-

tion, 220(3):306–311.

Sergeant, D., Boyle, R., and Forbes, M. (1998). Computer

visual tracking of poultry. Computers and Electronics

in Agriculture, 21(1):1–18.

Sherman, B. L. and Serpell, J. A. (2008). Training vet-

erinary students in animal behavior to preserve the

human–animal bond. Journal of veterinary medical

education, 35(4):496–502.

Smith-Akin, K. A., Bearden, C. F., Pittenger, S. T., and

Bernstam, E. V. (2007). Toward a veterinary infor-

matics research agenda: an analysis of the pubmed-

indexed literature. international journal of medical

informatics, 76(4):306–312.

Spink, A., Tegelenbosch, R., Buma, M., and Noldus, L.

(2001). The ethovision video tracking system—a tool

for behavioral phenotyping of transgenic mice. Phys-

iology & behavior, 73(5):731–744.

Stronach, S. and Wetherby, A. M. (2014). Examining re-

stricted and repetitive behaviors in young children

with autism spectrum disorder during two observa-

tional contexts. Autism, 18(2):127–136.

Sujatha, P., Kumarasamy, P., Preetha, S., and Balachandran,

P. (2018). Vetinformatics: A new paradigm for quality

veterinary services. Research & Reviews: Journal of

Veterinary Science and Technology, 5(2):16–19.

Talbot, R. (1991). Veterinary medical informatics. Jour-

nal of the American Veterinary Medical Association

(USA).

Tillett, R., Onyango, C., and Marchant, J. (1997). Us-

ing model-based image processing to track animal

movements. Computers and electronics in agricul-

ture, 17(2):249–261.

Van de Weerd, H., Bulthuis, R., Bergman, A., Schlingmann,

F., Tolboom, J., Van Loo, P., Remie, R., Baumans,

V., and Van Zutphen, L. (2001). Validation of a new

system for the automatic registration of behaviour in

mice and rats. Behavioural processes, 53(1):11–20.

Vas, J., Top

al, J., P

ech, E., and Mikl

osi, A. (2007). Measur-

ing attention deﬁcit and activity in dogs: a new ap-

plication and validation of a human adhd question-

naire. Applied Animal Behaviour Science, 103(1-

2):105–117.

Winslow, T., Payne, S. W., and Massoudi, K. A. (2018).

Functional analysis and treatment of problem behav-

ior in 3 animal shelter dogs. Journal of Veterinary

Behavior, 26:27–37.

Zamansky, A., Bleuer-Elsner, S., and van der Linden, D.

(2018). Automatic analysis of canine behavior is pos-

sible: a call for collaborative effort of the canine sci-

ence community. In Proceedings of the the 6th Canine

Science Forum.

Zamansky, A., van der Linden, D., and Baskin, S. (2017).

Pushing boundaries of re: Requirement elicitation for

non-human users. In Requirements Engineering Con-

ference (RE), 2017 IEEE 25th International, pages

406–411. IEEE.

Animal Health Informatics: Towards a Generic Framework for Automatic Behavior Analysis Position Paper

441