Smartphone-based Device for Checking Mental Status in Real Time

Mayumi Oyama-Higa

1, 2

, Wenbiao Wang

, Shigeo Kaizu

1, 2

, Terufumi Futaba

and Taira Suzuki

Chaos Technology Research Laboratory, Otsu, Japan

Kwansei Gakuin University, Nishinomiya, Japan

PricewaterhouseCoopers Aarata, Tokyo, Japan

Ryukoku University, Kyoto, Japan

J. F. Obern University, Tokyo, Japan

Keywords: Smartphone-based Device, Largest Lyapunov Exponent, Real-Time Mental Health Check-up for Highway

Drivers, Application to the Caring for Amyotrophic Lateral Sclerosis Sufferers.

Abstract: In this article, we present a smartphone-based device for checking mental status in real time, which for the

first time enables real-time check-up of mental status with a smartphone. With this device, by measuring

pulse waves, two important mental health indicators can be visualized at the same time: the largest

Lyapunov exponent obtained from non-linear analysis of pulse waves, and the autonomic nerve balance.

Before the development of this device, the measurement of these indicators had already been conducted in

thousands of experiments, and their relationship with individual’s mental status had been intensively studied

in recent years. This device enables users to conduct the measurement and capture the mental status

dynamically, without the limitation of place and time. It has the potential application in preventing accidents

due to failure of emotional management. The device is convenient to use and cost-effective.

1 INTRODUCTION

Problems related to mental disorder, or simply

mental changes, can lead to serious consequences. In

order to measure and analyse individual’s mental

status for early detection of potential mentally

related problems, we have performed a number of

studies in recent years, whose general description

can be found in Oyama’s 2012 book. It has been

discovered that chaotic fluctuations obtained from

the fingertip pulse waves contain information of the

central nervous system. In particular, the largest

Lyapunov exponent (LLE), which quantifies the

variation of the attractor trajectory, can serve as an

indicator of mental immunity. Besides, we have

developed the measuring device in order that a

measurement of fingertip pulse waves can be

conducted anywhere at any time.

For the results of our recent studies, specifically,

the method of measuring and analysing fingertip

pulse waves have been applied to various research

subjects concerning detection of mental changes

(Oyama-Higa et al., 2008; Wang et al., 2012) and

check-up for mental and cognitive disorders, such as

dementia (Oyama-Higa and Miao, 2006; Oyama-

Higa et al., 2008; Pham et al., 2015) and depression

(Oyama-Higa et al., 2008; Hu et al., 2011; Pham et

al., 2013).

This paper presents our improvement in the

measuring device: We have made the device more

convenient by connecting the pulse wave sensor to a

smartphone. Moreover, we calculate and display

values of autonomic nerve balance (ANB) at the

same time, which indicates whether sympathetic

nerve or parasympathetic nerve predominates. Thus,

we have made it possible to examine minute changes

in mental condition by graphic display of LLE and

ANB. Although this device still needs improving in

some aspects, it is so far the only existing device that

makes such mental check-up possible and

convenient with a smartphone.

2 CALCULATION METHOD

We introduce the two key indexes calculated and

displayed by our measuring device: LLE and ANB.

In addition, an index of vascular age will also be

shown.

Oyama-Higa, M., Wang, W., Kaizu, S., Futaba, T. and Suzuki, T.

Smartphone-based Device for Checking Mental Status in Real Time.

DOI: 10.5220/0005655401370142

In Proceedings of the 9th International Joint Conference on Biomedical Engineering Systems and Technologies (BIOSTEC 2016) - Volume 4: BIOSIGNALS, pages 137-142

ISBN: 978-989-758-170-0

137

2.1 Largest Lyapunov Exponent (LLE)

Time series data with deterministic chaos can be

constituted by fingertip plethysmograms (Tsuda,

Tahara and Iwanaga, 1992). Let

x (i), i = 1, 2, … (1)

denote the time series data. Using the method of

delays, the phase space is reconstructed with vectors

represented as

X(i) = ( x(i), x(i-τ), …, x(i-(d-1)τ) )

= { x

(i) }

k=1, …, d

(2)

where τ is a constant delay, d is the embedding

dimension and x

(i) is defined as

(i) = x(i-(k-1)τ), k = 1, …, d. (3)

In our study where the time series are recorded from

fingertip pulse waves, studies (Sano and Sawada,

1985; Sumida et al., 2000) have shown that the

optimal choices for the constant delay and the

embedding dimension are

τ = 50 ms (4)

and

d = 4. (5)

The largest Lyapunov exponent (LLE) is a measure

of complexity that reflects the divergence and

instability of the attractor trajectory. Let X(t) evolve

with time starting with an initial trajectory X(0).

Then, LLE can be calculated as

(6)

where the separation of the trajectories is

represented by

(7)

and the initial separation is represented by

(8)

in the phase space. In the measuring device in our

previous studies as mentioned above, the method

proposed by Rosenstein et al. in 1993 is applied for

estimating the LLE. For convenience, the value of

LLE is normalized to a range of 0-10 in the display

of our device.

Figure 1 shows the plethysmogram and attractor

obtained from the measurements, and LLE obtained.

Our previous studies have shown that the values of

LLE of a mentally healthy individual fluctuate

within a reasonable scope (from 2-7, centred at 5).

Figure 1: Plethysmogram (top), attractor (right) and LLE

(bottom).

When LLE is abnormally high, the mental

immunity of the individual is so strong that he or she

is likely to go to extremes: such individual can be

easily irritated and take unexpected actions. On the

other hand, when it is abnormally low, the mental

immunity is so weak that the individual is prone to

mental illnesses. In other words, a high LLE

indicates a mental status of adapting to the external

environment (we simply called it “external

adaptation” in some of our previous articles), while a

low LLE indicates a status of “internal focusing”.

2.2 Autonomic Nerve Balance (ANB)

Spectral analysis of heart rate variability can

evaluate the activity of the autonomic nervous

system. We consider the high frequency (HF, 0.15-

0.40 Hz) component which represents

parasympathetic nerve activity, and the low

frequency (LF, 0.04-0.15 Hz) component which is

an index of sympathetic nerve activity. In our study,

autonomic nerve balance (ANB) is defined as a

normalized index ranging from 0 to 10 as follows.

ANB = 10 B / 3.5, (9)

where

B = ln (LF) / ln (HF). (10)

From the ranges of LF and HF, we can clearly

observe that B ranges from 1, when both LF and HF

take the value 0.15, to approximately 3.5, when LF

is at the minimum while HF is at the maximum.

Therefore, ANB goes from approximately 2.86 to 10.

For convenience, in our current study, we still apply

a 0-10 valued graph to display the result of ANB.

ANB < 5 indicates predominance of parasympathetic

nerve while ANB > 5 indicates sympathetic

predominance.

The computation method has also been

embedded in the measuring device that has been

used so far in our studies, and it has been registered

as a U.S. patent (Higa, 2011).

BIOSIGNALS 2016 - 9th International Conference on Bio-inspired Systems and Signal Processing

138

2.3 Vascular Age

In addition to LLE and ANB, a normalized index of

vascular age will also be calculated and displayed.

This index is defined as

σ

1exp

2.8571  100



(11)

where N is the physical balance of the blood vessel,

computed using second derivative of

photoplethysmogram (SDPTG) (Sano et al., 1985).

N is comparable to the actual age: a high N indicates

unbalanced function of the vessel and vulnerability

to sclerosis, while a low N indicates plasticity of the

vessel. Note that σ is monotonically increasing with

respect to N, ranging approximately from 0 to 10 as

N goes from 0 to 100.

In this article concerning only LLE and ANB

will be discussed in details, so we omit further

explanation of the vascular age.

3 SYSTEM DESCRIPTION

3.1 General Description

The software that runs on Android phones is named

“Lady Alys”. It has two main parts: the engine part,

in charge of the computation process, and the driver

portion, for connecting the sensor. It can also be

connected to the Internet to save and deliver the

measured data. The main feature, as stated above, is

the real-time display of LLE and ANB during the

pulse wave measurement, which enables the users

(subjects) to check their mental status easily.

The sensor determines the haemoglobin levels in

the finger capillary vessels using infrared

spectroscopy (an 840 nm light-emitting diode is

equipped), and performs digital transformation that

converts 200 Hz to 12 bits.

Values of LLE calculated from the pulse wave,

for clarity, are displayed in 10 colours according to

the values, from red to blue in colour depth. The

closer the colour is to red, the higher the chaotic

fluctuation, and thus the external adaptability, is

reflected. On the contrary, a colour close to blue

represents a low fluctuation and in this case the user

is internally focused

Besides fingertips, pulse waves from earlobes

can also be used as inputs. Since the sensor is

portable, even long-time measurement will not cause

much inconvenience. Furthermore, equipped with

devices such as Bluetooth, it is also possible to take

the data wirelessly.

Figure 2: Smartphone and sensor.

The acquired data can be saved in the database

on the Internet. These data can be further analysed

with advanced analytical software “Dr. Lyspect”

(computer-based). “Dr. Lyspect” is an updated

version of “Lyspect” that we developed (described

in details in Oyama-Higa et al., 2012), which has

been used as a measuring device in our recent

studies cited in the Introduction section.

3.2 Main Screens

In this section, we introduce the main screens of the

smartphone-based device. Figures 3 and 4 show the

starting screen and the pulse wave display

respectively.

Figure 3: Starting screen.

Fluctuations in LLE due to emotional changes

are displayed in colour: the higher LLE, the closer to

red, while the lower the LLE, the closer to blue.

During the measurement, after the trial time period

of the first 17 seconds, values of LLE are recorded

and displayed in colour every second. It turns red

when the LLE is greater than 5, and darkens as the

value increases. On the other hand, when the LLE is

less than 5, the screen turns blue and grows lighter

Smartphone-based Device for Checking Mental Status in Real Time

139

as the value decreases.

Figure 4: Pulse wave screen.

Figure 5: Coloured display of LLE.

Figure 6: Time-series display of LLE (in red) and ANB (in

blue).

Values of LLE and ANB are also displayed in

time series. However, for ANB, the trial time period

at the beginning of the measurement requires 60

seconds.

Moreover, LLE and ANB are displayed in

Cartesian coordinates in real time, which visualizes

the correlation of the two indexes.

At the end of the measurement, three semi-

circular graphs are displayed, showing the results of

the average values of LLE, ANB and vascular age

respectively. As to Dr. Lyspect, these three graphs

will also be displayed, among other detailed results,

on the results window.

Figure 7: Display of the Cartesian coordinate plane (ANB:

X-axis, LLE: Y-axis).

Figure 8: Semi-circular graphs of Lady Alys (top; from

top to bottom: LLE, ANB and vascular age) and Dr.

Lyspect (bottom; from left to right: LLE, vascular age and

ANB).

BIOSIGNALS 2016 - 9th International Conference on Bio-inspired Systems and Signal Processing

140

4 APPLICATION EXAMPLES

There are various applications of the device. We

illustrate two typical ones.

4.1 Real-time Check-up of Mental

Status for Highway Drivers

One is the possible application for the enhancement

of driver safety.

We collected the real-time LLE data of a driver

who was driving on a highway (Figure 9).



Figure 9: Driving experiment on a highway.

We observe that his LLE suddenly increased at

several time points. These time points are actually

when the driver was driving past another car, when

he was passing a car that was stuck in the way due to

accident, and when he was checking a monitoring

camera. All these behaviours require carefulness and

concentration that make the driver tense up. In

Section 2.1, we have explained that a high value of

LLE indicates strong mental immunity, which is

inevitable in adapting oneself to the external

environment.

A too high LLE represents the status of excessive

stress that may give burden to the driver. On the

other hand, however, if his LLE keeps low under

such circumstances, the driver will be prone to

operation error.

Therefore, self-adjustment is necessary in both

extreme cases. “Lady Alys” can facilitate such self-

adjustment in that it makes the real-time mental

index visible to the driver who is driving. From this

viewpoint, this device can help enhance traffic

safety.

4.2 Caring for Amyotrophic Lateral

Sclerosis Sufferers

The other is the potential application in the caring

for the sick.

We conducted an experiment on the

communication between a male end-stage

Amyotrophic Lateral Sclerosis (ALS) sufferer and

his wife. The patient’s pulse waves were measured

when his wife was talking to him for 16 minutes.

The comprehensive result is obtained with “Dr.

Lyspect” (Figure 10) (Undoubtedly, “Lady Alys”

would be enough for the observation of time series

of LLE).

Figure 10: Conversation to end-stage ALS patients.

With end-age ALS, the patient was bedridden

without any ability to initiate and control any

voluntary movement, so he was not able to make

any physical reaction to wife’s talk.

However, from the graph of time series of LLE

(bottom of Figure 10), we observe significant mental

reaction of the patient. The LLE rose to the first

peak when the wife was talking about their daughter,

and then the second peak appeared when her topic

moved to their parents. Since a high value of LLE

indicates the mental status of external adaptation,

this result reveals that the patient was actually

listening to his wife attentively when she was talking

about his concern.

The device can not only enable us to observe the

emotional reaction of the patient, but also gratify the

carer, namely, the wife in this experiment who

realized that her husband did make response to her

efforts.

In addition to ALS, we also conducted similar

experiments with encephalopathy sufferers, and

obtained the same conclusion.

In such a way, when physical communication is

not possible, the simple measurement with “Lady

Alys” can be of help for both carers and carees.

5 CONCLUSIONS

With this device, measurement with Android tablet

or mobile phone can be easily conducted,

Smartphone-based Device for Checking Mental Status in Real Time

141

irrespective of the limitation of place and time. The

“Lady Alys” can help the users observe their real-

time changes in mental status. The display with

intuitive colours makes the values of LLE more

visible. By conducting the measurement during

various daily activities, the users are able to perform

real-time mental check-up and then self-control

according to his or her mental status.

The measurement is also possible to involve two

or more subjects, in order to observe the reactions in

their relationships, especially the carer-caree

interaction.

In addition, admittedly, there are several

drawbacks with the system. For example, instead of

showing changes in the screen colour, an acoustic

alarm would be more applicable, as it can make the

driver much easier to recognize the alarm without

sporadic attention to the smartphone screen. Such

improvement is under consideration and will serve

as a subject of our future study. Further development

of wireless devices and the improvement of sensors

are also in progress.

ACKNOWLEDGEMENTS

We would like to express our sincere thanks to Mrs.

Mitsuko Tanabiki, Director, and the nursing staff

members of Himeji Himawari Nursery School, for

her cooperation in our experiments.

REFERENCES

Higa, M. (2011) Autonomic nervous balance computation

apparatus and method therefor, U.S. Patent US

2011/0313303 A1.

Hu, Y., Wang, W., Suzuki, T. and Oyama-Higa, M. (2011)

‘Characteristic extraction of mental disease patients by

nonlinear analysis of plethysmograms’, AIP Conf.

Proc., vol. 1371, pp. 92-101.

Oyama-Higa, M. and Miao, T. (2006) ‘Discovery and

application of new index for cognitive psychology’,

2006 IEEE Conf. on Systems, Man, and Cybernetics

Proc., vol. 4, pp. 2040–2044.

Oyama-Higa, M., Miao, T., Kaizu, S. and Kojima, J.

(2012) ‘Mental health self-check system using

“Lyspect” ’, Proc. of the Sixth International

Symposium on e-Health Services and Technologies,

Sixth International Symposium on e-Health Services

and Technologies (EHST 2012), Geneva, pp.9-18.

Oyama-Higa, M., Miao, T., Tsujino, J. and Imanishi, A.

(2008) ‘Possibility of mental health self-checks using

divergence of pulse waves’, Proc. of the First

International Conference on Biomedical Electronics

and Devices, BIOSIGNALS 2008, Funchal, pp. 361-

370.

Oyama, M. (2012) Psychology of mental flexibility

(English edition, Kindle), Seattle: Amazon Services

International, Inc.

Pham, T.D., Oyama-Higa, M., Truong, C.T., Okamoto, K.,

Futaba, T., Kanemoto, S., Sugiyama, M. and Lampe, L.

(2015) ‘Computerized assessment of communication

for cognitive stimulation for people with cognitive

decline using spectral-distortion measures and

phylogenetic inference’, PLos One, vol. 10(3),

e0118739.

Pham, T.D., Thang, T.C., Oyama-Higa, M., Nguyen, H.X.,

Saji, H. and Sugiyama, M. (2013) ‘Chaos and

nonlinear time-series analysis of finger pulse waves

for depression detection’, Proc. of the International

Conference on Bio-inspired Systems and Signal

Processing, BIOSIGNALS 2013, Barcelona, pp. 298-

301.

Rosenstein, M.T., Collinsa, J.J. and De Luca, C.J. (1993)

‘A practical method for calculating largest Lyapunov

exponents from small data sets’, Physica D: Nonlinear

Phenomena, vol. 65, pp. 117-134.

Sano, M. and Sawada, Y. (1985) ‘Measurement of the

Lyapunov spectrum from a chaotic time series’, Phys.

Rev. Lett., vol. 55, pp. 1082.

Sano, Y., Kataoka, Y., Ikuyama, T., Wada, M., Imano, H.,

Kawamura, K., Watanabe, T., Nishida, A. and Osanai,

H. (1985) ‘Evaluation of peripheral blood circulation

to predict the possible impairment of circulatory

organs by quadratic differential plethysmography and

its application (in Japanese)’, Science of Labour, vol.

61, no. 3, pp. 129-143.

Sumida, T., Arimitu, Y., Tahara, T. and Iwanaga H.

(2000) ‘Mental conditions reflected by the chaos of

pulsation in capillary vessels’, International Journal of

Bifurcation and Chaos, vol. 10, pp. 2245-2255.

Tsuda, I., Tahara T. and Iwanaga. I. (1992) ‘Chaotic

pulsation in capillary vessels and its dependence on

mental and physical conditions’, International Journal

of Bifurcation and Chaos, vol. 2, pp. 313-324.

Wang, W., Hu, Y., Oyama-Higa, M., Suzuki, T., Miao, T.

and Kojima, J. (2012) ‘Analysis of

electroencephalogram and pulse waves during music

listening’, Proc. of the Sixth International Symposium

on e-Health Services and Technologies, Sixth

International Symposium on e-Health Services and

Technologies (EHST 2012), Geneva, pp. 31-35.

BIOSIGNALS 2016 - 9th International Conference on Bio-inspired Systems and Signal Processing

142