BREATH AND POSITION MONITORING

DURING SLEEPING WITH A DEPTH CAMERA

Meng-Chieh Yu

, Huan Wu

, Jia-Ling Liou

, Ming-Sui Lee

1,2

and Yi-Ping Hung

1,2

Graduate Institute of Networking and Multimedia, National Taiwan University, Roosevelt Road, Taipei, Taiwan

Department of Computer Science and Information Engineering, National Taiwan University,

Roosevelt Road, Taipei, Taiwan

Keywords: Non-contact breath measurement, Sleep monitoring, Sleep position, Sleep cycle, Depth camera.

Abstract: Sleep monitoring is increasingly seen as a common and important issue. In this paper, a depth analysis

technique was developed to monitor user’s sleep conditions without any physical contact. In this research, a

cross-section method was proposed to detect user’s head and torso from the depth images. Then, the system

can monitor user’s breathing rate, sleep position, and sleep cycle. In order to evaluate the measurement

accuracy of this system, two experiments were conducted. In the first experiment, eight participants with

various body shapes were asked to join the experiment. They were asked to change the sleep positions

(supine and side-lying) every fifteen breathing cycles in two circumstances (sleep with and without a thin

quilt) on the bed. The experimental results showed that the system is promising to detect the head and torso

with various sleeping postures. In the second experiment, a realistic over-night sleep monitoring experiment

was conducted. The experimental results demonstrated that this system is promising to monitor the sleep

conditions in realistic sleep conditions. To conclude, this study is important for providing a non-contact

technology to detect multiple sleep conditions and assist users in better understanding of their sleep quality.

1 INTRODUCTION

Sleep is essential for a person’s mental and physical

health. Studies indicate that sleep plays a critical role

in immune function (Born et al., 1997), metabolism

and endocrine function (Spiegel et al., 1999),

memory, learning (Maquet, 2001), and other vital

functions. However, there are some sleep disorders,

such as sleep apnea, insomnia, hypersomnia,

circadian rhythm disorders, which might interfere

with physical, mental and emotional functioning.

For better understanding of sleep problems, many

sleep centres and research groups are devoted to the

sleep study. Polysomnography (PSG) is a multi-

parametric test used in the study of sleep and as a

diagnostic tool in sleep medicine. It monitors body

functions including brain activity (EEG), eye

movement, muscle activity, heart rhythm, and

breathing while sleeping (Douglas, et al., 1992). In

this study, we focus on the research issues in sleep

cycle, sleep breathing, and sleep positions. For the

measurement of sleep cycle, EEG monitoring is one

of the most accurate methods to detect the period of

non-rapid eye movement (NREM) and rapid eye

movement (REM). However, it is not convenient to

use. In recent years, motion sensor and pressure

sensor array are widely used to monitor user’s sleep

conditions and body movement while sleeping

(Actiwatch, 1998; Fitbit, 2010; WakeMate, 2010), as

well as estimate the sleep cycle and evaluate the

sleep quality. For breath measurement while

sleeping, sleep apnea is one of the most important

sleep disorder characterized by abnormal pauses in

breathing or instances of abnormally low breathing

during sleep. For decades, the breath measurement

methods would direct contact to the user while

monitoring, and it might interfere with the user and

reduce the sleep quality. Although some non-contact

breath measurement methods were proposed in

recent years, such as ultra-wideband (UWB) and

structured light plethysmography (SLP), there still

have some measurement limitations. For sleep

position, in order to prevent sleep apnea, studies

showed that side-lying position is the best sleep

posture for individuals with sleep apnea (Cartwright

et al., 1984; Szollosi et al., 2002; Loord et al., 2007;

Yu M., Wu H., Liou J., Lee M. and Hung Y..

BREATH AND POSITION MONITORING DURING SLEEPING WITH A DEPTH CAMERA.

DOI: 10.5220/0003702000120022

In Proceedings of the International Conference on Health Informatics (HEALTHINF-2012), pages 12-22

ISBN: 978-989-8425-88-1

 2012 SCITEPRESS (Science and Technology Publications, Lda.)

Hoque et al., 2010). A study analysed six common

sleep positions, and concluded that supine positions

were more likely to lead to snoring and a bad night's

sleep (Idzikowski et al., 2003). However, to date,

there has been relatively little research conducted on

the measurement of sleep positions.

In this study, a sleep monitoring system using a

depth camera was proposed to monitor users’

breathing rate, body movement, and sleep position in

bed. Moreover, we evaluated the measurement

accuracy of the system, including the accuracy of

head and torso detection, breath measurement

(compared to RIP), and sleep movement (compared

to Actigraphy). Through the experimental results,

we confirmed that the system could accurately

monitor user’s sleep conditions. This paper is

structured as follows: The first section deals with the

introduction of present sleep studies. The second

section of the article is a review of several breath

measurement methods and activity monitoring while

sleeping. The proposed system design is described in

the third section. The experimental results are

demonstrated in section four followed by the

discussion on some important findings. Finally,

conclusions and suggestions are given for further

research.

2 RELATED WORKS

In this section, we discuss relevant literatures of

breath measurement and sleep cycle monitoring

while sleeping.

2.1 Breath Measurement

while Sleeping

Breathing is important while sleeping. There are

many breathing-related sleep disorders, such as

apnea and hyperventilation syndrome (HVS).

Currently, many methods are proposed to monitor

the breath conditions while sleeping. Most screening

tools consist of an airflow measuring device, a blood

oxygen monitoring device, and the respiratory

inductance plethysmography (RIP). Thermistor (TH)

measurements have been traditionally used to

determine airflow during PSG studies. It is placed

over the nose and mouth and infers airflow by

sensing differences in the temperature of the warmer

expired air and the cooler inhaled ambient air.

However, low accuracy in detecting hypopneas is a

major drawback (BaHamman, 2004). The pulse

oximeter is a medical device that monitors the

oxygen saturation of user’s blood, and changes in

blood volume in the skin. Low oxygen levels in the

blood often occur with sleep apnea and other

respiratory problems (Douglas, et al., 1992).

Respiratory Inductance Plethysmography (RIP)

measures the body movement of chest wall or

abdominal wall caused by breathing exercise (Whyte

et al., 1991; Cantineau et al., 1992), and then the

breathing conditions can be estimated accurately.

However, most of the breath measurement methods

are essential to directly contact to the user while

measuring, and it might affect the user and decrease

the sleep quality.

In recent years, some non-contact breath

measurement methods are developed. A study used

a CCD video camera to detect the optical flow of the

user in bed (Nakajima et al., 2001). PneumaCare

developed a non-invasive method called Structured

Light Plethysmography (SLP), which utilizes the

distortion with movement of a structured pattern of

light to calculate a volume or change in volume of a

textured surface. Another study conducted an

experiment and the results showed that SLP was

comparable in performance to spirometer (Wareham

et al., 2009). Moreover, slit lights projection (Aoki

et al., 2006) is another non-invasive method which

measures the breathing conditions by projecting the

near-infrared multiple slit-light patterns on the user

and measuring the breathing status.

In addition to computer vision-based methods,

there is a non-contact method which uses ultra

wideband (UWB) to measure the breathing status. A

study proposed an application of UWB radar-based

heart and breathing activities for intensive care units

and conventional hospital beds (Staderini, 2002).

Another study used UWB to measure baby’s

breathing and heart rate especially in terms of

opportune apnea detection and sudden infant death

syndrome prevention (Ziganshin et al., 2010).

2.2 Sleep Cycle

For monitoring the sleep activity through movement,

actigraphy has been used to study the sleep patterns

for over 20 years. Actigraphy is a non-invasive

method of monitoring human activity cycles (Sadeh,

et al., 1994). It is useful for determining sleep

patterns and circadian rhythms. The advantage of

actigraphy over traditional PSG is that actigraphy

can conveniently record the sleep activity (Ancoli-

Israel et al., 2003). In recent years, many

commercial products were developed, such as Fitbit,

WakeMate, and Actiwatch. In general, these

products detect the information of time to fall asleep,

time to wake up, and totally sleeping time. A study

BREATH AND POSITION MONITORING DURING SLEEPING WITH A DEPTH CAMERA

evaluated the measurement results of actigraphy and

compared to PSG, and the experimental results

showed that sleep parameters from actigraphy

corresponded reasonably well to PSG (Kushida et al.,

2001). In addition, there is a non-contact method

which uses a microphone and an infrared sensor to

monitor the sleep status. Moreover, some studies

utilize motion sensors (accelerometer, piezoelectric

sensor) inside the pillow (Harada et al., 2000) or bed

(Malakuti et al., 2010; Hoque et al., 2010) to

monitor the sleep movement and sleep positions.

However, none of related research in our survey

has a complete study to provide a non-contact and

multi-functioning sleep monitoring technique to

monitor the sleep conditions. In this study, we

developed a non-contact sleep monitoring system

which can monitor user’s sleep position, breathing

condition, and body movement in the same time.

3 SYSTEM DESIGN

In this study, a cross-section object detection method

is proposed to detect user’s head and torso using a

depth camera. The sleep position, body movement,

and breathing condition are monitored once the head

and torso is detected. The procedure of this method

is as follows: First, the view transformation is

estimated. Then, a median filter is adopted to reduce

the image noise after view transformation. Next, a

cross-section method is used to detect user’s head

and torso so that the sleep position and body

movement can be measured. Besides, a breath

Figure 1: System Framework.

measurement method is proposed to detect the

breathing conditions through the movement of the

torso. The system framework of this system is

shown in Figure 1.

3.1 System Environment

In this system, a depth camera (Microsoft, 2011) is

used to capture the sequence of depth images of the

user on the bed. The depth camera consists of an

infrared laser projector combined with a CMOS

sensor, which captures color images and a depth

images under ambient light condition. In addition,

the depth image can also be captured under the no-

light condition. For the reason of easy setup and

preventing the interference with the sight view of the

user while sleeping, in this study, the depth camera

is placed on the wall behind the head instead of

suspending from the ceiling. Besides, in order to

ensure that the user’s head and torso can be captured,

and for the issues of breath measurement distance

(the shorter the better), the limitation of sensing

distance (larger than 0.8m), the depth camera is

placed in the distance of 125 cm (49.2 inches) from

the bed. The diagram for the system is shown in

Figure 2. The region between gray dotted lines

indicates the sight view of the depth camera, and the

region between yellow lines indicates the sight view

of the user.

Figure 2: System Diagram.

3.2 Depth Image Processing

Although the skeleton of the user body can be

extracted easily through Microsoft Kinect SDK, the

skeleton of body while lying on the bed cannot be

extracted easily. It is because that the background is

too close to the user, and the body might be covered

by a quilt. In this study, a cross-section method is

proposed to detect user’s head and torso with a depth

camera. We process the depth image signals at the

resolution of 320 pixels in width and 240 pixels in

height, and the frame rate is 30 frames per second.

HEALTHINF 2012 - International Conference on Health Informatics

View Transformation

In order to determine the cross-sections of the depth

image, we would like to transform the camera view

from the side view to the top view. To do that, we

need to calculate bed’s normal vector first. In order

to rotate the camera view to the top of the bed, three

points and one rotate center point need to be

specified manually. After taking three 3D-points on

the bed and using cross product, the system could

get the normal vector of the bed. Then, these 2D-

points could project to 3D-points in the real world.

Then, we proceed to calculate rotation matrix for the

bed’s normal vector. Once we have the rotation

matrix, we can project all 2D points back to 3D

point-cloud. Again, we project it back to 2D depth

image. However, it will lose some information after

rotating the camera view, so a median filter is used

to fill empty holes. Figure 4b shows the original

depth image, and Figure 4c shows the depth image

after view transformation.

Cross-Section Method

We generate several binary images by setting

different thresholds starts from the shallowest point

of the depth image to the depth of the bed. We

generate cross-sections every 2 cm (0.787 inches)

from top to bottom. Generally, the distance between

the highest point of the human body and the bed is

around 18~28 cm, therefore, there would be 9~13

transverse sections of the person from top to bottom.

Figure 3 shows ten cross-sections (red line) from the

highest point of the red point to the bed.

Figure 3: Cross-sections of the lying user from top to

bottom. Red point indicates the highest point of the user.

Head and Torso Detection

By using connect-component analysis, the

components from each cross-section can be

extracted. The concept of this method is to find out

spheres in each cross-section. Once there is a circle

growing larger from top section to bottom section,

we assume that it might be a sphere there. So far,

this algorithm might find other spheres. To decide

the highest sphere, we collect each circle’s

contribution from each section. More circles at the

same location means higher probability to have

sphere there. If sphere candidates have n different

locations, the probability that might be a sphere at

location l is:

() =

∑

#



∑∑

#





(1)

In addition to detecting head from single depth

image, we need to leverage the advantage of video

sequence. Hence, we push every head location found

by each frame into a queue. Then, we use the same

idea to re-locate the highest probability head-like

sphere. This will avoid some occasional misleading

failed detection.

Figure 4: Depth image Processing Procedure of our system.

(a) Captured color image. (b) Captured depth image. (c)

View transformed image. (d) Filtered image. (e) Cross-

section image. (f) Final result of head/torso detection.

Once the head is detected, the next step is to

detect the torso’s ROI (region-of-interest). We adopt

almost the same way as detecting the head, but this

time we track cuboids rather than spheres. However,

there is a problem that the pillow might be

recognized as a torso. Therefore, we reject cuboids if

there is a head on it. Figure 4 shows the processing

BREATH AND POSITION MONITORING DURING SLEEPING WITH A DEPTH CAMERA

procedure of head and torso detection in this system.

Head and Torso Detection Algorithm

Inputs:

C := Set of circles from each sections

Cu := Set of cuboid from each sections

Th := threshold distance to determine two different cluster

Outputs:

Head and torso positions

Steps:

1.Classify_components(C,Th)

{

//Classify C into clusters according to distance Th.

1.1 clusters_num = SeqPartition(C,Th)

1.2 voting[cluster_num] //# of member in each

component

1.3 leader[cluster_num] //the biggest sphere size in

individual cluster

1.4 if( clusters_num > 0)

{

1.4.1 Loop for each C

element in C

1.4.1.1 num = cluster_number(i)

1.4.1.2 if(voting[num] == 0 OR leader[num]’s

size < Ci’s size )

leader[num] = C

End if

1.4.1.3 voting[num] = voting[num] + 1

1.4.2 End loop

1.5 }

1.6 Sort voting and leader array

1.7 Return array and # of cluster

}

2. Q

head

:= a queue that collects head’s position and

location in the video sequence.

Find_head(C)

{

2.1 Th = head_boundary.

2.2 Head = Classify_components(C,Th).

2.3 Push Head into queue Q

head

2.4 Final_Head = Classify_components(Q

head

,Th).

}

3. Q

torso

:= a queue that collects torso’s position and

location in the video sequence.

Find_torso(Cu,Head)

{

3.1 Th = torso_bounday

3.2 Remove cuboid from Cu if it intersects with Head.

3.3 Torso = Classify_ components(Cu,Th).

3.4 Push Torso into queue Q

torso

3.5 Remove cuboid from Q if it intersects with Head.

3.6 Final_torso = Classify_ components(Q

torso

,Th).

}

4. Detect_body()

{

4.1 Collect C and Cu from each sections

4.2 if (Head = Find_head(C))

4.2.1 Torso = Find_torso(Cu)

4.3 End if

}

Breath Measurement

The breathing signal can be extracted from the torso

ROI once we detect the head and torso. While the

user is inhaling, his chest wall will expand, and the

average depth value of the torso ROI will decrease;

on the contrary, the average depth value of the torso

ROI will increases while the user is exhaling.

Therefore, the sequential of the average depth value

of the torso ROI is considered as the breathing signal

under the premise that the user is sleeping. For

breath measurement, a turning point detection

algorithm is proposed. At first, a mean filter is used

for reducing the noises caused by the sensing

deviation and body movements. Then, the turning

points of the breathing signal are detected using the

second derivative method. Finally, in order to

eliminate redundant turning points, a dynamic

threshold is applied to find the exact peak points and

valley points. Figure 5 shows a fragment of the

breathing signal (blue line) and the measurement

results (vertical gray line) during a realistic

overnight sleep.

Figure 5: Breath Measurement. The blue line indicates the

raw breathing signals, and the gray lines indicate the

turning points which we detected.

While the turning points of the breathing signals

are detected, the information of the breathing

conditions can be figured out easily. The breathing

conditions include the breathing rate, breathing

depth, breathing stability, inhalation time, exhalation

time, inhalation/exhalation ratio, and sleep apnea

symptoms.

1193

1193,5

1194

1194,5

1195

1195,5

1196

012345678910111213

Average Distance

(cm)

Time (sec)

HEALTHINF 2012 - International Conference on Health Informatics

Body Movement

The body movement is defined as the sum of head

movement and torso movement. The HR

t indicates

the average depth value of the head ROI in time t,

and the TR

t indicates the average depth value of the

head ROI in time t. Then, the absolute difference

value between two adjacent images frames could be

calculated (Equation 2). M

indicates the movement

value of the user in time t.





−HR





−TR





(2)

Sleep Position

In this system, two main sleep positions (supine

position and side-lying position) can be recognized.

After the head and torso are detected, the highest

point of head ROI and torso ROI can be found. Then,

the ratio of the highest head point to torso point is

calculated. Figure 6 shows the highest point of head

ROI (blue dot) and torso ROI (red dot) in the side-

lying position and supine position.

Figure 6: Sleep Positions. Red dot indicates the highest

point of torso ROI, and blue dot indicates the highest point

of head ROI.

Next, the sleep position can be classified

according to the ratio defined in equation 3. In order

to find out the threshold to classify the sleep position,

an experiment was conducted to record five

participant’s (two females and three males) highest

points of head ROI and torso ROI in two sleep

positions (side-lying and supine) and two conditions

(sleep with no quilt and sleep with a thin quilt)

(Figure 7). The results revealed that the distance of

the highest head point does not change significantly

in different sleep positions and conditions. However,

the distance of the highest torso point changed

significantly in different sleep positions. The

average ratio is -0.02633 in supine position, and it is

0.0652 in side-lying position. Therefore, the

detection threshold of sleep position is set to the

median value: 0.01. While the ratio is larger than the

threshold value, the sleep position is defined as the

supine position. Otherwise the sleep position is

defined as the side-lying position (equation 4). From

Figure 7, we can observe that the standard deviation

of the body distance is bigger in side-lying position

than others. It is because the highest torso points are

different for female and male. However, our method

can also distinguish the sleep position accurately no

matter no matter the gender.

 =













(3)



 = 



 > 0.01

ℎ

(4)

Figure 7: The distance between the depth camera and the

highest point of head ROI and torso ROI in two positions

(supine and side-lying) and two conditions (sleep with no

quilt and sleep with a thin quilt).

3.3 Measurement Limitations

There are some measurement limitations in this

system. First, according to the law of rectilinear

propagation of light, the depth value cannot be

detected while the IR patterns are blocked by objects.

From the experiment results, we found that the most

common problem is that the hand would block some

of the depth IR patterns while side-lying. It might

affect the accuracy of torso detection. Second, the

breathing amplitude of torso movement would be

decreased with the increase of the thickness of quilt.

According to our test, the average breathing

amplitude of the torso movement with no quilt is 0.5

cm and it is 0.35 cm while sleeping with a thin quilt.

The thickness of the thin quilt in our test is 0.6 cm.

However, while sleeping with a thick quilt, such as

thick silk-padding quilts, the system might not

accurately detect the torso movement caused by

breathing exercise.

4 EXPERIMENTS

Two experiments were conducted to evaluate the

measurement accuracy of head/torso detection, sleep

1234

1224

1235

1237

1270

1153

1263

1147

1000

1050

1100

1150

1200

1250

1300

Supine Side Lying Supine Side Lying

Sleep without quilt Sleep with a thin quilt

Distance (cm)

Head Body

BREATH AND POSITION MONITORING DURING SLEEPING WITH A DEPTH CAMERA

position, body movement, and breath measurement

of this system. First experiment was mainly

designed to evaluate the measurement reliability for

different users. Second experiment was designed to

evaluate the measurement accuracy in realistic

overnight-sleep condition.

4.1 Experiment I

Experimental Design

Eight participants volunteered to participate in this

experiment (five males and three females). The

average age is 33.8 years old (SD = 17.6), including

two sixty-year old participants, five young

participants (25~30 years old), and a ten-year old

participant. The body mass index (BMI) of them is

in the range between 18.6~29.75. In this experiment,

participants were asked to lie down on the pillow,

and a breathing sensor, RIP (Thought Technology

Ltd., 2010), was used to record the breathing

conditions as the ground truth. During the

experimental procedure, they were asked to change

the sleep position every fifteen breathing cycles. The

procedure of this experiment is in the sequence of

supine, lying on the right side, supine, lying on the

left side, supine, and lying on the right side. Totally,

the participant needed to change the sleep position

five times. Besides, the experimental procedure

needed to be done twice, including a condition that

the participants sleep with a thin quilt, and a

condition that they sleep with no quilt. Before each

task, participants were reminded not to breathe

deliberately.

Experimental Results

The sleep measurement were divided into four

different conditions in this experiment, including

two sleep positions (side-lying and supine) and two

circumstances (sleep with a thin quilt and sleep with

no quilt). For each condition, the total numbers of

correct head detection frames were calculated

manually as well as the total numbers of correct

torso detection frames. The average of accurate rate

and standard deviation in each condition are listed

below. The experimental results showed that while

participants slept with no quilt, the measurement

accuracy of head detection was 98% (SD = 0.036)

while in the side-lying position, and it was 99.3%

(SD = 0.018) in the supine position. Moreover, the

measurement accuracy of torso detection was 91.5%

(SD = 0.16) in the side-lying position, and it was

99.3% (SD = 0.01) in the supine position. Besides,

while participants slept with a thin quilt, the

measurement accuracy of head detection is 96.7%

(SD = 0.11) in the side-lying position, and it was

99.5% (SD = 0.02) in the supine position. Moreover,

the measurement accuracy of torso detection was

94.5% (SD = 0.1) in the side-lying position, and it

was 99.5% (SD = 0.008) in the supine position.

Overall, the average accurate rate was 98.4% in head

detection and 96.4% in torso detection. The

experimental results of head and torso detection are

shown in Figure 8.

Figure 8: Measurement results of head and torso detection

in experiment I.

For breath measurement, the measurement

accuracy is defined as the ratio of the totally

breathing cycles we detected to the totally breathing

cycles the RIP system detected. The measurement

results of the RIP system was regarded as the ground

truth of the breathing conditions. The experimental

results show that while the user sleeps with no quilt,

the measurement accuracy of breathing rate was

81.9% (SD = 0.11) in the side-lying position, and it

was 90.4% (SD = 0.07) in the supine position.

Moreover, while the user sleeps with a thin quilt, the

measurement accuracy of breathing rate was 84.1%

(SD = 0.05) in the side-lying position, and it was

88% (SD = 0.08) in the supine position. Overall, the

Figure 9: Measurement results of breath measurement in

experiment I.

98.0%

99.3%

96.7%

99.5%

91.5%

99.3%

94.5%

99.5%

20%

40%

60%

80%

100%

Side lying Supine Side lying Supine

Sleep without quilt Sleep with a thin quilt

Head Detection Torso Detection

81.9%

90.4%

84.1%

88.0%

20%

40%

60%

80%

100%

Side lying Supine Side lying Supine

Sleep without quil

Sleep with a thin quil

HEALTHINF 2012 - International Conference on Health Informatics

average accurate rate of breath measurement was

86.3%. The experimental results of the breath

measurement are shown in Figure 9.

For sleep position, the experimental results

showed that in the circumstance of sleeping with a

thin quilt, the detection accuracy was 100% (N=24)

in the side-lying position, and it was 100% (N=24)

in the supine position. Besides, while the user slept

with no quilt, the detection accuracy was 95.8%

(N=24) in the side-lying position, and it was 100%

(N=24) in the supine position.

4.2 Experiment I I: Realistic

Overnight-Sleep Monitoring

Experimental Design

The experiment was conducted to ensure that the

system could monitor the realistic overnight-sleep

conditions accurately. A male participant (28 years-

old) volunteered to participate in this experiment.

The same as the first experiment, the breathing

sensor (RIP) was used to measure the breathing

conditions as the ground truth. In addition, an

actigraphy was used to measure the movement of the

non-dominant hand while sleeping (Figure 10).

There was only one limitation that the participant

was asked to lie on the pillow. In this experiment,

the participant was asked to participate in a ten-day

overnight-sleep monitoring experiment. The

experiment did not specify the time to go to the bed,

the time to getting up, and the totally sleeping time.

Besides, we required participants to sleep with a thin

quilt for five days, and to sleep with no quilt for

another five days. Participant’s breathing rate, body

movement, and sleep position were monitored by

our method and compared to the RIP and actigraphy.

Figure 11 shows one of a realistic overnight-sleep

monitoring results in day 3. Figure 11a shows the

measurement results of breathing rate. Red curve

indicates the measurement results of RIP system,

and the blue curve indicates the measurement results

of our system. Lower part of Figure 11a shows the

Figure 10: Experimental Diagram. The breathing sensor

(RIP) and actigraphy are used to detect the breathing rate

and body movement.

sleep positions we detected (blue) and real condition

(red). Figure 11b shows the movement level

detected by an actigraphy, and Figure 11c shows the

movement level detected by our system. In this day,

the participant slept with a thin quilt from 1:30 AM

to 4:53 AM.

Experimental Results

The same with experiment I, the sleep measurement

were divided into four different conditions,

including two sleep positions (side-lying and supine)

and two sleep circumstances (sleep with a thin quilt

and sleep with no quilt). Totally, the participant slept

42 hours in ten nights.

Following shows the experimental results. In the

circumstance of sleeping with no quilt, the

measurement accuracy of head detection was 89.4%

(SD = 0.14) in the side-lying position and it was

99.9% (SD = 0.0007) in the supine position.

Moreover, the measurement accuracy of torso

detection was 89.3% (SD = 0.014) in the side-lying

position and it was 89.3% (SD = 0.0003) in the

supine position. Besides, in the circumstance of

sleeping with a thin quilt, the measurement accuracy

of head detection was 99.9% (SD = 0.007) in the

side-lying position and it was 98.8% (SD = 0.17) in

the supine position. Moreover, the measurement

accuracy of torso detection is 99.4% (SD = 0.0003)

in the side-lying position and it is 99.9% (SD =

0.003) in the supine position. The experimental

results of head and torso detection are shown in

Figure 12. Overall, the average accurate rate of head

detection was 96.7% (SD = 0.073), and the average

accurate rate of torso detection was 96.8% (SD =

0.031).

For body movement, the times of the movement

events in our method and actigraphy were compared.

According to the observation, we observed that big

body movement can be measured both in our system

and actigraphy, such as the event of turning over the

body. Besides, micro-movement could be measured,

(see Figure 11b and 11c).

For breath measurement, the measurement

accuracy of breathing rate was 89.7% (SD = 0.05) in

the side-lying position and it was 92.8% (SD = 0.05)

in the supine position in the circumstance of

sleeping with no quilt Moreover, the measurement

accuracy of breathing rate was 92.4% (SD = 0.07) in

the side-lying position, and it was 92.7% (SD = 0.07)

in the supine position. Overall, the average accurate

rate of breath measurement was 92.03% (SD =

0.044). The comparison of the breath measurement

in these different conditions is shown in Figure 13.

For sleep position, the experimental results

showed that in the circumstance of sleeping with a

BREATH AND POSITION MONITORING DURING SLEEPING WITH A DEPTH CAMERA

Figure 11: A Realistic Overnight-Sleep Monitoring. Red color indicates the ground truth measured by RIP and actigraphy,

and the blue color indicates the results of our system. (a) The results of breathing rate and sleep positions. (b) The

movement level detected by an actigraphy. (c) The movement level detected by our system.

Figure 12: Measurement results of head and torso

detection in experiment II.

Figure 13: Measurement results of breath measurement in

experiment II.

thin quilt, the detection accuracy was 94.7% (N=19)

in the side-lying position, and it was 100% (N=21)

in the supine position. Besides, while the user slept

with no quilt, the detection accuracy was 88.23%

(N=17) in the side-lying position, and it was 100%

(N=20) in the supine position.

5 DISCUSSIONS

The aim of this section is to summarize, analyse and

discuss the results of experiments and give

guidelines for the future developments.

5.1 Head/Torso Detection

From the experimental results of head and torso

detection, we observed some issues worthy of

discussion. First, while the user slept with a thin

quilt, the overall detection accuracy of torso was

better than uncovered. One reason might be that the

thin quilt could smooth the shape of the torso, and

enhance the measurement accuracy. Second, we

found that the gesture might affect the head

detection. In this system, the head and torso could be

detected accurately under the premise that the shape

of the head or torso is not overlapped by hand or

other objects. Figure 14a and 15b show two special

sleep gestures that can be detected accurately. It is

89.4%

99.9%

98.8%

89.3%

99.4%

99.9%

20%

40%

60%

80%

100%

Side lying Supine Side lying Supine

Sleep without quilt Sleep with a thin quilt

Head Detection Torso Detection

89.7%

92.8%

91.9%

93.4%

20%

40%

60%

80%

100%

Side lying Supine Side lying Supine

Sleep without quilt Sleep with a thin quilt

HEALTHINF 2012 - International Conference on Health Informatics

because that the shape of the head is not overlapped.

However, there are some conditions that the head or

torso could not be detected well. According to the

observation, we found that the head could not be

detected well while the user is scratching (Figure

14c and d). In this condition, the shape of the head

might be changed and no longer a sphere contour. In

this case, the system could not recognize it as a head.

One possible solution method is to detect the hand

position, and then we can estimate the head position

while the head is overlapping by hand.

Figure 14: Special Sleep Postures.

In addition, there are some sleep conditions or

sleep positions that we did not discuss. First, the

algorithm of head and torso detection we proposed

can be applied to detect multiple heads and torsos.

Moreover, a shortest distance pairing procedure is

used to pair the head and torso of specified sleeper.

However, we still have the detection problem while

the users are overlapping. Second, the head and

torso can be detect in the prone position. Howerer,

this system can not recogize whether the user is in

the supine position or prone position now. Figure 15

shows the detection result of the head and torso

detection in the prone position and multiple users.

Figure 15: Other Sleep Conditions. Left: multiple sleepers.

Right: prone position.

5.2 Breath Measurement

From the experimental results of breath

measurement, we observed some phenomenon

which was similar to the conditions of head and

torso detection. First, the measurement accuracy of

breathing is higher while sleeping in the supine

position than in the side-lying position. We observed

that there are more noises in the side-lying position

than in the supine position. Besides, according to our

measurement results, the average amplitude of the

breathing signals is 0.8 cm in the supine position.

However, the average amplitude of the breathing

signals is about 0.5 cm in the side-lying position.

Overall, less signal noises and more breathing

amplitude would increase the measurement accuracy

while sleeping in the supine position. Second, the

overall measurement accuracy of breathing while

sleeping with a thin quilt was better than the

accuracy while sleeping with no quilt. We speculate

that it might because the reason that the thin quilt

reduces the wrinkles of the torso surface. Third, the

movement of the torso is seen as the breathing signal,

but the system cannot identify whether the

movement is caused by breathing exercise or other

exercises. According to the observations of the

measurement results in experiment I, we found that

there were more detection errors while the user is

moving or turning around in bed. In experiment I,

the participants were asked to turn over the body

frequently, and the participant was almost in static

condition in experiment II. That’s the reason that the

measurement accuracy of breathing is lower in

experiment I than in experiment II. One possible

solution method is to suspend the breathing

measurement while the user is moving.

6 CONCLUSIONS AND FUTURE

WORK

In this study, we proposed a depth image sequence

analysis technique to monitor user’s sleep position,

body movement, and breathing rate on the bed

without any physical contact. A depth image-based

processing method is proposed to monitor the

sleeping conditions. The results of experimental I

showed that the proposed method is promising to

detect the head and torso with various sleeping

postures and body shapes. The results of

experimental II showed that the system can

accurately monitor the sleeping conditions.

Therefore, we confirm that the system could provide

relevant sleep information and sleep report to the

user. Furthermore, the sleep parameters which we

detected can provide to the sleep centre to diagnose

the sleep problems. This study is important for

BREATH AND POSITION MONITORING DURING SLEEPING WITH A DEPTH CAMERA

providing a non-contact technology to measure the

sleep conditions and assist users in better

understanding of his sleep quality.

In the future, we expect to detect more sleeping

conditions and solve some measurement limitations,

such as the problems of overlapping. Besides, we

will develop a multimedia feedback sleep-assisted

system which can detect the breathing status and

provide appropriate sleeping guidance in real time to

help users shorten the time to fall asleep. In addition,

a web-based browser will be developed to provide

the personalized sleeping information to the user.

ACKNOWLEDGEMENTS

This work was supported by the National Science

Council, Taiwan, under grants NSC 98-2221-E-002-

127-MY3. And many thanks to HealthConn Corp.

(http://hcc.healthconn.com/), who provided many

assistances of the professional knowledge.

REFERENCES

Aoki, H. and Koshiji, K., 2006. Non-contact Respiration

monitoring method for screening sleep respiratory

disturbance using slit light pattern projection. World

Congress on Medical Physics and Biomedical

Engineering, 14(7), 680-683.

Ancoli-Israel, S., Cole, R., Alessi, C., Chambers, M.,

Moorcroft, W., Pollak, C. P., 2003. The role of

actigraphy in the study of sleep and circadian rhythms,

Sleep , 26(3), 342-392.

BaHammam, A., 2004. Comparison of nasal prong pressure

and thermistor measurements for detecting respiratory

events during sleep. Respiration, 71(4), 385-390.

Idzikowski, C., 2003. Sleep position gives personality clue.

BBC News, 16 September 2003.

Born, J., Lange, T., Hansen, K., Molle, M. and Fehm, H.

L., 1997. Effects of sleep and circadian rhythm on

human circulating immune cells. The Journal of

Immunology, 158(9), 4454-4464.

Cantineau, J. P., Escourrou, P., Sartene, R., Gaultier, C.,

and Goldman, M., 1992. Accuracy of respiratory

inductive plethysmography during wakefulness and

sleep in patients with obstructive sleep apnea. Chest,

102(4), 1145-1151.

Cartwright, R. D., 1984. Effect of sleep position on sleep

apnea severit. Sleep, 7(2), 110-114.

Douglas, N., Thomas, S., and Jan, M., 1992. Clinical value

of polysomnography, Lancet, 339, 347-350.

Fitbit, http://www.fitbit.com/ , retrieved on 2011/5.

Harada, T., Sakata, A., Mori, T., and Sato, T., 2000. Sensor

pillow system: monitoring respiration and body movement

in sleep, Intelligent Robots and Systems, 1, pp. 351-356.

Hoque, E., Dickerson, R. F., Stankovic, J. A., 2010.

Monitoring body positions and movements during

sleep using WISPs. Wireless Health, 44-53.

Kushida, C. A., Chang, A., Gadkary, C., Guilleminault,

C., Carrillo, O., and Dement, W.C., 2001. Comparison

of actigraphic, polysomnographic, and subjective

assessment of sleep parameters in sleep-disordered

patients. Sleep Med., 2(5), 389-396.

Microsoft, Kinect, http://www.xbox.com/kinect, retrieved on

2011/3.

Loord, H., and Hultcrantz, E., 2007. Positioner - a method

for preventing sleep apnea. Acta Oto-laryngologica,

127(8), 861-868.

Maquet, P., 2001. The role of sleep in learning and

memory. Science, 294, 1048–1052.

Malakuti, K. and Albu, A. B., 2010. Towards an

intelligent bed sensor: Non-intrusive monitoring of

sleep irregularities with computer vision techniques.

International Conference on Pattern Recognition,

4004-4007.

Nakajima, K., Matsumoto, Y., and Tamura, T., 2001.

Development of real-time image sequence analysis for

evaluating posture change and respiratory rate of a

subject in bed. PHYSIOLOGICAL MEASUREMENT,

22, N21-N28.

Philips Actiwatch, http://www.healthcare.philips.com/ ,

retrieved on 2010/8

PneumaCare, http://www.pneumacare.com/ , retrieved on

2011/6.

Wareham,R., Lasenby,J., Cameron, J., Bridge, P. D., and

Iles, R., 2009. Structured light plethysmography (SLP)

compared to spirometry: a pilot study, European

Respiratory Society Annual Congress.

Sadeh, A., Sharkey, K. M., and Carskadon, M. A., 1994.

Activity-based sleep-wake identification: an empirical

test of methodological issues. Sleep, 17(3), 201-207.

Staderini, E. M., 2002. UWB radars in medicine. Aerospace

and Electronic Systems Magazine, 17(1), 13-18.

Spiegel, K., Leproult, R., and Cauter, D. V., 1999. Impact

of sleep debt on metabolic and endocrine function.

The LANCET, 354(9188), 1435-1439.

Szollosi, I., Roebuck, T., Thompson, B., Naughton, M. T.,

2002. Lateral sleeping position reduces severity of

central sleep apnea/cheyne-stokes respiration, Sleep,

29 (8), 1045-1051.

Thought Technology Ltd., http://www.thoughttechnology.

com/, retrieved on 2010/6.

Wang, C., Ahmed, A., and Hunter, A., 2007. Locating the

upper body of covered humans in application to

diagnosis of obstructive sleep apnea. Proceedings of

the World Congress on Engineering, 662-667.

Whyte, K. F., Gugger, M., Gould, G. A., Molloy, J.,

Wraith, P. K., and Douglas, N. J., 1991. Accuracy of

respiratory inductive plethysmograph in measuring

tidal volume during sleep, Journal of Applied

Physiology, 71(5), 1866-1871.

WakeMate, http://www.wakemate.com/ , retrieved on 2011/5.

Ziganshin, E. G., Numerov, M. A., and Vygolov, S. A.,

2010. UWB baby monitor. Ultrawideband and

Ultrashort Impulse Signals, 159-161.

HEALTHINF 2012 - International Conference on Health Informatics