Feasibility of Eye-tracking based Glasses-free 3D Autostereoscopic

Display Systems for Medical 3D Images

Dongwoo Kang, Seok Lee, Hyoseok Hwang, Juyong Park, Jingu Heo, Byongmin Kang,

Jin-Ho Lee, Yoonsun Choi, Kyuhwan Choi and Dongkyung Nam

Multimedia Processing Lab, Samsung Advanced Institute of Technology, Suwon-si, South Korea

Keywords: Three-dimension, 3D, Glasses-free 3D Autostereoscopy, Eye-tracking, Medical 3D, Cardiac CT, Coronary

CTA, 3D Heart, Display.

Abstract: Medical image diagnosis processes with stereoscopic depth by 3D display have not been developed widely

yet and remain understudied Many stereoscopic displays require glasses that are inappropriate for use in

clinical diagnosis/explanation/operating processes in hospitals. An eye-tracking based glasses-free three-

dimensional autostereoscopic display monitor system has been developed, and its feasibility for medical 3D

images was investigated, as a cardiac CT 3D navigator. Our autostereoscopic system uses slit-barrier with

BLU, and it is combined with our vision-based eye tracking system to display 3D images. Dynamic light field

rendering technique is applied with the 3D coordinates calculated by the eye-tracker, in order to provide a

single viewer the best 3D images with less x-talk. To investigate the feasibility of our autostereoscopic system,

3D volume was rendered from 3D coronary CTA images (512 by 512 by 400). One expert reader identified

the three main artery structures (LAD, LCX and RCA) in shorter time than existing 2D display. The reader

did not report any eye fatigue or discomfort. In conclusion, we proposed a 3D cardiac CT navigator system

with a new glasses-free 3D autostereoscopy, which may improve diagnosis accuracy and fasten diagnosis

process.

1 INTRODUCTION

Three-dimensional (3D) medical imaging techniques

are increasingly employed for evaluation of not only

identifying complex organ structures but also

diagnosing abnormalities. Recent advanced 3D

imaging techniques such as MR, CT and Ultrasound

showed the usefulness and evoked the demand of 3D

medical imaging displaying system. Also 3D

graphics techniques have been developed fast, which

enables high quality 3D medical volume rendering

(Chan et al., 2013, Ferroli et al., 2013, Langdon et al.,

2014). However, the advanced and complex 3D

medical images are displayed with 2D monitors,

where 3D objects are projected into 2D plane.

3D displays have become available these days.

Especially, three-dimensional movies presented via

stereoscopic displays have become more popular in

recent years aiming at a more engaging viewing

experience. However, medical image diagnosis

processes with stereoscopic depth by 3D display have

not been developed widely yet and remain

understudied.

Additionally, many stereoscopic displays require

glasses that are inappropriate for use in clinical

diagnosis/explanation/operating processes in

hospitals. A few studies of 3D autostereoscopy in

medical image analysis area exist (Narita et al., 2014).

A glass-free 3D autostereoscopic display monitor

has been developed by Samsung Advanced Institute

of Technology (SAIT) (Park and Nam, 2012, Park et

al., 2013) which provides almost same 3d quality as 3D

display that requires 3D glasses: ≤5% crosstalk.

We investigated "3D display needs" from medical

doctors in various medical departments from

Samsung Medical Center (SMC). Based on the

collected opinions, we developed a new application

of glass-free medical 3D: Cardiac CT 3D Navigator.

One of the medical imaging systems that require

3D display system is cardiac CT due to complex

anatomy of a heart and isotropic 3D volume of

cardiac CT. 3D display for cardiac CT diagnosis

without glasses techniques may enable physicians

and medical doctors to make a detailed/accurate

134

Kang, D., Lee, S., Hwang, H., Park, J., Heo, J., Kang, B., Lee, J-H., Choi, Y., Choi, K. and Nam, D.

Feasibility of Eye-tracking based Glasses-free 3D Autostereoscopic Display Systems for Medical 3D Images.

DOI: 10.5220/0005826901340138

In Proceedings of the 9th International Joint Conference on Biomedical Engineering Systems and Technologies (BIOSTEC 2016) - Volume 2: BIOIMAGING, pages 134-138

ISBN: 978-989-758-170-0

diagnosis and facilitate diagnosis processes.

In this paper, we introduce our glasses-free

autostereoscopic display to medical society, and

discuss the feasibility as a medical 3D display, which

may help improved diagnosis accuracy and fastened

diagnosis process, with an application with cardiac

CT data.

Figure 1: Slit-barrier with BLU composition.

2 METHOD

We developed a new 3D diagnosis system with a new

glasses-free display technique. The new glasses-free

display system is based on eye-tracking technique and

slit-barrier with back light unit (BLU) device

technique. The new system was applied to 3D cardiac

CT, for a new glass-free cardiac CT 3D navigator

system.

2.1 Eye-tracking based Glasses-free 3D

Autostereoscopic Display

Our 3D display consists of display panel, optical

element for 3D and camera. The 4K liquid crystal

display (LCD) panel was used. The optical element

for 3D device is a device that controls the direction of

lights which pass through the panel, where our

display system is based on slit-barrier with BLU

technique. The slits in the barrier allow only left

image pixels to pass into left eye position, right image

pixel to pass into right eye position. Our slit-barrier

locates behind LCD panel and in front of BLU

(Figure 1).

The camera is used for eye-tracking, where

machine learning based eye-tracking algorithm was

applied. With this eye-tracking algorithm, the viewer

doesn’t have to find the good position to see the 3D

properly, but can see the 3D in any position that

tracking is allowed. The eye-tracker identifies the

viewer’s the 3D coordinates of the pupil center, the

subpixel values for the left and right views in display

panel are adjusted to be seen correctly by viewer's

eyes.

The real-time machine learning based eye-

tracking algorithm starts from face detection using the

AdaBoost learning algorithm (Freund and Schapire,

1997). From the detected face region, subregions with

eyes are extracted by shape alignments using

Supervised Descent Method (SDM) (Xiong et al.,

2013). 23 landmark points of eyes, nose and mouth

were used for SDM shape alignments.

Figure 2: 3D rendering techniques. Our autostereoscopy

uses dynamic light field mapping method.

3D Light ray image is generated by a 3D

rendering algorithm, where each pixel’s light ray

direction was determined by the slit-barrier.

Especially, based on eye tracking algorithm,

Dynamic Light Field Mapping (DLFM) is applied for

the 3D rendering (Park and Nam, 2012, Park et al.,

2013). Using the position and direction information

of each light, each light is mapped to the 3D eye

coordinates. While Dynamic View Mapping (DVM)

3D rendering method (Boev et al., 2008), which many

vendors uses currently just switches left/right images

according to the eye position, DLFM maps each light

ray to eye coordinates (Figure 2). Because a DLFM

technique utilizes the photorefractive effect of each

ray, viewer can see the 3D images without the

limitation of viewing distance. Also DLFM can solve

Feasibility of Eye-tracking based Glasses-free 3D Autostereoscopic Display Systems for Medical 3D Images

135

the image degradation problem of large 3D displays.

The overall specification of our 3D

autostereoscopic system is shown in Table 1, and the

prototype of our display is shown in Figure 3.

Table 1: Eye-tracking based autostereoscopic display

system specification.

Panel size 31.5 Inch

2D resolution 3840x2160

3D resolution 1536x720

Viewing distance

100cm

±50cm

Viewing Angle

H60

°/V40°

User 1 person only

Frame Rate 60Hz

Figure 3: Prototype of eye-tracking based autostereoscopic

display system for medical 3D.

2.2 3D Cardiac CT Navigator: 3D

Autostereoscopy Visualization

Feasibility

Figure 4: 3D cardiac CT navigator concept.

3D coronary CT angiography (CTA) images were

visualized with our 3D eye-tracking based 3D

autostereoscopic display system. Under advice of

medical doctors at SMC, we made a 3D cardiac CT

navigator software. A 3D CTA anonymized image

data was obtained from SMC. The CTA dataset was

acquired on the dual-source 64-slice CT scanner

(Definition Siemens Medical Solution, Germany)

with a gantry-rotation time of 330mms and standard

collimation of 0.6mm, and had excellent image

quality. 3D CTA scan parameters were 512x512

matrix, voxel size 0.38x0.38x0.3mm

, and 412 slices.

The patient did not have any luminal stenosis or

plaque.

The proposed navigator system aims identifying

the 3D structure of the complex organs easily, and we

developed a 3D cardiac CT navigator proto as an

example (Figure 4). With help of enhanced 3D depth

perception in our 3D display, viewers can recognize

the complex 3D structure in depth.

Figure 5: 3D cardiac CT navigator S/W proto. Original CT

volume is rendered without any segmentation (up) and

whole segmentation (down).

For 3D visualization, we followed the standard

cardiac CT image and graphics processing from a

coronary CTA image dataset: (1) whole heart

segmentation, (2) coronary artery segmentation, and

(3) 3D volume rendering. User can adjust the color

and transparency of 3D volumes by option. Also,

multi-planar reconstruction (MPR) is aligned with 3D

volume rendering by side (Figure 5). Further, our

software has an option of converting the volume to

3D mesh, which graphics artists can decorate

manually.

One expert reader from SMC was asked to

BIOIMAGING 2016 - 3rd International Conference on Bioimaging

136

identify the structure of heart with our 3D

autostereoscopy. In a segmented heart only 3D

volume, the expert reader identified main heart

structures including three main coronary artery

structures (LAD, LCX and RCA).

3 RESULTS AND DISCUSSION

Figure 6: Eye-tracking for 3D autostereoscopy.

A 3D coronary CTA image was tested with our 3D

glasses-free autostereoscopic display system and 3D

cardiac CT navigator software. The average error of

the eye-tracking was 2mm, which was calculated as

Euclidean distance between the center of pupils and

tracked eye coordinates (Figure 6). Also the tracking

time of eyes was 16ms in average. The average

crosstalk of our 3D display was 4.9%, which was

similar 3D quality to that of 3D display with glasses.

With dynamic light field rendering method, 2 stereo

images (left and right images) were generated for 3D

display, making the depth 20cm (Figure 7). A

standard 2.5 GHz personal computer running

windows 7 was used for running the cardiac software.

The rendered 3D heart volume is 3D light field

rendered by dynamic light field mapping method for

displaying in our 3D autostereoscopy. All the

processes ran in the real time.

The expert reader visually assessed the quality of

our 3D autostereoscopy. He did not report any eye

fatigue or discomfort. Also, he identified the 3D heart

structure with our 3D autostereoscopy, including 4

chambers, aorta, main coronary arteries (LAD, LCX

and RCA). He didn’t provide the quantity assessment

but reported he could identify the 3D structure faster

and easier. Further investigation is required with

quantification for testing usefulness of our 3D

autostereoscopy: this is a limitation of our study. Also

number of cases

Figure 7: Stereo images from 3D mesh for a 3D CT

volume(up) and generated anaglyph (down). A graphics

artist decorated the heart manually.

4 CONCLUSIONS

We proposed a 3D cardiac CT navigator system with

our new glasses-free 3D autostereoscopy. We

introduced the feasibility of 3D autostereoscopy for

medical image diagnosis. It may improve diagnosis

accuracy and fasten diagnosis process. Our 3D

autostereoscopic system can be applied any 3D

volumetric medical images.

ACKNOWLEDGEMENTS

We thank Dr. Jinho Choi at Samsung Medical Center

(SMC) for all the advice for the medical 3D navigator

system.

REFERENCES

Chan, S., Conti, F., Salisbury, K. & Blevins, N. H., 2013,

Virtual reality simulation in neurosurgery: technologies

Feasibility of Eye-tracking based Glasses-free 3D Autostereoscopic Display Systems for Medical 3D Images

137

and evolution, Neurosurgery, 72(1), 154-164.

Ferroli P., Tringali, G., Acerbi, F., Schiariti, M., Broggi, M.

Aquino, D. & Broggi, G., 2013, Advanced 3-

dimensional planning in neurosurgery, Neurosurgery,

72(1), 54-62.

Langdon, W. B., Modat, M., Petke, J., Harman, M., 2014,

Improving 3D medical image registration CUDA

software with genetic programming, Proceedings of the

2014 Annual Conference on Genetic and Evolutionary

Computation, 951-958.

Narita, Y., Tsukagoshi, S., Suzuki, M., Miyakita, Y., Ohno,

M., Arita, H., Saito, Y., Kokojima, Y., Watanabe, N.,

Moriyama, N. & Shibui, S., 2014, Usefulness of a glass-

free medical three-dimensional autostereoscopic

display in neurosurgery, International Journal of

Computer Assisted Radiology and Surgery, 9(5), 905-

911.

Park, J., Nam, D., 2012, Active light field rendering in

multi-view display systems, SID International

Symposium, 36-39.

Park, J., Nam, D., Choi, S. Y., Lee, J., Park, D. S., & Kim,

C. Y., 2013, Light Field Rendering of Multi-view

Contents for High Density Light Field 3D Display, SID

International Symposium, 667-670.

Freund, Y., Schapire, R.E., 1997, A Decision-Theoretic

Generalization of on-Line Learning and an Application

to Boosting, Journal of Computer and System Sciences,

119-139.

Xiong, X., Torre, F. D. L., 2013, Supervised Descent

Method and its Applications to Face Alignment, CVPR,

532-539.

Boev, A., Georgiev, M., Gotchev, A. & Egiazarian, K.,

2008, Optimized Single-viewer Mode of Multiview

Autostereoscopic Display, EUSIPCO, 25-29.

BIOIMAGING 2016 - 3rd International Conference on Bioimaging

138