![](bg3.png)
sensitive enough for precise touch detection. The
hand/finger can be separated from its shadow by the
intensity value. If the intensity of a pixel in the
foreground is lower than some threshold, the pixel is
regarded as a part of the shadow. The threshold
value must account for the level of ambient light
around the system, and doing so is not an easy task.
For environments in which shadow segmentation
cannot be done with sufficient accuracy, there are
several methods to detect a touch by tracking the
movement of the hand/finger (Kjeldsen et al., 2002;
Borkowski, 2004; Borkowski 2006; Audet, 2012).
In Audet (2012), the tip of the finger is tracked.
If it enters a button region and stops, the system
detects a touch. For real-time processing, the system
utilizes simple technologies such as frame
differences for moving-object tracking and a simple
template matching for locating the tip of the finger.
If more than one tip is recognized, the most distant
tip from the user’s body is selected for touch
detection. In Borkowski (2004) and Borkowski
(2006), an elongated foreground (i.e. a finger) is
detected in the button region by monitoring changes
in the region’s luminance. The button region is
virtually split into two or more subregions, one
fingertip-sized subregion is located in the center of
the button region, and the other subregions surround
this central subregion. A touch by a finger tip on the
center of the button is detected when luminance
changes are observed at only some of the subregions
and at the central subregion.
These methods can efficiently recognize an
elongated foreground shape, but it is difficult for the
system to distinguish a finger touching the button
from a hovering finger or the elongated shadow of
something else. Therefore, such methods may be
suitable for a small personal display, but they are
problematic for a large interactive display with
hand-sized large buttons.
2.3 Touch Detection by Shadow
One of the characteristics of a projector-based
system is that a shadow unavoidably appears when a
user is touching or about to touch the screen,
because the hand intercepts the projected light; the
camera, owing to its perspective different from that
of the projector, sees this shadow. This particular
feature can be utilized for detecting a touch.
If a front camera can reliably distinguish
between a hand and its shadow, a screen touch can
be detected by calculating the relative proportions of
the hand and the shadow in the camera image (Kale,
2004; Song, 2007). When a hand is about to touch
the screen but still hovering on it, the shadow is
relatively larger than when the hand is touching.
This method depends on accurate separation of the
shadow from the hand through the use of only
optical color information.
The shadow color might be recognized more
easily than the hand color because it is not affected
by the projected light and is almost the same as the
unilluminated screen color. The intensity
(brightness) value is often used as a threshold
criterion (Brutzer, 2011; Winkler, 2007). The pixels
with lower intensity value than this threshold are
extracted as shadow. However, this simple criterion
may falsely accept more than a few non-shadow
pixels, and the threshold must be adjusted whenever
the ambient light changes.
In Wilson (2005) and Kim (2010), the shape of
the finger shadow is the metric for touch detection.
To generate a clean shape, some morphological
operations must be performed before analysis of the
shadow shape.
2.4 Proposed Method – Measuring
Shadow Area
As stated in Section 1, our target system is a large
interactive display. Virtual touch buttons are as large
as a hand and are supposed to be touched with the
whole hand. Therefore, we do not rely on the shape
and direction of the user’s hand for touch detection.
To ensure real-time performance, we try to avoid
time-consuming operations, such as morphological
ones and shape analysis. Therefore, we use an area
proportion metric in the button region.
The shadow color on the screen can be known in
the system by taking an image with the projector off
during initialization. However, ambient light may
change throughout day.
One way to know the shadow area in the touch
detection process is to intercept the projected light.
So, by altering the color of projected light in the
button area, we can find those pixels that do not
change color after the projected light changes. As
both the button region without interception and the
touching hand are affected by the color change, this
scheme works out except when the user wears a
black or very dark glove (Figure 1). Thus, under the
following two conditions, we can achieve touch
detection even in places where the ambient light
changes.
(a) The proportion of the foreground (non-
background) area exceeds a threshold.
(b) The proportion of shadow is below a separate
threshold.
VISAPP2014-InternationalConferenceonComputerVisionTheoryandApplications
36