ARE HAPTICS-ENABLED INTERACTIVE AND TANGIBLE

CINEMA, DOCUMENTARIES, 3D GAMES, AND SPECIALIST

TRAINING APPLICATIONS OUR FUTURE?

Miao Song and Peter Grogono

Concordia University, Montreal, Quebec, Canada

Keywords:

Interactive cinema, Computer graphics, Computer-human interaction, Softbody simulation, Topological me-

dia, Haptic-enabled environments, Augmented reality.

Abstract:

Interactive augmented reality via tangible media and cinema enabled with responsive environments through

haptic devices providing the physical feedback from a virtual space to the real space to the interacting audience

is becoming more common especially now that computer graphics and other visual stimuli are becoming so

photorealistic and where they fail, the brain makes up the missing details. This covers games, cinema, virtual

reality applications and education applications. It seems that so common that we will all live in a couple of

decades in an interactive virtually-enhanced world where we won’t be able some times to tell apart the virtual

and the real. How will this affect the upcoming generations and child education? Will we be out of touch with

the reality like in the Matrix (Wachowski and Wachowski, 1999)? While possibly accepting the upcoming

inevitable change, can we improve on it early on and turn it to our advantage? If so, how? We present the

initial research steps as a part of this work to start answering these questions.

1 INTRODUCTION

Position Statement. In many technological soci-

eties today, people are constantly enhancing their

sensory input with virtual things, objects, and char-

acters through either video, gaming, virtual reality,

augmented reality, or any other graphically-enhanced

human-device interaction. Devices such at more-and-

more sophisticated gaming consoles, such as Wii, or

haptic devices along with high-deﬁnition displays,

stereoscopy, or virtual-reality glasses are becoming

increasingly cheap and accessible to medium-lower

income families and educational institution facilitat-

ing access to them by the children and a younger gen-

eration overall. Not to mention the traditional PCs,

laptops, cellphones, PDAs are ﬁtted with cameras and

all sorts of gadgets that are now defacto standards.

In a couple of decades, we will live in a virtually-

enhanced world where it will be difﬁcult at times to

separate reality from computer generated enhance-

ments. It is imperative that such “enhancements” will

impact the way the children are educated and what

they will have taken for granted and the norm. While

there will absolutely be the cases of various inten-

tional or not intentional abuse of technology, we be-

lieve we still can make the advances very useful in

education and perhaps even combat the negative ef-

fects of it. Thus, we present, as space permits, the

effects of such virtualization, and how it can be used

effectively, and our very ﬁrst steps if the research in

that direction using 3D graphics techniques, interac-

tive cinema methodologies, and others of how they

can further improveeducation and training of children

and adults in various subject areas.

Audience. The intended audience of this work cov-

ers a wide spectrum of people who may engage in the

interactive (or even passive at the moment) activities

in front of any types of screens, displays, enhanced

glasses as well as enabled controls in PCs, TVs, gam-

ing consoles (e.g. Wii), mobile phones, and personal

digital assistants (PDAs). We cover the education, en-

tertainment, and training spectra. All involve the au-

dience interaction with the surrounding visual envi-

ronment with computer-generated imagery. We will

ﬁrst discuss the general issues in a human society and

then narrow the scope down to more concrete sectors.

393

Song M. and Grogono P.

ARE HAPTICS-ENABLED INTERACTIVE AND TANGIBLE CINEMA, DOCUMENTARIES, 3D GAMES, AND SPECIALIST TRAINING APPLICATIONS OUR FUTURE?.

DOI: 10.5220/0001821803930398

In Proceedings of the Fourth International Conference on Computer Graphics Theory and Applications (VISIGRAPP 2009), page

ISBN: 978-989-8111-67-8

Related Work. There is a signiﬁcant number of the

related work materials we reference here that would

require a lot of space to describe. They all mostly

relate to the techniques of interactivity, computer

graphics, dynamic physically-based real-time soft-

body simulation, haptics technology, augmented re-

ality, tangible media and so on. Some of them are re-

cited here: (Song and Grogono, 2008a), (Wikipedia,

2008), (Conti and Khatib, 2005), (Kocher, 2008),

(Clua et al., 2006), (Matusik and Pﬁster, 2004), (Dav-

enport et al., 1991).

2 “COST” AND “BENEFITS”

Haptic devices are becoming more accessible. Fam-

ilies acquire larger and bigger television systems

including home theaters. Not to mention ever-

enhancing gaming consoles and PC games in general,

the Wii gaming console reaches out not only to game

geeks, but also to couples, to entire families, and more

interestingly to the female market with their exercise

programs and the like.

Stereoscopy plays another role of creating holo-

grams or visual stereo effects in the cinema and com-

puter programs for animation and games.

Augmented reality is a natural way to bring vir-

tual objects into the real world where we live. With

advances of the AR techniques development, AR has

been applied on medical imaging, where doctors can

access data from patients; aviation, where tools show

pilots important data about the landscape they are

viewing; training, in which technology provides stu-

dents or technicians with necessary data about spe-

ciﬁc objects they are working with; in museums,

where artifacts can be tagged with information such

as the artifact’s historical context or where it was

discovered (EDUCAUSE Learning Initiative (ELI),

2005); military and people’s everyday life. For ex-

ample, through AR handheld devices such as tablet

PC’s, PDA’s, or camera mobile phones, users hold the

device up and “see through” the display to view both

the real world and the superimposed virtual objects.

With the GPS in one’s PDA or a cellphone, one can

move around and see the virtual objects, models, ani-

mations, or game from different views as the AR sys-

tem performs alignment of the real and virtual cam-

eras automatically (Cawood and Fiala, 2008). The

latter is can be useful to guide people in the unknown

area towards their desired destination they have pre-

speciﬁed.

Problems. There are a numberof inherent problems

with the virtualization of our realities.

Abuse in terms of being addicted to the machines

producing the virtual enhancements, games, and so on

resulting in psychological dependence and the corre-

sponding medical consequences and health problems,

reduction of motor functions, physical lack of exer-

cise. The same reason applies to being out of touch

with others and being antisocial preferring the virtual

people and characters and being ignorant and cold

about the real people and friends. Losing human val-

ues and acquiring virtual values with no substance as

well as social aspects and virtual characters in pref-

erence of the real, especially for introvert people can

impair the entire new generation. Lack of touch of

physical touch with another person can make it for-

eign to acquire a family.

Some virtual environments, particularly games,

can “train” violence and make it an ordinary behav-

ior that could potentially spill into the real. This was

known for long time.

Will learning to drive or to pilot with crashes

knowing that’s just a virtual crash, not real, make

safer drivers and pilots? Does it promote the learn-

ers to be more careless and reckless? Do they lessen

the value of life?

Current augmented reality products more focus

on individual users and may not lead themselves to

team activities or group learning. Furthermore, even

though it may raise the pedagogical value, the stu-

dents are also in the risk of becoming infatuated with

such technology.

Most of these problems are actually not new at

all as they were pointed out as soon as ﬁrst addic-

tive computer games started coming out and children

preferred a lot more to stay home and play the games

instead of going outside and playing with others or

doing other activities.

Solutions. Technological advancement is inevitable

from the hardware and software points of view. Can

we turn it to our advantage and make it beneﬁcial?

Interactive feel and touch should, while approach-

ing reality, never reach it, to allow conscious controls

and cues with the interaction interface to hint the vir-

tual from the real.

It’s been shown in scientiﬁc research that the chil-

dren that played computer games had better logic de-

velopment. Education can be effective just as games

if not more, more efﬁciently delivered.

Anger management in the virtually-enhanced en-

vironment can release the anger there to virtual char-

acters, instead of to the real people.

Participative i.e. active, rather than passive, in-

volvement of the audience with the virtual or aug-

mented realities helps teaching and learning: It is pos-

GRAPP 2009 - International Conference on Computer Graphics Theory and Applications

394

sible that some humans hired as teachers do not actu-

ally have the skills to teach effectively, so they can be

(1) complemented with the virtual education as well

as (2) trained themselves to be better educators using

the same virtual and augmented reality tools. This is

in a way similar to tradition expert systems approach,

where the expert system does not entirely replace a

human expert (e.g. doctors) but rather reminds or sug-

gests alternatives or helps to train novice doctors the

expertise until they become more acquainted with the

profession. The virtual educator in this setting with

not only display knowledge in text and graphical user

interface (GUI), but actually would appear and be re-

sponsive in the complete environment with vocal and

emotional support, which are even today technolog-

ically quite possible. Such an educator can evaluate

and correct the real novice educator about gestures,

facial expressions, speech, and effective communica-

tion according to the social norms at the time by eval-

uating several sensory inputs, such as a video and au-

dio signal with some pattern recognition techniques,

e.g. for face and speech recognition followed by natu-

ral language understanding and natural language gen-

eration techniques as well as generation of the facial

expressions by the virtual educator. Responses and

expressions and voice can be compared with a good

set of stored ones or the bad ones, measuring their

distance and then making a suggestion based on that

measurement.

Learning and training can take even further steps,

some may seem superﬂuous today and unnecessary,

as the authors agree, but may very well become a part

of our lives in the future.

The haptic devices can grow in the virtual real-

ity worlds to the sizes of training gear in a gymna-

sium. This can counter the lack of exercise effect. A

the moment such a technology or a similar exists, but

expensive and mostly found in space agencies or the

military.

Haptic gloves and joystick-like devices can sim-

ulate some responsive feedback environments on a

smaller scale, e.g. a hand of a surgeon touching a

softbody organ. Such sense of touch can also be prop-

agated while interacting with the virtual characters in

the environment.

Learning through games can be entertaining and

effective. Hide-and-seek in the augmented reality can

develop a better sense of the environment, say when

both virtual and real people take part in hide-and-seek

with the augmented glasses, it adds an extra twist and

places to look where one would not normally look for

someone hiding, yet not loosing a touch with the re-

ality at the same time.

3 APPROACH TO THE

PROPOSED RESEARCH

With our research we study and contribute the effects

of virtualization of different aspects and enhancing

them with the interactivity while trying to maintain

the balance of the virtual and real. At these very early

and preliminary stages of the research we cannot tell

how it will play out in the end but we lay out a path

towards better augmented and virtual reality interac-

tive and responsive systems for education and training

via interactive cinema (interactive documentaries can

teach the subject of history, law, politics, negotiation,

business, etc.) or games for younger generation.

Focus. The research on haptics-enabled interactive

and tangible cinema, documentaries, 3D games, and

specialist training applications, involves theoretical

and practical experience in ﬁlm and video production,

extensive 3D computer graphics knowledge and the

related programming skills, and experience in digi-

tal multimedia ﬁeld. The author has been very lucky

through their PhD program to be able to blend the

multiple disciplines throughout the research to expe-

rience ﬁrst hand. As mentioned, the research area will

cover various ﬁelds, people who enable the research

at this present moment need to have some expertise

and qualiﬁcations in these areas to design, develop,

and experiment with such a fusion of environments.

One has to have a vast background in com-

puter science, programming and algorithms, com-

puter graphics; and some cinema. Then, bridging the

computer science and visual and digital arts, deﬁn-

ing topology of media as well as the tangible aspect

of the media and its responsive environments as a

form of interactive digital art. A strong creativity in

interactive media is another factor that enables the

research and experiments such as we describe. Re-

search projects, such as CitySpeak (Lewis and OBX

Labs, 2008), allow putting interactive art of text into

the scale of masses in the installations in public places

such as airports and train stations, with a graphically

impeding visualized text that any person could con-

tribute to by interacting with it via SMS messaging.

One also has to have an extensive cinematography

background, which enables them to judge the efﬁcacy

and aesthetics of moving images, and therein lies their

interest in the potential of “new visualization” with

the new media. Such background and extensive expe-

rience is a great help for the research not only in the

theory aspect of cinema and documentary, but also in

the production practice techniques of the careful de-

sign on the scenarios of the movie, which will suit the

interactive cinema techniques.

ARE HAPTICS-ENABLED INTERACTIVE AND TANGIBLE CINEMA, DOCUMENTARIES, 3D GAMES, AND

SPECIALIST TRAINING APPLICATIONS OUR FUTURE?

395

It is a vast undertaking to accumulate all this

knowledge and experience in one person. To be feasi-

ble and efﬁcient, one who is conducting the research

has to bring together people across several disciplines.

The author is grateful to have gathered a supervisory

committee of ﬁve such people to enable the present

research.

Real-time Responsive Virtual Environments and

Tools. In this work we explore a variety of inex-

pensive techniques for human-computer interaction

in various ﬁelds as well as advanced 3D modeling

and animation in a rich spectrum of application do-

mains. Thus, we attempt to come up with a set

of tools, techniques, methodologies using haptics-

enabled and other interface devices such that the tool-

box/framework developed is affordable and applica-

ble with least number of the modiﬁcations to the do-

mains, such as classical cinema and documentaries,

specialist training applications (e.g. surgeons, divers,

and astronauts in preparation of operations and mis-

sions that involvedifﬁcult physical activities), interac-

tive 3D games using haptics-enabled devices and their

applications for home cinema production, 3D games

development, and other application domains.

The ﬁrst area of focus for the research is lean-

ing towards the real-time softbody simulation (Song

and Grogono, 2008a; Song and Grogono, 2008b)

in haptics-enabled environment for specialist-training

and interactive cinema. We begin by using a joystick-

like and a sensor glove-like haptic devices to interact

and deform a softbody in an OpenGL (OpenGL Ar-

chitecture Review Board, 2008) 3D environment that

provides feedback back to the user restricting their

movements based on the elasticity and shape of the

modeled virtual objects (e.g. the softbody tissue of an

organ) acting as inverse sensors. We then proceed to

the game and cinema scenes to enrich them with the

interactive features.

The current research approaches concentrated on

the user interaction are more concentrated on involv-

ing the users to participate in the screen play altering

the script, make the decision for the characters in the

movie scene, similarly as it is done in role-playing

games (RPGs).

Another goal of our research is to extend that

and give the users the opportunity to interact with

the computer graphics model or cinema scenes at

the more realistic level with feedback. The audience

would feel the force, temperature, and elastic defor-

mation of the objects or could experience the charac-

ters surrounding environment of the ﬁlm, rather than

stay at the observing level.

Therefore, the intended research gradually and in-

crementally will focus on the mentioned aspects and

will include the following: enhanced softbody simu-

lation, specialist training, augmented reality, and in-

teractive cinema as the applications of our approach.

Interactive Cinema. Interactive cinema is the evo-

lutionary approach to traditional movie, which gives

the audience an active role in the showing movies.

Compared to traditional movie,the interactive cinema

allows viewers to interrupt the movie from time to

time, and to choose among different possibilities of

how the story goes on. Viewers may interact with the

haptic devices to feel the feedback forces and touch

the objects in the scene. This new and emerging tech-

nique will give viewers a realistic experience and en-

roll as part of the scene for the ﬁlm. The ﬁlm elements

in interactive cinema are controlled by real-time con-

ditions and algorithm behaviors.

Interactive cinema, evolving from the mathemat-

ical models and procedural programming, becomes

alive when some of its parameters are controlled by

a viewer. The real-time challenges of the aspect are

the most interesting to solve in a less expensive way

than traditional approaches.

The focus is on continuous story play environ-

ments, story authoring systems, and scenarios for in-

teraction. Enhanced with haptic devices and stereo-

scopic effects, the feeling of the interaction will con-

sume the audience to be an integral part of the story-

line in the movie, which is a lot much more than a

game.

The perceptual experiences can be different each

time the movie is played. The interactive movie ap-

proach is more suited for smaller rather large audi-

ences in traditional cinema theaters as the interactiv-

ity aspect in movies does not scale very well. The

home and small audience aspect, the technologies

studies and developed in this work should be afford-

able by regular home users, small education groups in

schools, colleges, universities, etc.

During the duration of this project the knowledge

of how to adapt inexpensive haptic devices, algo-

rithms, techniques, and methodologies to interactive

cinema in order for it to be more accessible to digi-

tal media artists, augmented reality researchers, and

computer scientists alike to further research in the

area, and make it accessible to audiences at home and

school. The knowledge acquired from making the in-

stallations, setups, software development and re-use,

and the mathematics behind would be made available

as a guideline of how to make such setups in order

to solicit feedback and improve the process and select

the appropriate tools and techniques.

Such contributions are important in the age of dig-

GRAPP 2009 - International Conference on Computer Graphics Theory and Applications

396

ital media not only for artists and small audiences, but

also for educational purposes of children through in-

teraction, as it is widely known the children develop

better their abilities when one interacts with them

more. Telling interactive stories, teaching decision

making in various life situations are speciﬁcally im-

portant applications of this research.

Enhanced Softbody Simulation. The research and

development of the 3D simulation of softbody de-

formation still has a lot of aspects that can be ex-

plored in the softbody deformation simulation area

because it has emerged as a new challenge in com-

puter graphics, and, therefore, was not fully exhausted

yet. Meanwhile, the application of softbody defor-

mation in computer graphics has signiﬁcant value for

medical research and media arts achievement. The re-

sulting models from the simulated models, will help

to create feedback environments that will map the vir-

tual feedback to physical through an haptic interface.

In our real physical world there exist not only rigid

bodies but also soft bodies, such as human and ani-

mal’s soft parts and tissue, and other non-living soft

objects, such as cloth, gel, liquid, and gas. Soft body

simulation, which is also known as deformable object

simulation, is a vast research topic and has a long his-

tory in computer graphics. It has been used increas-

ingly nowadays to improve the quality and efﬁciency

in the new generation of computer graphics for char-

acter animation, computer games, and surgical train-

ing. So far, various elastically deformable models

have been developed and used for this purpose.

Specialist Training. Surgical training is one of the

most promising ﬁelds in medicine where computer

graphics and virtual reality techniques are emerg-

ing. The computer generated visual virtual environ-

ment imitates the reality of medical operations and

organ construction to fulﬁll the training purpose. This

new application improves surgical outcomes and de-

creases the research costs. However, the reality and

accuracy of the software always require high-end

knowledge of physics, mathematic and heavy com-

putation. It makes it difﬁcult for users to interact with

virtual objects in real-time.

Furthermore, another medical application of elas-

tic object deformation is computer-assisted surgery,

which guides the surgeon during the real-time surgery

operation. An haptic simulator will help novice sur-

geons to practice their suturing skills. The surgical

simulation system includes the research topics of the

real-time interaction with 3D elastic objects, physics

aspects of the deformation behaviors of elastic ob-

jects, photo-realistic visualization,and haptic force-

feedback algorithm. Users interact with the digital

model through physical suturing tools that are at-

tached to an haptic device. The digital model deforms

relative to the weight, pressure, impact, such as push-

ing, pulling, cutting applied on the tissue. The goal

of this research is to give surgeons the force feedback

accurately through the haptic model. A similar ap-

proach can be used in training astronauts to manipu-

late the instruments for their space servicing missions

in environments that are generally more difﬁcult to

operate in especially when wearing a highly-resistive

space-suit gloves under pressure. Commonly, astro-

nauts do the training for that underwater requiring

special tanks and the necessity of the availability of

the spacesuit. Having an haptic environment with the

simulated geometry will allow the astronauts a more

accessible training tool to practice their dexterity and

strength skills prior getting into the spacesuits more

often.

Augmented Reality. In the past few years, Virtual

Reality, which has attracted a great deal of media

attention is to immerse a user inside an imaginary,

computer-generated “virtual world”. The user is cut

off from any view of the real world outside. Later,

some attention has been paid to the ﬁeld of Aug-

mented Reality, in which the user can see the real

world around him, with computer graphics superim-

posed or composited with the real world. The real

and virtual objects coexisted. User wear an optical

see-through Head-Mounted Display or other digital

devices, such as PDA and mobile phones which al-

low user to see a combination of the real world and a

virtual world. Within the academy, educators are pro-

viding students with deeper and more meaningful ex-

periences by linking the educational content with real

places and objects. Moreover, PDAs or other portable

devices can use GPS data to provide users with visual,

audio, or text-based information about the place and

objects in our real world.

3D Games. Games are a natural place for the topics

of interactive cinema, education, and responsive envi-

ronments to be used. There has been a lot of focus in

the industry to make the games with fancy stereo an-

imation or augmented reality or virtual environment

spaces, and cinema re-done in to the game plots. The

games are a natural outcome of the above listed top-

ics and techniques, so we trim the discussion about

the games here.

ARE HAPTICS-ENABLED INTERACTIVE AND TANGIBLE CINEMA, DOCUMENTARIES, 3D GAMES, AND

SPECIALIST TRAINING APPLICATIONS OUR FUTURE?

397

4 CONCLUSIONS

It seems that our lives are becoming more and more

entangled with the augmented or virtual interactive

environment spaces enhanced with advanced com-

puter graphics techniques. As any technology can be

used for peaceful and educational purposes, we are

striving to identify the right technology in the haptics

world affordable by the masses for the entertainment,

creation, and education through the study of ﬁlm,

computer graphics, augmented reality techniques, as

well as their application in the industry.

ACKNOWLEDGEMENTS

We would like to acknowledge Professors Jason

Lewis of Visual Arts, OBX Labs and Hexagram, Xin

Wei Sha of Topological Media Lab and Hexagram,

and Marielle Nitoslawska of School of Cinema. Also

a token of appreciation for helpful advice and com-

ments go to Alison R. Loader, Jean-Claude Bustros,

and Alice Jim. This work was sponsored in part by

the Special Individualized Program (SIP), Humani-

ties, and Faculty of Engineering, and Computer Sci-

ence (ENCS), Concordia University, Montreal, Que-

bec, Canada.

REFERENCES

Cawood, S. and Fiala, M. (2008). Augmented Reality: A

Practical Guide. Pragmatic Bookshelf.

Clua, E., Feij, B., Rocca, J., Schwartz, J., das Graas, M.,

Perlin, K., Tori, R., and Barnes, T. (2006). Game

and interactivity in computer science education. In

Proceedings of the International Conference on Com-

puter Graphics and Interactive Techniques. ACM

SIGGRAPH. ISBN 1-59593-364-6.

Conti, F. and Khatib, O. (2005). Spanning large workspaces

using small haptic devices. In WHC, pages 183–188.

IEEE Computer Society.

Davenport, G., Smith, T. A., and Pincever, N. (1991). Cin-

ematic primitives for multimedia. IEEE Computer

Graphics and Applications, 11(4):67–74.

EDUCAUSE Learning Initiative (ELI) (2005). 7 things

you should know about augmented reality.

http:

//connect.educause.edu/Library/

Kocher, D. (2008). Convergence culture: Where old and

new media collide by henry jenkins. First Monday,

13(5).

Lewis, J. and OBX Labs (2008). CitySpeak.

http://

cspeak.net/

Matusik, W. and Pﬁster, H. (2004). 3D TV: a scalable

system for real-time acquisition, transmission, and

autostereoscopic display of dynamic scenes. ACM

Trans. Graph, 23(3):814–824.

OpenGL Architecture Review Board (1998–2008).

OpenGL. [online].

http://www.opengl.org

Song, M. and Grogono, P. (2008a). A framework for

dynamic deformation of uniform elastic two-layer

2D and 3D objects in OpenGL. In Proceedings

of C3S2E’08, pages 145–158, Montreal, Quebec,

Canada. ACM. ISBN 978-1-60558-101-9.

Song, M. and Grogono, P. (2008b). An LOD control inter-

face for an OpenGL-based softbody simulation frame-

work. In Proceedings of CISSE’08, University of

Bridgeport, CT, USA. Springer. To appear.

Wachowski, L. and Wachowski, A. P. (1999). The Matrix.

Warner Bros. and Village Roadshow Pictures. [Mo-

tion picture].

Wikipedia (2008). Haptic technology — Wikipedia, The

Free Encyclopedia. [accessed 20-August-2008].

GRAPP 2009 - International Conference on Computer Graphics Theory and Applications

398