EMBEDDING MULTIMEDIA CONTENT WITHIN VIRTUAL
ENVIRONMENTS
The OGRE Approach
Paulo N. M. Sampaio
1,2
, Roberto Ivo C. de Freitas
2
and Gonçalo Nuno P. Cardoso
2
1
Centre for Informatics and Systems of the University of Coimbra (CISUC)
2
Laboratory of Distributed Systems and Networks (Lab-SDR)
University of Madeira (UMA), Campus da Penteada 9000-390, Funchal, Madeira, Portugal
Keywords: Virtual Reality, OGRE, Multimedia Documents.
Abstract: Most of the tools and languages for modeling Virtual Reality environments, such as VRML, X3D, Java3D,
etc. do not provide means of describing the synchronized presentation of multimedia content inside these
environments. Multimedia has demonstrated its capabilities of motivating users and capturing their atten-
tion, which is an interesting characteristic when we want to provide a higher degree of immersion inside
Virtual Reality applications. This paper presents a robust and generic solution for the integrated presenta-
tion of different kinds of media objects inside virtual environments based on the Graphical Engine OGRE.
1 INTRODUCTION
The integration of multimedia content inside Virtual
Environments (VEs) is a promising and interesting
trend in the development of Virtual Reality (VR)
applications since interaction can be enhanced, and
through the addition of audio and video, the user´s
immersion inside the VE can be improved. Indeed,
we understand the integration of multimedia content
as the temporal and logical synchronization of dif-
ferent media objects (with at least one audio or
video) rendered inside a VE.
Many VR systems have been proposed in the lit-
erature developed within different application do-
mains: e-learning (Chee, 2001), (McArdle et al.,
2004), (Halvorsrud et al., 2004), collaboration
among workgroups (Bochenek and Ragusa, 2003),
augmented collaborative spaces (Pingali and Su-
kaviriya, 2003), Multimodal VR applications (Car-
rozino et al., 2005), among others. Indeed, the rapid
prototyping, modeling and authoring of VEs has
been a major concern to many authors, as presented
in (Rodrigues and Oliveira, 2005), (Osawa et al.,
2002), (Ficheman et al., 2005) and (Garcia et al.,
2002). Although, most of the systems propose the
development of VEs, few of them (or none) explore
the presentation of integrated multimedia content
inside VEs (Walczak et al., 2006).
This paper presents a robust solution to provide
the integration of multimedia content inside a VE
based on the Graphical Engine OGRE
(http://www.ogre3d.org/). The API implemented is
called OGRE-Multimedia, and can be applied to any
VR application to allow their customization with
multimedia content.
OGRE (Object-Oriented Graphics Rendering En-
gine) is a scene-oriented, flexible 3D engine written
in C++ designed to make it easier and more intuitive
for developers to produce applications using hard-
ware-accelerated 3D graphics. The class library
abstracts all the details of using the underlying sys-
tem libraries like Direct3D and OpenGL and pro-
vides an interface based on world objects and other
intuitive classes. When comparing OGRE with other
different languages for describing virtual worlds
(such as VRML (http://www.w3.org/MarkUp/
VRML), X3D (Extensible 3D (X3D)) or Java3D
(http://java.sun.com)), the choice was not hard to
make, specially when it comes to favoring design
quality, flexibility and clear documentation.
This paper is organized as follows: Section 2 pre-
sents a solution to customize a VE with multimedia
presentations; Section 3 presents the main architec-
ture of the API developed; Section 4 illustrates a
Multimedia and Virtual Reality application, and;
Finally, Section 5 presents some conclusions.
524
N. M. Sampaio P., Ivo C. de Freitas R. and Nuno P. Cardoso G. (2008).
EMBEDDING MULTIMEDIA CONTENT WITHIN VIRTUAL ENVIRONMENTS - The OGRE Approach.
In Proceedings of the Third International Conference on Computer Graphics Theory and Applications, pages 524-531
DOI: 10.5220/0001098905240531
Copyright
c
SciTePress
2 CUSTOMIZING MULTIMEDIA
PRESENTATIONS WITH XML
The main goal of this project is to propose and de-
velop an API for providing the presentation of mul-
timedia content within an OGRE´s virtual environ-
ment (VE). The multimedia content is related to the
presentation of any multimedia player (such as
RealPlayer (http//www.real.com/player), GRiNs
(www.oratrix.com/GRiNS/SMIL2.0), etc.), Flash
(Adobe – Flash Player) or a web-browser. The idea
is to customize and render the multimedia presenta-
tion as textures over any 3D object inside the VE.
However, an issue is considered when it comes
to the integration of a multimedia presentation inside
a virtual environment (VE). This issue is related to
the characterization of the temporal and logical
structures of the multimedia document, and how to
respect these synchronization constraints inside the
VE. Besides, these synchronization constraints still
may depend on external events to the VE, such as
user interactions or events notifying the availability
of resources.
The solution for that relied on the proposal of an
XML-based meta-language for describing multime-
dia documents inside virtual environments, or as we
called the meta-multimedia document. The meta-
multimedia document is applied as a multimedia
authoring language where users can customize the
virtual environment and describe what is going to be
presented, where and when they will be presented.
Briefly, it describes all the components of the mul-
timedia presentation and their temporal and logical
synchronization. The particularity about this presen-
tation is that all the media objects (multimedia
documents, flash, web-browsers, and primitive me-
dia objects such as video, image, text, audio, etc.)
are synchronized and rendered anywhere inside the
virtual environment. The interpretation and coordi-
nation of this document presentation inside the VE
is done by the API developed for OGRE, the OGRE-
Multimedia.
As presented in Figure 1, the structure of the
meta-multimedia document is composed of four
main elements: panel, trigger, eventHandler and
event.
Each panel element describes a presentation
panel for media objects inside the virtual environ-
ment. The container called panels is a set of the
panel objects that must be rendered inside a VE.
Each element trigger characterizes an object inside
the VE which controls the activation and deactiva-
tion of a multimedia presentation. The container
called triggers is a set of all the trigger objects that
will be used to control the multimedia presentations
inside a VE.
Each element EventHandler characterizes how
the presentation of the media objects associated with
a given trigger will be controlled (e.g., start their
presentation when the user clicks on the trigger or
when he approximates it). The container called
eventHandlers is a set of all multimedia presentation
described by all the elements eventHandler.
Figure 1: Example of a meta-multimedia document.
EMBEDDING MULTIMEDIA CONTENT WITHIN VIRTUAL ENVIRONMENTS - The OGRE Approach
525
Each element event characterizes how and when
the presentation of each media object of a given
eventHandler will be carried out.
The structure of the meta-multimedia document
was defined to make the process of authoring the
multimedia document easier and intuitive. We con-
sider the meta-multimedia document as the key-
solution for the integration of multimedia content
inside VEs. Indeed, with this document, the author
of the application is able to customize his virtual
environment without changing a single line of his
code.
3 OGRE-MULTIMEDIA:
INTEGRATING MULTIMEDIA
WITHIN VIRTUAL
ENVIRONMENTS
Most of the libraries available for creating Virtual
Reality applications do not have appropriate APIs
for the integration of multimedia content inside a
VE. Some languages and platforms such as VRML,
X3D, Java3D and OGRE, provide only APIs for the
presentation of single media objects (such as video
or audio) without integrating or synchronizing these
objects.
The solution proposed with OGRE-Multimedia
is to provide an API to integrate the presentation of
different multimedia objects (rendered by different
plug-ins or APIs) around the definition of the meta-
multimedia document. In this sense, the meta-
multimedia document describes all the synchroniza-
tion relations among all the components (media
objects, Flash presentation, web-browsers, etc.) of
the document. Taking advantage of the OGRE´s
component-based architecture, this API can be eas-
ily instantiated and integrated with the remaining
available library. This section presents the main
architecture of OGRE-Multimedia
The architecture of OGRE-Multimedia describes
the integration of the meta-multimedia document
with the Virtual Environment, which are supported
by the implemented software modules and some
existing APIs. This architecture is depicted in Fig. 2.
The architecture of OGRE-Multimedia is composed
of four main components: (1) External modules
(TinyXML, OIS, OgreAL, DevIL, Navi and
OGRE), (2) Elementary modules (Meta-multimedia
document and Virtual Environment), (3) System
startup, and (4) System Update.
Figure 2: Architecture of OGRE-Multimedia.
GRAPP 2008 - International Conference on Computer Graphics Theory and Applications
526
External Modules. The external modules are repre-
sented by those APIs developed by other projects, or
which were already provided by the OGRE´s li-
brary. These modules were helpful for the imple-
mentation of OGRE-Multimedia, and demonstrate
how this API can be integrated straightforward with
other plug-ins to enable the presentation of multi-
media content inside VEs. These modules are:
TinyXML: This API (TinyXML) provides an
XML syntactic analyzer (parser) developed in
C++. TinyXML is applied for reading informa-
tion described on the meta-multimedia docu-
ment;
OIS: This module (OIS) describes a simple solu-
tion for managing the interaction through the in-
put devices (such as mouse, joystick, keyboard,
etc.) and feedback devices (vibrating commands,
etc.);
OgreAL: This module (OgreAL Documentation)
is related to a library, based on the library Ope-
nAL ( www.openal.org), which provides func-
tionalities for the presentation of audio inside an
OGRE application. OgreAL is applied in con-
junction with OGRE-Multimedia to provide the
presentation of all the audio information declared
on the meta-multimedia document.
DevIL: This API (DevIL) is related to the library
used by OGRE for the presentation of images in-
side VEs. DevIL is applied by OGRE-
Multimedia for the presentation of images de-
clared on the meta-multimedia document;
Navi: This API (Simmons) enables the presenta-
tion and interaction of a web browser window
(based on Firefox 2.0) in an OGRE application.
This API is integrated with OGRE-Multimedia
to provide the presentation of flash (.swf) con-
tent such as video or animations;
OGRE: This module represents the graphic en-
gine OGRE which is the main module of the sys-
tem being responsible for creating, managing
and updating the tri-dimensional model.
Elementary Modules. The elementary modules
describe the non-functional components of the archi-
tecture, which are used as a support for the applica-
tion. These modules are:
Meta-multimedia document: Which describes
all the information about the media objects and
their temporal relations, as presented in section
2;
Virtual Environment: This module is related to
the tri-dimensional models (panels and triggers)
applied to support the multimedia presentation
inside the Virtual Reality application.
System Startup. This component is in charge to set
up the presentation of multimedia content in VEs.
Therefore, the modules of this component are exe-
cuted only once when the application is launched.
These modules are numbered following the se-
quence they are executed:
1) Loading the meta-multimedia document: This
modules reads and loads into memory the multi-
media document (XML), building a DOM
(http://www.w3.org/DOM) out of this file. The
API TinyXML is used for this purpose;
2) 5) and 8) Search by the first element of a par-
ticular type: This module analyses DOM and
search for the first element of a given type. This
step is executed three times, each one searching
for the first occurrence, respectively, of a panel,
a trigger and an eventHandler;
3) 6) and 9) Extraction and definition of the pa-
rameters for each presentation: These modules
extracts and loads all the parameters necessary
for the creation of each panel, trigger and even-
tHandler (and events);
4) 7) and 10) Creation and rendering each panel,
trigger and defined audio: These modules create
the respective panel, trigger and prepare the
presentation of the audio, based on the parame-
ters defined on the meta-multimedia document;
11) Startup of the Navi plug-in: This module pre-
pares the system for the utilization of texture-
based videos, animations and web pages, using
the functionalities of Navi.
System Update. This component is in charge to
control and update the multimedia presentation
inside the VE according to possible user interac-
tions. Therefore, the modules of this component are
executed several times, once the modules of the
component startup are executed. Similar to the
modules of startup, the modules of this component
are also numbered following the sequence of execu-
tion inside the architecture:
12) Checking for user interaction on triggers: This
module verifies if a user interacted with a trigger
by clicking over it. This module uses the func-
tionalities of OIS and OGRE;
13) Controlling of the state of each presentation:
which updates the control parameters related to
the state of each presentation (active or inactive),
EMBEDDING MULTIMEDIA CONTENT WITHIN VIRTUAL ENVIRONMENTS - The OGRE Approach
527
according to their triggering type (selection or
proximity) and user interaction;
14) Updating the presentation: This module is re-
sponsible for timing each multimedia presenta-
tion and controlling each media object being pre-
sented. This module also monitors and activates
a trigger object based on the proximity of the
user;
15) Media startup: This module launches the pres-
entation of a media object. For this purpose, the
appropriate plug-in is applied for the presenta-
tion (by the creation of a multimedia texture for
visible objects), such as DevIL for images, Navi
for video, animation and web pages, and OgreAL
for audio;
16) and 19) Media finalization: This module inter-
rupts the presentation of a media object, elimi-
nating the respective texture for visible media
objects. This interruption is done based on the
presentation duration for this object declared on
the meta-multimedia document;
17) Calculation of the degree of transparency of a
texture: This module calculates the degree of
transparency of a given texture, in case there is a
appearing/fading effect associated with it;
18) Deactivation of presentation: This module is
responsible for interrupting a given presentation
by invoking the module media finalization to in-
terrupt the presentation of the media objects cur-
rently being presented;
20) Checking for user interaction on Navi: This
module verifies if there was a user interaction
with a panel whose texture is a Navi web
browser. This module is useful to determine, in
the future, the exact interaction point with the
web browser;
21) Calculation of the intercepted panel character-
istics: This module determines the characteristics
of the 3D object target of the user interaction.
These characteristics include the number poly-
gons, position of polygons, etc. This information
is useful to determine which polygon the user in-
teracted with;
22) Calculation of the exact point of interception:
This module determines the exact point of a user
interaction with the Navi web site presentation.
This is helpful to follows hyper-links defined in
a web site.
23) and 24) Navi update: which is in charge of up-
dating a Navi web browser texture. The texture
is updated as a result of a web page navigation,
introduction of text in the web site, etc.
Some implementation details and issues are further
discussed on the next section.
4 THE PROTOTYPE
IMPLEMENTED
This section illustrates the prototype implemented
by the presentation of an OGRE-Multimedia appli-
cation. This application applied a virtual world (also
called map) result of another project (Cardoso,
2007). This map describes a three-store building
where the capabilities (distribution, communication,
physics, Artificial Intelligence, enhanced textures,
emission of particles, etc.) of OGRE and its related
APIs were exploited. Figure 3 illustrates a global
view of this application.
OGRE-Multimedia was easily integrated to this
OGRE application, where the methods of OGRE-
Multimedia were invoked to enable the presentation
of multimedia content previously defined on the
meta-multimedia document, called “MMDocu-
ment.mmc”. In the case of this VR application, the
first floor of the building is reserved for the presen-
tation of multimedia content is an exposition-like
fashion. This floor is an open-wide area, where in
front of each wall there is a sensitive column which
can be activated by the user’s click or proximity
(depending on its previous configuration on the
meta-multimedia document) in order to trigger the
multimedia presentation on the wall (Figures 3a and
4a).
GRAPP 2008 - International Conference on Computer Graphics Theory and Applications
528
(a) 1
st
Floor (b) 2
nd
Floor (c) 3
rd
Floor
Figure 3: Global view of the virtual three-store building.
(a) (b)
(c) (d)
(e) (f)
Figure 4: Presentation of the multimedia content inside the virtual environment.
Figure 4(b) illustrates the interactive column in the
virtual environment. In particular, this column
(which is represented by a trigger element on the
meta-multimedia document) is able to launch a pres-
entation by a user´s click.
The multimedia presentation is launched after
the user interacts (by clicking) with the sensitive
column (Figures 4c). When the media objects start
to be presented, their level of transparency is gradu-
ally changed producing the effect of fading-in. As
we can see in Figure 4(d), a web-browser is also
presented as one of the multimedia textures. This
browser is rendered by the API Navi which allows
EMBEDDING MULTIMEDIA CONTENT WITHIN VIRTUAL ENVIRONMENTS - The OGRE Approach
529
the user to navigate on the Web. Figures 4(d), 4(e)
and 4(f) present an example of this navigation.
All the media objects presented inside the VE are
synchronized and managed by OGRE-Multimedia
which keeps pace of each presentation starting and
interrupting all the media objects according to their
previous configuration on the meta-multimedia
document. OGRE-Multimedia enables the multime-
dia presentation inside the VE making of it a more
realistic environment and, above all, keeping the
user´s focus.
5 CONCLUSIONS
This paper presented the development of an API for
the presentation of integrated multimedia content
inside Virtual Environments based on the Graphical
Engine OGRE, called OGRE-Multimedia. The inte-
grated presentation of multimedia content inside
VEs relied on the proposal of an XML-based repre-
sentation to describe all the media objects to be
presented and their synchronization relations, the
meta-multimedia document. OGRE-Multimedia can
be applied straightforward in different OGRE Vir-
tual Reality applications since it is the result of an
open-architecture where different APIs were applied
in conjunction to provide the presentation of differ-
ent kinds of media objects including the traditional
images, audio, video, animations, etc., and also
multimedia documents such as FLASH, and web-
browsers as well. Indeed, the combination of Multi-
media and Virtual Reality can be successfully ap-
plied to the design of robust applications where
users feel more comfortable and have their focus
inside the VE, definitely augmenting their feeling of
immersion.
REFERENCES
Chee, Y.S., Network Virtual Environments for Collabora-
tive Learning. Invited talk. In Proceedings of
ICCE/SchoolNet 2001—Ninth International Confer-
ence on Computers in Education, Seoul, S. Korea.
ICCE/SchoolNet (2001) 3–11.
McArdle, G.; Monahan, T.; Bertolotto, M.; Mangina, E.:
A Web-Based Multimedia Virtual Reality Environ-
ment for E-Learning. Proceedings Eurographics 2004,
July 2004, Grenoble, France.(2004)
Halvorsrud, R.; Hagen, S.: Designing a Collaborative
Virtual Environment for Introducing Pupils to Com-
plex Subject Matter. NordiCHI '04: Proceedings of the
third Nordic conference on Human-computer interac-
tion. Tampere, Finland, (2004), 121-130, ISBN:
1581138571.
Bochenek, G.M.; Ragusa, J.M.: Virtual (3D) collaborative
environments: an improved environment for integrated
product team interaction?: Proceedings of the 36th
Annual Hawaii International Conference on System
Sciences, 2003. (2003), 10 pp.
Pingali, G.; Sukaviriya, N.: Augmented Collaborative
Spaces. In proceedings of the 2003 ACM SIGMM
workshop on Experiential telepresence, International
Multimedia Conference. Berkeley, California, (2003).
Carrozino, M.; Tecchia, F.; Bacinelli, S.; Cappelletti, C.;
Bergamasco, M.: Lowering the development time of
multimodal interactive application: The real-life ex-
perience of the XVR Project.: In ACM SIGCHI Inter-
national Conference on Advances in Computer Enter-
tainment Technology, (2005).
Rodrigues, S. G. & Oliveira, J. C.: ADVICe - um Ambien-
te para o Desenvolvimento de ambientes VIrtuais
Colaborativos. XI Simpósio Brasileiro de Sistemas
Mutimídia e Web - WebMedia2005, Poços de Caldas,
MB, Brasil, (2005).
Osawa, N.; Asai, K.; Saito, F.: An interactive toolkit
library for 3D applications: it3d. In proceedings of the
workshop on Virtual environments 2002, ACM Inter-
national Conference Proceeding Series; Vol.23, Barce-
lona, Spain, (2002). 149 - 157
Ficheman, I. K.; Pereira, A. R.; Adamatti, D. F.; Oliveira,
I. C. A.; Lopes, R. D.; Sichman, J. S.; Amazonas, J. R.
A.; Filgueiras, L. V. L.: An interface usability test for
the editor musical. In proceedings: International Con-
ference on Enterprise Information System - ICEIS
2005. Miami USA v. 5, (2005) 122-127.
Garcia, P.; Montalà, O.; Pairot, C.; Skarmeta, A.G.:
MOVE: Component Groupware Foundations for Col-
laborative Virtual Environments. Proceedings of the
4th international conference on Collaborative virtual
environments, Bonn, Germany, (2002) 55 – 62.
Walczak, K., J. Chmielewski, M. Stawniak, S.
Strykowski.: Extensible Metadata Framework for De-
scribing Virtual Reality and Multimedia Contents.
Proceedings of the 7th IASTED International Confer-
ence on Databases and Applications DBA 2006. Inns-
bruck, Austria¸ (2006) 168-175.
OGRE 3D: Open source graphics engine. URL:
http://www.ogre3d.org/
VRML: Virtual Reality Modeling Language. URL:
http://www.w3.org/MarkUp/VRML/
ISO/IEC 19775:2004 — Extensible 3D (X3D). URL:
http://www.web3d.org/x3d/specifications/ISO-IEC-
19775-X3DAbstractSpecification/
Java3D API, URL: http://java.sun.com/products/java-
media/3D/
RealPlayer - Real Networks, 2004. URL: http//
www.real.com/player/.
GRiNS – SMIL 2.0 Player, Home Page. URL: http://
www.oratrix.com/GRiNS/SMIL2.0/.
Adobe – Flash Player. URL:
http://www.macromedia.com/ software/flash/about/
GRAPP 2008 - International Conference on Computer Graphics Theory and Applications
530
TinyXML – XML Parser. URL: http://
www.grinninglizard.com/tinyxml/
OIS – Object Oriented Input System. URL: http://source-
forge.net/projects/wgois/
OgreAL Documentation. URL: http://
www.mooproductions.org/ogreal/documentation/
OpenAL – Cross-Platform 3D Audio. URL: http://
www.openal.org/
DevIL Development Team. DevIL – A full featured cross-
platform Image Library. URL: http://
openil.sourceforge.net/
Simmons, A. Main Page – NaviWiki. URL: http://
navi.agelessanime.com/wiki/index.php/Main_Page
DOM – Document Object Model. URL: http://
www.w3.org/DOM/
Cardoso, G. “UMa Virtual”. B.Sc. dissertation. University
of Madeira (UMa), 2007.
EMBEDDING MULTIMEDIA CONTENT WITHIN VIRTUAL ENVIRONMENTS - The OGRE Approach
531