NORMATIVE VIRTUAL ENVIRONMENTS

Integrating Physical and Virtual under the One Umbrella

Anton Bogdanovych

, Simeon Simoff

and Marc Esteva

School of Computing and Mathematics, University of Western Sydney, NSW, Australia

Artiﬁcial Intelligence Research Institute (IIIA-CSIC), Campus UAB, Barcelona, Catalonia, Spain

Keywords:

Normative Multiagent Systems, Virtual Institutions, Causal Connection.

Abstract:

The paper outlines a normative approach to the design of distributed applications that must consistently inte-

grate a number of environments (i.e. form-based interfaces, 3D Virtual Worlds, physical world). The appli-

cation of the described ideas is illustrated on the example of a ﬁsh market, which is an application that can

simultaneously be accessed by the people from the physical world, people using form-based interfaces and

people embodied in a 3D Virtual World as avatars. The Normative Virtual Environments approach in this case

allows for maintaining a consistent causal connection amongst all these environments.

1 INTRODUCTION

In order to survive in today’s highly competitive mar-

ket it has become a requirement for the employees

of many companies to be able to conduct their busi-

ness not only directly at the work place, but also at

home, using portable computers, or even “on the go”,

with the help of mobile devices (Umar, 2005). The

fact that many people continuously have to switch be-

tween conducting their activities in the physical envi-

ronment and within various virtual environments has

generated a demand for consistent integration of these

environments and the corresponding interfaces. The

need for bridging the gap between physical and virtual

has become a major concern of researchers working

in such areas as ubiquitous computing (Weiser, 1999)

and augmented reality (Azuma, 1997).

Both ubiquitous computing and augmented reality

applications primarily focus on creating a mixed real-

ity space and hiding the fact that virtual and physical

environments are in fact logically different spaces. In

some situations, however, a clear logical separation

between such spaces is highly desirable.

To illustrate this demand we offer an example of

a ﬁsh market accessible via three different types of

interfaces. First of all, there is a possibility to par-

ticipate in the actual physical ﬁsh market, where sell-

ers can demonstrate the quality of their ﬁsh and let

buyers purchase it, both acting in the physical world.

Those, who can’t be physically present in the actual

ﬁsh market may decide to buy their ﬁsh through the

Internet using a form-based interface. This is partic-

ularly useful for mobile devices, which are employed

by users who are moving around and do not require

a high level immersion from the provided interface.

Finally, there is also a possibility to have a life-like

3-dimensional representation of a ﬁsh market, where

the location of the ﬁsh market and the ﬁsh being auc-

tioned are reconstructed as realistic 3D models based

on the provided speciﬁcations. Such an interface so-

lution is useful for participants wishing to access the

ﬁsh market remotely, while still having a high degree

of realism to help them making informed decisions.

Clearly, each of the three types of interfaces pre-

sented above is oriented towards a separate class of

users. For the users accessing the ﬁsh market re-

motely it is not an option to be present at the physical

location and, therefore, technological integration of

the corresponding virtual environments with the phys-

ical world is not possible. The application of aug-

mented reality or ubiquitous computing techniques is

not beneﬁcial in this case, while the need to logically

integrate all the three environments so that they all

maintain a single logical state is evident. For the par-

ticipants acting within the ﬁsh market remotely it is

absolutely vital to be updated on the changes of the

environment states (i.e. that the ﬁsh they are prepared

to buy was already purchased in the physical world

and is not in stock anymore) as well as it is important

to have a logically consistent environment.

For the development of systems requiring a logi-

cal integration of the physical world with a number of

233

Bogdanovych A., Simoff S. and Esteva M. (2008).

NORMATIVE VIRTUAL ENVIRONMENTS - Integrating Physical and Virtual under the One Umbrella.

In Proceedings of the Third International Conference on Software and Data Technologies - PL/DPS/KE, pages 233-236

DOI: 10.5220/0001900602330236

 SciTePress

virtual environments we propose a new conceptual so-

lution called normative virtual environments (NVE).

These are environments that provide a uniﬁed way

of controlling the interactions of participants through

normative regulation of these interactions and are ca-

pable of maintaining a causal connection (Maes and

Nardi, 1988) between all the existing environments.

This means that the state of any environment could be

changed through acting upon any other environment.

Further we present the architectural solution sup-

porting such causal connection. The proposed ap-

proach is quite general and has a wide range of ap-

plications from electronic markets to e-procurement

systems. For the clarity of this presentation, how-

ever, we limit the problem domain to ﬁsh markets.

The remainder of the paper is structured as follows.

Section 2 introduces the ﬁsh market domain. Sec-

tion 3 explains the concept of normative virtual en-

vironments and outlines the underlying technological

solution. Section 4 shows the implementation of the

ﬁsh market as a normative virtual environment. Fi-

nally, Section 5 presents some concluding remarks.

2 PROBLEM DOMAIN

In the Mediterranean fresh ﬁsh has been traditionally

sold through auction houses. There, ﬁsh is grouped

into sets of boxes (lots) and auctioned following the

Dutch protocol: price is progressively and quickly de-

creased by a small amount until a buyer submits a bid

or the price descent reaches a withdraw price.

Some contemporary ﬁsh markets automate their

selling methods via information technology. Figure 1

presents an example of such contemporary ﬁsh mar-

ket in Spain. Here the auction process is visualized on

the big electronic board. For each lot the board shows

the identiﬁcation number of the product, type of prod-

uct, number of items in the lot and the current price.

The buyers are supplied with infrared devices that can

communicate with the main server. Pressing the “bid”

button on this device would stop the clock, ﬁnish the

auction and announce the buyer who has placed the

bid as the auction winner.

Figure 1: Contemporary Fish Market in Spain.

Even in such contemporary ﬁsh markets the pres-

ence of human buyers at the auction houses is still

necessary. This imposes two main barriers. First, it

restricts the potential buyers to those present in the

auction house. Second, it makes the participation in

several auctions simultaneously costly, as companies

have to send a representative to each one. The elim-

ination of such limitations would be very proﬁtable

for both buyers and sellers. Increasing the number of

buyers makes the market more competitive and thus

increases the buying price to the beneﬁt of sellers. It

also permits the participation of buyers without inter-

mediaries saving costs to the buyers. Next, we outline

the normative virtual environments approach offering

a general solution for eliminating these constraints.

3 NVE

The need for remote participation in ﬁsh markets was

expressed by the MASFIT project (Cun

ı et al., 2004),

which suggests using normative multiagent systems

for accomplishing this task. In particular, the partic-

ipation of buyers in ﬁsh market auctions is mediated

by a trusted third party called the electronic institu-

tion. An electronic institution formalizes the rules of

the ﬁsh market environment and establishes what the

participants are permitted and forbidden to do.

Such rules include the roles the participants can

play, the activities each role can engage into, the in-

teraction protocols associated to each of the activities

and a set of actions that can be performed. The tar-

get environment is separated into a number of logi-

cal groups of activities (scenes). Scenes are intercon-

nected to form a network that represents sequences

of activities, concurrency of activities or dependen-

cies among them. Only participants playing partic-

ular roles are admitted to a given scene, where they

should follow the interaction protocol speciﬁed for

the scene. Once the rules associated with an Elec-

tronic Institution are formalized, a software compo-

nent called AMELI can be used to launch the institu-

tion, let participants join it and communicate within

and will maintain the correct institutional state.

The MASFIT project wasn’t focused on integrat-

ing physical and virtual environments, instead it de-

veloped the mechanisms for software agents to par-

ticipate in a ﬁsh market simulation with similar con-

ditions as the humans do in the physical world. In our

approach we follow the path taken by the MASFIT

project for consistent integration of the virtual end

physical environments. Next, we present our solution

to extending the MASFIT approach.

ICSOFT 2008 - International Conference on Software and Data Technologies

234

3.1 Causal Connection

Our solution to integrating the physical world and

a number of virtual environments is to utilize the

AMELI system and causally connect it to each of the

environments. We call this approach Virtual Institu-

tions. Figure 2 outlines the ﬁsh market scenario im-

plemented using Virtual Institutions. Here a number

of ﬁsh auctions are conducted in the physical world

by a number of auction houses. Each auction house

connects to the virtual institution through the Auc-

tion Software running on their servers. The same Vir-

tual Institution is used by the participants in a num-

ber of virtual environments. On the picture these are

3D Virtual World (3-dimensional representation of a

ﬁsh market), 2D Form-Based Environment (a 2D rep-

resentation suitable for portable and stationary com-

puters) and 2D Mobile Environment (a simpliﬁed 2-

dimensional representation for mobile devices).

Having a causal connection allows any participant

to connect to the system through any environment and

the participants’ representation in every other envi-

ronment will be automatically created and the result

of any action will be propagated there. The tech-

nological component that enables the causal connec-

tion of all the environments to a single institution is

the “Causal Connection Server”. It utilizes the func-

tionality of the AMELI system. For every partici-

pant entering some environment the Causal Connec-

tion Server sends a request to AMELI requesting the

permission to join the institution. Then, every partic-

ipant’s action is ﬁrst validated with AMELI and if ac-

cepted – its performance is propagated to every con-

nected environment. Figure 3 illustrates the causal

connection mechanism for a 3D Virtual World.

The ﬁgure presents an example of how the institu-

tion controls the admission of participants to certain

activities (rooms). An event is generated as the result

of the participant positioning the mouse pointer over

the door handle and clicking the left mouse button (re-

questing the avatar in the Virtual World to open a door

by pushing the door handle). Each event that requires

institutional veriﬁcation has an associated script and

the name of this script is stored in the Action/Message

table. The Causal Connection Server consults with

Auction

Server

Auction

Server

Physical

World

Physical

World

Figure 2: The system architecture example.

Figure 3: Causal Connection.

Figure 4: Causal Connection Server Architecture.

the Action/Message Table to ﬁnd the institutional

message that represents this event in AMELI. In case

such a message is accepted by the AMELI, the re-

sponse message is sent back and the Causal Connec-

tion Server again consults the Action/Message table

to transform this response into the name of the action

that has to be executed in the Virtual World. Next, the

action is performed by executing the corresponding

script. In the given example this action will result in

opening the door and moving the avatar through it.

3.2 Technological Solution

Figure 4 outlines the Virtual Institutions approach

with the detailed focus on the architecture of the

Causal Connection Server. The AMELI system is

featured with two additional components: Federation

Monitor and Institution Monitor. These components

offer an interface to AMELI, allowing the observation

of all messages within a single Electronic Institution

platform. The Causal Connection Server is connected

to AMELI through sockets provided by these moni-

tors and collects available messages.

The Causal Connection Server is supplied with the

number of components. The Agent Launcher com-

ponent creates agents representing humans inside the

AMELI. The Communicator is a gateway between

AMELI, Agent Launcher and the connected environ-

ments. The StringToMessageProcessor translates the

institutional messages into a special Message class

that can be used for easy access to the message param-

eters. Two Message Monitors observe the incoming

messages and forward them for further processing.

The processing of the messages received from

NORMATIVE VIRTUAL ENVIRONMENTS - Integrating Physical and Virtual under the One Umbrella

235

Figure 5: Physical World and Form-Based Environment.

Figure 6: Fish Market as a 3D Virtual World.

AMELI is done by the Action Composer. This com-

ponent consults the Action/Message table to the se-

lected environment. The Message Composer does

the opposite: it transforms events received from

the Atmosphere Player into messages that should be

sent to AMELI for veriﬁcation. In order to create

the correspondence between avatars and agents, the

AvatarID/AgentID table stores their identiﬁers and

helps in making the mapping between them. Such a

mapping is necessary for being able to correctly dis-

patch messages and actions to recipients.

4 FISH MARKET INSTITUTION

To demonstrate how the above-described architecture

can be enacted we illustrate its application to the ﬁsh

market scenario. The developed system consistently

connects all the environments outlined in Figure 2.

4.1 Physical World

Physical world is the environment where several

ﬁsh auctions are conducted by a number of auc-

tion houses. The participants perform their activities

through speciﬁc devices, which communicate with

the auction software through an infrared port. All the

responses of the virtual institution (associated with

the messages of participants from other connected en-

vironments) are communicated back to the auction

software, which then decides which of those to dis-

play on the auction board. Figure 5 a) gives an im-

pression about the interface used by participants.

4.2 Form-based Environments

There are also two form-based environments con-

nected to the system: a simplistic version for partici-

pants with mobile devices and the environment used

by users with personal computers. Apart from the

more functional interface the users of the non-mobile

environment can employ software agents to act in the

system on their behalf. The detailed presentation of

this functionality is given in (Cun

ı et al., 2004). Fig-

ure 5 b) outlines the form-based interface.

As in the previous case the actions of the partic-

ipants are ﬁrst veriﬁed with the institution and only

then propagated to all the connected environments.

The validated actions of the participants from other

environments also result the update of the interface.

4.3 3D Virtual World

The 3D Virtual World is presented in Figure 6. Here

all the participants are visualized as avatars and can

observe each other’s actions. In the current limited

implementation we do not present a detailed 3D vi-

sualization of the auctioned lots, but simply put its

picture on the auction board (together with all the re-

lated information and the price). When a buyer de-

cides to purchase the lot – its avatar is raising a hand

to give a visual clue to other participants. The details

of the auction winner are then displayed on the auc-

tion board. To maintain the causal connection, every

participant from any connected environment is visu-

alized as an avatar. Participants’ actions are also con-

sistently mapped to the avatars and the environment.

5 CONCLUSIONS

The paper has presented normative virtual environ-

ments as an approach for consistent logical integra-

tion of physical and virtual environments. The tech-

nological solution supporting this concept was illus-

trated on the example of a ﬁsh market, which can be

jointly used by participants from the physical world

and a number of remote virtual environments.

REFERENCES

Azuma, R. (1997). A survey of augmented reality. Pres-

ence, 6(4):355–385.

Cun

ı, G., Esteva, M., Garcia, P., Puertas, E., Sierra, C.,

and Solchaga, T. (2004). MASFIT: Multi-Agent Sys-

tems for Fish Trading. In Proceedings of the European

Conference on Artiﬁcial Intelligence, pages 710–714.

Maes, P. and Nardi, D. (1988). Meta-Level Architectures

and Reﬂection. Elsevier Science Inc., NY, USA.

Umar, A. (2005). IT Infrastructure to Enable Next Gen-

eration Enterprises. Information Systems Frontiers,

7(3):217–256.

Weiser, M. (1999). Some computer science issues in ubiq-

uitous computing. SIGMOBILE: Mobile Computing

and Communications Review, 3(3):12.

ICSOFT 2008 - International Conference on Software and Data Technologies

236