AN APPROACH TO MULTI-AGENT VISUAL COMPOSITION

WITH MIXED STYLES

Joaquim Reis

Departamento de Ciências e Tecnologias de Informação, ISCTE, Avenida das Forças Armadas, 1600 Lisboa, Portugal

Keywords: Agents, Creativity, Design, Shape Grammars.

Abstract: Applications of computer systems that mix Art, Science and Engineeering have appeared as a result of the

evolution of information technologies in the last three decades. Frequently, they involve the use of Artificial

Intelligence techniques and they have appeared in the fields of music, literary arts and, more recently, visual

arts. This article proposes a computational system based on creative intelligent agents that, by making use of

the shape grammar formalism, can support visual composition synthesis activities. In this system, each

agent gives its creative contribution through a style of its own. Different modes of agent contribution can be

put into perspective like, for instance, cooperative or non-cooperative modes, the resulting composition

emerging from these contributions.

1 INTRODUCTION

We can say that the degree of evolution of the

human species can be measured by the degree of

sophistication of the technology it uses, as well as by

the degree of social organisation. There has always

been a relationship among technology evolution,

human evolution and human societies evolution. The

importance of information technologies has been

increasing progressively. Presently they impact in

every area of society, not only in engineering or

management, just to cite the most evident cases, but,

and more recently, in art too.

Of course that any form of art involves always

som

e kind of technology. To produce his work, the

painter uses tools like brushes, dyes, canvas, for

instance, as well as certain techniques to work with

these tools. But, in this era of information and

computers, the mix of techonology and art is

becoming more and more common, not just at the

level of execution, but also at the level the creative

process. The contribution of Artificial Intelligence,

an area of computer science born in the 1950s (

Rich

1991, Russell 2003), has been significant for this.

This article makes a very brief revision of the

plication of computer systems to the artistic

creative field, next centers around the visual arts,

then introduces the theme of shape grammars, and

finally it proposes a system based on creative agents

to support the visual composition activity.

2 ART, COMPUTERS AND

ARTIFICIAL INTELLIGENCE

There has been, in the last three decades, an

increasing use of information technologies in the

musical arts. Due to the digital technology of

computers it is nowadays possible to store, analyse,

modify and synthesize with an accuracy higher than

that of the human ear system, and even generate

musical compositions. It was in the musical arts that

information technologies have penetrated more

rapidly, perhaps due to the facts that music is of a

more quantitative nature, was more theoretically

mature and the needs of memory and processing

capacity aren’t excessive (at least when compared to

those of visual arts). With the progression of

Artificial Intelligence and the appearance of the

knowledge based systems, in particular expert

systems, new methods of musical composition have

appeared. The process of composition now involves

an interaction between the composer and the

computer system in which the composer generates,

and concentrates on, the original ideas for a

composition, and has support from the computer

system that was programmed with knowledge about

the composition process (

Kurzweil 1990, Miranda

2001).

In the literary arts comp

uters are presently very

useful. For instance, text processors are

indispensable software nowadays and together with

them, there are also a set of software tools like

357

Reis J. (2006).

AN APPROACH TO MULTI-AGENT VISUAL COMPOSITION WITH MIXED STYLES.

In Proceedings of the First International Conference on Software and Data Technologies, pages 357-362

DOI: 10.5220/0001310603570362

 SciTePress

spelling correctors, grammar correctors, style

correctors, and dictionaries, thesaurus and other

kinds of linguistic data bases. In particular, in the

field of natural language processing, Artificial

Intelligence has had an important role (

Winograd

1983

, Allen 1987), in understanding (analysis) as well

as in generation (synthesis). Automatic translation,

data base interfaces in natural language and other

kinds of systems involving dialogue, story

understanding and generation and poetry generation,

are some examples of the rich contribution of

Artifical Intelligence.

In the visual arts, due to a slower evolution of

hardware, in particular respecting to graphic

capabilities (limited graphic resolution of output

devices), only more recently (when compared to the

musical and literary arts) the use of information

technologies has become more attractive. From the

decade of 1990 on we could watch a progression in

the investigation, with its products migrating

progressivelly to software tools to support the

creative activity of designers, architects and

graphical artists in general. The computer supported

visual arts methods of creation vary from free hand

drawing by using the computer screen as it were

paper or canvas, to the most complex image

processing involving the generation of shadows,

surfaces, shapes, colors, and the execution of

translation, rotation, scaling repetition and distortion

operations on shapes. Mixing with the cognitive

aspects, in the field of the creative process,

mathematical processes and Artificial Intelligence

techniques like fractals, chaos theory, genetic

algorithms, rule based systems and artificial life

(

Kurzweil 1990), the latter related to the idea of

agents of Distributed Artificial Intelligence (

Ferber

1999

), have been used. Some of these techniques,

namely artificial life, have migrated to the animation

and cinema field. As examples of the application of

these methods to the visual arts we can point the

works of the painter Harold Cohen, with his

AARON program (

Cohen 1999), and of Leonel

Moura with his system of painter robots, that were

both participants in an recent art exposition in

Lisbon (

Bioart 2005).

3 THE GRAMMARS OF SHAPE

Shape grammars were introduced in the 1970s by

Stiny and Gips (

Stiny 1972). They are similar in

principles to the grammars used in the area of

Artificial Intelligence in natural language

understanding and generation (

Allen 1987), with the

difference of being based not on symbols, but on

shapes (points, lines, two-dimensional and

three-dimensional geometric shapes) as well as, by

extension, also other parameters like dimensions,

colors, etc..

A shape grammar is composed by a basic

vocabulary of shapes, an initial shape and a finite set

of rules that specify how shapes can be generated

from other preexistent shapes, similarly to the

lexicon, the initial symbol and the grammar rules of

a language used in a natural language processing

system. The rules of a shape grammar specify how,

in a composition in progress, shape existing in the

composition can be replaced by new shapes.

Each rule has a left side, pre-condition, or

antecedent, and a right side, action, or consequent.

The left side specifies the pattern for which the rule

is applicable and the right side the respective pattern

for substitution. Briefly, a rule is applicable if there

is a similarity transformation (i.e., an isometry or a

transformation of scale) leading to a match of the

shape of the left side of the rule with a shape

existing in the composition. Rules are applied in a

forward manner (from antecedent to consequent),

like in the production/rule-based forward-chaining

systems of Artificial Intelligence, which perform a

kind of forward inference. When applied, a rule

replaces the shape(s) matched in the composition

with the shapes in its right side. Symbolically, we

can express a rule in the following schematic form:

<shape(s)-to-match> =>

<shape(s)-for-substitution>

Special markers (labels) that aren’t part of the

composition can be used for rule application control

and application termination condition specification.

-> ->

...

a) rule b) resulting composition

Figure 1: Shape grammar example.

For instance, in Figure 1 we show a shape

grammar with only one rule, as well an example of a

possible result from the repeated rule application.

Even a simple shape grammar like this can show

some emergence behavior. This feature depends on

the degree of detail and hierarchy accomodated by

the internal computational representation used for

the geometric shapes in the system (which

influences the ability to consider and process certain

shapes as non-atomic ones, and to recognise shapes

in the composition that weren’t explicitly included in

it).

ICSOFT 2006 - INTERNATIONAL CONFERENCE ON SOFTWARE AND DATA TECHNOLOGIES

358

-> ->

...

a) rule b) resulting composition

Figure 2: Shape grammar example.

For instance, in Figure 2, we show possible

results with the same shape grammar of

Figure 1 with

a system able to recognise emergent shapes.

evaluate candidate

solutions

generate candidate

solutions

yes

goals

accomplished?

Figure 3: Solution generation for a design problem.

Shape grammars with various rules allow for a

greater diversity of alternative compositions. The

process of finding a solution for a design problem

can be seen as a search process which generates

candidate solutions for the problem. This process

takes place in the search space of alternative

possible compositions resulting from the application

of the rules of the shape grammar (

Mitchell 1998). In

its most basic form, this is an iterative

generate-and-test search process that, in each

iteration, produces a possible solution and evaluates

it, testing if the goals of the design problem were

attained. In

Figure 3 we represent the basic structure

of this process.

A system for the execution of this search process

must have a generation component (which is the

shape grammar), a test component (which filters the

acceptable solutions) and, additionally, a control

strategy (rules for determining which way the

system will follow next). This control strategy can

be more than a simple depth-first or breadth-first

search and include heuristics to make the search

process an informed one, causing the system to give

priority to the best alternatives. Such a system could

include specific design knowledge in any of those

three components. In terms of interaction with the

user, different configurations are possible: in one

extreme all functionalities are carried on by a user,

in the other extreme all the process would be

automatic.

The use of shape grammars has been exploited in

applications in different design problems, in the

context of syntesis (generation) and analysis

(interpretation) of visual compositions and also as

means to the description and the representation of

styles, including for didactic purposes and also other

specific applications, for instance in architectural

drawings (

Gips 1999, Tapia 1999, Knight 2000,

Mitchell 1998).

A style is a way of someone doing something

(

Simon 1971) and shows up when that someone

chooses an alternative or a process for generating a

solution. In the field of design, a style is a king of

design knowledge which is a characteristic of a

product, or a set of products, of design, and is

recognisable through the presence of some visual

elements like shape, color, relative position, texture,

dimension, orientation (

Dondis 1973, Bonsiepe

1983

, Wong 1993), as well as certain ways of

combining those elements. Visual compositions can

be generated automatically according to specific

styles. Each style is implemented by the set of rules

of the shape grammar specific to the style.

4 AGENTS, STYLES AND

AGENTS WITH STYLE

The Intelligent Agents technology has been applied

to complex problems involving intelligence and

interaction among agents, human included, namely

in the field of the Internet and in animation in

cinema. It is an intelligent systems technology from

the area of Artificial Intelligence that proposes a

problem solving approach where the problem

solving effort is distributed to a group of intelligent

agents (

Weiss 1999, Ferber 1999). In this context, an

agent is defined as an entity situated in an

environment, that perceives the environment, and

acts in the environment. Not rarely, an agent based

system is composed by a certain number of agents,

i.e., it is a multi-agent system, each with a certain

level of intelligence and a certain level of

communication capabilities. In terms of problem

solving, the idea behind a multi-agent system is to

approach the solution of problems that an isolated

agent wouldn’t be able to solve.

Respecting to the complexity of the internal

architecture, we can classify agents in cognitve and

reactive agents. Cognitive agents tipically have an

architecture containing internal representation

mechanism that allow them to maintain an

up-to-date model of their environment and of

themselves, to reason about the environment and

AN APPROACH TO MULTI-AGENT VISUAL COMPOSITION WITH MIXED STYLES

359

about the results of its own actions in the

environment. This kind of agents is the most

sophisticated, autonomous and intelligent, and they

have capabilities to maintain goals and plan their

actions in order to be able to attain those goals. In

general, multi-agent systems with cognitive agents

are composed by a small number of agents and the

activity in the system comes more from the

cognitive activity of the agents than from the

interaction amongst them. Communication can be

very sophisticated and is usually based on message

passing, and may inlcude standardised protocols.

Agent activity coordination mechanisms can vary

and can go from a total cooperation among agents to

antagonistic forms involving competition and

negotiation.

Reactive agents have a very simple architecture.

They don’t have any internal representation and

reasoning mechanisms (and thus, they can’t have

goals, and they aren’t able to plan their actions) and

its behavior is based on stimulus-response patterns.

In general, multi-agent systems based on reactive

agents are composed by a great number of agents

and the activity of the system stems more from the

interaction amongst them. Communication is very

primitive, and is carried through signal exchange or

by operating and recognising changes in the

environment.

agents

composition

Figure 4: A system for multi-agent generation of visual

composition.

The system we propose in this article is a

multi-agent system based on creative Intelligent

Agents in which shape grammars are used to support

an activity of visual composition. In this system,

sketched in

Figure 4, each agent (see Figure 5) gives

its own creative contribution through a style of its

own. A style is represented by a set of rules

implementing the shape grammar for the style. Each

agent has the set of rules of its own style, and tries to

apply them, depending of the present state of the

composition. Given a set of these agents, each one

with its own style, and given a set of shapes

recognisable in the context of those styles each agent

will apply the rules of its style, whenever possible,

together progressivelly generating a visual

composition, or a set of alternative visual

compositions.

(composition)

inference engine

(forward-chaining)

rule memory

working memory

an agent

Figure 5: Architecture of an agent of the system depicted

in the previous figure (simplified).

As the agents of this system are rule-based they

are, essentially, reactive. However, the rules contain

knowledge, i.e., problem domain knowledge (styles

of visual composition) so, in a certain perspective,

the agents can be considered cognitive. The

interaction among the agents occurs through changes

in their environment, which means the visual

composition in progress. Control of the agent

activity can involve human external intervention.

Several forms of agent activity coordination can be

put in perspective, some involving more cooperative

agent contributions others more competitive in

nature. The resulting composition will emerge from

these contributions.

a) agents and grammars b) alternative compositions

Figure 6: Shape grammar example.

For example, in Figure 6, we represent a set of

agents with simple shape grammars and two possible

visual composition results, one of them is the result

of a less cooperative process (the result of the style

application shows an apparent disorder), and the

ICSOFT 2006 - INTERNATIONAL CONFERENCE ON SOFTWARE AND DATA TECHNOLOGIES

360

other of a more cooperative one (in this case the

result of the style application shows more order).

Possible applications for this kind of system are

visual composition generation with mixed styles,

free generation or controlled and goal oriented

generation (e.g., technical drawing, Web page layout

design).

5 WORK IN PROGRESS AND

FUTURE WORK

The system we have described is still being

conceptualised but there is already some work in

progress leading to its realisation. This work in

progress includes an embryo of an ontology of a

geometric two-dimensional domain, including

geometric points and relative position relations

between geometric points on the x and on the y

direction, line segments and relative position

relations between line segments on the x and on the

y direction, and rectangular shapes and relative

position relations between rectangular shapes on the

x and on the y directions in the plane. In the

following we very briefly show some aspects of this

ontology.

a) horizontally b) verticallly

Figure 7: Relative position relations between two

geometric points on the same line.

Starting from the relative position relations

between two geometric points on the same line

shown in

Figure 7, we have built the 13 basic

relations between two line segments horizontally (x

axis) and vertically (y axis). For reasons of space

economy we only show, in

Figure 8, the relations for

the first case (x axis); for the other case there is an

equal number of similar relations but in the vertical

direction (y axis). These relative position relations

on an axis are inspired by the temporal interval

relations of (

Allen 1983).

Combining these relations pairwise we can get

the relative position relations between rectangular

shapes in the two-dimensional plane, in number 169

(13x13).

x-before(a,b)

x-meets(a,b)

x-overlaps(a,b)

x-starts(a,b)

x-inside(a,b)

x-ends(a,b)

x-equals(a,b)

x-equals(b,a)

x-after(a,b)

x-met-by(a,b)

x-overlapped-by(a,b)

x-started-by(a,b)

x-contains(a,b)

x-ended-by(a,b)

a) direct relation b) inverse relation

Figure 8: Relative position relations between two line

segments horizontally (in the x axis).

For reasons of space economy we only show, in

Figure 9, 13 of those relations.

x-before(a,b)

x-after(b,a)

x-before(a,b)

x-after(b,a)

y-before(a,b)

y-meets(a,b)

y-overlaps(a,b)

y-starts(a,b)

y-inside(a,b)

y-ends(a,b)

y-equals(a,b)

y-equals(b,a)

a b

y-after(a,b)

y-met-by(a,b)

y-overlapped-by(a,b)

y-started-by(a,b)

y-contains(a,b)

y-ended-by(a,b)

b a

a) direct relation b) inverse relation

Figure 9:

Relative position relations between two

rectangular shapes in the two-dimensional plane.

A different aspect in the work in progress is a

small prototype program, being developed in

AN APPROACH TO MULTI-AGENT VISUAL COMPOSITION WITH MIXED STYLES

361

Common Lisp, to implement the forward-chaining

inference engine for experimenting with the set of

rules the shape grammar of an agent. We also intend

to look for software tools that allow suitable domain

knowledge representation and reasoning, preferably

implemented in, and with good integration with the

symbolic environment of, Common Lisp. Possible

candidates are the rule and logic based tools LOOM

(

LOOM 2005) and LISA (LISA 2005), for instance.

In the near future, more immediate work to be

developed has to do with computational geometry

(e.g., geometric shape manipulation, including

similarity transformations, particularly for

recognizing shape patterns in the left side of shape

grammar rules). But the most significant aspects to

give special attention to are the control mechanisms

for rule application, which can be viewed in the

intra-agent and in the inter-agent perspectives, the

latter involving the coordination of the activity of

the agents in the multi-agent system.

REFERENCES

Allen 1987. Allen, James, Natural Language

Understanding, The Benjamin/Cummings Publishing

Company, Inc., 1987.

Allen 1983. Allen, James F., Maintaining Knowledge

about Temporal Intervals, Communications of the

ACM 26 (11), 1983, 832-843.

Bioart 2005. Bioart (Bioarte) 2005 - Lisboa, Galeria

António Prates, Lisboa, Portugal, 2005 (exhibition

catalog, ISBN: 972-8639-03-1).

Bonsiepe 1983. Bonsiepe, Gui, Teoria e Pratica del

Disegno Industriale, Elementi per una Manualistica

Critica, Giangiacomo Feltrinelli Editore, Milano,

1983.

Cohen 1999. Cohen, Harold, http://crca.ucsd.

edu/%7Ehcohen/.

Dondis 1973. Dondis, Donis A., A Primer of Visual

Literacy, The Massachusetts Institute of Technology,

1973.

Ferber 1999. Ferber, Jacques, Multi-Agent Systems, An

Introduction to Distributed Artificial Intelligence,

Addison-Wesley, 1999.

Gips 1999. Gips, James, Computer Implementation of

Shape Grammars, Workshop on Shape Computation,

MIT, 1999 (

http://www.shapegrammar.org/

implement.pdf).

Knight 2000. Knight, Terry, Shape Grammars in

Education and Practice: History and Prospects,

Department of Architecture, MIT, 2000

(

http://www.mit.edu/~tknight/IJDC/).

Kurzweil 1990. Kurzweil, Raymond, The Age of

Intelligent Machines, Chapter Nine: The Science of

Art (http://www.kurzweilai.net/meme/frame.html?

main=/ articles/art0307.html).

LISA 2005. http://lisa.sourceforge.net/.

LOOM 2005. http://www.isi.edu/isd/LOOM/.

Miranda 2001. Miranda, Eduardo Reck, Composing Music

with Computers, Music TECHNOLOGY Series, Focal

Press, Series Ed. Francis Rumsey, 2001.

Mitchell 1998. Mitchell, William J., The Logic of

Architecture, The Massachusetts Institute of

Technology, 1998.

Rich 1991. Rich, Elaine; Knight, Kevin, Artificial

Intelligence, McGraw-Hill, 1991.

Russell 2003. Russell, Stuart; Norvig, Peter, Artificial

Intelligence, A Modern Approach, Prentice Hall, 2003.

Simon 1971. Simon, H.A., Style in Design, Proceedings of

the Second Annual Environmental Design Research

Association Conference, 1971, 1-10.

Stiny 1972. Stiny, G. and Gips, J., Shape Grammars and

the Generative Specification of Painting and

Sculpture, in C. V. Freiman, ed., Information

Processing 71 (North Holland, Amsterdam, 1972), pp.

1460-1465.

Tapia 1999. Tapia, M., A Visual Implementation of a

Shape Grammar System, Environment and Planning

B: Planning and Design, 1999, vol.26, pp.59-73.

Weiss 1999. Weiss, Gerhard (ed.), Multiagent Systems, A

Modern Approach to Distributed Artificial

Intelligence, The MIT Press, 1999.

Winograd 1983. Winograd, Terry, Language as a

Cognitive Process, Syntax, Addison-Wesley

Publishing Company, Inc., 1983.

Wong 1993. Wong, Wucius, Principles of Form and

Design, John Wiley & Sons, Inc., 1993.

ICSOFT 2006 - INTERNATIONAL CONFERENCE ON SOFTWARE AND DATA TECHNOLOGIES

362