A Multi-level Model of Motivations and Valuations for Cognitive Agents

Samer Schaat, Klaus Doblhammer and Dietmar Dietrich

Institute of Computer Technology, Vienna University of Technology, Gusshausstrasse 27-29, Vienna, Austria

Keywords:

Cognitive Architectures, Artiﬁcial General Intelligence, Cognitive Agents, Motivations, Valuations, Emotions.

Abstract:

In developing cognitive agents using a functional model of the human mind as their decision unit, a model

of motivations and valuations is needed as the basis for the agents’ decision making. This enables agents

to cope with their internal and external world while pursuing their own agenda. We show that a technical

model based on the psychoanalytical drive concept and Damasio’s neuro-biological ﬁndings is appropriate for

human-inspired cognitive agents. In particular, after overcoming the hurdles of interdisciplinary work between

hermeneutic and axiomatic approaches, a transformation of psychoanalytical and neuro-biological concepts

into an consistent and deterministic model solves the problem of motivations and valuations in artiﬁcial cog-

nitive agents. This multi-level model is presented, in which multiple principles and inﬂuences of valuation are

used to incrementally generate and decide an agenda for the agent’s behavior.

1 INTRODUCTION

When following a holistic and integrated approach

to develop a functional model of the human mind as

a decision unit for cognitive agents, basic questions

are (1) what is the source for the agent’s agenda and

(2) how can the agent cope with the external world

while pursuing this agenda? These are key ques-

tions since a cognitive agent is also an autonomous

agent, i.e. a system situated within an environment

which it acts on in pursuit of its own agenda (Franklin,

1997). Additionally, when using the human mind as

an archetype, cognitive processes are dependent on

motivations and valuations, e.g. decision making is

based on emotions (Damasio, 2003). The ﬁrst ques-

tion above can thus be phrased as the problem of mo-

tivation, i.e. what should agents do and why should

they do anything at all? The second question can

be formulated as the problem of valuation in cog-

nitive agents, i.e. the determination of relevance of

processed data (e.g. objects, plans, actions) for the

agent’s motivations. Hence, these two questions can

be regarded as interdependent, since valuation is a

means of fulﬁlling motivations.

When considering a holistic model of the human

mind as a decision unit in cognitive agents, embodi-

ment must be considered when tackling the problem

of motivation and valuation, in particular the inter-

action between the psyche and body. In this sense,

autonomy and embodiment are foundations of higher

cognitive functions such as planning, and motivations

are based on the agent’s bodily state on the one hand

while serving as the basis for planning on the other

hand. Hence, the model presented follows a genera-

tive approach, i.e. motivations are generated based on

the agent’s bodily needs and in turn are the source for

generating goals for planning.

As a cognitive theory the ARS project has chosen

the second topographical model of Sigmund Freud

(Freud, 1915), since it is the only appropriate holis-

tic and functional model of the human mind (Dietrich

et al., 2009) (see 2) and is also backed up by neuro-

scientiﬁc data (Solms and Turnbull, 2002). However,

it can only be used as a framework and general basis

and is not sufﬁcient to develop a human-inspired cog-

nitive architecture. For instance, Damasio’s concept

(Damasio, 2003) is used to support the concretization

of motivations and valuations in the ARS model.

2 RELATED WORK

Motivations and valuations are key methods used by

humans to cope with and adapt to the world. Some

robot architectures, such as (Parisi and Petrosino,

2010)(Konidaris and Barto, 2006), consider drives for

motivational purposes. Nevertheless, a bionic and

functional approach to motivations and valuations are

often neglected in current cognitive architectures. Es-

tablished cognitive architectures, such as ACT-R (An-

255

Schaat S., Doblhammer K. and Dietrich D..

A Multi-level Model of Motivations and Valuations for Cognitive Agents.

DOI: 10.5220/0004916502550260

In Proceedings of the 6th International Conference on Agents and Artiﬁcial Intelligence (ICAART-2014), pages 255-260

ISBN: 978-989-758-016-1

 2014 SCITEPRESS (Science and Technology Publications, Lda.)

derson et al., 1997), focus on ‘higher’ cognitive func-

tions without considering the basis for these higher

functions, i.e. generation and valuation of goals. Only

a few cognitive architectures, such as MicroPsi (Bach,

2011) and CLARION (Sun, 2007), also focus on mo-

tivational aspects. However, these approaches do not

consider a generative and embodied approach. For

instance, MicroPsi distinguishes physiological, cog-

nitive and social needs, which are all hard-wired. In

the embodied approach of the ARS model all needs

are grounded in physiological needs. Based on so-

cial rules and the agent’s memories of how to sat-

isfy physiological needs in the short and long term,

they are transformed into complex motivations and

goals. CLARION in fact considers the derivation of

‘secondary’ drives from ‘primary’ drives, but uses the

drive-concept more in terms of a behavioristic ap-

proach, since it is based on Hull’s concept of drives

(Hull, 1951). In contrast to the concept of drives in

ARS, the drives’ strength in CLARION is determined

by an internal deﬁcit and an external stimulus (e.g.

food in case of hunger). In this regard, the ARS multi-

level approach enables a more ﬂexible motivational

system: generally speaking, a ﬁrst level of motivation

and valuation considers only how to satisfy its motiva-

tion best according to the agent’s memories and with-

out consideration of the external world. After further

levels of motivation and valuation, it is only in the

ARS secondary process that the reality imposed by

the external world is considered.

Emotions are a central aspect of motivations and

valuation; they can be seen in general terms as a rep-

resentation of an agent’s internal state (emotions as

embodied information of valuation and importance)

(S. C. Marsella and Petta, 2010). Recently, various

stand-alone computational models of emotion have

been developed, i.e. they are usually not integrated

into a full-ﬂedged cognitive architecture. Due to its

focus on the connection between emotion and cogni-

tion (S. C. Marsella and Petta, 2010), cognitive ap-

praisal theory is currently the dominant theory for

computer models of emotions. In this theory, emo-

tions emerge from the appraisal of external events

and situations under the consideration of the agent’s

beliefs, desires and intentions. The result of such

appraisal is the triggering of cognitive responses, in

particular coping strategies (e.g. planning, procrasti-

nation) (S. C. Marsella and Petta, 2010). Appraisal

theories focus on determining a sufﬁcient set of ap-

praisal criteria to explain the elicitation and differen-

tiation of emotions. A widely used model of cog-

nitive appraisal theory is the OCC model (Ortony

et al., 1990) (e.g. used in EMA (Marsella and Gratch,

2009)), with appraisal criteria such as unexpected-

ness, level of appeal and desirability. In appraisal

theory, emotions are only elicited by evaluations of

external events and hence only considered for interac-

tion purposes. This is a major difference to the ARS

model, where the generation of emotions is inﬂuenced

by external events on the one hand and based on the

agent’s drives on the other hand. In particular, this

means the agent may be in an emotional state even

without consideration of the external world. Another

category of emotion theories follows a dimensional

approach and models emotions not as discrete enti-

ties but as points in a continuous dimensional space

(S. C. Marsella and Petta, 2010). A typical example

for this is the three-dimensional PAD model (Mehra-

bian and Russell, 1974), with pleasure (a measure of

valence), arousal (indicating the level of affective ac-

tivation) and dominance (a measure of power or con-

trol) as dimensions.

3 ARS APPROACH

The leitmotif of the ARS approach (Dietrich et al.,

2009) is to model the functions behind the desired ca-

pabilities that generate behavior instead of simply de-

scribing behavior. This complies with the generative

and broad approach of Artiﬁcial General Intelligence

(AGI).

In the ARS project, the human cognitive architec-

ture (i.e. the brain) is considered as an information

processing system that stores, manipulates and trans-

fers information. Following the standard approach in

computer technology, it is described in a top-down

design process using a layered model, starting with

three functional layers. The ﬁrst layer, the neurons,

can be described as hardware under consideration of

the laws of physics. The next layer is called the neu-

rosymbolic layer, which handles the symbolization of

the neural layer. The third layer represents the psy-

chic layer, which is described in functional terms on

an algorithmic level. Following a monistic view, psy-

che and brain are of course the same, with only differ-

ent models being used in their respective descriptions.

Since only a functional description is relevant for ar-

tiﬁcial systems (but not how these functions are im-

plemented), the ARS project focuses on the descrip-

tion of the psychic apparatus. As mentioned in the in-

troduction, the second topographical model of Freud

(Freud, 1915) was chosen as a general framework,

which uses the abstract functions Id, Ego and Super-

Ego to describe the human psychic apparatus. The Id

represents drives, which are in effect bodily demands

coming from internal sensors, the Super-ego repre-

sents internalized moral demands and the Ego me-

ICAART2014-InternationalConferenceonAgentsandArtificialIntelligence

256

diates between the Id and the Super-Ego under con-

sideration of the external environment. These three

abstract functions are concretized incrementally until

their description can be used for a functional model

that can be implemented in a computer simulation.

The most concrete level of the ARS functional model

is shown in Fig. 1 using a track view in which func-

tion modules are summarized into tracks.

Figure 1: ARS functional model - track view.

4 PSYCHIC INTENSITY

Following an interdisciplinary approach, the general

psychoanalytical framework is described based on

terminology used in the ARS project, and then con-

cretized by a technician in the subsequent sections to

a coherent and deterministic multi-level model of mo-

tivations and valuations.

4.1 Economic Point of View

The economic point of view describes functionalities

that focus on circulation, distribution and transforma-

tion of quantities of excitation. Freud called these

quantities ’psychic energy’ , which became a funda-

mental term in psychoanalysis (Freud, 1940, p. 158)

to designate a representation of a bodily drive-tension,

which is created by differences in homeostatic levels.

The quantiﬁcation of ’psychic energy’ in the psychic

apparatus is called quota of affect. With this value,

psychic content can be used for cathexis, i.e. ’psychic

energy’ is assigned to certain psychic content via its

quota of affect. This represents the valuation of psy-

chic content. Following the so-called pleasure prin-

ciple and the principle of reality, cathexes are essen-

tial for a psychic economy since they provide the di-

rection for decisions, actions and plans that an agent

executes or wants to execute. The pleasure principle

says that the entirety of psychic activity is aimed at

avoiding unpleasure and generating pleasure, while

the principle of reality is a modiﬁcation of it as as-

pects of reality are involved in this design of activ-

ity. Pleasure arises from the discharge of ’psychic en-

ergy’; unpleasure is equivalent to the sum of all quotas

of affect within the psychic apparatus. Since emotions

are experienced as an aspect of pleasure and unplea-

sure, the term ’psychic energy’ is also a basic term

for the generation of emotion and feeling, which addi-

tionally inﬂuence decision making and bodily states.

An adult psyche does not assign all ’psychic en-

ergy’ to psychic content. Instead, a certain amount

of it is available for particular psychic functions, the

so-called Ego-functions. This ’neutralized drive en-

ergy’ can be regarded as an engine for these functions,

which deal with thinking, focus of attention, planning,

adaption to reality and other concepts. This implies

discharge and produces pleasure as well.

4.2 Psychic Intensity in the ARS Model

For an embodied approach, the connection between

body and psyche, i.e. between the different descrip-

tion languages of physics and information theory (see

2), must therefore be considered. This is done by use

of the neurosymbolic layer. For the questions of mo-

tivation and valuation, this means that we must con-

sider the bodily source of motivation and valuation

(i.e. the connection between body and psyche), and

in particular the symbolization of the bodily source

and its psychic representation. The use of the term

‘psychic energy’ shows that these aspects are not con-

sidered in classical psychoanalysis. Nevertheless, in

a concrete technical model they must be concretized

and structured using the appropriate level and lan-

guage of description.

But the term ‘psychic energy’ is misleading and

not compliant with the ARS approach, since it uses

the description language of the physical world which

is only appropriate for the ﬁrst layer (see above). But

it is misleading even as a metaphor for describing the

concept’s functionality, e.g. the abstract and func-

tional concept of money is better suited as a metaphor.

Since the essence of the concept is valuation, i.e. the

quantiﬁcation of the importance of processed content,

the term psychic intensity was chosen as an appropri-

ate representation of the underlying functionality. As

with money, the physics of psychic intensity is irrele-

vant for its description; only a functional description

can cover the essence of the concept. In particular,

psychic intensity serves as an information value. In

the algorithmic level of the ARS model, i.e. the psy-

chic layer, the concept of psychic intensity is used as

the link between bodily needs (i.e. homeostatic needs)

and motivations, in particular as a valuation system

for determining the relevance of memories and per-

ception and as a foundation for decision making.

AMulti-levelModelofMotivationsandValuationsforCognitiveAgents

257

Next, beginning with the concept of psychic in-

tensity we describe the coherent multi-level model of

motivations and valuations.

5 A MULTI-LEVEL MODEL

The foundation for motivations and valuations in the

ARS model is provided by the concept of drives. This

is true not only for the ARS primary process, their

principal domain, but also for the secondary process,

which has to cope with the demands generated by the

drives under consideration of social rules, the agent’s

current plans and the external world. Based on the

drives, an extended representation of motivations and

valuations is given by emotions, which additionally

consider the agent’s perception. Hence motivations

and valuations are represented in the ARS agent us-

ing a multilevel model, which breaks the complexity

of the motivation system up into several levels. This

allows an incremental generation of motivations and

valuations, where motivations and valuations are ex-

tended and/or adapted using different principles (rules

of the ARS primary and secondary process) and inﬂu-

ences (memories, perception, plans). Psychic inten-

sity is thus extended to - and concretized on - multiple

levels, which nonetheless all serve the same purpose:

motivation and valuation.

Such a multi-level model of motivations and valu-

ations based on the agent’s body and memories com-

plies with Damasio’s model (Damasio, 2003), where

the homeostatic mechanisms for the regulation and

control of the organism’s processes are considered to

be multiple but coherent levels which (in terms of evo-

lution theory) evolved as adaptations for coping with

the world.

Figure 2: Different valuation-principles and inﬂuences

(memories and perception) are considered.

Each homeostatic need is represented by an or-

ganic tension value (e.g. the volume of stomach con-

tent). As described above, this value is represented as

psychic intensity in the psychic layer. A range of 0-1

is used for the quantiﬁcation of psychic intensity. A

personality parameter determines the mapping from

drive tension to psychic intensity, i.e. in which in-

tensity the agent’s psyche represents the bodily need.

Next, in the process of generating a drive representa-

tion (see Ch. 5), psychic intensity is represented as

a quota of affect which is used to valuate memories,

e.g. objects and actions that have previously satisﬁed

the drive, and hence represents the ﬁrst level of valua-

tion. This ﬁrst level of valuation lead to the ﬁrst level

of motivations, drive representations. This valuation

is used under the consideration of the pleasure prin-

ciple, in particular to determine objects that have pro-

vided the best satisfaction of the drive (i.e. the bodily

need) according to the agent’s memories without the

consideration of reachability. The valuation of con-

tent may be changed by defense mechanisms, which

take into account internalized rules such as social and

moral rules (e.g. a rule that forbids eating in a spe-

ciﬁc situation may result in changing the valuation of

food objects). Next, through the function of ‘neutral-

ization’ , an amount of the drive’s quota of affect, de-

termined by a personality parameter, is reduced and

used as neutralized intensity for a possible extension

of already valuated content, using the reality principle

in the ARS secondary process as a valuation criterion.

This is primarily the consideration of the compatibil-

ity of the valuated objects and actions with the current

reality provided by the external world and with the

agent’s current plans. In general, logic and time are

considered in this valuation process. This additional

valuation transforms a drive representation to a valu-

ated goal, where the goal’s object, action and plan are

valuated. The valuation of goals can be extended by

emotions, which are another level of valuation and are

generated based on the remaining quota of affect (af-

ter neutralization) and memorized emotions that are

activated by perception. An overview of this incre-

mental multi-level valuation is provided in Fig. 2.

In the next section, the key elements of motiva-

tions and valuations, drives and emotions will be out-

lined.

5.1 Drive Representations: Motivations

in the ARS Model

The representation of a drive, i.e. a somatic organ

tension, in the psyche is provided by a drive repre-

sentation, which consists of a drive source (the or-

ganic origin), a drive aim (an activity that decreases

the organic tension) and a drive object (with which

the drive can achieve its aim). The action given by

the drive aim is always a strategic action, which is

implemented as a tactical plan. In particular, how this

action is fulﬁlled (the tactic) is the domain of reason-

ing in the ARS secondary process. Self-preservation

and sexual drives are distinguished. The drive source

for the former is the respective organ (e.g. stom-

ICAART2014-InternationalConferenceonAgentsandArtificialIntelligence

258

ach), the drive source for the latter is called libido,

a value that represents hormonal sources. Depend-

ing on erogenous zones on four different body parts,

four so-called partial sexual drives are distinguished:

anal, phallic, oral and genital. Only the last of these

drives reproduction, while the others may be regarded

as intrinsic motivations , i.e. pleasure gain from sat-

isfying sexual drives is independent from the satis-

faction of bodily needs. To complete the represen-

tation of a drive, the agent has to choose an object

and action with which the bodily need, represented by

the drive, can be satisﬁed. Psychoanalytic theory de-

scribes this process as cathexis, which corresponds to

the valuation of objects and actions. Drive objects and

aims that were best valuated in the past (i.e. brought

the amounts of highest satisfaction) are valuated best.

In the ARS agent, only valuated data can be consid-

ered in the function modules. The valuation done by

cathexis is based on (1) the memorized valuation and

(2) the quota of affect available. Memorized valuation

is represented by an association between a memorized

drive representation and an object. The valuation pro-

cess consists of the following steps: (1) Activation of

memorized drive representations with the search cri-

teria of drive source and drive component using an

associative search algorithm, (2) Assignment of the

drive representation’s provided quota of affect to the

data associated to the memorized drive representa-

tions, (3) Consideration of accumulation by multiple

valuations.

5.2 Emotions and Feelings

Based on the agent’s drives and inﬂuenced by mem-

ories activated by perception, emotions are an addi-

tional representation of the agent’s internal state un-

der consideration of the external world - in particu-

lar, the agent’s experience with similar situations. The

transformation of emotions to feelings, based on con-

scious perception thereof (Damasio, 2003), forms a

new valuation-level, which may change the valuation

of goals.

Currently basic emotions (anger, mourning, anxi-

ety, joy, saturation, and elation) are considered in the

ARS agent, which are generated based on the agent’s

actual drives and perception, in particular the memo-

rized emotions associated with perception. They are

determined by four factors, namely unpleasure (the

sum of the quotas of affect), pleasure (the value gen-

erated after reducing psychic intensity), the sum of all

aggressive quotas of affect and the sum of all libidi-

nous quotas of affect. Two sources for these factors

are considered: (1) drive representations and (2) the

factors of memorized emotions that are activated by

perception. Dependent on the dominance of a subset

of these four factors, which also determine the inten-

sity of the emotions, different emotions are generated.

Figure 3: Feedback on body or psyche.

5.3 Motivation Control Loop

In terms of a control loop, the feedback on the source

of motivations must be considered. In the embodied

approach of the ARS model, this can occur in two

ways: the ﬁrst possibility is the implicit reduction of

psychic intensity (the source of motivations) through

reduction of the respective organ tension via a speciﬁc

action. For instance, eating inﬂuences the state of the

stomach, which in turn changes drive tension and the

psychic representation thereof, psychic intensity. In

case of actions that do not produce direct feedback on

the bodily source (e.g. beating), a second possibility

of feedback is provided by the perception of the cur-

rently executed action. Since this feedback occurs on

the psychic layer, it affects psychic intensity directly.

6 SIMULATION

To ﬁnd the functions behind any given behavior, a

use-case driven approach is employed: the behav-

ior to be generated by the functions of the ARS

model and their corresponding inner processes are de-

scribed together with psychoanalysts and structured

as use-cases. Thus requirements are speciﬁed that are

then fulﬁlled by functions and data which generate

the described behavior in an agent-based simulation.

For simulation the multi-agent simulation framework

MASON is used. After presenting a simulation of

an agent-object interaction in, here an overview of

preliminary results of simulating a social interaction

is given next. The use-case deﬁnes a situation with

two agents (a green one and a red one) and a Wiener

Schnitzel as food source.

The green agent’s drive state, especially the dom-

ination of aggressive drives (see Fig. 4), the mem-

orized valuation of the Schnitzel and the red agent,

and the green agent’s personality parameters result in

the green agent beating the red agent. In particular,

due to the green agent’s personality parameters, the

quota of affect of aggressive drive representations is

AMulti-levelModelofMotivationsandValuationsforCognitiveAgents

259

higher than that of libidinous drive representations,

and the quota of affect is only reduced minimally by

increasing neutralized intensity. Thus valuation by

the green agent occurs primarily via cathexis follow-

ing the pleasure principle, and due to the green agent’s

memories, the drive object ‘red agent’ and -aim ‘beat’

(which are transformed into the respective goal), are

valuated the highest.

Figure 4: Beating in simulation step 21 reduces aggressive

drives, which increases pleasure. Together with memorized

valuations activated by perception this leads to the genera-

tion of the shown emotional state.

The emotions shown in Fig. 4 represent the

drive representations and memories activated within

the green agent by perceiving the Schnitzel and the

red agent. Since the red agent looks similar to the

red agent’s memorized brother - who, in combina-

tion with a food source, is associated with anger -

and aggressive drive representations dominate, anger

also dominates; the low libidinous drive representa-

tions, together with the memorized joy activated by

perceiving the Schnitzel, lead to a low generation of

joy compared with beating the red agent. Beating

the red agent leads to reduction of the corresponding

psychic intensity and hence to generation of pleasure

within the green agent, which in turn increases joy

and reduces anxiety.

7 CONCLUSIONS

Valuation of objects and actions using the concept

of psychic intensity lead to the generation of drive

representations from bodily needs, which together

with emotions represent the agent’s motivations and

ground its autonomy in its body. This shows that

the development of a consistent and deterministic

model of psychoanalytic and neuro-biological con-

cepts for the simulation in agent-based systems is pos-

sible. In particular, the connections between body,

drives, emotions and goals and the feedback on the

body are considered in a deterministic and cyclic fash-

ion. The presented multi-level model, in which dif-

ferent valuation criteria are considered, shows how

human-inspired complex valuation enables an incre-

mental determination of the relevance of objects, ac-

tions, goals, and plans to reach them.

REFERENCES

Anderson, J. R., Matessa, M., and Lebiere, C. (1997). Act-

r: A theory of higher level cognition and its rela-

tion to visual attention. Human-Computer Interaction,

12(4):439–462.

Bach, J. (2011). A motivational system for cognitive ai. In

J. Schmidhuber, K. R. T. and Looks, M., editors, Pro-

ceedings of the 4th international conference on Artiﬁ-

cial general intelligence. Springer, Berlin.

Damasio, A. (2003). Looking for Spinoza: Joy, Sorrow, and

the Feeling Brain. Harvest Books.

Dietrich, D., Fodor, G., Zucker, G., and Bruckner, D.

(2009). Simulating the Mind - A Technical Neuropsy-

choanalytical Approach. Springer, Wien.

Franklin, S. (1997). Autonomous agents as embodied ai.

Cybernetics and Systems, 28(6).

Freud, S. (1915). The Unconscious, volume XIV (1914-

1916) of On the History of the Psycho-Analytic Move-

ment, Papers on Metapsychology and Other Works.

Vintage.

Freud, S. (1940). An outline of psycho-analysis. Interna-

tional Journal of Psycho-Analysis, 21:27–84.

Hull, C. (1951). Essentials of Behavior. Yale University

Press, New Haven.

Konidaris, G. and Barto, A. (2006). An adaptive robot moti-

vational system. In From Animals to Animats 9, pages

pp 346–356. Springer, Berlin.

Marsella, S. C. and Gratch, J. (2009). Ema: A process

model of appraisal dynamics. Cognitive Systems Re-

search, 10(1):70–90.

Mehrabian, A. and Russell, J. (1974). An approach to envi-

ronmental psychology. MIT Press, Cambridge.

Ortony, A., Clore, G. L., and Collins, A. (1990). The Cog-

nitive Structure of Emotions. Cambridge University

Press, New York.

Parisi, D. and Petrosino, G. (2010). Robots that have emo-

tions. Adaptive Behavior, 18(6):453–469.

S. C. Marsella, J. G. and Petta, P. (2010). Computational

models of emotion. In K.R. Scherer, T. B. and Roesch,

E., editors, Blueprint for Affective Computing. Oxford

University Press, New York.

Solms, M. and Turnbull, O. (2002). The Brain and the Inner

World: An Introduction to the Neuroscience of Sub-

jective Experience. Karnac/Other Press, Cathy Miller

Foreign Rights Agency, London, England.

Sun, R. (2007). The motivational and metacognitive control

in clarion. In Gray, W. D., editor, Integrated Models

of Cognitive Systems, pages 63–75. Oxford University

Press, New York.

ICAART2014-InternationalConferenceonAgentsandArtificialIntelligence

260