Organizational Structure and Responsibility

Lamb

er Royakkers

, Davide Grossi

and Frank Dignum

Eindhoven University of Technology, The Netherlands

Utrecht University, The Netherlands

Abstract. We analyze the organizational structure of multi-agent systems and

explain the precise added value and the effects of such organizational structure

on the involved agents. To pursue this aim, contributions from social and organi-

zation theory are considered which provide a solid theoretical foundation to this

analysis. We argue that organizational structures should be seen along at least

three dimensions, instead of just one: power, coordination, and control. In order

to systematize the approach, formal tools are used to describe the organizational

structure as well as the effect of such structures on the activities in multi-agent

systems, and especially the responsibilities within organizations of agents. The

main aim of the research is to provide a formal analysis of the connections be-

tween collective obligations to individual responsibilities. Which individual agent

in a group should be held responsible if an obligation directed to the whole group

is not fulﬁlled? We will show how the three dimensions of an organizational

structure together with a speciﬁc task decomposition determine the responsibili-

ties within a (norm-governed) organization.

1 Introduction

The concept of responsibility is a central concept to all legal systems and norm-governed

organizations. Analyzing this concept is therefore fundamental if we aim at improving

the behavior of these systems or organizations. Obtaining a formal representation of

responsibility, however, is quite complex because of the very different meanings of this

concept can take. Our concept of responsibility is restricted to the analysis of organiza-

tional performance. Therefore, we clarify and classify some meanings of responsibility

and we relate them to the three relevant dimensions of an organizational structure we

isolated in [1]. These three relevant dimensions are power, coordination and control,

with their matching actions ’to delegate’, ’to inform’ and ’to monitor’. The coordi-

nation actions are actually only one type of meta actions that should be considered.

Besides the plan to achieve the content of the obligation the group should create that

plan, allocate agents to parts of the plan, create a plan for what to do when the original

plan fails, etc. These meta actions should also be coordinated again creating in the end

an inﬁnite regression of meta actions. In this paper we will not take all these layers into

account, but will limit us to the coordination actions that are necessary to indicate the

several notions of responsibility.

In this article we will import some of the studies in organizations and social theory

to describe a more rigorous foundation of organizational structures in MAS, which will

Royakkers L., Grossi D. and Dignum F. (2006).

Organizational Structure and Responsibility.

In Proceedings of the 3rd International Workshop on Computer Supported Activity Coordination, pages 65-74

DOI: 10.5220/0002482000650074

 SciTePress

be informally and formally exposed in Section 2. In order to describe organizational

structures we have to ﬁrst describe exactly what the meaning is of the relations that

form the structure. E.g. what is the meaning of an “power” relation and, maybe even

more importantly, what are the consequences of the existence of such a relation between

two agents? We will introduce a modal logic for this characterization. Several notions of

responsibility (given a plan) will be discussed formally in Section 3. How the individual

responsibilities relate to the underlying structure of an organization will be discussed

in Section 4. In the last section, we will draw some conclusions and give directions for

future research.

2 Organizational Structure and its Logic

The thesis we hold in this paper, which is inspired by foundational work on social and

organization theory like [2–5], is that organizations do not exhibit one single structural

dimension, but that they are instead multi-structured objects. In particular, we view or-

ganizational structure as hiding at least three relevant dimensions which we call: power,

information and control. We will analyze power in relation with the delegation activity,

coordination in relation with the knowledge and information issues, and control in re-

lation with the monitoring and recovery issues. As a result of this analysis, organizations

will be represented as explicitly displaying a triple structure constrained on the basis of

the interplay between the three notions of power, coordination, and control. It is the

structure based on goal or task decomposition and related to power and delegation ca-

pabilities between the roles. Although we do not pretend to give full deﬁnitions of these

relations (see [6,7] for some more elaborate deﬁnitions of the delegation and power

relations) we will characterize these relations in terms of some of their consequences

for the agents, enacting the roles, between which these relations are deﬁned.

To describe an organization and its structure we will use a (typed) multi-modal

propositional logic. The organizational structures are denoted through the special propo-

sitions Power(r, s) to indicate that ‘the agent enacting role r has the agent enacting role

s in its power’ (i.e. the agent playing role r can delegate goals to the agent playing role

s), Coordination(r, s) to indicate that ‘the agent enacting role s has access to the in-

formation that is accessible to the agent enacting role r’, and Control(r, s) to indicate

that ‘the agent enacting role r controls the agent enacting role s’ (i.e. the agent playing

role r is responsible for the agent playing role s). Note that these relations are deﬁned

on roles. We denote the fact that agent i enacts role r, i.e., is a role enacting agent

([8]), by the special proposition rea(i, r). Furthermore we use a modal operator K

for

knowledge accessible to an agent i. For the characterization of the organization struc-

tures we build on dynamic logic ([9]). Dynamic formulas such as [ξ]φ, meaning that

after each execution of ξ formula φ holds, where ξ is a parameterized construct of the

type i : α denoting the performance of action α by agent or role i, or a composed

construct such as: i : α

; j : α

(subsequent performance), i : α

&j : α

(parallel

performance),

i : α (i refrains from performing α). The formal semantics is given by

means of a Kripke structure where there are accessibility relations R

i:α

associated with

each parameterized action i : α.

We can now give a full formal deﬁnition of the syntax of our description language

Org:

Deﬁnition 1. (Syntax of Org)

Given a ﬁnite set AR of role names, a ﬁnite set Ag of agent names, a countable set P

of atomic propositions, a ﬁnite set of parameterized actions A (in general the elements

of A are denoted by i : α with i in Ag the performing agent of α) containing at least i :

achieve(φ), i : delegate(j, φ), i : inform(j, φ) and i : monitor(φ), the propositional

language L

that is built up from atoms P

, the countable set P = L

∪ {Power(r, s),

Coordination(r, s), Control(r, s), rea(i, r) | r, s ∈ AR, i ∈ Ag, φ ∈ L

}, the

admissible formulas are recursively deﬁned as follows:

– P ⊆ Org

– If φ and ψ ∈ Org, then φ ∧ ψ, ¬φ ∈ Org

– If φ ∈ Org and i ∈ A, then K

(φ) ∈ Org

– If φ ∈ Org and i : α ∈ A, then [i : α]φ ∈ Org

– If i : α ∈ A, then DON E(i : α), DO(i : α), O(i : α) and Can(i : α) ∈ Org

Binary connectives → and ∨, and nullary connective ⊥ can be deﬁned as usual. For

the knowledge operators (K

) we assume the axiomatization characterizing S5. The

assertions DONE(i : α) stands for “α has just been performed by agent i”, DO(i : α)

stands for “α is going to be the next action performed by agent i”, and CAN

(α)) stands

for “α lies in the capabilities of agent i”. O(i : α) is the deontic assertion to the effect

that agent i ought to perform action α.

The semantics of Org will be given in two steps. First we deﬁne the semantics of the

special relations Power , Coordination and Control through a multi-digraph deﬁned

on the set of roles. This deﬁnes a tuple OS which will be part of the Kripke model

given after. We will only introduce some basic elements which are strictly of use for the

development of this article.

Deﬁnition 2. (Organizational structures)

OS is characterized by the following:

hRoles ∪ Agents, {R

P ower

, R

Coordination

, R

Control

, Rea}i

where Roles ∪ Agents is the ﬁnite set of roles and agents;

and {R

P ower

, R

Coordination

, R

Control

} are three irreﬂexive binary relations on Roles

characterizing the Power, Coordination and Control structures. Rea indicates which

agents play which roles.

The semantics of Org is deﬁned in terms of Kripke models (cf. [1]).

3 A Formal Analysis

3.1 Task Allocation

In order for organizations to fulﬁll their objectives, subtasks are isolated via a form of

organizational planning and distributed in a way which deﬁnes the roles agents can play

in contributing to the performance of the organization. We call this designing process

of the activity of an organization task-allocation. Roles can then be seen as sort of

placeholders in a rationally designed activity of an organizations: an agent taking part

to the organization will occupy one of these places, that is, will play a role. In this work,

agents playing a role in an organization are called, following [8], role enacting agents

or rea’s.

The distribution of the sub-tasks in an organization in order to achieve a certain

goal or collective task τ depends on a plan of the organization, i.e., a concrete manner

to achieve the goal (collective task). We can deﬁne a plan to achieve a certain goal

τ as a decomposition of the complex action achieve(τ ) by a sequence of (possible

simultaneous) individual actions. Besides task division, task allocation is needed, which

indicates which role of the organization has to achieve which sub-task of the complex

task. We use the following deﬁnition for task allocation:

Deﬁnition 3. (Task allocation)

A task allocation for a task τ within the set of roles AR is deﬁned as follows:

: achieve(τ

) • r

: achieve(τ

) • . . . • r

: achieve(τ

such that r

: achieve(τ

)•r

: achieve(τ

)•. . .•r

: achieve(τ

)]τ. We refer to the

task allocation of τ within AR as P lan(AR, τ ). To indicate that task achieve(τ

) has

been allocated to role r

in P lan(AR, τ) (for j = 1, 2, . . . , n), we use the following

notation: hr

: τ

i ∈ P lan(AR, τ)

where • stands for either the simultaneous operator ’&’ or the sequential operator

’;’. r

: achieve(τ

)&r

: achieve(τ

) stands for the simultaneous performance of

achieve(τ

) by an agent enacting role r

and achieve(τ

) by an agent enacting role

, and r

: achieve(τ

); r

: achieve(τ

) stands for the sequential composition of

: achieve(τ

) and r

: achieve(τ

We need the simultaneous operator, since some actions have to be performed at the

same time. The sequential operator is needed because some actions might depend on

other ones: a certain action can only be performed if an other action is done. So, the

plan must at least determine the order of sub-actions. For example, the notiﬁcation of

acceptance of a certain paper by an Editorial Board can only be done if it is reviewed

by some members of the Editorial Board.

We will use the concept of task allocation as a starting point for framing the various

notions we are interested in. In particular, as we will see in the coming section, it plays

an essential role for the deﬁnition of the notion of task-based responsibility. Besides,

we will analyze the notion of “failure” in the accomplishment of a task understanding

it as an organizational variant of the notion of social harm described in [10]. In our

context, we deﬁne the untoward event Dτ

as the impossibility, or the reduction of the

possibility to achieve the goal τ allocated to role r. The performance of an action α by

an agent i enacting role r determining social harm can then be represented as [i : α]Dτ

that means, after each execution of action α by agent i the social harm represented by

Dτ

is the case.

Note that the function of the numeric index j consists in denoting the position within the task

allocation sequence.

3.2 Responsibilities in Form

So far we have dealt with organizations at their role level, where the task-allocation and

the organizational structure range. Responsibilities concern agents and arise in relation

with task-allocation and structure once there are agents enacting the roles of a given

organization.

Given a task-allocation allocating a speciﬁc subtask to a role, and given that an agent

is enacting that role, the agent is then said to be responsible for that task or task-based

responsible. In other words, the allocation of subtasks to roles determines a distribution

of what we call task-based responsibilities over the set of agents enacting the roles

of the organization. Being autonomous, agents can independently decide whether to

perform the subtasks to which they are allocated or not, and whether to perform them

in the expected way. In this case the fulﬁllment of the organizational objectives is put

in jeopardy by the conduct of some agent that is said then to be causally responsible for

the failure occurred.

In organizations an agent can happen to be causally responsible of some failure

without actually being blamed by the organization. This can happen if an agent i which

is task-based responsible for performing a task, delegates the performance to a sub-

ordinate agent j which fails or jeopardizes the execution of the delegated task. This

observation reveals an interplay between the notions of responsibility isolated above,

and dimensions of social structure such as the possibility to delegate allocated tasks,

i.e., what we called power relation in the previous section. Social structure in relation

with responsibility will be discussed in detail in Section 4. Here it sufﬁces to notice that

the presence of a power structure within an organization causes a difference between

the two notions of task-based and causal responsibility: ‘I may have not performed the

task you delegated to me, but you were the one appointed to it’. Therefore, if an orga-

nizational task is not performed, the one being socially responsible in front of the orga-

nization, the one who gets the blame for the failure, is not necessarily the one causally

responsible for it, but it is the one to which that task was appointed. The acknowledg-

ment of such a gap calls for the distinction of yet another meaning of the notion of

responsibility which we call failure-based responsibility: who should control the per-

formance of an agent to check whether a failure occurs and take countermeasures if that

is the case?

We can now provide an action logic representation of the notions of responsibility.

Causal responsibility.

An agent is said to be causally responsible when it does something (or fails to do some-

thing) that causes the untoward event D

. We formalize causal responsibility as follows:

Deﬁnition 4. (Causal responsibility)

For all i ∈ Ag: R

(Dτ) := [i : α]Dτ ∧ DO(i : α) ∧ ¬Dτ

meaning that agent i is causally responsible for the untoward event if and only if agent i

performs an action which necessarily determines the occurrence of the untoward event

and, ﬁnally, the untoward event is not the case before the agent performs the action.

Causal responsibility can also be attributed to nonhuman events, for example, that

a house is severely damaged in a storm. In this article, we restrict ourselves to agents

in an organizational context. Notice that an agent which is causally responsible, may

not be considered blameworthy. For example, if the chairman of the Editorial Board

has forgotten to inform a member i to review some papers in one week, and agent i did

not review the papers in one week, then the achievement of the goal of the Editorial

Board to notify of the results of the reviews within the deadline will be reduced. The

member i would be considered responsible in the sense of having caused the situation,

but he would not be responsible in the sense of blameworthy. An agent does something

blameworthy, if he knows (or could have known) that the action he performs leads to

the impossibility or the reduction of the possibility to achieve a goal τ:

Deﬁnition 5. (Causal blameworthiness)

For all i ∈ Ag: Bl

(Dτ) := [i : α]Dτ ∧ DO(i : α) ∧ ¬Dτ ∧ K

([i : α]Dτ )

The importance of the knowledge component in the dynamics of responsibilities within

organizations is analyzed in detail in Section 4.

Task-based responsibility.

The notion of task-based responsibility is somehow interchangeable with duty and

refers to what individuals are expected to do in virtue of their social roles. We as-

sume that task-based responsibility is a consequence of role adoption: an agent who

accepts to play a given role in an organization takes a responsibility with regard to the

accomplishment of that role, i.e., with the tasks associated to it [10]. In this article, this

notion of responsibility completely depends on the position an agent occupies in the

performance of the organization.

Deﬁnition 6. (Task-based responsibility)

For all i ∈ Ag and a task allocation P lan(AR, τ ):

(τ

) := rea(i, r

) ∧ hr

: τ

i ∈ P lan(AR, τ)

Intuitively, we want that the following property holds:

Property 1. For all i, j ∈ Ag and a task allocation P lan(AR, τ ):

(τ

) → O(i : achieve(τ

)) ∧ [

i : achieve(τ

)]Dτ

The obligation O(i : achieve(τ

)) expresses that the organization entrusts agent i

with his task τ

(rea(i, r

) ∧ hr

: τ

i ∈ P lan(AR, τ)), and [i : achieve(¬τ

)]Dτ

expresses the empowerment of i to prevent the reduction of the possibility or the im-

possibility to achieve goal τ

. So, an agent i fails to fulﬁll his task-based responsibility

(τ

) if he violates the norm O(i : achieve(τ

)) which leads to the untoward event

Dτ

. However, the agent is considered blameworthy when he actually knows (or could

have known) that he has this obligation and that he can perform the action to achieve

his task. For example, he has not received the information needed for the performance

of his task, or the achievement of his task depends on an earlier task in the task alloca-

tion which is not performed. This notion of blameworthy can formally be described as

follows:

Deﬁnition 7. (Task-based blameworthiness)

For all i ∈ Ag and a task allocation P lan(AR, τ ):

(τ

) := R

(τ

) ∧ K

(τ

)) ∧ CAN

(achieve(τ

))

Social responsibility.

The notion of social responsibility builds on the notion of task-based responsibility, and

it is somehow analogous to a notion of violation in standard deontic logic.

Deﬁnition 8. (Social responsibility)

For all i ∈ Ag and a task allocation P lan(AR, τ ): R

(τ

) := R

(τ

) ∧ D

that is to say, agent i has the responsibility to achieve τ

and the achievement of τ

impossible or jeopardized. Notice that this notion of responsibility is very simple and is

independent from the notion of causal responsibility.

4 Responsibilities and Organization Structure

We cannot hope to provide a full account of all interactions between responsibilities

and organizational structures. However, in the rest of this section we aim to capture

some essential traits of those interconnections. We understand those relations essen-

tially as guaranteeing some effects to the basic actions of delegate, inform and monitor,

which play an essential role with respect to responsibilities and their development in

organizations.

The following deﬁnitions characterize the inﬂuence of the organization relations on

the actions above. Through these basic properties we can also formally analyze some

consequences of them on the notions of responsibilities studied in the previous section.

Deﬁnition 9. (Power)

For all i, j ∈ Ag s.t. i 6= j and r, s ∈ AR:

(Power(r, s) ∧ rea(i, r) ∧ rea(j, s)) → [i : delegate(j, φ)]O(j : achieve(φ))

If a power relation exists between roles that are enacted by two agents then a delegate

action will have as effect an obligation for the recipient, that is, a form of “your wish

is my command” principle. Intuitively, if a power relation holds between roles r and

s, all delegation acts performed by an agent i enacting role r on agents enacting role s

succeed in creating an obligation for these agents.

Task-based responsibility cannot be delegated. If Agent i has, according the task

allocation, to achieve task φ and has a power relation with agent j, he can delegate his

task to j, but he remains task-based responsible for the achievement of φ. Since φ is

not the original task of agent j according to the given task allocation (see deﬁnition

6). Agent j, however, can be causally responsible if he fails to fulﬁll his delegated

obligation.

A difference between an individual task and a collective task is that in an individual

task all information is readily available and can be reasoned about. However, when a

collective task is divided over the individuals of that collective, they might not know

the whole plan, typically do not have information about actions that are performed, etc.

Therefore, we need a coordination structure.

Deﬁnition 10. (Coordination)

For all i, j ∈ Ag s.t. i 6= j, r, s ∈ AR:

((Coordination(r, s) ∧ rea(i, r) ∧ rea(j, s)) ∧ DON E(i : monitor(φ))) →

((K

φ → O(i : inform(j, φ))) ∧ [i : inform(j, φ)]K

φ)

If a coordination relation holds between roles r and s, all information acts performed

by agents enacting role r to agents enacting role s are successful in the sense that they

create, in these last agents, the knowledge they acquired via monitoring the occurrence

of a certain fact: the inform action will automatically lead to the corresponding epis-

temic state in the recipient. Further, there is a normative aspect: agent i should inform

another agent j about φ if they are connected through a coordination link and if agent i

has monitored (checked) φ.

On this basis, a coordination-related type of responsibility can be deﬁned.

Deﬁnition 11. (Coordinational responsibility)

For all i, j ∈ Ag and a task allocation P lan(AR, τ ):

coor

(inform(j, φ)) := K

([

i : achieve(K

φ)]Dτ

) ∧ R

(τ

)

On the basis of the coordination structure, there is a speciﬁc allocation of the infor-

mation actions, which is needed for the achievement of the individual tasks in the task

allocation. Given this deﬁnition, we can say agent i is responsible to inform agent j,

when the knowledge of φ is a necessary means to the achievement of τ

and that agent

j does not have that knowledge.

We state the if someone is coordinationally responsible to inform an agent about

φ, he is also obliged that the agent will be informed about φ. This can be expressed as

follows:

Property 2. For all i, j ∈ Ag and a task allocation P lan(AR, τ ):

coor

(inform(j, φ)) → O(i : achieve(K

, φ))

The responsibility of an agent i to inform some agent j about a certain aspect φ can

follow from the coordination link between these agents if the knowledge of φ is neces-

sary for the achievement of the task of agent j according the task allocation and i can

monitor or check φ. This shows, in particular, how a given task allocation needs to be

integrated with a suitable allocation of coordinational responsibilities in order to guar-

antee the information necessary for the correct functioning of the organization. This

property can be formalized as follows:

Property 3. For all i, j ∈ Ag s.t. i 6= j, r

, r

∈ AR and task allocation P lan(AR, τ ):

Coordination(r

, r

) ∧ rea(i, r

) ∧ rea(j, r

) ∧ hr

: τ

i ∈ P lan(AR, τ) ∧

(¬K

φ → ¬CAN(j : achieve(τ

))) ∧ CAN (i : monitor (φ)) → R

coor

(inform(j, φ))

So, agent i is responsible to inform agent j about φ if there is a coordination link

between the roles r

and r

they respectively enact, and without the information about

φ agent j cannot perform his task according to the task allocation. If agent i does not

inform agent j, it follows that agent j cannot perform his task, which can lead to Dτ

So, agent i can be causally responsible if he does not inform agent j about φ (see

deﬁnition 4). Note, that agent j is still task-based responsible with respect to τ

, but not

blameworthy, when he does not get the information necessary for the achievement of τ

(see deﬁnition 7).

Finally, we get to a characterization of the dimension of control in organizational

structure:

Deﬁnition 12. (Control)

For all i, j ∈ Ag s.t. i 6= j and r

, r

∈ AR:

(Control(r

, r

) ∧ rea(i, r

) ∧ rea(j, r

)) →

[i : monitor (DONE(j : achieve(φ))](Dτ

→ O(i : achieve(τ

))

If a control relation exists then the monitor action will have as further consequence the

generation of an obligation for the controller in case the controlled actor did not achieve

the relevant state causing the untoward event. On this basis, the notion of failure-based

responsibility can be deﬁned.

Deﬁnition 13. (Failure-based responsibility)

For all i, j ∈ Ag s.t. i 6= j and r, s ∈ AR:

control

(monitor(j, φ)) := Control(r, s) ∧ rea(i, r) ∧ rea(j, s)

This type of responsibility depends completely on the control relation.

The control responsibility has another normative aspect: if an agent has control

over another agent he is obliged to monitor the controlled agent whenever he knows the

controlled agent has an obligation. Formally,

Property 4. For all i, j ∈ Ag and a task allocation P lan(AR, τ ):

control

(monitor(j, φ)) ∧ K

(O(j : achieve(φ))) →

O(i : monitor (DON E(j : achieve(φ))))

We can imagine that an agent who has delegated his task to agent j, has the obligation

to monitor whether the delegated agent has done the task, since he might be responsible

to monitor agent j and he knows that the delegated agent j has the obligation.

5 Conclusions

We have provided some elementary notions of responsibility in its interconnection with

the structure of an organization. We argued that organizations are deﬁned through sev-

eral structural relations. Although people refer to these structures they still lack a precise

formal deﬁnition. In this article these relations have been given a solid foundation. This

allows us to check desirable properties of the structures and how they (should) interact.

Now we have a characterization and can proof properties given some structural proper-

ties of these relations. In future work we will look at more elaborate deﬁnitions of the

power, coordination and control relations.

Responsibilities are closely related to the speciﬁc task allocation within an organi-

zation. Although the task allocation can be determined dynamically through the process

of delegation, some of it is predetermined through the role structure of the organization

which assigns typical tasks to certain roles. The organizational structure plays an even

greater role in the monitoring and control of execution of the tasks for which the agents

are responsible. The logical framework we presented offers a semantics for the notions

of responsibility that is necessary for determining at least some interconnections be-

tween organizational structure and responsibilities. It gives some insides into when an

agent can really be held responsible for when tasks are not (or wrongly) performed.

These observations might lead to guidelines for the design of an organizational struc-

ture given that one wants some responsibilities to be covered at all times. In this article

we just offered a glance of these observations through the example. However, we hope

to extend this area in future work, e.g., to combine our work with the work done in [11]

about the representation of organized interaction with action concepts.

References

1. Grossi, D., Royakkers, L., Dignum, F., Dastani, M.: Foundations of organizational structure

in multi-agent systems. In Dignum, F., ed.: Proceedings of AAMAS’05, Fourth International

Joint Conference on Autonomous Agents and Multiagent Systems, ACM Press (2005) 690–

697

2. Selznick, P.: Foundations of the theory of organization. American Sociological Review 13

(1948) 25–35

3. Morgenstern, O.: Prolegomena to a theory of organizations. Manuscript (1951)

4. Giddens, A.: Social Theory and Modern Sociology. Polity Press (1984)

5. Alberts, D., Hayes, R.: Power to the Edge: Command ... Control ... in the Information Age.

CCRP Publication Series, Washington (2003)

6. Castelfranchi, C., Falcone, R.: From task delegation to role delegation. In Lenzerini, M., ed.:

LNAI 1321. AI*IA 97: Advances in Artiﬁcial Intelligence. Springer Verlag, Berlin (1997)

278–289

7. Jones, A.J.I., Sergot, M.: A formal characterization of institutionalised power. Journal of the

IGPL 3 (1996) 429–445

8. Dignum, V.: A Model for Organizational Interaction. SIKS Dissertation Series (2003)

9. Harel, D.: Dynamic logic. In Gabbay, D., Guenthner, F., eds.: Handbook of Philosophical

Logic: Volume II: Extensions of Classical Logic. Reidel, Dordrecht (1984) 497–604

10. Conte, R., Paolucci, M.: Responsibility for societies of agents. Journal of Artiﬁcial Societies

and Social Simulation 7 (2004)

11. Santos, F., Jones, A., Carmo, J.: Responsibility for action in organisations: a formal model.

In Hintikka, G., Tuomela, R., eds.: Contemporary Action Theory, volume Synthese Library

267, Deventer, Kluwer (1997) 333–350