On using Theorem Proving for Cognitive Agent-oriented Programming

Alexander Birch Jensen

1 a

, Koen V. Hindriks

2 b

and Jørgen Villadsen

1 c

DTU Compute, Department of Applied Mathematics and Computer Science, Technical University of Denmark,

Richard Petersens Plads, Building 324, DK-2800 Kongens Lyngby, Denmark

Social AI Group, Boelelaan 1111, Vrije Universiteit (VU) Amsterdam, 1081 HV Amsterdam, The Netherlands

Keywords:

Agent-oriented Programming, Cognitive Multi-Agent Systems, Software Reliability, Proof Assistants.

Abstract:

Demonstrating reliability of cognitive multi-agent systems is of key importance. There has been an exten-

sive amount of work on logics for verifying cognitive agents but it has remained mostly theoretical. Cognitive

agent-oriented programming languages provide the tools for compact representation of complex decision mak-

ing mechanisms, which offers an opportunity for applying a theorem proving approach. We base our work

on the belief that theorem proving can add to the currently available approaches for providing assurance for

cognitive multi-agent systems. However, a practical approach using theorem proving is missing. We explore

the use of proof assistants to make verifying cognitive multi-agent systems more practical.

1 INTRODUCTION

Reliability is of key importance for software systems

in general and multi-agent systems (MAS) and cog-

nitive MAS (CMAS) in particular (Dix et al., 2019).

Cognitive multi-agent systems are systems that con-

sist of agents that use cognitive notions such as beliefs

and goals to decide on their next action. Dedicated

cognitive agent programming languages have been

developed for engineering these systems. Because

these languages have incorporated these high-level

concepts of beliefs and goals as ﬁrst-class citizens

they can rather compactly represent quite intricate de-

cision making mechanisms. However, demonstrating

the reliability of such CMAS remains very challeng-

ing because of the complex behaviour these mecha-

nisms are able to generate. We often observe com-

plex behaviour patterns of agents that exceed those of

traditional procedural programs (Winikoff and Crane-

ﬁeld, 2014).

Testing methods for CMAS have been explored

by (Koeman et al., 2018; Dastani et al., 2010), but

testing alone is unlikely to provide the levels of assur-

ance required by stakeholders. To provide this level

of assurance for complex CMAS, formal veriﬁcation

techniques are needed for demonstrating their relia-

bility. The fact that several agent languages have been

https://orcid.org/0000-0002-7298-2133

https://orcid.org/0000-0002-5707-5236

https://orcid.org/0000-0003-3624-1159

speciﬁed by means of a formal semantics, e.g. 3APL,

GOAL (Hindriks et al., 1999; Hindriks et al., 2001),

and AgentSpeak(L) languages such as Jason and Ja-

CaMo (Rao, 1996; Bordini et al., 2007; Boissier et al.,

2020), provides a suitable starting point for devel-

oping these techniques. The initial developments of

BDI logics such as by (Rao and Georgeff, 1991) and

(Cohen and Levesque, 1987) did not connect well

with the more practically oriented work done in en-

gineering and programming of agents such as (Hin-

driks et al., 1999). These logics were signiﬁcantly

more complicated which provided a barrier, and only

little progress was made to close the gap it created.

Although work such as (Jongmans et al., 2010; Bor-

dini et al., 2004) has provided more practical tools

to demonstrate reliability using a model checking ap-

proach, we notice that the work on veriﬁcation logics

has remained mostly theoretical thus far.

The success of state-of-the-art proof assistants has

been demonstrated in the literature for more tradi-

tional software development paradigms different from

CMAS. They have proved quite successful for veri-

ﬁcation of various software (and hardware) systems

(Ringer et al., 2019). One major branch is based on

simple type theory (higher-order logic) with prime ex-

amples being HOL Light, HOL4 and Isabelle/HOL

(Harrison, 1996; Slind and Norrish, 2020; Nipkow

et al., 2002). Another major branch contains those

based on dependent type theory such as Agda, Coq

and Lean (The Agda Developers, 2020; Bertot and

Cast

eran, 2004; Avigad et al., 2020). Built from a

446

Jensen, A., Hindriks, K. and Villadsen, J.

On using Theorem Proving for Cognitive Agent-oriented Programming.

DOI: 10.5220/0010349504460453

In Proceedings of the 13th International Conference on Agents and Artiﬁcial Intelligence (ICAART 2021) - Volume 1, pages 446-453

ISBN: 978-989-758-484-8

small, trusted kernel proof assistants provide tools for

ensuring reliability of systems that are also trustwor-

thy (Paulson et al., 2019). Arguably, these systems re-

main accessible mostly for specialists and have been

applied in practice to relatively limited cases. The

answer to why this is the case seems obvious: be-

cause only then it pays off. Applying it to much

larger programs is too complicated and requires too

much effort. But we believe that here there is ac-

tually a beneﬁt of CMAS that still needs to be ex-

plored: CMAS are typically much more compact pro-

grams, and agent programming languages are typi-

cally very high-level programming languages that in-

troduce high-level concepts such as beliefs and goals.

At least at ﬁrst sight this appears to make them more

suitable candidates for a theorem proving approach.

Given the success of proof assistants in other ar-

eas, there should be something for us here to explore.

To the best of our knowledge, there is no work thus far

that has looked into the viability of a theorem proving

approach. More speciﬁcally, we aim to explore the

use of proof assistants as a practical alternative for

demonstrating the reliability of CMAS.

In this paper, we want to explore a practical ap-

proach as an answer to our main research question:

how can a theorem proving approach contribute to

reliable engineering of cognitive agent systems?

We argue that proof assistants can be useful and

play an important role in verifying cognitive multi-

agent programs. To support our argument, we use

the GOAL agent programming language as our pri-

mary example. We do so partly because the core

of this language captures the core concepts of cog-

nitive agent programming but also is relatively simple

(when we leave out more advanced concepts for struc-

turing agent programs such as modules) and a formal

semantics is available which facilitates the process of

formalization in a proof assistant.

The paper is structured as follows. Section 2 high-

lights some of the relevant existing work in the lit-

erature. Section 3 gives a brief introduction to the

basic concepts of the programming language GOAL.

Section 4 discusses the challenge of ensuring relia-

bility of CMAS and compares the current available

approaches for demonstrating reliability. Section 5

explores how a theorem proving approach can con-

tribute to demonstrating reliability. Section 6 serves

as an appetizer for early work on formalizing GOAL

in Isabelle/HOL. Finally, Section 7 makes some con-

cluding remarks.

2 RELATED WORK

Agent systems, especially in the multi-agent set-

ting, are notoriously hard to test. The ﬁndings of

(Winikoff and Craneﬁeld, 2014) show that belief-

desire-intention (BDI) agents indeed have a large

number of paths through the program — something

that has often been presumed. In the case of white box

testing, covering all paths thus becomes increasingly

impossible. Perhaps surprisingly, it is also found

that adding failure handling to agents signiﬁcantly in-

creases the size of the behavior space.

Traditionally, temporal logic has been the pre-

ferred language for speciﬁcation of agents and cor-

rectness properties, but it falls short when consider-

ing e.g. error-prone environments and capturing that

agents succeed where possible. In such cases the

straight-forward statement that the agent will eventu-

ally succeed is too strong and not provable. (Winikoff,

2010) suggest an approach that combines testing and

formal veriﬁcation as neither in practice succeeds to

obtain assurance of the reliability of the system.

The focus of formal veriﬁcation of CMAS has

mostly been on testing with somewhat limited logi-

cal languages: (Jongmans et al., 2010) suggest an ap-

proach using a model checker on top of the program

interpreter. The model checker is implemented for the

interpreter of the GOAL language, but the proposed

framework is generally applicable to other agent pro-

gramming languages as well. (Koeman et al., 2018)

propose an automated testing framework for auto-

matically detecting failures in cognitive agent pro-

grams. Although the work shows promise, more work

is needed particularly towards localization of failures.

(Koeman et al., 2017) explore an approach apply-

ing the concept of omniscient debugging to cognitive

agent programs. The work is based on the idea of re-

versing the program’s execution rather than trying to

debug the program by reproducing failure in a rerun.

In agent programs this becomes especially useful as

they usually have both non-deterministic aspects and

rely on agent environments both of which make the

reproduction of the circumstances of failure difﬁcult.

Some work on theorem proving was done by

(Alechina et al., 2010) and in particular by (Shapiro

et al., 2002) which explored veriﬁcation of agents

speciﬁed in Cognitive Agents Speciﬁcation Language

(CASL). However this work was focused mostly on

the speciﬁcation level and thus did not connect well

with agent-oriented programming (AOP). As such, it

still left a gap in terms of practical applicability for

CMAS.

The present paper closely relates to our work in

(Jensen, 2021b) where the formalization of GOAL in

On using Theorem Proving for Cognitive Agent-oriented Programming

447

Isabelle/HOL (to be presented in Section 6) is dis-

cussed in details; (Jensen, 2021a) explores a method

for transforming GOAL agents’ program code to an

agent logic of a veriﬁcation framework — unlike this

paper, a theorem proving approach is not considered.

3 THE PROGRAMMING

LANGUAGE GOAL

We use GOAL as our primary example of a cogni-

tive AOP (c-AOP) language. This section brieﬂy in-

troduces the basic concepts of GOAL. For a more

detailed overview of the GOAL agent programming

language we refer to (Hindriks et al., 2001; Hindriks,

2009; Hindriks and Dix, 2014).

The language GOAL is an agent-oriented pro-

gramming language based on the BDI logic that draws

inspiration from concurrent programming in general

and from the programming language UNITY in par-

ticular (Misra, 1994). However, unlike UNITY, the

basic concepts of the GOAL language are cognitive

concepts such as beliefs and goals instead of simple

assignments. GOAL uses a declarative notion of goals

that agents use to decide their next action.

An agent’s beliefs and goals are drawn from a

classical, propositional language and are stored in a

belief base and goal base, respectively. A belief base

Σ and goal base Γ constitute an agent’s mental state

hΣ, Γi. The constraints imposed on mental states are

a consequence of the blind commitment strategy that

agents use by default (Rao and Georgeff, 1993):

• Σ is consistent, i.e. does contain a contradiction,

• for all γ ∈ Γ:

– γ is not entailed by the agent’s beliefs, and

– γ is satisﬁable.

Formulas over mental states are used to express con-

ditions on beliefs and goals. The language of mental

state formulas is formed by Boolean combinations of

the belief and goal modalities B and G, respectively. It

may be helpful to think of these modalities as queries

for inspection of the mental state:

• BΦ: is Φ entailed by the belief base Σ?

• GΦ: is Φ a goal (or subgoal) of the goal base Γ?

Besides these more complex notions for deﬁning and

reasoning about an agent’s state, the GOAL language

provides rules and capabilities (or basic actions) for

deriving an agent’s choice of action from its mental

state. In contrast to other agent programming lan-

guages, as the goals of an agent in GOAL are declar-

ative, they specify (conditions on) states the agent

wants to achieve and not how to achieve them. The

speciﬁcation of rules provides means to program the

agent such that it rationally selects actions for achiev-

ing its goals. As such, the language is built on the

core principle that agents always derive their choice

of action from their current beliefs and goals.

4 ENSURING RELIABILITY

The paradigm of c-AOP draws many parallels with

concepts of concurrent programs which are noto-

riously difﬁcult to both test and formally verify.

The primary reason is their complex behaviour pat-

terns and by extension the number of possible paths

through the program. Consequently, covering every

path of the program through testing is practically im-

possible and any proof attempt will explode in size.

For languages that have a formal semantics, such

as GOAL, formal techniques can be used, but the fo-

cus has been mostly on limited logical languages and

model checking. Although these works have shown

that they can be applied in practice, the work on ver-

iﬁcation logics has remained mostly theoretical thus

far, and there has not been much work to show that

proof assistants can contribute to demonstrating reli-

ability of CMAS.

We want to pursue the idea that a proof assis-

tant can contribute to reliability of CMAS. (Calegari

et al., 2020) recently reviewed logic-based technolo-

gies for MAS and provide a nice overview of the

available, and actively developed, agent technologies

including means of agent veriﬁcation. Notably, while

model checking is highly represented, the literature

review reveals no mention of theorem proving nor

proof assistants. In conclusion, it seems that these

tools and methodologies have not been adopted in the

MAS community. The question remains why this is

the case. At the time MAS became trending, model

checking had manifested itself as an effective formal

technique for veriﬁcation of software (Clarke et al.,

2001). As such, adapting model checking to MAS

seemed an obvious choice of direction for veriﬁca-

tion. Going back to the initial work on BDI agent

logics, a lot of the work was promising. However,

the practical applications were not explored further,

possibly due to the need for automatic tools to reduce

the effort of proof, and working with such tools may

have seemed a daunting task. However, the AOP ap-

proach does not require as complex a logic as the orig-

inal work in BDI and intention logics (Hindriks and

van der Hoek, 2008).

Some of the typical challenges for engineering

CMAS reliably are: knowledge representation (KR),

ICAART 2021 - 13th International Conference on Agents and Artiﬁcial Intelligence

448

Table 1: Programming concepts of multi-agent systems and possible approaches for demonstrating their reliability.

Programming concept Debugging Automated testing Model checking Theorem proving

Knowledge representation X X X ?

Cognitive concepts X X X ?

Decision rules X X X ?

Multiple agents X X X ?

Environments X × × ?

cognitive concepts such as beliefs and goals, deci-

sion rules, multiple agents, environments (the exter-

nal world). Each of these contributes to the complex-

ity of demonstrating the reliability, either by adding to

the complexity or making it easier. We have listed the

key programming concepts of MAS in Table 1 and an

indication of which approaches we believe currently

are state-of-the-art (X) or not well established (×).

Theorem proving for MAS is something we are cur-

rently exploring (?).

Debugging approaches remain the most versatile

as they cover all CMAS concepts, but their weakness

is that debugging only truly makes sense once an er-

ror has been observed — they do not inherently aid

us in ﬁnding (obscure) error states. To overcome this,

it often makes sense to devise a number of automated

tests, e.g. by checking that certain properties hold be-

fore and after execution of modules. Outside of the

immediate difﬁculties of devising testing schemes, a

great difﬁculty lies in deterministic reproduction of

error states when connected to external environments.

The highest degree of reliability is currently achieved

by applying model checking techniques. However, a

potential pitfall is the often necessary introduction of

assumptions to reduce the size of the state space.

5 A THEOREM PROVING

APPROACH

In this section, we want to advocate an approach us-

ing theorem proving to ensure reliability of MAS. We

already mentioned how the use of proof assistants has

shown success in other applications of software.

We are particularly interested in what could be

gained from a theorem proving approach for c-AOP

for which we see some particular beneﬁts: the cog-

nitive concepts deﬁne primarily single agents and

only in second instance the multi-agent level because

agents communicate about their beliefs and goals. Es-

pecially the proof assistants based on higher-order

logic are appealing. Here, we focus on Isabelle/HOL.

Primarily because this is the proof assistant we have

most experience working with. The strengths of

Isabelle/HOL include ﬁrst and foremost a power-

ful automatic proof search, including an automatic

search for counterexamples (Isabelle will inform us

that a particular proof goal cannot be solved due to

the found counterexample) (Paulson, 2017). Further-

more, Isabelle facilities a readable language for struc-

tured proofs: Isar. The practicality of Isabelle rests on

its LCF-style foundation: every proof goes through

the kernel using abstract types to ensure its soundness

(Paulson, 2019).

Much of the work with formal techniques for ver-

iﬁcation of CMAS has been with limited logical lan-

guages. A reason for the lack of experimentation

with higher-order logic (HOL) languages could be

that ﬁnding the proper level of abstraction is difﬁcult:

the veriﬁcation of agents is clearly bound to the con-

text in which they are deployed. If we make our mod-

els too general, veriﬁcation by proof will likely fail.

At the other end of the spectrum, if we fail to sufﬁ-

ciently generalize the proof goal, the proof result may

be too weak. It is clear that using a proof assistant

will not inherently provide the answer to how a HOL-

based model of CMAS could look like. Yet still, the

practicalities should make the approach more feasible

from an engineering perspective. We are interested

to see if a new and fresh perspective could emerge

through the use of HOL, and what a general sketch of

a HOL-based approach using theorem proving could

look like.

We need to consider how to demonstrate the effec-

tiveness of a theorem proving approach using proof

assistants. The nature of veriﬁcation can be very rig-

orous: the proof attempt either fails or succeeds. One

way to alleviate this absoluteness is to compare the

approach to similar approaches such as model check-

ing: where do we achieve the highest degree of pre-

cision in the model, which approach produces the

strongest theoretical results, how efﬁciently do we

reach these results and how much effort is required

to obtain them?

On using Theorem Proving for Cognitive Agent-oriented Programming

449

6 FORMALIZING GOAL IN

ISABELLE/HOL

To demonstrate the feasibility of the suggested ap-

proach, we have started work on a formalization of

the GOAL agent programming language and a veriﬁ-

cation framework. We take as our starting point and

frame of reference the work by (de Boer et al., 2007).

In the following we will illustrate our suggested

theorem proving approach using Isabelle/HOL. For

further details see (Jensen, 2021b).

The Isabelle ﬁles are publicly available online:

https://people.compute.dtu.dk/aleje/public/

The theory ﬁle Gvf PL.thy formalizes the proposi-

tional logic that serves as foundation. The the-

ory Gvf GOAL.thy imports Gvf PL.thy and formalizes

GOAL (up to the point of proving soundness of the

presented proof system).

The Isabelle/HOL formalization is around 400

lines of code in total and loads in less than a second

on a MacBook Pro with a 2.4 GHz 8-core i9 processor

and 32 GB memory.

The website also links to demonstrations related

to other work we have done on veriﬁcation of GOAL

agents (Jensen, 2021a).

In Isabelle/HOL expressions to be parsed within

the context of the (embedded) higher-order language

are encapsulated in the so-called “cartouches”

···

Note that they generally can be omitted for atomic

expressions.

Cognitive concepts of agents are modelled

through mental states. Mental states consists of a be-

lief base and goal base that can be set up as a datatype

over formulas of propositional logic (the type Φ

type-synonym mst =

(Φ

set × Φ

set)

The command type-synonym introduces a new type

name as an abbreviation.

Certain constraints apply: not every tuple of sets

of formulas qualify as mental states. We think of the

declarative goals as achievement goals which is (par-

tially) captured by the following deﬁnition:

deﬁnition is-mst ::

mst ⇒ bool

(

∇

) where

∇ x ≡ let (Σ, Γ) = x in

Σ ¬|=

⊥

∧ (∀γ∈Γ. Σ ¬|=

γ ∧ {} ¬|=

γ)

The command deﬁnition introduces a new deﬁnition.

Unlike abbreviations, deﬁnitions are not automati-

cally expanded by Isabelle which is helpful when a

higher level of abstraction is desired. In our deﬁni-

tion above, the type is

mst ⇒ bool

: given a mental

state, an evaluation of the expression will result in a

Boolean value. We introduce ∇ as a shorthand for the

deﬁnition. The variable x is a pair of sets, as we de-

ﬁned earlier, and let (Σ, Γ) = x in . . . introduces local

variables Σ and Γ in . . . by unfolding the x deﬁnition.

We have detached the type for the mental states

(mst) from the constraints (is-mst). This gives some

extra footwork: we need to introduce the deﬁnition

into statements concerning mental states. On one

hand we may need the deﬁnition as an assumption

for a proof to go through; on the other hand we may

need to prove that alterations preserve the mental state

properties.

For reasoning about mental states, a language is

embedded into Isabelle that allows for inspection of

mental states through the belief and goal modalities,

B and G, respectively:

datatype Φ

B Φ

G Φ

Neg Φ

(

) |

Imp Φ

(inﬁxr

−→

60) |

Dis Φ

(inﬁxl

∨

70) |

Con Φ

(inﬁxl

∧

80)

The command datatype introduces a new datatype.

Each option has a constructor followed by one or

more input types. Multiple options are divided by

the special | character. The constructors may be re-

cursive as is the case for the Boolean operators. The

keywords inﬁxl and inﬁxr signify that the follow-

ing shorthand is left- or right-associative, respectively.

The numbers deﬁne the order of precedence.

Next we deﬁne the semantics of mental state for-

mulas. The belief modality requires that the formula

is entailed by current beliefs. The goal modality is

more complicated: we either require that the formula

is a goal or a subgoal (not entailed by current beliefs):

primrec semantics ::

mst ⇒ Φ

⇒ bool

(inﬁx

50)

where

M |=

(B Φ) = (let (Σ, -) = M in Σ |=

Φ)

M |=

(G Φ) = (let (Σ, Γ) = M in

Σ ¬|=

Φ ∧ (∃γ∈Γ. {} |=

γ −→

Φ))

M |=

(¬

Φ) = (¬ M |=

Φ)

M |=

(Φ

−→

) = (M |=

−→ M |=

)

M |=

(Φ

∨

) = (M |=

∨ M |=

)

M |=

(Φ

∧

) = (M |=

∧ M |=

)

The command primrec deﬁnes a primitive recursive

function. We introduce inﬁx syntax (|=

) similar to

textbook deﬁnitions. The semantics for Boolean op-

erators can be deﬁned by using Isabelle’s built-in op-

erators.

As a slight deviation from (de Boer et al., 2007),

we have baked the subgoal property into the seman-

tics of the language. In the original work, the goal

base is deﬁned such that it includes all subgoals. This

means that the goal base is an inﬁnite set of formulas.

We have opted for a constructive approach.

ICAART 2021 - 13th International Conference on Agents and Artiﬁcial Intelligence

450

Now that we have deﬁned the language and its se-

mantics, we turn to prove some interesting properties

of the goal modality. The following lemma justiﬁes

to call G a logical operator:

lemma G-properties:

shows

¬ (∀Σ Γ Φ ψ. ∇ (Σ, Γ) −→ (Σ, Γ) |=

G (Φ −→

ψ) −→

G Φ −→

G ψ)

and

¬ (∀Σ Γ Φ ψ. ∇ (Σ, Γ) −→ (Σ, Γ) |=

G (Φ ∧

(Φ −→

ψ)) −→

G ψ)

and

¬ (∀Σ Γ Φ ψ. ∇ (Σ, Γ) −→ (Σ, Γ) |=

G Φ ∧

G ψ −→

G (Φ ∧

ψ))

and

{} |=

Φ ←→

ψ −→ ∇ (Σ, Γ) −→ (Σ, Γ) |=

G Φ ←→

G ψ

The command lemma instantiates a proof with the

given proof goal(s) that are to be solved. In our case,

we have stated four individual goals, each stating a

different property of the G modality. The proof de-

tails are omitted. We may later use the result of the

lemma by referencing its name G-properties.

The properties of the modalities form the basis for

an inductively deﬁned proof system for mental state

formulas:

inductive derive ::

⇒ bool

(

) where

R1:

conv (map-of T) ϕ

= ϕ =⇒ {#} `

=⇒ `

R2:

{#} `

Φ =⇒ `

(B Φ)

A1:

(B (Φ −→

ψ) −→

(B Φ −→

B ψ))

A2:

(¬

(B ⊥

))

A3:

(¬

(G ⊥

))

A4:

((B Φ) −→

(¬

(G Φ)))

A5:

{#} `

(Φ −→

ψ) =⇒ `

(¬

(B ψ) −→

(G Φ)

−→

(G ψ))

The command inductive introduces an inductive def-

inition. The use of =⇒ signiﬁes that side conditions

apply (the result is to the far right).

In the inductive deﬁnition above, `

denotes

derivability in classical logic. The ﬁrst condition of

R1 enforces that ϕ

and ϕ are structurally equivalent.

As such, R1 transfers any proof rules for classical

logic via `

. R2 captures that agents believe tautolo-

gies to be true. A1-A5 are the axioms and have no side

conditions.

We can prove the proof system sound with respect

to the semantics of mental state formulas (again the

proof is omitted, but we note that it in its current form

requires around 80 lines):

theorem derive-soundness:

assumes

∇ M

shows

Φ =⇒ M |=

The command theorem is internally similar to

lemma. The distinction is reminiscent of writing for-

mal proofs on paper, i.e. theorem signals an impor-

tant result.

In summary, we acknowledge that our work is still

in its early stages. Nevertheless, we think that the in-

termediate results show good promise that a full for-

malization of the GOAL programming language and

the veriﬁcation framework is possible.

7 CONCLUSION

We have considered reliability of cognitive multi-

agent systems (CMAS) which consist of agents that

use cognitive notions such as beliefs and goals.

The work on ﬁnding and improving approaches to

demonstrating reliability of CMAS has in the multi-

agent system community been dominated by model

checking, debugging and testing. While all of these

approaches have shown to be effective and practical,

efforts towards new and better strategies are ongoing.

We have observed that while much of the

early work on agent and veriﬁcation logics showed

promise, it has remained mostly theoretical and with

limited practicality. We question if the reason could

be due to the trends at the time work with CMAS

gained traction, and not because the approach is in-

ferior to others.

The early work on theorem proving as an ap-

proach to demonstrate reliability of CMAS has left

a gap between theory and practice. We ﬁnd that proof

assistants provide the practical tools to close this gap,

and that they can contribute to demonstrating relia-

bility, primarily by means of powerful automation for

proof search.

To illustrate the feasibility of the suggested

approach, we have started work on formalizing

the GOAL agent programming language in the

Isabelle/HOL proof assistant. Our early ﬁndings

show good promise for the approach.

ACKNOWLEDGEMENTS

We would like to thank Asta Halkjær From for

Isabelle related discussions and for comments on

drafts of this paper.

REFERENCES

Alechina, N., Dastani, M., Khan, A. F., Logan, B., and

Meyer, J.-J. (2010). Using Theorem Proving to Ver-

ify Properties of Agent Programs. In Speciﬁcation

and Veriﬁcation of Multi-agent Systems, pages 1–33.

Springer.

On using Theorem Proving for Cognitive Agent-oriented Programming

451

Avigad, J., Moura, L. d., and Kong, S. (2020). Theorem

Proving in Lean. https://leanprover.github.io/theorem

proving in lean/.

Bertot, Y. and Cast

eran, P. (2004). Coq’Art: The Calculus

of Inductive Constructions. Springer.

Boissier, O., Bordini, R. H., Hubner, J., and Ricci, A.

(2020). Programming Multi-Agent Systems Using Ja-

CaMo. MIT Press.

Bordini, R., Fisher, M., Wooldridge, M., and Visser, W.

(2004). Model Checking Rational Agents. Intelligent

Systems, IEEE, 19:46–52.

Bordini, R., H

ubner, J., and Wooldridge, M. (2007). Pro-

gramming Multi-Agent Systems in AgentSpeak Using

Jason. Wiley.

Calegari, R., Ciatto, G., Mascardi, V., and Omicini, A.

(2020). Logic-based technologies for multi-agent sys-

tems: a systematic literature review. Autonomous

Agents and Multi-Agent Systems, 35.

Clarke, E., Biere, A., Raimi, R., and Zhu, Y. (2001).

Bounded Model Checking Using Satisﬁability Solv-

ing. Formal Methods in System Design, 19:7–34.

Cohen, P. and Levesque, H. (1987). Intention = Choice +

Commitment. In Proceedings of AAAI-87, volume 42,

pages 410–415.

Dastani, M., Brandsema, J., Dubel, A., and Meyer, J.-

J. (2010). Debugging BDI-Based Multi-Agent Pro-

grams. In Programming Multi-Agent Systems, pages

151–169.

de Boer, F. S., Hindriks, K. V., van der Hoek, W., and

Meyer, J.-J. (2007). A veriﬁcation framework for

agent programming with declarative goals. Journal

of Applied Logic, 5:277–302.

Dix, J., Logan, B., and Winikoff, M. (2019). Engineer-

ing Reliable Multiagent Systems (Dagstuhl Seminar

19112). Dagstuhl Reports, 9(3):52–63.

Harrison, J. (1996). HOL Light: A tutorial introduction. In

Srivas, M. and Camilleri, A., editors, Formal Methods

in Computer-Aided Design, pages 265–269, Berlin,

Heidelberg. Springer Berlin Heidelberg.

Hindriks, K. and van der Hoek, W. (2008). GOAL Agents

Instantiate Intention Logic. In Programming Multi-

Agent Systems, pages 196–219.

Hindriks, K. V. (2009). Programming rational agents in

goal. In El Fallah Seghrouchni, A., Dix, J., Dastani,

M., and Bordini, R. H., editors, Multi-Agent Program-

ming: Languages, Tools and Applications, pages 119–

157. Springer US, Boston, MA.

Hindriks, K. V., Boer, F., van der Hoek, W., and Meyer, J.-

J. (1999). Agent programming in 3APL. Autonomous

Agents and Multi-Agent Systems, 2:357–401.

Hindriks, K. V., de Boer, F. S., van der Hoek, W., and

Meyer, J.-J. (2001). Agent Programming with Declar-

ative Goals. In Intelligent Agents VII Agent The-

ories Architectures and Languages, pages 228–243.

Springer.

Hindriks, K. V. and Dix, J. (2014). GOAL: A Multi-agent

Programming Language Applied to an Exploration

Game. In Agent-oriented software engineering, pages

235–258. Springer.

Jensen, A. B. (2021a). Towards Verifying a Blocks World

for Teams GOAL Agent. In ICAART 2021 - Proceed-

ings of the 13th International Conference on Agents

and Artiﬁcial Intelligence. SciTePress. To appear.

Jensen, A. B. (2021b). Towards Verifying GOAL Agents

in Isabelle/HOL. In ICAART 2021 - Proceedings of

the 13th International Conference on Agents and Ar-

tiﬁcial Intelligence. SciTePress. To appear.

Jongmans, S.-S., Hindriks, K., and Riemsdijk, M. (2010).

Model Checking Agent Programs by Using the Pro-

gram Interpreter. In CLIMA, pages 219–237.

Koeman, V., Hindriks, K., and Jonker, C. (2017). Omni-

scient debugging for cognitive agent programs. In

Proceedings of the Twenty-Sixth International Joint

Conference on Artiﬁcial Intelligence, IJCAI-17, pages

265–272.

Koeman, V., Hindriks, K., and Jonker, C. (2018). Automat-

ing failure detection in cognitive agent programs. In-

ternational Journal of Agent-Oriented Software Engi-

neering, 6:275–308.

Misra, J. (1994). A Logic for Concurrent Programming.

Technical report, Department of Computer Sciences.

The University of Texas at Austin. Austin, Texas,

USA.

Nipkow, T., Paulson, L., and Wenzel, M. (2002).

Isabelle/HOL — A Proof Assistant for Higher-Order

Logic. Springer.

Paulson, L. (2017). Computational logic: Its origins and ap-

plications. Proceedings of the Royal Society A: Math-

ematical, Physical and Engineering Science, 474.

Paulson, L., Nipkow, T., and Wenzel, M. (2019). From

LCF to Isabelle/HOL. Formal Aspects Comput.,

31(6):675–698.

Paulson, L. C. (2019). Formalising mathematics in simple

type theory. In Centrone, S., Kant, D., and Sarikaya,

D., editors, Reﬂections on the Foundations of Mathe-

matics: Univalent Foundations, Set Theory and Gen-

eral Thoughts, pages 437–453. Springer International

Publishing, Cham.

Rao, A. S. (1996). AgentSpeak(L): BDI Agents Speak

out in a Logical Computable Language. In Proceed-

ings of the 7th European Workshop on Modelling Au-

tonomous Agents in a Multi-Agent World: Agents

Breaking Away: Agents Breaking Away, MAAMAW

’96, pages 42––55, Berlin, Heidelberg. Springer-

Verlag.

Rao, A. S. and Georgeff, M. P. (1991). Modeling Ratio-

nal Agents within a BDI-Architecture. In Proceedings

of the Second International Conference on Principles

of Knowledge Representation and Reasoning, KR’91,

pages 473–484. Morgan Kaufmann Publishers Inc.

Rao, A. S. and Georgeff, M. P. (1993). Intentions and Ratio-

nal Commitment. In Proceedings of the First Paciﬁc

Rim Conference on Artiﬁcial Intelligence (PRICAI-

90). Citeseer.

Ringer, T., Palmskog, K., Sergey, I., Gligoric, M., and Tat-

lock, Z. (2019). QED at Large: A Survey of En-

gineering of Formally Veriﬁed Software. Founda-

tions and Trends

 in Programming Languages, 5(2-

3):102–281.

ICAART 2021 - 13th International Conference on Agents and Artiﬁcial Intelligence

452

Shapiro, S., Lesp

erance, Y., and Levesque, H. J. (2002). The

Cognitive Agents Speciﬁcation Language and Veriﬁ-

cation Environment for Multiagent Systems. In Pro-

ceedings of the First International Joint Conference

on Autonomous Agents and Multiagent Systems: Part

1, AAMAS ’02, pages 19—-26, New York, NY, USA.

Association for Computing Machinery.

Slind, K. and Norrish, M. (1998-2020). HOL-4 manuals.

https://hol-theorem-prover.org/.

The Agda Developers (2020). The Agda Wiki. https://wiki.

portal.chalmers.se/agda/pmwiki.php.

Winikoff, M. (2010). Assurance of Agent Systems: What

Role Should Formal Veriﬁcation Play? In Dastani,

M., Hindriks, K. V., and Meyer, J.-J. C., editors, Speci-

ﬁcation and Veriﬁcation of Multi-agent Systems, pages

353–383. Springer US, Boston, MA.

Winikoff, M. and Craneﬁeld, S. (2014). On the Testability

of BDI Agent Systems. Journal of Artiﬁcial Intelli-

gence Research, 51:71–131.

On using Theorem Proving for Cognitive Agent-oriented Programming

453