Cognition Capabilities and the Capability-affordance Model

Vaughan Michell

Informatics Research Centre, Henley Business School, University of Reading, Whiteknights, Reading, RG6 6UD, U.K.

v.a.michell@ henley.reading.ac.uk

Keywords: Cognitive Informatics, Distributed Cognition Theory, Capability-affordance Model, Affordance

Mechanism, Affordance Path.

Abstract: Much research has been done on physical and cognitive affordances in designed objects, but little has been

done on human cognitive capabilities. This paper applies the capability-affordance model to cognitive agent

capabilities and affordances. It develops a cognition-affordance model by identifying cognition resources

using the SRK model and cognitive task analysis. It proposes four cognition mechanisms and suggests

cognitive capability depends on cognitive mechanisms interacting with knowledge. Affordance possibilities

depend on different knowledge paths where existing or new agent knowledge is applied/grown by copying

or mutation. Mutation may use existing logic creating new knowledge directly applicable to the real world,

or, new theoretical knowledge affordances of imagination. We propose a two axis model to link cognitive

affordance and imagination. We propose how perceived and cognitive affordances relate to the perception-

action axis and that epistemic-axiological axis relates to the theoretical models of thought to account for

creativity in human-agent cognition.

1 INTRODUCTION

Our work focuses on the capability of the object in

the environment in terms of people and natural and

human designed objects/systems ie ‘what could the

object do?’ and ‘how do we measure what it could

do? The term ‘cognitive affordance’ defined by

Norman et al., (1988) is widely used to explain

combinations of object uses in relation to what is

perceived in the environment. Cognitive affordance

theory is often used to design human computer

interfaces (Hartson, 2003) and in ecological design

of systems. However, firstly it does not explain the

interactions of internal human resources that provide

the cognitive capability (Norman, 1988). Perceived

affordance depends on representations in long term

memory and the way the agent brain processes these.

Secondly Norman’s approach can confuse the

perception of how objects and features of objects in

the real world could be used by the agent, with our

focus, the way knowledge can be used in the brain in

terms of the affordance of reasoning and creativity

to produce new ideas and thoughts (Albrechtsen,

2001). We therefore use the term cognition

affordance to explain the possible interactions of

mental resources that produce creative thought

possibilities.

1.1 The Capability-affordance Model

Our previous papers introduced the idea of the

capability resource model (Michell, 2011) and

capability-affordance model using the work of

Gibson and Norman. We reasoned that the capability

of two interacting resources Ri was dependent on an

affordance mechanism AM and the possible 4d

space-time path AP to execute the affordance

(Michell, 2013).

Capability = f(AM(Ri) x AP(Ri)) and R = f {Aij}

This was used to explain physical resource

combinations or directly perceivable affordances

acting in 3 dimensions plus time (Barentsen and

Trettvik, 2002). However, we paid little attention to

the way knowledge can be used in the brain to

explain cognition affordance. We used Stamper et al.

(2000) to differentiate between two types of

behaviour substantive (or physical action) eg a

doctor injecting a patient and semiological action of

the doctor making sense of signs using his

knowledge and cognition to diagnose the patient.

1.2 Definitions

Further details and explanations can be found in

(Michell, 2013).

Michell V.

Cognition Capabilities and the Capability-affordance Model.

DOI: 10.5220/0004774300860095

In Proceedings of the Third International Symposium on Business Modeling and Software Design (BMSD 2013), pages 86-95

ISBN: 978-989-8565-56-3

Table 1: Definitions.

1.3 Objectives

In this paper we explore how the capability-

affordance model can be used to describe cognition

and semiological actions with five objectives:

a) To set out the reasoning for semiological

affordances and their relationship to cognitive

affordances

b) To develop a model of semiological affordances

and their relationship to existing work

c) To identify semiological mechanisms

d) To identify semiological paths

e) To account for creativity.

Section 2 explores the definition of cognitive

affordance and its role in semiological resource

interaction and defines cognitive capability as an

interaction of agent cognitive and knowledge

resources. Section 3 reviews cognitive resources

using human cognitive behaviour. Section 4 reviews

agent knowledge resources based on cognitive

architecture research and identifies cognitive

mechanisms based on semiotics models. Section 5

proposes an integrated cognition model. Section 6

discusses the implications of the model and Section

7 and 8 concludes and identifies future work.

2 SEMIOLOGICAL CAPABILITY

As mentioned before, this section considers

cognitive affordance and its role in semiological

resource interaction, and defines cognitive

capability.

2.1 Cognitive vs Semiological

An action (a) is a transformation of resources

(Michell, 2011). A semiological action uses signs

perceived by the agent from the environment to

process possible actions (Stamper et al., 2000)

Semiological action depends on information and

knowledge from sensors and cognitive actions in the

mind of the agent. The interaction between the

cognitive mechanisms of the brain and knowledge

create possible ideas for action – cognition

capabilities Ccog. Using the capability affordance

model the cognitive affordance mechanism relates to

the interaction of cognitive resources (Boy, 1998),

one of which must be an agent’s mind and its

cognitive mechanisms Rcog. The second represents

tangible or intangible resources ie data, information

and Knowledge: Rk, obtained from the environment

as only this is able to interact mentally in an

affordance pair in the agent.

Ccog = f(agent cognitive mechanism x data,

information, knowledge interaction) = f(Rcog x Rk)

To understand these affordances we need to

understand theories of cognitive brain function.

However, while the interaction mechanism of non-

human objects is well understood, the mechanism of

the brain is not.

2.2 Distributed Cognition Theory

Distributed cognition theory provides a framework

for cognition spaces based on cognitive psychology

(Zhang and Patel, 2006). We extend Zhang’s model

using a physical internal space Sp and a cognitive

space Sc. Sp comprises the biochemistry and

physiology resources of the body (B) ie the

mechanism of biochemical reactions and the

physical structures such as bone and flesh,

synapses., muscle (P) that provide the path for the

animal to work and select physical affordances. The

cognitive space Sc comprises the mechanisms of

perception and cognition (C) and the data

information and knowledge (K) and relates to

semiological affordances. The External space

represents the environment and the natural and

manmade (technology) structures (S) than provide

physical and cognitive affordance possibilities as a

result of agent sensor information (I) and actuators

such as hands.

We now identify the agent cognition and

knowledge resources and the mechanisms AMcog

(C ) and affordance paths APcog relating to data,

information and knowledge (K) that enable

semiological capability.

Cognition Capabilities and the Capability-affordance Model

Figure 1: Affordance Spaces (adapted from Zhang).

3 AGENT COGNITIVE

RESOURCES

3.1 Cognitive Behaviour Modelling

To develop an understanding of the cognition

resources and their mechanisms AMcog we

investigate applied psychology cognitive behaviour

literature. Norman’s 7 stages theory of action can be

used to model cognition (Zachary et al., 1998). A

worker will have specific goals G and actions to

execute E to achieve them eg a surgeon examining a

patient. This may involve a cascade of sub-goals eg

visually examine patient talk to them, feel them.

Perception P involves recognising patterns of

speech, images from sensors and haptic patterns.

Figure 2: Norman’s 7 Stages of Action (adapted).

The agent interprets the perception Ip in terms of

how it relates to a planned course of action. The

agent evaluates options for action E vs the goals and

selects the best plan of action to achieve the goal.

The user then plans his actions in his mind makes

the intension In to act and specify the sensor-motor

driving actions S. The agents actuators eg hands,

limbs execute the action E. The result of the action is

then perceived through the senses and compared to

the goal and any corrective action applied. This P,

Ip, E, G, In, S, E loop can be considered to be

carried out at different levels of cognition.

3.2 The SRK Model

Rasmussen identified three types of cognitive

behaviour in the skills, rules, knowledge in the SRK

model of human decision making in high risk work-

system domains (Rasmussen, 1983). Skill based

behaviour corresponds to sensory motor

performance during unconscious actions and

unconscious control where the human sensor-motor

system acts automatically based on the agents tacit

knowledge of learnt tasks using learnt perceived

patterns (Albrechtsen, 2001 ). Continuous 4d space-

time data signals from agent sensors update a model

of the environment in episodic memory. Skill based

behaviour SBB relates to learnt sensory motor

patterns based on previous experience, eg changing

gear when driving etc and is termed unconscious

control because of its automatic response directly

from perception through the 7 stage model to sensor-

motor action. Conscious control involves greater

cognition using rule based behaviour RBB where

recognition of signs and cues from the sensors drive

an ‘if-then rule’ behaviour based on stored action

rules related to cues perceived from the

environment. This depends on perceptions of

familiar patterns in a familiar environment matching

the necessary cues/signs associated with the action

rule conventions (Vicente and Rasmussen, 1992).

Cues/signs relate either to experiences or

learnt/cultural behaviour encoded as rules labelled

for ‘states/situation or goals and tasks’ (Rasmussen,

1983). Rule based behaviour cannot generate new

rules, but this occurs in highest level of behaviour,

knowledge based behaviour KBB where the agent’s

mental model of the world enables the formulation

of new rules, goals and strategies and predictions of

the response of the environment. KBB can use both

sensor and history information to construct the

conceptual mental model (Albrechtsen, 2001).

Rasmussen’s SRK model identifies the three

mechanisms of cognitive processing as data signals

(SBB), as rules or cue-rule-action mappings or

symbol/concept based problem solving and analysis

actions (Vicente and Rasmussen, 1992).

Third International Symposium on Business Modeling and Software Design

Table 2: Cognitive tasks and actions.

3.3 Cognitive Task Analysis

Work on analysis of cognitive tasks (CTA) for

human computer interfaces (Hall et al., 1995)

(Norman, 1986) and ecological interface design

(Wong et al., 1998) provide additional models that

relate to cognition. We use the convention of

underlined letters as shorthand for cognitive tasks.

Miller’s vocabulary of actions for mental processes,

human information processing resources and their

related task agents (Lee and Sanquist, 1995), (Table

1a) represents the output of human cognition

resources. Bloom’s analysis of learning (Anderson et

al., 2005) identifies cognitive behaviours related to

cognition and learning. Recalling and remembering

knowledge is considered the lowest cognitive level

of activity that we can relate to SBB. With the

action of comprehending or understanding

knowledge at a higher level, that then enables the

ability to apply the knowledge as rules and actions.

The analysis activity is considered a higher level

activity still with evaluate and finally create (new

knowledge) and problem solving as the highest level

of cognitive activities in terms of complexity and

abstraction in the cognitive process. This relates to

KBB (Table 1b), where cognitive tasks relate to

transformation actions of agent mental cognition

resources.

4 AGENT KNOWLEDGE

RESOURCES

4.1 Cognitive Architectures

There are three main types of knowledge. Knowing

what or ‘learning by using relates’ to the use of

systems or technology as encoded in human episodic

memory and can be loosely approximated to SBB.

Knowledge from ‘learning by doing’ ‘know how’

corresponds to RBB (Carud, 1997). Rules are

evident in the external environment as procedures,

policies, processes and as tacit codified rules in

agents. KBB relates to ‘know why’ or knowledge

gained by ‘learning by studying’ as well as concepts

and relationships and tacit mental models created in

the mind (Carud, 1997). Cognitive architecture

theories; SOAR, EPIC, ACT-R, PARI) (Laird et al,

1987), can provide insight into knowledge

interaction. We use the COGNET knowledge

framework (Zachary et al., 1998) based on cognitive

psychology and goal oriented models based on

Rasmussen’s and Norman’s approach (Figure 3).

Managesattention

between cogni tiv e

tasks

DeclarativeKnowle dge(Kdec)

ProceduralKnowledge(Kproc)

PerceptualKnowledge(Kper)

Coded

symbols

Environment

ActionKnowledge(Kact)

Exter na l Acti o n

Acti vati on

Context

knowledge

Taskacti va tio n

Agent

New task based

con tex t

knowledge

Symbol

signs

4ddata

Sensory‐motor action

Sense/Perception

process

AxiologicalDecisionprocess

Epistemic

Evaluationprocess

Internal

Semiosi s Acti on

Acti vati on

Figure 3: COGNET Knowledge Framework (adapted).

4.2 Perceptual Knowledge and

Reasoning

In the COGNET model information from the agent’s

visual, aural haptic etc sensors is converted into 4d

space time signal information. Sensory cues are

perceived as visual (eg images) and auditory (eg

conversation) patterns (Zachary et al, 1998). This

sensory cue information in the form of signs

becomes meaningful by the process of perception

recognising patterns in the signals. COGNET uses

the term ‘sensory demons’ to refer to system

interface displays or patterns of natural phenomena

eg a red spotted mole disease pattern. Auditory

demons include speech terms and acts that have

specific semantic significance. The interpretation of

Cognition Capabilities and the Capability-affordance Model

signs in the environment is the process of semiosis

(Pierce, 1935) and a cognitive abduction process of

inference to select the best semantic meaning of a

sign eg for visual and aural recognition of patterns

(Magnani and Bardone, 2006) as in perception

related to a knowledge base of experience. In

Pierce’s process of semiosis, sensor signals for

objects in the environment are perceived as signs

and symbols representing objects and their meaning

for the agent are decoded in a process of connotation

(Benfell et al., 2013). The resulting perceptual

knowledge Kper relates to pattern recognition

models that link meaning to a visual model based on

working memory and with experience from long

term memory these provide ‘coded symbols’ that

can be used by other cognitive processes.

4.3 Declarative Knowledge and

Reasoning

Declarative knowledge, Kdec, includes the agent’s

mental conceptual model of the world based on the

concepts identified through symbols and semantics

mentioned earlier (Clark and Feldon, 2006) ie

‘knowledge as a conceptual structural model’

(Albrechtsen, 2001). It comprises abstract construct

symbols which unlike rules cannot be reduced to

signs (Rasmussen, 1983). It focuses on what and

why and is based on propositions, facts and is

hierarchically structured and uses episodic memory

to record environment events and model them (Clark

and Feldon 2006). The knowledge ranges from the

real world ie 2D/3D models of the physical

environment to logical relations and logic rules,

facts, beliefs and solution strategies and cases to

behavioural models and abstract models that have no

equivalent in the real world environment.

Declarative knowledge is used for problem solving

and includes the history of relevant objects to the

task and also plans and solution strategies to achieve

a goal (Zachary et al., 1998). KBB relies on the

individual tacit declarative mental model that the

agent constructs that differs from agent to agent.

Cognitive interpretation processes operate on the

declarative model schema to make connections

between symbols that enables insight and new

knowledge to be developed to support problem

solving. As Pirolli et al asserts Information=>

schema=> insight=> solution (Pirolli and Card,

2005). The power, capability and reliability of agent

KBB depends on the power, capability and

reliability of the symbolic mental model and

conceptual processing and related affordances. The

range and complexity of declarative conceptual

model covers predicates, definitions semantic

relations (structural, causal, functions) and simple

and complex associations as well as rules, facts and

beliefs (Aamodt, 1991).

4.4 Procedural Knowledge and

Reasoning

Procedural knowledge resources (Kproc) relates to

models of rules, for example national language rules,

job context rules eg clinical rules, mathematics rules

etc. The rules are encoded with cues for when a task

is relevant. Organisational semiotics models these

rules in the form of norms (Stamper et al., 2000).

The COGNET model suggests the goal and

procedural knowledge form a cognitive task which

directs the use of the knowledge through the

cognitive mechanisms to execute a semiological or

substantive task if the goal cue is recognised.

Tcog = f(goal, procedural knowledge) = f(G,Kproc)

Cognitive tasks are managed at a meta-cognitive

level of reasoning to decide which course of action

to take and when to take it. These evaluation

mechanisms are axiological mechanisms (Benfell et

al., 2013) for decision making and selection of

strategies related to agent internal resources

affording decision making and evaluation. Meta-

cognitive tasks adjust the priority of these tasks if

interrupted by perceived events in the external

environment eg bells ringing, 4d scene changes etc.

Interpretation Mechanisms. Reasoning involves

interpreting information (sign/rule processing and

symbols) about an environmental situation.

This could be from sensors and processing this

information in conjunction with knowledge to

produce a given goal. Reasoning may be rule based

as in RBB. Sign/rule processing relates to the if-then

reasoning using procedural knowledge ie selection

of the best rule according to sign cues from the

external environment. Alternatively it may be based

on models and cases in the symbolic conceptual

model as in KBB. Symbolic processing relates to the

processing of symbols and read, write and update of

the declarative knowledge model. This knowledge

based processing capability unlike rule based

processing is adaptive to new environments where

new knowledge and rules can be developed by

knowledge based reasoning (Albrechtsen, 2001). At

higher level this involves processing mechanisms for

inference methods. These methods may be deductive

based on logical mental models and theories or

inductive (knowledge of events and instances)

developed from experience, education and training

Third International Symposium on Business Modeling and Software Design

Physical

Empi ri cs

Syntactics

Semantics

Pragmati cs

Structure ofsigns, language,syntax,codes

Formofsignseg physicalstructure

Rulesforuseofsigns

Meaningofthesign

Actionsbasedonmeaningofthe sign

Soc i aleffects

Socialimplicationsofthe sign

• Logicmodel s

• Semanticmodel s

•

Belief



model

• Intentions

• Responsibilities

• Cultural models

• Culturerules

• businessrules

• Behavi ourrules/no

• Languagerul es

eg English,maths

• Heuri stics

• Sensor‐motorrul es

• Sensor‐motorheuri

• Natura lworl dactio

• Tec hnic alworl dact

• 2D,3Dshapes

• 4Dspace‐ti me

• Fa cts

• Abs tra ctpa tterns

• Domai npa tterns

• Pl ans,strategies,cases

Figure 4: Semiotic knowledge hierarchy (after Stamper).

(Aamodt, 1991). Other forms of reasoning include

induction and deduction based on neural connections

made during learning these techniques from others.

Deductive reasoning is based on logic models that

may be learnt or culturally developed. Inductive

reasoning involves testing against hypotheses.

4.5 Action Knowledge and Reasoning

Action relates to substantive action tasks on the

technical or natural environment. For example,

pressing buttons, moving objects. Alternatively they

may relate to semiological actions such as

communication (Zachary et al., 1998) or thinking

that changes the state of the mind but has no external

impact. Action knowledge relates to sensor-motor

knowledge of how to drive and control the agent

actuator bio-mechanics to control movement eg of

hands/body. This relates to skills under automatic

control in terms of hand-eye coordination driven by

environmental cues for physical tasks. Action

knowledge includes sensory-motor knowledge for

natural objects as well as man-made technology eg

hand-eye coordination for drug injection or routine

mental maths calculations.

4.6 Knowledge Summary

We have seen an agent’s knowledge covers a range

of semiotic ladder levels. Rules and schemas may

relate to the physical world. They may relate to use

of language and the syntax for example sentence

construction or mathematical expressions. They may

relate to expected behaviour encoded from

experience or business rules to cultural rules about

behaviour. Rules may relate to formal models eg

laws or they may be informal may be developed

from experience, for example rules of thumb. Rules

may relate to both physical and mental behaviours

(Benfell et al., 2013). See Figure 4.

5 THE

COGNITION-AFFORDANCE

MODEL

Cognition affordance relates to a) the interaction of

cognitive reasoning processes of the brain with

knowledge to model potential actions and strategies

for the real or imaginary agent world and b) the

selection the best course of action either physical or

mental. Cognition affordance depends on semiosis

of perceived signs interpreted as sensor signals from

the real world and/or from the conceptual world of

the imagination of the agent using these signals,

signs, symbols to model the real and imaginary

worlds to plan actions. Cognition capability depends

on the mechanism of cognitive tasks (Ct) acting on

cognitive knowledge ie Ccog = f(Ct xCknow). The

Cognition Model (figure 5) based on organisational

semiotics EDA model (Liu et al., 2013) identifies

and integrates cognitive processes in conjunction

with the models discussed earlier.

PERCEPTUAL

REASONING

EPISTEMIC

REASONING

ACTION

REASONING

AXIOLOGICAL

REASONING

SBB

KBB

RBB

Mode ls (s ymbols)

• Domain3d,4d

• Facts/beliefs

• Logi ca l  relations/rules

• Strategies,cas es

• Goals

• Conceptual relations

Rules(s i gns ‐cues)

• Cog/meta cognitiverules

• Culturerules

• businessrules

• Behavi ourrules/norms

• Language rules

• heuri s tics

Patterns

• 2D,3Dshapes

• 4Dspace‐time

• Abs trac tpa ttern s

• Domainpatterns

• Sensor‐motorrules

• Sensor‐motorheuristics

• Natural worl dacti ons

• Techni cal worl dactions

Signs /cues

Whatispossible andwhen?

What?

Why?

How?

Interpet

Perce

Evaluate

Act

Kdec

Kper

Kproc

Kact

Figure 5: Cognition model.

Perception relates to pattern recognition

reasoning on perception knowledge Kper. Epistemic

Cognition Capabilities and the Capability-affordance Model

reasoning relates to inference processes and

reasoning about conceptual declarative knowledge

Kdec. Axiological reasoning relates to rule based

decisions about behaviour operating based on

procedural knowledge Kproc. See Figure 5.

5.1 Perception Capability and

Reasoning

Perceiving patterns in the environment relates to

lower order cognitive tasks such as recalling RE and

comprehending CO. This depends on matching

perceptive knowledge ie the range of pattern

databases the agent possesses or making sense of

new patterns via epistemic reasoning. For example a

clinician in seeking to identify a disease needs to

match the cues from the patient in terms of

visual/aural/haptic information after examining and

talking with the patient and reading their notes. The

clinician may build a perceptual model of the patient

based on disease patterns, physiology patterns and

models of consequences (Chapman et al., 2002).

This produces a number of affordance options as

possible disease models that need to be interpreted

based on their plausibility relative to perception.

This represents a cycle of Norman’s model to meet

the goal and intention of identifying the disease

model for the patient.

Cper= f (perceptual reasoning tasks x Kper)

Audio perception is a function of the ability to both

record and to match aural patterns sensed in the

environment as a language of sounds with meaning.

A language affordance eg ‘can speak English’

depends on the action of the cognitive processing

resource recalling (RE) and matching information

from the language patterns and the quality; depth,

range of the agent vocabulary Klang.

Cper= f(Alang) = f(RE x Klang)

Similarly, visual perception affordances Avip are a

function of the ability to both record and to match

visual patterns sensed in the environment to

meanings. For example visual perception affords

recognition of disease patterns by recalling (RE) and

comprehending (CO) disease patterns and cues that

best match the sensed disease pattern which depends

on the cognitive action of recall and its interaction

with disease knowledge.

5.2 Interpretation Capability Cepi

Cepi - Epistemic reasoning involves higher

cognitive tasks such as analysis AN, synthesis SN,

problem solving PS and creativity CR as useful

strategies in unfamiliar situations. Reasoning

strategies such as induction, abduction and

deduction may be used. The use of epistemic

reasoning in medicine is often referred to as

hypothetico-deductive reasoning (Chapman et al.,

2002) and involves establishing a hypothesis for the

problem illness, gathering data to support or refute

the hypothesis followed be evaluation to establish

the best causal reasoning (ie know why’) for the

symptoms. This requires conceptual knowledge

models of illness, disease functionalities,

mathematics etc. Affordance options relate to the

different problem- solution models and their

plausibility vs goal/evidence ie Cepi = f (prob

solving reasoning x Kdec). The capability of

epistemic reasoning as in expertise, is complex and

in any individual will vary with the ability to reason

and conceptually model the world and the depth,

specificity and form of the knowledge the agent is

able to develop (Aamodt, 1991).

5.3 Evaluation Capability Caxi

Caxi refers to: Axiological reasoning.

It relates to cognitive tasks using RBB and

decision processes to select the best rule given

environmental or mental cues. Affordance options

here relate to the permutations of the possible meta-

cognitive actions and their sequences and the

cognitive task based are different rule models and

their plausibility vs cues and the action goal.

Caxi = f (rule reasoning x Kproc)

Cognitive tasks relate to actions on procedural and

declarative knowledge. Cognitive tasks include

understanding and problem solving where obvious

rules can’t be invoked and declarative knowledge is

required. This may include information processing

strategies such as Miller’s cognitive tasks planning

PL and controlling CT (Lee and Sanquist, 1995).

The rule evaluation takes place at different levels in

the semiotic ladder. From the evaluation of laws and

policy rules down to process and action rules. The

rules act as a constraint on the possible actions. A

clinician has many different rule sets to follow. At

high level may be policies and WHO guidelines at

the process level clinical pathways can be selected to

guide possible team actions. At the action level

algorithms (eg for inserting catheters) and the

clinicians own heuristic rules developed from

experience. The affordance options relate to

different disease/illness rule models and the

cognitive process involves the clinician deciding

Third International Symposium on Business Modeling and Software Design

which rules to apply by assessing a series of

permutations of cues and disease patterns.

5.4 Action Capabilities Cact

Action capabilities relate to automatic actions ie

unconscious thought and skill based actions. This

includes human-environment sensory motor skills eg

grasping, human-technology skills eg using a mouse

to drag and drop and human-human interactions eg

shaking hands etc. A medical example might be

identifying the actions and behaviours to stabilise an

emergency patient (Chapman et al., 2002).

Affordance options are different action models and

their plausibility vs cue/stabilisation goal. Here the

clinicians react instinctively to act based on

experienced action knowledge of the steps to take

how to behave and use equipment and human

resources based on the cues for action from the

patient, colleagues, technology resources and the

situation ie Cact= f (action reasoning x Kact).

6 DISCUSSION

Cognition axes are proposed in this section as well

as cognitive capabilities.

6.1 Cognition Axes: Real Vs Imaginary

We can say cognition capability is a tuple of these

four capabilities:

Ccog = f (Cper, Cepi, Caxi, Cact)

In all the above cases the cognitive capability of

the clinician will depend on experience, cognitive

ability and cognitive resources. As Gibson notes:

interpretation depends on the agents culture,

experience and intentions (Benfell et al., 2013). The

following sections discuss differences in cognitive

capability. The agent cognitive affordance space Sc

relates to how the mental models and reasoning

process can provide the agent with alternative action

possibilities. This represents the interaction of the

cognitive reasoning mechanism resources with

memory and knowledge. The axis of linkage

between perception-action represents affordance

possibilities in the real world of seeing and doing ie

perceived and cognitive affordances. HCI design

depends on making possibilities of using the

technology as obvious as possible so they can

directly be used for action. We suggest the linkage

between interpretation and evaluation represents the

imagination where possible concepts and possible

actions can be modelled and the implications tested

before action is decided. Another aspect of cognitive

affordance is the possibility to imagine or model

new imaginary interactions and imaginary logics and

languages. Imagination suggests a conceptual

environment that can model a) the real world and its

features and use it to identify possible future states

and secondly b) to model imaginary worlds with

different rules, logic and beliefs. This capability

enables great works of literary fiction (Harry Potter),

art (Salvador Dali), science (relativity theory).

Techniques such as brainstorming and creative

methods where normal logic and beliefs are

suspended can sometimes highlight new possibilities

where the imaginary world highlights a new

possibility or creative affordance applicable to real

world logic. Perhaps the process of dreaming is

nature’s mechanism for trying out possible illogical

affordances that would not have occurred to the

conscious animal having to make sense of a real

physical world!

Real worl d

Model axis

Imaginaryworl d

Modelaxis

Real world rules

&constra i nts

Modelled

worl d

Per

Act

Eval

Int

Perception

‐actionworld

seeing

doing

imagining

Figure 6: b) real vs imagi ned world intersections

Imaginaryworld

model

Realwor ldmodel

Brainstormingsolutions

Figure 6: Cognition Axes – real vs. imaginary.

6.2 Comparing Cognitive Capabilities

Experience and practice in each of the cognitive

mechanisms will be determined by the roles and

work the agent carries out and how much involves

real world vs imaginary world models. Some job

roles involve a greater focus on seeing and doing eg

a nurse, artist. Other roles focus more on

imagination and conception without action as in the

theoretical world eg novelist, scientist. Different

roles will exercise and focus different cognitive

functions. Some involve a combination eg a knee

surgeon, may have good levels of perception of

disease, 3d structures and have good haptic

Cognition Capabilities and the Capability-affordance Model

perception with much experience of manipulating

joints. He may have lots of logic and problem

solving experience, some experience of using

clinical pathway rules, but will have less experience

in policy rules and in reasoning about them

compared to a Medical Director. An anaesthetist

may have better abilities for reasoning about drugs

based on repeated experience. In contrast an artist

may be more creative than the surgeon because they

focus their life on painting which involves seeing-

doing action experience in terms of visual perception

and painting heuristics. Their imagination is less

structured and may involve creating and using

imaginary rules and concepts. In contrast an

architect’s creativity will be more structured as it is

limited by the rules of physics. Natural abilities to

perceive, follow rules, to reason will also depend on

the brain physiology as well as experience.

Ccog = f(Experience Ex x Reasoning Ability Ab)

6.3 Knowledge Paths

Knowledge is developed by the process of learning

from experience and/or use of cognitive capabilities

and mental modelling or taught or communicated by

others. In each and every case both the knowledge

and the cognitive capabilities is potentially growing

depending on the brain physiology and individual

cognitive capabilities ie intelligence of the

individual. Benfell et al., (2013) mentions the link

between affordance and memes in which

affordances ie ideas are communicated ‘by reading,

watching television etc’ It is this exposure to

affordance examples that enables direct copying.

Alternatively we may use our existing knowledge

and cognitive skills to playing with and mutate

ideas. We may use existing logic to extend our

knowledge or, depending on our capabilities, create

new logic to produce new knowledge. This happens

at different levels of the semiotic ladder from the

knowledge of the physical world through formal to

informal abstractions such as culture knowledge

with learned formal and or informal rules/heuristics.

See Figure 7.

existi ng

pa tterns ,

logic

alternative

imaginary

logic/play

Exi sting

Knowledge

Appl y

Copy

Mutate

Build

Extend

Figure 7: Knowledge paths.

7 SUMMARY AND

CONCLUSIONS

This paper has shown how the capability-affordance

model and others can provide an approach to explain

the possible interactions of mental resources that

produce creative thought possibilities and cognition

affordances and meet our objectives. Section 2

explained the reasoning for semiological affordances

by identifying the internal cognitive space and its

relationship to cognitive affordance - objective a).

To develop a model of semiological affordances

(objective c) in Section 3 we identified cognition

resources using the SRK model and cognitive task

analysis. In section 4 we proposed agent knowledge

resources based on cognition architecture and we

suggested how semiotics and Peirce’s model relate

to cognition mechanisms. In section 5 we showed

that Cognitive capability depends on cognitive

functions interacting with knowledge and proposed

4 mechanisms for cognition. We proposed in section

6 that the cognitive path (objective d) depends on the

knowledge paths where existing knowledge is

applied and grown by copying or mutation. Mutation

can occur as a result of mind games, mental playing

and imagination. This mutation may use existing

logic of Ccog to creative new knowledge directly

applicable to the real world. or, to create new

knowledge that is not directly applicable as

theoretical knowledge. To account for creativity

(objective e) we proposed how perceived and

cognitive affordances relate to the perception-action

axis and the epistemic-axiological axis relates to

mental theoretical models to extend the real world

model or create new imaginary worlds as in

creativity.

8 FUTURE WORK

The complexity of cognition (the process of using

cognitive actions and knowledge) means we have

only scratched the surface. Further work is needed to

identify the detailed mechanisms of cognitive

affordance permutation with specific examples using

cognitive task analysis. For example how does a

novelist or artist think compared with a surgeon.

However whilst methods such as Cognitive task

analysis can provide useful insights into the

processual mechanisms, the detailed models rely on

developments in cognitive psychology and medical

research.

Third International Symposium on Business Modeling and Software Design

REFERENCES

Aamodt, A. (1991). A knowledge-intensive, integrated

approach to problem solving and sustained learning.

Knowledge Engineering and Image Processing Group.

University of Trondheim, 27-85.

Albrechtsen, H., Andersen, H. H. K., Bødker, S., &

Pejtersen, A. M. (2001). Affordances in activity theory

and cognitive systems engineering.

Anderson, L.W., Krathwohl D.R.,Bloom B.S.. (2005) A

taxonomy for learning, teaching, and assessing.

Longman.

Barentsen, K., and Trettvik, J. (2002). An Activity Theory

Approach to Affordance, Second Nordic Conference

on Human Computer Interaction: ACM, pp. 51-60.

Benfell A., Williams R., Liu K. (2013)Enriching

enterprise resource planning systems for strategic

advantage: A semiotic motivated approach.

Proceedings of ICISO, Stockholm.

Boy, G. A. (1998). Cognitive function analysis (Vol. 2).

Praeger Pub.

Carud, R. (1997). On the distinction between know-how,

know-why, and know-what. Advances in strategic

management, 14, 81-101.

Chapman, D. M., Char, D. M., & Aubin, C. D. (2002).

Clinical decision making. Rosen1s Emergency

Medicine concepts and clinical practice, 125-133.

Clark RE., Feldon D.F.(2006): "Instructional implications

of cognitive task analysis as a method for improving

the accuracy of experts’ self-report." Avoiding

simplicity, confronting complexity: Advances in

studying and designing (computer-based) powerful

learning environments 109-116.

Hartson, R. (2003). Cognitive, physical, sensory, and

functional affordances in interaction design. Behaviour

& Information Technology, 22(5), 315-338.

Hall, E. P., Gott, S. P., & Pokorny, R. A. (1995). A

Procedural Guide to Cognitive Task Analysis: The

PARI Methodology (No. AL/HR-TR-1995-0108).

ARMSTRONG LAB BROOKS AFB TX HUMAN

RESOURCES DIRECTORATE.

Laird, .J. E., Newell, A., Rosenbloom P.S..(1987) Soar:

An architecture for general intelligence. Artificial

intelligence 33.1 1-64.

Lee, J. D., & Sanquist, T. F. (2000). Augmenting the

operator function model with cognitive operations:

Assessing the cognitive demands of technological

innovation in ship navigation. Systems.

Liu S., Liu K., Li W. (2013) A multi-agent system for

pervasive healthcare. Proceedings of ICISO,

Stockholm.

Magnani, L., & Bardone, E. (2006). Designing human

interfaces. The role of abduction. Computing,

Philosophy and Cognition. College Publications.

Michell, V.A.(2011) A Focused Approach to Business

Capability. First International Symposium on Business

Modelling and Software Design – BMSD 2011, Sofia,

Bulgaria, pp. 105–113.

Michell V. (2013) The Capability Affordance Model:

Comparing Medical Capabilities. In: B. Shishkov

(Ed.) Business Modeling and Software Design –

BMSD’12 Revised Selected Papers, Springer-Verlag –

Lecture Notes in Business Information Processing,

Berlin-Heidelberg.

Pierce C. S. (1931-35), Collected Papers, (6 volumes),

Hartshome C. & P. Weiss (eds.), Cambridge, Mass.

Harvard U.P.

Pirolli, P., & Card, S. (2005, May). The sense making

process and leverage points for analyst technology as

identified through cognitive task analysis. In

Proceedings of International Conference on

Intelligence Analysis (Vol. 5).

Rasmussen, J. (1983). Skills, rules, and knowledge;

signals, signs, and symbols, and other distinctions in

human performance models. Systems, Man and

Cybernetics, IEEE Transactions on, (3), 257-266.

Stamper, R., Liu, K., Hafkamp, M., Ades,

Y.(2000)Understanding the Role of Signs and Norms

in Organisations. Journal of Behaviour & Information

Technology 19(1), 15–27.

Vicente, K. J., & Rasmussen, J. (1992). Ecological

interface design: Theoretical foundations. Systems,

Man and Cybernetics, IEEE Transactions on, 22(4),

589-606.

Wong, W,B,L.Sallis P.J., O'Hare. D., (1998) The

Ecological Approach to interface design: Applying the

Abstraction Hierarchy to intentional domains

Computer Human Interaction Conference,

Proceedings. Australasian. IEEE.

Zachary, W., Ryder, J. M., &Hichinbothom, J. H. (1998).

Cognitive task analysis and modelling of decision

making in complex environments. Making decisions

under stress: Implications for individual and team

training, 315-344.

Zhang, J., & Patel, V. L. (2006). Distributed cognition,

representation, and affordance. Pragmatics &

Cognition, 14(2), 333-341.

Cognition Capabilities and the Capability-affordance Model