Bringing Psychological, Affective and Motivational Relevance

Frameworks to Real Information Retrieval Systems

Oswald Barral

Helsinki Institute for Information Technology HIIT,

Department of Computer Science, University of Helsinki, Helsinki, Finland

1 RESEARCH PROBLEM

The concept of relevance has been widely studied in

the ﬁeld of information science and a strong theoret-

ical background has been developed around it. It is

recurrent to ﬁnd distinctions between objective rele-

vance, which is intrinsically dependent on the item to

evaluate; and objective relevance, which is dependent

on the perception that the user has of such item. In-

formation systems, namely information retrieval sys-

tems, are commonly based on objective, algorithmic

relevance metrics. These metrics are especially well

suited as they are easily quantiﬁable measures of rel-

evance. Nevertheless, objective relevance alone is

far from indicating the real relevance of an item in

a given context. It is evident that subjective relevance

assessments play a strong role in the success of an

information-seeking task. Subjective relevance has

multiple facets, and in the proposed research we aim

to address a component that has been deﬁned in the

literature as affective, psychological or motivational

relevance. It can be understood as the relationship

between the user’s current emotional and cognitive

state and the information item. By studying this com-

ponent, we tackle directly the information need and

its relationship between the user’s current emotional,

cognitive and affective state. To give a simple exam-

ple, during a complex information-seeking task, an

item can be perceived as relevant when the user is en-

gaged into the search task, but perceived as irrelevant

when the user feels overwhelmed, frustrated or bored.

2 OUTLINE OF OBJECTIVES

In the present research we aim to apply state-of-the-

art physiological computing techniques in order to

model psychological, affective, and motivational rel-

evance. That is, we want to infer the users’ cogni-

tive, affective and emotional state while engaged in

an information-seeking task by analyzing their psy-

chophysiological responses associated with the pre-

sented information items. Such states will be modeled

in the system allowing for an online bio-cybernetic

loop between the user and the system (adapting the

information space, retrieval engine and user interface

accordingly). By doing so, we aim to enhance the per-

formance of current information retrieval systems in

complex search scenarios such as exploratory search

tasks, while improving user experience.

Figure 1: Overview of the proposed ecosystem. The psy-

chophysiological responses of the user while interacting

with the system are fed into the affective relevance model,

which is inputted to the system. The information presented

to the user is therefore inﬂuenced by his/her own cognitive,

affective and motivational state.

3 STATE OF THE ART

In information science, relevance has been theoret-

ically analyzed in depth over the past forty years.

Saracevic (Saracevic, 1975) carried out the ﬁrst com-

prehensive review in information science around the

concept of relevance back in 1975. He compiled rele-

vant work mainly from the two previous decades, ad-

dressing relevance from philosophy all the way to rel-

evance in information retrieval systems. Additionally

he aimed at providing a framework to study relevance

within the ﬁeld of information science. Later on,

Saracevic updated his review as the ﬁeld was growing

and the discussion was enlarging (Saracevic, 2007a;

Saracevic, 2007b). However, Saracevic has not been

the only researcher to study the multifaceted aspects

of relevance. Schamber (Schamber, 1994) or Mizzaro

Barral O..

Bringing Psychological, Affective and Motivational Relevance Frameworks to Real Information Retrieval Systems.

 2015 SCITEPRESS (Science and Technology Publications, Lda.)

(Mizzaro, 1997), are examples of authors that made

a substantial contribution during the 90’s by writing

comprehensive and historical reviews on relevance.

We encourage interested readers to go through their

work in order to gain a deeper insight in the history of

relevance in information science.

A large consensus exists within the community in

considering relevance as multifaceted, dynamic and

with a high subjective component (Saracevic, 1997;

Mizzaro, 1998; Ingwersen, 1996; Cosijn and Ingw-

ersen, 2000; Borlund, 2003; Borlund and Ingwersen,

1998). In the following sections, we will tackle some

of the subjective aspects of relevance, leaving out of

the analysis objective and algorithmic measures of

relevance. Speciﬁcally, we will go through three rel-

evance components addressed in the literature by dif-

ferent authors, that are highly related and intercon-

nected, and which we believe can be studied together.

These components deﬁne the relationship between the

user state, the information need and the information

item.

3.1 Psychological Relevance

The concept of psychological relevance was ﬁrst

raised in cognitive sciences, in the popular book “Rel-

evance: Communication and Cognition” by Sperber

and Wilson (Sperber and Wilson, 1986). In the book,

the authors deﬁne a framework for relevance in com-

munication and cognition theory, based on the as-

sumption that human cognition only pays attention

to potentially relevant information. They argue that

relevance is framed in the cognitive system accord-

ing to comparisons between assumptions and a par-

ticular cognitive state. In 1992, based on their theo-

retical framework, and borrowing some of their theo-

retical deﬁnitions, Harter deﬁned psychological rele-

vance applied to the ﬁeld of information science (Har-

ter, 1992). He used the concept of relevance of a phe-

nomena, elaborated in the above-mentioned book by

Sperber and Wilson, as a mainstay for his deﬁnition.

In plain words, this concept refers to the fact that phe-

nomena have a direct impact on the person’s cognitive

state, and therefore affect his assumptions by making

them stronger or more manifest.

Harter deﬁned psychological relevance in infor-

mation science as follows: “Psychological relevance

allows us to talk about an ‘information need’ as the

current context –the cognitive state at a given time–

of an individual who consults an information system”

(Harter, 1992). In his work, he illustrates through ex-

amples how the cognitive state of an individual has

a direct impact on the perceived relevance of infor-

mation items. Moreover, he makes a distinction be-

tween relevance and “weak relevance”. In Harter’s

view, relevance occurs when the fact of accessing an

information item implies a direct modiﬁcation of the

user’s cognitive state. However, weak relevance oc-

curs when the user anticipates a change on his cogni-

tive state, even though accessing the information item

does not imply a direct modiﬁcation of the cognitive

state. Other variants of the concept of psychological

relevance are well illustraded by Harter through well-

elaborated examples.

Harter’s deﬁnition has been present in the work of

other researchers, mainly when reviewing the litera-

ture, but it has not had so far a direct implication in

the implementation of information retrieval systems.

In the following paragraph, we would like to cite other

authors when rephrasing Harter’s deﬁnition of psy-

chological relevance. By doing so, we hope the reader

will gain a clearer understanding of the concept.

In her review, Schamber (Schamber, 1994)

rephrased Harter’s concept of psychological rele-

vance as follows: “Psychological relevance assumes

that users are actively seeking information that will

change their internal context or store of knowledge,

and that if an information item does this, users will

perceive it to be relevant”. Saracevic (Saracevic,

1996) pointed at Harter’s deﬁnition in the following

terms: “Psychological relevance is viewed as a dy-

namic, ever changing interpretation of information

need in relation to presented texts. It is based on an

assumption (stated as fact) that the ‘searcher’s cog-

nitive state changes and evolves with the discovery of

each relevant citation’.” In fact, Saracevic proposed

the term “cognitive relevance” for the concept, as he

argued that the focus was on cognition, rather than on

psychology. Finally, to summarize, it is pertinent to

cite Mizzaro’s reference to Harter’s work (Mizzaro,

1997): “Harter (1992) applies the theory of psycho-

logical relevance, proposed by Sperber and Wilson, to

the concept of relevance in information science. He

obtains an elegant framework and draws some very

interesting conclusions for IR and bibliometrics.”

3.2 Affective and Motivational

Relevance

As we just discussed, psychological relevance de-

scribes the relationship between the users’ prior

knowledge and cognitive state, and the information

item. Instead, affective or motivational relevance

is deﬁned as the relationship between the users’

intents goals and motivations, and the information

item (Saracevic, 1996; Cosijn and Ingwersen, 2000).

Therefore, affective or motivational relevance is goal

oriented, and is dependent on the user’s current cogni-

PhyCS2015-DoctoralConsortium

tive context (Borlund and Ingwersen, 1998). Sarace-

vic (Saracevic, 1996; Saracevic, 1997) ﬁrst adapted

this term from Schutz’s concept of relevance in phi-

losophy (Schutz and Zaner, 1970). Schutz named

“motivational relevance” as one of the three ba-

sic types of relevance, together with “topical rele-

vance” and “interpretational relevance”, that he de-

ﬁned when studying human relationships in the social

world. According to Schutz, this kind of relevance is

the one that deﬁnes the course of action, the action

to be executed, the selection of a speciﬁc alternative.

Saracevic extends and applies the notion to the ﬁeld of

information science. Moreover, he couples it with the

notion of affective relevance, resulting in the follow-

ing deﬁnition: “Motivational or affective relevance is

the relation between the intents, goals, and motiva-

tions of a user, and texts retrieved by a system or in

the ﬁle of a system, or even in existence. Satisfaction,

success, accomplishment, and the like are criteria for

inferring motivational relevance.”

Having said that, Saracevic’s deﬁnition of affec-

tive relevance has not had an unanimous acceptation

by the community. Researchers such as Cosijn and

Ingwersen (Cosijn and Ingwersen, 2000) or Borlund

(Borlund, 2003) have argued that the concept does

not refer to a speciﬁc type or kind of relevance but,

instead, should be seen as an attribute of relevance

that inﬂuences all other types of relevance. For in-

stance, Cosijn and Ingwersen argue that affective rel-

evance might act as an additional dimension, inﬂu-

encing other subjective relevance types (such as situ-

ational relevance, utility, etc.). Following Borlund’s

point of view, which is in the same direction, motiva-

tional or affective relevance is actually the cause for

users to search. Therefore, according to her, intent,

goals and motivations have to be seen as a character-

istic of all the types of relevance: “Thus, the ‘drive’ to

want information is not an independent, speciﬁc type

of relevance, but an inherent characteristic of rele-

vance behavior in general” (Borlund, 2003).

Interestingly, Cosijn and Ingwersen (Cosijn and

Ingwersen, 2000) discuss the fact that affective and

motivational relevance should be considered as two

different types of relevance. In their view, motiva-

tional relevance is indeed deﬁning the goals and mo-

tivations of the user, hence is seen as an independent

characteristic inﬂuencing all the other kinds of rele-

vance. Affective relevance, instead, is related to the

affects and emotions a user experience when in an

information-seeking task. Namely, they link their un-

derstanding of affective relevance with Barry’s empir-

ical investigation (Barry, 1994), and her types of rel-

evance identiﬁed as “criteria pertaining to the user’s

beliefs and preferences”. Speciﬁcally, Cosijn and In-

gwersen link affective relevance to Barry’s affective-

ness, that she deﬁnes as any kind of emotional re-

sponses to any aspect of a given document.

3.3 Physiological Computing in

Information Retrieval

Physiological computing has gained importance in

the recent years. Researchers have studied how mea-

suring psychophysiological measures can enhance in-

formation systems, by making the system aware of

the user’s cognitive, emotional and affective state (Al-

lanson and Wilson, 2002; Fairclough, 2009; Fair-

clough and Gilleade, 2014). Eye tracking, pupillom-

etry, electroencephalography, cardiovascular mea-

sures, electro-dermal activity or facial electromyog-

raphy are some of the techniques used to measure and

infer user emotional and cognitive state (Cacioppo

et al., 2007). For example, recent research have

shown how it is possible through different physio-

logical channels (electro-dermal activity and pressure

sensors, among others) to detect frustration (Kapoor

et al., 2007). Electroencephalography has been stud-

ied to detect cognitive workload (Gevins and Smith,

2003) or motivational intensity and fatigue (Lorist

et al., 2009), for instance. We believe that some of

these metrics and computational techniques can be

useful in order to address psychological, affective,

and motivational relevance in the scope of a complex

information retrieval system.

Studies have approached relevance prediction

through psychophysiological signals, namely using

eye gaze data (Puolam

aki et al., 2005; Ajanki et al.,

2009; Loboda et al., 2011). Others have used pupil

size (Oliveira et al., 2009; Barral et al., ) and even

recent studies have tried to predict perceived rele-

vance of text items using EEG (Eugster et al., 2014)

or EDA and fEMG (Barral et al., 2015). We would

like to stress that in the present research, we are not

so much interested in predicting relevance directly

from psychophysiological data but focus instead on

studying the relationship between cognitive states and

perceived relevance, in order to model affective, psy-

chological, and motivational relevance. This will in-

directly allow for relevance prediction through psy-

chophysiological data, but in a more informed way,

as we will study which are the cognitive and affec-

tive states that have an impact on perceived relevance

(Barral and Jacucci, 2014).

BringingPsychological,AffectiveandMotivationalRelevanceFrameworkstoRealInformationRetrievalSystems

4 METHODOLOGY

The concepts of affective, psychological and moti-

vational relevance have remained in the theoretical

frameworks of information science as they have never

been applied to information retrieval systems. In

order to address this challenge, and in light of the

broad literature regarding physiological computing

and the strong theoretical background regarding rel-

evance in information science, we base our research

around the research questions stated below. We pro-

pose a methodology to answer each of these research

questions, which are essential to address the above-

mentioned research problem. These questions are

presented in chronological order as, prior to solving

each of the questions, the previous ones need to be

addressed.

Q1. Which physiological measures and which

cognitive states are best suited to indicate psycho-

logical, affective, and motivational relevance? We

plan to thoroughly review the literature on physiolog-

ical measures used to infer cognitive states in order

to comprehensively identify which are the cognitive

states that might potentially inﬂuence perception of

relevance, and which are the physiological measures

that are able to indicate them.

Q2. Which are the information-seeking scenarios

where the modeling of psychological, affective, and

motivational relevance is more pertinent in order to

enhance current information retrieval solutions? We

hypothesize that complex search tasks are best suited

for taking advantage of psychological, affective, and

motivational relevance metrics. Through user stud-

ies, we will measure different cognitive states such

as level of frustration, motivation/engagement, con-

fusion, etc. and their relationship with perceived rele-

vance in different search scenarios in order to identify

the search-tasks where the modeling of psychologi-

cal, affective, and motivational relevance is more per-

tinent.

Q3. How to model psychological, affective, and

motivational relevance? Once the information-search

tasks and the cognitive states have been identiﬁed, we

will deﬁne, implement and test several user models

based on psychological, affective, and motivational

relevance in order to discuss how these models can

be applied to currently available information retrieval

systems.

Q4. To what extend modeling and implementing

psychological, affective, and motivational relevance

in an information-seeking system enhances current

information retrieval solutions? Once the scenarios

and measures have been identiﬁed, and the models

have been implemented, we plan to merge everything

together into a real information retrieval system that

will make use of psychological, affective, and motiva-

tional relevance measures. We will test such system

against current I.R systems in order to study weather

the users’ performance in the previously identiﬁed

information-seeking tasks (Q2) improves.

5 EXPECTED OUTCOME

The main outcome of this research will be an in-

formation retrieval system especially designed to ad-

dress complex information-seeking tasks. This sys-

tem will make use of the users’ cognitive state de-

tected through physiological computing techniques in

order to feed its psychological, affective, and moti-

vational relevance model. The research will lead to

a system that will outperform current information-

seeking solutions by making use of such relevance

model.

Additionally, several outcomes will arise from the

research process while addressing Q1-Q4; namely, a

comprehensive understanding of the different cogni-

tive states suitable to model psychological, affective,

and motivational relevance in order to better support

users and enhance their performance while undertak-

ing information seeking tasks in real scenarios.

6 STAGE OF THE RESEARCH

We have recently explored how relevance of informa-

tion items can be inferred through electroencephalog-

raphy (Eugster et al., 2014) and through electrodermal

activity and facial electromyography (Barral et al.,

2015). The outcomes of this research shows that,

even if such physiological measures carry informa-

tion regarding perception of relevance, their predic-

tive power is still relatively low. We believe that one

way to enhance the applicability of such techniques is,

as exposed above, to study how the relevance of infor-

mation items is perceived in relation to the cognitive,

emotional, and affective states of the users. Thereof,

the current research is focused in this direction.

Firstly, we are deﬁning and designing a review of

the literature regarding psychophysiological inference

of cognitive states in HCI, in order to address Q1.

Also, we are studying how motivational intensity af-

fects the cognitive processing abilities when present-

ing text items in different layouts (eg. list, clusters,

etc.), and how it is reﬂected in the physiology, by

measuring electroencephalography, eye movements

and pupillometry, electrocardiography and electroder-

mal activity. Next steps, and based on the outcomes

PhyCS2015-DoctoralConsortium

of the literature review, we will conduct further stud-

ies regarding users’ cognitive states while engaged in

information seeking tasks, that might be related with

affective, psychological and motivational relevance

such as arousal, frustration, confusion, etc.

REFERENCES

Ajanki, A., Hardoon, D. R., Kaski, S., Puolam

aki, K., and

Shawe-Taylor, J. (2009). Can eyes reveal interest? im-

plicit queries from gaze patterns. User Modeling and

User-Adapted Interaction, 19(4):307–339.

Allanson, J. and Wilson, G. M. (2002). Physiological com-

puting. In CHI’02 Extended Abstracts on Human Fac-

tors in Computing Systems, pages 912–913. ACM.

Barral, O., Eugster, M. J., Ruotsalo, T., Spap

e, M. M.,

Kosunen, I., Ravaja, N., Kaski, S., and Jacucci, G.

(2015). Exploring peripheral physiology as a predic-

tor of perceived relevance in information retrieval. In

Proceedings of the 20th International Conference on

Intelligent User Interfaces (in Press), IUI ’15, New

York, NY, USA. ACM.

Barral, O. and Jacucci, G. (2014). Applying physiologi-

cal computing methods to study psychological, affec-

tive and motivational relevance. In Symbiotic Interac-

tion, Third International Workshop, Symbiotic 2014.

Springer.

Barral, O., Kosunen, I., and Jacucci, G. Inﬂuence of read-

ing speed on pupil size as a measure of perceived

relevance. In Proceedings of the Joint Workshop on

Personalized Information Access (PIA 2014), in con-

junction with the 22nd conference on User Modeling,

Adaptation and Personalization (UMAP 2014).

Barry, C. L. (1994). User-deﬁned relevance criteria: an ex-

ploratory study. JASIS, 45(3):149–159.

Borlund, P. (2003). The concept of relevance in ir. Journal

of the American Society for information Science and

Technology, 54(10):913–925.

Borlund, P. and Ingwersen, P. (1998). Measures of relative

relevance and ranked half-life: performance indicators

for interactive ir. In Proceedings of the 21st annual

international ACM SIGIR conference on Research and

development in information retrieval, pages 324–331.

ACM.

Cacioppo, J. T., Tassinary, L. G., Berntson, G. G., et al.

(2007). Handbook of psychophysiology, volume 2.

Cambridge University Press New York.

Cosijn, E. and Ingwersen, P. (2000). Dimensions of

relevance. Information Processing & Management,

36(4):533–550.

Eugster, M. J., Ruotsalo, T., Spap

e, M. M., Kosunen, I.,

Barral, O., Ravaja, N., Jacucci, G., and Kaski, S.

(2014). Predicting term-relevance from brain signals.

In Proceedings of the 37th International ACM SIGIR

Conference on Research & Development in Informa-

tion Retrieval, SIGIR ’14, pages 425–434, New York,

NY, USA. ACM.

Fairclough, S. H. (2009). Fundamentals of physiological

computing. Interacting with computers, 21(1):133–

145.

Fairclough, S. H. and Gilleade, K. (2014). Advances in

Physiological Computing. Springer.

Gevins, A. and Smith, M. E. (2003). Neurophysiologi-

cal measures of cognitive workload during human-

computer interaction. Theoretical Issues in Er-

gonomics Science, 4(1-2):113–131.

Harter, S. P. (1992). Psychological relevance and informa-

tion science. Journal of the American Society for in-

formation Science, 43(9):602–615.

Ingwersen, P. (1996). Cognitive perspectives of information

retrieval interaction: elements of a cognitive ir theory.

Journal of documentation, 52(1):3–50.

Kapoor, A., Burleson, W., and Picard, R. W. (2007). Auto-

matic prediction of frustration. International Journal

of Human-Computer Studies, 65(8):724–736.

Loboda, T. D., Brusilovsky, P., and Brunstein, J. (2011). In-

ferring word relevance from eye-movements of read-

ers. In Proceedings of the 16th international confer-

ence on Intelligent user interfaces, pages 175–184.

ACM.

Lorist, M. M., Bezdan, E., ten Caat, M., Span, M. M.,

Roerdink, J. B., and Maurits, N. M. (2009). The inﬂu-

ence of mental fatigue and motivation on neural net-

work dynamics; an eeg coherence study. Brain re-

search, 1270:95–106.

Mizzaro, S. (1997). Relevance: The whole history. Jour-

nal of the American society for information science,

48(9):810–832.

Mizzaro, S. (1998). How many relevances in information

retrieval? Interacting with computers, 10(3):303–320.

Oliveira, F. T., Aula, A., and Russell, D. M. (2009). Dis-

criminating the relevance of web search results with

measures of pupil size. In Proceedings of the SIGCHI

Conference on Human Factors in Computing Systems,

pages 2209–2212. ACM.

Puolam

aki, K., Saloj

arvi, J., Savia, E., Simola, J., and

Kaski, S. (2005). Combining eye movements and col-

laborative ﬁltering for proactive information retrieval.

In Proceedings of the 28th annual international ACM

SIGIR conference on Research and development in in-

formation retrieval, pages 146–153. ACM.

Saracevic, T. (1975). Relevance: A review of and a frame-

work for the thinking on the notion in information sci-

ence. Journal of the American Society for Information

Science, 26(6):321–343.

Saracevic, T. (1996). Relevance reconsidered. In Proceed-

ings of the second conference on conceptions of li-

brary and information science (CoLIS 2), pages 201–

218. ACM Press.

Saracevic, T. (1997). The stratiﬁed model of information

retrieval interaction: Extension and applications. In

Proceedings of the annual meeting-American Society

For Information Science, volume 34, pages 313–327.

Learned Information (Europe) LTD.

Saracevic, T. (2007a). Relevance: A review of the literature

and a framework for thinking on the notion in infor-

mation science. part ii: Nature and manifestations of

relevance. Journal of the American Society for Infor-

mation Science and Technology, 58(13):1915–1933.

BringingPsychological,AffectiveandMotivationalRelevanceFrameworkstoRealInformationRetrievalSystems

Saracevic, T. (2007b). Relevance: A review of the liter-

ature and a framework for thinking on the notion in

information science. part iii: Behavior and effects of

relevance. Journal of the American Society for Infor-

mation Science and Technology, 58(13):2126–2144.

Schamber, L. (1994). Relevance and information behavior.

Annual review of information science and technology

(ARIST), 29:3–48.

Schutz, A. and Zaner, R. (1970). Reﬂections on the Problem

of Relevance. Greenwood Press.

Sperber, D. and Wilson, D. (1986). Relevance: Communi-

cation and Cognition. Harvard University Press, Cam-

bridge, MA, USA.

PhyCS2015-DoctoralConsortium