Feeling Hungry: Association of Dietary Patterns

with Food Choices using Scene Perception

Shoma Berkemeyer

a,∗

and Julius Sch

oning

b,∗

Osnabr

uck University of Applied Sciences, Osnabr

uck, Germany

Keywords:

Scene Perception, Dietary Patterns, Food Choices, Nutritional Behavior, Gaze Analytics, Multimodal Data

Analysis, Visualization, Nutritional Patterns.

Abstract:

Studies on nutrition have historically concentrated on food-shortages and over-nutrition. The physiological

states of feeling hungry or being satiated and its dynamics in food choices, dietary patterns, and nutritional

behavior, have not been the focus of many studies. Currently, visual analytic using easy-to-use tooling offers

applicability in a wide-range of disciplines. In this interdisciplinary pilot-study we tested a novel visual an-

alytic software to assess dietary patterns and food choices for greater understanding of nutritional behavior

when hungry and when satiated. We developed software toolchain and tested the hypotheses that there is no

difference between visual search patterns of dishes 1) when hungry and when satiated and 2) in being vegetar-

ian and non-vegetarian. Results indicate that food choices can be deviant from dietary patterns but correlate

slightly with dish-gazing. Further, scene perception probably could vary between being hungry and satiated.

Understanding the complicated relationship between scene perception and nutritional behavioral patterns and

scaling up this pilot-study to a full-study using our introduced software approaches is indispensable.

1 INTRODUCTION

Hunger and forms of under-nutrition has been one

of the nutritional research focus, investigating food

and/or nutrient shortage and their impact on diseases

and survival (Rubin, 2018; Berkemeyer, 2012). Sci-

entiﬁc inquiry into over-nutrition has been extremely

well-documented, with studies on obesity, metabolic

syndrome, diabetes, cancers, and other diseases of

civilization (Berkemeyer, 2009). Little though is

known about food choice and food perception in the

state of being hungry or satiated, a complex of phys-

iological and behavioral processes. The attitude to-

wards food, including food culture, and individual

emotional relation to, perception of and preference

of food, would co-determine food choice. Short-term

food choices and long-term dietary patterns affect nu-

tritional status, all which have bearing on etiology

and therapy of nutrition-based diseases. (Breer et al.,

2017; Payne et al., 2010; MacCormack and Lindquist,

2019).

The research ﬁeld of visual analytic provides ap-

plications capturing gaze-data and visualizing the

https://orcid.org/0000-0002-2125-7468

https://orcid.org/0000-0003-4921-5179

∗

Both author contribute equally

gazes as perception of the scene. In the ﬁeld of nutri-

tion the use of visual analytic is in its nascent stages.

Thus, we combined these disciplines in this paper,

opening up attractive opportunities to observe the vi-

sual search pattern on food, thereby ﬁnding applica-

tion in nutritional research.

Next to the development of an easy-to-use

toolchain for conducting and analyzing studies on

scene perception of dishes for nutritional research, a

pilot study was conducted. Our pilot study objective

was to investigate scene perception with food gaze on

dishes when hungry (before lunch) and the food gaze

at satiety (after lunch), taking into account the prior

knowledge of dietary patterns. The underlying as-

sumption of our pilot-study was that involuntary gaze

movements on food would be an index for sponta-

neous nutritional choices, with stated priors of nutri-

tional patterns to augment our current understanding

of nutritional behavior.

In the following sections we report our pilot study

goals, the tooling developed for this pilot study, and

state the hypotheses. In Methods we describe the set-

ting, participants, development of the visual analytic

tooling for assessing gaze-data to index scene percep-

tion and the data-analysis conducted. This is followed

up by presentation of the Results and Discussion.

188

Berkemeyer, S. and Schöning, J.

Feeling Hungry: Association of Dietary Patterns with Food Choices using Scene Perception.

DOI: 10.5220/0010146101880195

In Proceedings of the 4th International Conference on Computer-Human Interaction Research and Applications (CHIRA 2020), pages 188-195

ISBN: 978-989-758-480-0

2 GOALS FOR PILOT STUDY

In an interdisciplinary team comprising of nutritional,

computer, and behavioral scientists, two goals were

identiﬁed. Firstly, creating software for conduct-

ing surveys with eye-tracking at the site of food-

consumption. Second goal was to test the analysis

toolchain for ﬁrst insights on before and after lunch

scene perception. We thus set up our pilot study to-

wards these two goals and for testing our developed

software under realistic conditions.

2.1 Tooling

This application of scene perception in nutritional

studies required data-collection directly at food-

eating locations, e.g., canteens or food-courts. For

our study, the university canteen, Mensa, setting was

chosen due to nearness and easy feasibility of pilot-

testing. We assumed that people would have lim-

ited time for lunch-break, as stipulated by employ-

ers, classically half-an-hour. Thus, the study’s imple-

mentation goal was identiﬁed as a mobile hardware-

software solution that would work without special

equipment, such as chin-rests, and allow running each

experiment at minimum time.

The second challenge, next to data-gathering, was

data-storage for an interdisciplinary team. The con-

cept of using multimedia container formats for stor-

ing survey-data (Sch

oning et al., 2017c,b), would al-

low ﬁrst exploratory visual data analysis without any

special tool. By improving our existing implementa-

tion, a more meaningful visualization was to be pro-

grammed, cf. Figure 4. The current approach and

the future programming requirement would allow nu-

tritional scientists to explore the data collected with-

out additional requirement of special self-compilation

or costly proprietary software, enabling newer dimen-

sions in nutritional analytics.

2.2 Hypotheses

The prototyped survey tool enabled us to collect

questionnaire-based information and gaze point on

stimuli types such as photos, video footage, and an-

imations.

For this pilot-study, the ﬁrst null-hypothesis was

deﬁned as 1) that there is no difference between vi-

sual search patterns of dishes when participants were

hungry and when satiated. As the second hypothesis,

we expected 2) no difference between visual search

patterns and a priori stated dietary pattern.

(a) Step I:

answering a

questionnaire (10s)

(b) Step II:

recalibrate the low-cost eye

tracking device (2sec)

gazing on stimuli —

different dishes (40s)

(d) Step V: having lunch

Step V: gazing on stimuli

again (40sec)

Figure 1: Steps of the study designed; not illustrated: Step

IV have lunch.

3 METHODS

The study was conducted in January 2020 under the

speciﬁcations of the declaration of Helsinki (World

Medical Association, 2013). Informed consent was

obtained prior to study participation. Ethical clear-

ance was invalid for this pilot-study as no unique

person-related, and identiﬁable data was gathered.

3.1 Setting

The pilot-study was based at a tertiary educational

setting. Study participants included students, staff,

or visitors of the tertiary educational institution. On

the day of survey, four different main dishes and sev-

eral side dishes were on offer at the canteen namely:

Dish 1: organic spaghetti with organic soya bolog-

nese, Dish 2: chicken schnitzel with peach and hol-

landaise sauce, Dish 3: gemstone pumpkin curry,

Dish 4: beef sliced Esterhazy in vegetable sauce, Dish

5: side dishes i.e., potatoes, salad, soup, and noodles.

3.2 Study Design, Variables and Tooling

A cross-sectional study design was used for this ex-

ploratory study. For this early-stage feasibility study,

Feeling Hungry: Association of Dietary Patterns with Food Choices using Scene Perception

189

a total of ten participants were recruited, which gen-

erated a medium volume of gaze-data. The recruit-

ment ensued random selection of people visiting the

canteen on the survey-day, given participant willing-

ness to take part in the study and informed writ-

ten study-consents were available. The inclusion

of ten participant for a pilot feasibility was deter-

mined by the period of lunch-break, of about half an

hour per person, over an entire lunch period of two

hours between 12.00 noon and 14.00 p.m. on the

day of data-collection and the use of a single com-

puter based terminal equipped with visual analytic for

data-gathering. Study participation, even so random

stochastically determined, can not rule out study par-

ticipation bias by those, who were inherently more

interested in nutrition, compared to those, who were

less interested in nutrition. The study was imple-

mented by one study-personnel on the survey day, the

software developed for this pilot study integrated a

self-training sequence for those conducting the study

allowing easy use and applicability.

Dietary patterns included non-vegetarian and veg-

etarian; non-vegetarian was deﬁned as omnivore,

healthy omnivore nutrition (Deutsche Gesellschaft f

Ern

ahrung e. V., 2020), and paleo-diet. Vegetarian

was deﬁned as ovo-lacto-vegetarian, ovo-vegetarian,

lacto-vegetarian and vegan diets; raw-food, whole-

food and ﬂexitarian diets; and pesco-vegetarian diet.

Dietary pattern was assumed to indicate long-term nu-

tritional pattern and behavior. Food choices were de-

ﬁned by canteen menus that were chosen by the study

participants on the survey-day. Food choices were

assumed to indicate short-term nutritional behavior.

There were ﬁve menus for choice on the study-day.

Background data were collected on age, gender, and

country of birth.

For the mobile hardware platform, a standard mid-

class laptop with a built-in, detachable low-cost eye-

tracking device was used. Estimating the computing

power required for mobile software solution, it was

determined that a low-cost laptop would not be able

to simultaneously handle multi-threading, necessary

for synchronous visualization of the stimuli, and data-

storage of gaze data. We, hence, used a standard mid-

class laptop for the pilot study, laptop speciﬁcations

were Intel i5 CPU equipped with a Tobii EyeX eye

tracker.

As seen in Figure 1, our software

covered the

whole acquisition process of data, including human-

machine-interaction, e.g., for the questionnaire and

for calibration of the eye tracking device. We provide

this software as free, open-source software, which is

The source code can be downloaded at https://www.

hs-osnabrueck.de/prof-dr-julius-schoening/chira2020

capable of providing a timed graphical user interface

for presenting the stimuli at a deﬁned time and in-

teroperable to different low-cost eye tracking device,

in the programming language of C++. The software

given as open-source for public research is also pro-

vided with an open-source free widget toolkit QT.

A simple calibration routine was implemented for

the use of low-cost eye-tracking device along with the

minimization of time spent for the study by each par-

ticipant. As shown in Figure 1(b), the participant was

therefore asked to ﬁxate on a cross in the middle of the

screen. The computer recognized a ﬁxation over one

second within an offset of 50 pixels, and the software

re-calibrated the gaze tracking.

The study-design required that the stimuli presen-

tation, as illustrated in Figure 1(c) would be ﬂex-

ible showing the day’s dishes available at the can-

teen. The stimuli were not compiled with the soft-

ware. Rather the stimuli, e.g., photos of dishes, were

placed in a speciﬁed folder next to the executable

built-in software. In this way, this pilot-study devel-

oped an easy, simple, and ﬂexible solution for inter-

changeable stimuli. By naming convention, the order,

as well as the duration of the stimulus, was set. For

example, the photograph titled “dish1 3000.jpg” was

shown for 3000msec followed by “dish2 1242.jpg”

and so on. To avoid restarting the software repeat-

edly, a guided dialog was provided for the survey in-

structor, allowing back and forth movements with in-

struction repeats. Thus, the software applicability al-

lowed easy, on-site training for people conducting the

study, thereof saving time in study-speciﬁc schooling

and training of personnel.

To regain participants after lunch we had a reward

for the study participants built in to the study-design.

Participants received a visualization of their gaze in

the format compatible with standard multimedia play-

ers (Sch

oning et al., 2017a). This is described in detail

in the following section.

3.3 Analysis

For scene perception analysis, the gaze trajectories

on each dish were visualized for our investigation.

Therefore static heat-maps, as shown in Figure 2, for

each shown dish per participant, were generated by a

simple Python script. Next to the ﬁve different dishes,

cf. Subsection 3.1, a white screen was presented as

well for zero-error correction calibration. For further

analyses, the differences in the gaze patterns when

hungry and when satiated were visualized.

According to the hypotheses 1), the resulting heat-

maps were composed next to each other for visual in-

spection. In addition to the heat-maps per participant,

CHIRA 2020 - 4th International Conference on Computer-Human Interaction Research and Applications

190

(a) dish 4: before having lunch (b) dish 4: after having lunch

Figure 2: Cumulated gaze patters off all participants on stimulus dish 4 and dish 5 , after having lunch the participants explore

the outer areas of the stimulus.

cumulative heat-maps per dish for all participants was

generated, cf. Figure 2. Finally 2), cumulative heat-

maps over vegetarians and non-vegetarians were gen-

erated as well.

We compared the visual search pattern when hun-

gry and when satiated as an assessment of nutritional

behavior and compared it to participant’s food choice

of mid-day meal. Thereby, we visually examined,

thereby testing, the possibility of using visual search

pattern heat-maps for assessing nutritional behavior.

4 RESULTS

The overview of all study participants (cf. Appendix

Table 1) shows that of the ten study participants,

seven were non-vegetarian and three were vegetar-

ians, thereof one vegan, one vegetarian, and one

pescovegetarian. Age-range of the participants was

23-40 years. There were three women and seven men.

Of the seven non-vegetarians, two were women and

ﬁve men, and of the three vegetarians one was man

and two were women.

Testing hypothesis 1), Figure 3 suggests that there

was no signiﬁcant difference between visual search

patterns when hungry and when satiated using in-

dividual gaze-data. Visual inspection indicated that

there could be gaze-deviation in cumulative gaze-

patterns, i.e., when hungry and when satiated. How-

ever, this observation could also be due to, that the

same photos were presented before and after lunch,

which remains a study confounder.

Dish 1 Dish 2 Dish 3 Dish 4 Dish 5 white screen

4.14%

7.76%

5.12%

2.48%

9.72%

0.53%

similarity in visual search pattern %

Figure 3: Similarity of visual search pattern when hungry

and when satiated, accumulated over all participants.

Feeling Hungry: Association of Dietary Patterns with Food Choices using Scene Perception

191

Of the seven non-vegetarians, three gazed at

dish 2, and two chose dish 2 (chicken schnitzel with

peach and hollandaise sauce), one chose an undis-

closed dish 5; two gazed at dish 4, and one chose

it and the other chose dish 2. The seventh non-

vegetarian was undecided and chose dish 1 (organic

spaghetti with organic soy bolognase). Of the seven

non-vegetarians, six were decided and one undecided

at study-begin, and of the six only three chose what

they gazed at. Of the three vegetarians, the vegan

participant gazed and chose dish 3 (gemstone curry).

One was undecided and chose an undisclosed dish 5.

The third vegetarian (pescovegetarian) gazed, and

chose dish 2. Of the three vegetarians, two chose dif-

ferent dishes than at onset; one food-choice was dif-

ferent from the dietary pattern, one remained undis-

closed.

In total, only four (one vegetarian and three non-

vegetarian) of the ten participants made food choices

that coincided with what they gazed at. Two were

undecided in food-choice from onset on (one veg-

etarian and one non-vegetarian). Of the remain-

ing four participants (one vegetarian and three non-

vegetarian), three food choices were different from

what was gazed at, yet within the dietary pattern (non-

vegetarian) and in one case food choice was different

from the stated dietary pattern (vegetarian). We pro-

vide above-trend results for our hypothesis 2), which

would require a full-scale study with large sample

size.

5 DISCUSSION

Our pilot study indicates that individual visual search

patterns were similar when hungry and when satiated,

though cumulative visual search patterns suggest that

participants could have gazed on food (central plate)

when hungry and peripheral dish-area when satiated,

which though remains confounded by the presenta-

tion of same before-after pictures. Initial analysis in-

dicates that food choices tended to vary within dietary

patterns, and at times food choices could be different

from dietary patterns. The chance of greater undis-

closed food choices was probably related to higher

dietary pattern deviant choices, indicating nutritional

behavior pattern to be more complex than the stated

long-term dietary pattern or short-term food choice.

Our pilot study further provides that the development

of a low-cost tooling device can be used for nutri-

tional studies with the development of more easy-to-

use software for visual analytic.

Exploring and analyzing the temporal behavior of

the scene perception, the idea of using any multime-

dia player for exploratory analysis (Sch

oning et al.,

2017a,c,b) was adopted and improved in our study. In

this process we developed and improved the use of vi-

sual analytic to its stated purpose Thomas and Cook

(2005), i.e., to derive insight from massive, dynamic,

ambiguous, and often conﬂicting data, to detect the

expected and discover the unexpected.

We further added transparent heat-map images as

subtitle tracks instead of the use of the pure text-based

universal subtitle format (USF), thereby we increased

the meaning of gaze points per frame without leaking

the ability to be played by the use of standard multi-

media player such as the VLC Player. With our novel

approach, refer Figure 4 (b) with the USF in Figure

4 (a), the information content was increased, because

several gaze sampling points on the pixel were able to

be visualized with colors. Despite these extensions,

our approach still conformed to the multimedia con-

tainer format and provided both instantaneous visual-

ization with multimedia plays and the storage of all

data in raw formats.

Our pilot study indicates that food choices and

dietary pattern in frame of nutritional behavior pat-

tern show variance. Very little is known about the

decision-making process in food choices across the

human life-span. Ogden and Roy-Stanley (2020) re-

ported in children that food decision was automatic

or considered or sanctioned. Greater autonomy in

food decision-making does not automatically imply

healthy food choices (Breer et al., 2017). Further,

sanctioned behavior might also present a viable op-

tion in nutritional advice with diseases (Berkemeyer,

2009). MacCormack and Lindquist (2019) hypothe-

sized that people experience hunger as emotions and

misattribution process (Payne et al., 2010), to demon-

strate that hunger (Rubin, 2018; Berkemeyer, 2012)

shifts affective perceptions in negative but not in neu-

tral or positive contexts. A role for intuitive eating to

promote health has also emerged requiring further in-

vestigation (Rubin, 2018; Keirns and Hawkins, 2019).

A further application of our study results can be in

the practice of how restaurants prepare food. Much of

the processes are pre-deﬁned and much of food prepa-

ration,such as, chopping, slicing, base sauce prepara-

tions are done in advance. Everything has its place

and order in a professional kitchen to facilitate dif-

ferent people accessing same objects with these stan-

dardizations. Use of visual data analytic in scene per-

ception in this practice-ﬁelds could help, for exam-

ple, in training staff in simulations or real-life practice

evaluations, which would be additional covariates in

food choices. Food presentation has been shown to

affect food choice (Ogden and Roy-Stanley, 2020).

Our limitations include the number of partici-

CHIRA 2020 - 4th International Conference on Computer-Human Interaction Research and Applications

192

(a) universal subtitle format (USF) for visualization (b) transparent image as sub station alpha overlays as

subtitles for visualization

Figure 4: Gaze visualization with standard multimedia player; (a) former approach (Sch

oning et al., 2017a) and (b) our new

approach providing a heatmap in each frame.

pants in our pilot study, which can be addressed in

a full-study. Dichotomous grouping into vegetarians

and non-vegetarians might be inadequate, requiring a

third intermediate-grouping of semi-non-vegetarians

including pescovegetarians and/or ﬂexitarians. Fur-

ther, our before-after study can be improved with

assessment of food cultural and emotional factors,

which co-determine food choices. Acquisition of

healthy food-habits can be improved with reeduca-

tion, incorporating emotional association with food,

which too remains an additional dimension applica-

tion of the current pilot-study requiring future focus.

Finally, even so participants were randomly recruited

on the survey day we can not rule out participation

bias by those, who are interested in nutrition, which

remains a study-limitation. The use of cumulative

heat-maps would require greater optimization, which

can be conducted in future research in study-samples

similar to and diverse from the current pilot samples,

in order to generate long-term valid and reliable re-

sults.

This pilot-study indicates that by understanding

scene perception and gaze-data, a greater understand-

ing of nutritional patterns can be facilitated, includ-

ing the future development of innovative nutritional

therapy and innovative ways of preparing and serving

food. Thus, a small unobtrusive bay leaf at the right

place could probably instinctively help to make better

food decisions, e.g., as a food-nudging intervention.

Our pilot-study shows that such future goals can be

the mainstay of innovative interdisciplinary research

providing new, open, easy-to-use tools and exchange

of different scientiﬁc approaches and perspectives.

REFERENCES

Berkemeyer, S. (2009). Acid–base balance and weight

gain: Are there crucial links via protein and organic

acids in understanding obesity? Medical Hypotheses,

73(3):347–356.

Berkemeyer, S. (2012). Starvation versus calorie restric-

tion: Our road to food insecurity or health. Journal of

Nutrition & Food Sciences, (S1).

Breer, N., Gendig, C., and Berkemeyer, S. (2017). The re-

lationship of migration, age, income and dietary pat-

terns with body mass index in a cross-sectional analy-

sis of ebagil-study. In Eighth EUSPR Conference and

Members Meeting.

Deutsche Gesellschaft f

ur Ern

ahrung e. V. (2020).

Vollwertige ern

ahrung. https://www.dge.de/

ernaehrungspraxis/vollwertige-ernaehrung/ (2020-

01-30).

Keirns, N. G. and Hawkins, M. A. W. (2019). Intuitive eat-

ing, objective weight status and physical indicators of

health. Obesity Science & Practice, 5(5):408–415.

MacCormack, J. K. and Lindquist, K. A. (2019). Feeling

hangry? when hunger is conceptualized as emotion.

Emotion, 19(2):301–319.

Ogden, J. and Roy-Stanley, C. (2020). How do children

make food choices? using a think-aloud method to ex-

plore the role of internal and external factors on eating

behaviour. Appetite, 147:104551.

Payne, B. K., Hall, D. L., Cameron, C. D., and Bishara,

A. J. (2010). A process model of affect misattri-

bution. Personality and Social Psychology Bulletin,

36(10):1397–1408.

Rubin, O. (2018). The precarious state of famine research.

The Journal of Development Studies, 55(8):1633–

1653.

Sch

oning, J., Faion, P., Heidemann, G., and Krumnack, U.

(2017a). Providing video annotations in multimedia

containers for visualization and research.

Feeling Hungry: Association of Dietary Patterns with Food Choices using Scene Perception

193

Sch

oning, J., Gert, A., Ac¸ik, A., Kietzmann, T., Heide-

mann, G., and K

onig, P. (2017b). Exploratory mul-

timodal data analysis with standard multimedia player

— multimedia containers: a feasible solution to make

multimodal research data accessible to the broad au-

dience. In Proceedings of the 12th Joint Conference

on Computer Vision, Imagingand Computer Graphics

Theory and Applications (VISAPP), pages 272–279.

SCITEPRESS.

Sch

oning, J., Gundler, C., Heidemann, G., K

onig, P., and

Krumnack, U. (2017c). Visual analytics of gaze data

with standard multimedia player. Journal of Eye

Movement Research, pages 1–14.

Thomas, J. J. and Cook, K. A., editors (2005). Illuminating

the Path: The Research and Development Agenda for

Visual Analytics. IEEE Computer Soc., Los Alamitos,

Calif.

World Medical Association (2013). World medical associ-

ation declaration of helsinki. JAMA, 310(20):2191.

CHIRA 2020 - 4th International Conference on Computer-Human Interaction Research and Applications

194

APPENDIX

Table 1: Overview of all study participants of the pilot-study.

ID Age (years) Gender Brith Country Mensa-Type Nutritional Pattern Food Gaze (when

hungry)

Food choice (satiated)

P280413 31 Woman Germany Staff NP-NV Beef Sliced Esterhazy

in Vegetable Sauce

Chicken schnitzel

with peach and

hollandaise sauce

P281612 30 Man Netherlands Staff NP-NNV (Vegan) Gemstone Pumpkin

Curry

Gemstone Pumpkin

Curry

P283111 23 Man Germany Student NP-NV Chicken schnitzel

with peach and

hollandaise sauce

A different dish

P283312 38 Man Germany Staff NP-NV Chicken schnitzel

with peach and

hollandaise sauce

Chicken schnitzel

with peach and

hollandaise sauce

P283811 25 Man Germany Student NP-NV Undecided Organic spaghetti

with organic soya

bolognese

P284111 31 Woman Germany Student NP-NNV (Vegetar-

ian)

Undecided A different dish

P284811 32 Woman Germany Student NP-NV Chicken schnitzel

with peach and

hollandaise sauce

Chicken schnitzel

with peach and

hollandaise sauce

P285111 32 Man Germany Student NP-NNV (Vegetarian

with ﬁsh)

Organic spaghetti

with organic soy

bolognese

Organic spaghetti

with organic soy

bolognese

P285611 40 Man Germany Staff NP-NV Chicken schnitzel

with peach and

hollandaise sauce

Chicken schnitzel

with peach and

hollandaise sauce

P285710 34 Man Germany Staff NP-NV Beef Sliced Esterhazy

in Vegetable Sauce

Beef Sliced Esterhazy

in Vegetable Sauce

Feeling Hungry: Association of Dietary Patterns with Food Choices using Scene Perception

195