Food Image Presentation System that Discharge Smell Through

Screen and Psychological Effect

Akira Tomono

, Mana Tanaka

, Rei Shu

and Keisuke Tomono

Department of Information Media Technology, School of Information and Telecommunication Engineering,

Graduate School of Information and Telecommunication Engineering, Tokai University,

2-3-23 Takanawa, Minato-ku, Tokyo 108-8619 Japan.

Keywords: Food Image, Smell, Presence, Saliva Central Nerves, Blood Flow, Virtual Reality.

Abstract: The author et al. are currently engaged in a project for an image displaying system with a screen from which

smells are discharged for users, together with images, aiming at applying it to digital signage and for other

purposes in order to enhance the realistic sensation of the food images. The author et al. conducted

experiments in presenting food images and discharging smells from the same position, and analyzed the

users’ psychological impact. A subject questionnaire and a cerebral blood-flow meter were used for the

analysis. In the first experiment, it was clarified that when an airflow and a smell were discharged in

conformity with the image of cooking with a steaming hot pot, an inhaling action occurred and smell

perception rate was enhanced. In the second experiment, when a smell fit with a food image was discharged,

the realistic sensation and the oxyhemoglobin rose in the vicinity of the temple because the salivation

central nerves became active.

1 INTRODUCTION

Recently, digital signage with a large screen is

prevailing now (Burke R.R.,2009). It is more

flexible than conventional advertisement using

photographs and can be fixed in underground

walkways and at the entrance of a shop. It can be

used for the advertisement of various products, and

among them, food advertisement is very prospective.

It is known that the sense of smell has a strong effect

on human appetite (Nakamoto T., et al.,2012). We

often experience that we enter a restaurant allured by

the smell of food. Therefore, if the image of the food

shown in the digital signage discharges the smell of

the food, the advertisement can be expected to have

a high effect (Sakaino A., 2008).

At the moment, as for the method of displaying

an image with smell, it is common to install a smell

emission device beside an image display device.

However, in the case of large screens such as digital

signage, because the position displaying the image

and that discharging the smell are separated from

each other, it is difficult to give the realistic

sensation as if the displayed food image is

discharging its smell. It is also a problem that a

space for installing the smell emission device is

needed.

In order to overcome these problems, the author

et al. are developing a device called the Kansei

Multi-Media Display (KMMD) that can discharge a

smell from the same place as the image (Tomono. A.,

et al.,2008). Thin display panels have small holes

through which gas passes. The smell emission

device installed behind the screen creates airflow in

the front of the screen and sends a smell through the

airflow to the nose of users (Tomono. K., et al.,2011).

As researches based on a similar concept,

Sawada et al. proposed the interface that brought the

new communication medium of "wind" into the

bidirectional interaction between the virtual

environment and the real environment by integrating

the graphic presentation with the input and output of

wind on a special screen (Sawada et al., 2008). Also,

Matsuura et al. proposed the system that generated

an airflow directed toward the user from a certain

position on the screen by making the wind that rose

from four fans collide on the screen (Matsukura et

al., 2012).

One task was for the technology to effectively

send the discharged smell to users. It is also

necessary to clarify its psychological effect.

Conventionally, the multimodal psychological

261

Tomono A., Tanaka M., Shu R. and Tomono K..

Food Image Presentation System that Discharge Smell Through Screen and Psychological Effect.

DOI: 10.5220/0005024102610268

In Proceedings of the 11th International Conference on Signal Processing and Multimedia Applications (SIGMAP-2014), pages 261-268

ISBN: 978-989-758-046-8

 2014 SCITEPRESS (Science and Technology Publications, Lda.)

process caused by the presentation of multiple

senses is studied in the field of brain science. The

taste, the smell, and the relation to the saliva center

of the brain are clarified because of the advancement

of the brain blood flow measurement technique that

uses NIRS recently (Bembich et al., 2010). For

example, Kokan N et al. showed that NIRS

combined with a questionnaire is a useful method

for studying the functional neuroanatomy of OFC in

terms of olfaction (Kokan et al., 2011). However,

there are only a few studies aiming at the application

of an information system including the realistic

sensation and appetite sense of when a food image is

presented together with a smell.

In this paper, firstly, the KMMD system used for

experiment was explained and two kinds of

experiments were conducted by discharging a smell

from the position where the image of food was

displayed. The first experiment explained how to

generate an inhaling behavior using the feeling of

wind and effectively make users perceive the smell.

The second experiment investigated the

psychological condition of when a cooking scene

was projected and a scent was discharged in

conformity with the scene. It was evaluated by a

questionnaire and measured the changes in cerebral

blood flow around temple where the salivation

central nerve is said to exist.

2 STRUCTURE OF THE KMMD

EXPERIMENTAL SYSTEM

2.1 Image Displaying System Through

which a Gas Passes

Fig. 1 shows a KMMD experimental system with a

screen with holes and a projector. The size of the

screen was 1800 mm in width and 1200 mm in

length, and nine panels (400x200 mm) with many

holes were mounted on the screen. The panels were

1.5 mm in thickness, the holes were 2 mm in

diameter, and the distance between each hole was

3.5 mm. A high-brightness projector (Solid Ray Co.,

Sight3D U27) was used for image display. This

experimental system was produced for the purpose

of simulating the LED-type thin and large-screen

digital signage in the future.

The reason for this design was explained in this

paragraph. Conventionally, it is often the case that

digital signage with an LED pixel screen consists of

1-2 mm small-type three-color LEDs arranged in

array. Pixel spacing is determined by resolution and

the size of screen. For example, if a screen around

3.5 m in width is presumed, even if the pixel pitch is

set at 3.5 mm, a solution of about 1000 pixels in the

horizontal direction is possible. Although the holes

made on the screen are generally considered to

lower the resolution, if the screen is large, the holes

on the screen may not directly lead to the lowering

of resolution. It is desirable that the depth of holes is

shallower from the viewpoint of airflow property.

Therefore, the thickness that was available by

current thin-type panel manufacturing technology

was adopted.

Figure 1: KMMD using air cannon and smell presentation

experiment for food image.

2.2 Smell Emission Device

A smell emission device was installed behind the

panel. This device consisted of a box discharging

airflow by increasing atmospheric pressure behind

the holes, a mechanism increasing atmospheric

pressure. Two modes can be used for discharging

airflow. As for mode 1, an air cannon is installed

behind the holes and it can discharge a gas vortex

SIGMAP2014-InternationalConferenceonSignalProcessingandMultimediaApplications

262

ring by instantly increasing atmospheric pressure. As

for mode 2, it can discharge an airflow-like wind by

increasing the atmospheric pressure in the box using

a blower (Showa Denki Co., Ltd. SB-201-R3A4). If

a smell element is placed in the box, it is carried by

the vortex ring or an airflow-like wind and presented

to users. In the experiment in Chapter 3, mode 1 was

used, and for the experiment in Chapter 4, mode 2

was used.

Fig. 1 (b) shows that the gas is discharged in a

forward direction roughly vertically to the screen by

a blower installed behind the screen. It is visualized

using smoke

3 EXPERIMENT OF

PRESENTATION OF FOOD

IMAGE WITH SMELL (1)

3.1 Purpose of Experiment

The KMMD proposed by this research can

simultaneously discharge a smell with an image.

However, if the users are not inhaling when the

smell is discharged, a small amount of smell passes

near the users along with the air flow, so that it is

difficult for the users to perceive the smell. In order

to enhance the smell perception rate, it is desirable

to control discharge of the smell in time with the

users’ breathing. Therefore, the author et al. created

a situation where positive inhaling behavior

naturally occurred due to the contents use of diverse

sensation-presenting functions and discussed

methods for discharging a smell (Tomono. K., et

al.,2012).

Humans tend to take a positive inhaling-behavior

to effectively perceive the smell when they perceive

air movement. Therefore, it was hypothesized that if

a smell is discharged in a timely manner, it would

create a scene as if the smell were discharging from

the food, thereby enhancing the smell perception

percentage.

3.2 Experimental Environment

An air cannon with an aperture diameter of 9 cm

was installed behind the panels with holes attached

to the center of screen. When this system was

operated, the gas within the cannon passed through

the holes of the panels, formed an air vortex and

advanced to the subjects. The subjects were

positioned in front of the screen and the distance

between the subjects and the system was 150 cm.

The cannon was directed to the subjects’ face. The

velocity of the air vortex at this position was

measured by an anemometer and its velocity at the

time when it reached the subject was 1-2 m/s.

Because the air cannon was large, the airflow struck

the entire face. The subjects were asked whether or

not they felt the wind pressure, and all subjects

responded that it “ felt like wind.” When a scent

element was placed in the cannon, it was carried by

the air vortex ring and presented to the subjects.

The image was of a hot pot being cooked. A pot

was displayed in the center of the screen. When the

lid was opened, steam rose up from the pot and

sukiyaki-like ingredients appeared. Because the

camera was pointed at the image where the lid was

opened during shooting, it was as if steam were

flowing at the camera. In other words, if one

watches this image, one will feel as if the steam is

flowing into one’s direction.

3.3 Experimental Method

In order to link the airflow and smell to the image of

steam rising up from the pot when the lid was

opened, a vaporized sukiyaki source was placed in

the cannon and discharged at that scene. The

presentation of the wind feeling combined with the

smell was called Condition 1. As the comparison

with the conditions above described, in order to

realized the condition presenting only smell without

causing wind feeling, a small-type air cannon type

smell emission device was installed near the users.

Because the airflow was weak, it was almost

unnoticeable to the subjects. The presentation of

only the smell was called Condition 2. The

presentation of only the image, without the smell

and airflow, was called Condition 3.

Under conditions 1 and 2, in order to investigate

the smell perception rate, the subjects were asked to

hold an input device (counter button) and respond

when they perceived the smell. In order to detect the

breathing of subjects, PVDF Nasal/Oral Airflow

Sensor of Dymedix Diagnostics Co., Ltd.

9),

respiration sensor, was attached near the nasal cavity

of the subjects. This sensor was used to investigate

whether or not the inhaling behavior occurred when

the airflow struck the subjects’ face.

The experimental images described above

included scenes of opening the lid four times/m.

Accordingly, the interval between the scenes was

approximately 15 seconds. These images were

presented repeatedly. The smell was presented 15

times in Condition 1 and in Condition 2 and the

number of times when the smell was perceived was

FoodImagePresentationSystemthatDischargeSmellThroughScreenandPsychologicalEffect

263

obtained. After the completion of the experiment,

the subjects were asked for open feedback

concerning the airflow feeling and inhaling

behavior.

Under conditions 1 and 3, the impression of the

contents was investigated by a questionnaire. The

investigation items were “Impression of smell” and

“Impression of wind,” “Realistic sensation” and

“Feeling of tension,” and each item was evaluated

on a 5-point scale ranging from “Not felt” to

“Strongly felt.” The subjects were students aged

from 21-24 without any abnormal visual or smell

senses.

3.4 Experimental Results

Fig. 2 shows the average smell perception rate under

conditions 1 and 2. Although the smell perception

rate under Condition 2 was less than 50%, that under

Condition 1 increased by more than 30%. As a result

of the test, there was a significant difference

between both conditions at a significance level of

5%, and it was confirmed that the smell perception

rate under Condition 1 was higher. From the

questionnaire, a highly realistic sensation was

reported, such as “Although it was a familiar image

scene, I felt the steam from the pot due to the wind

and I inadvertently smelled the scent.”

The analysis of the output waveform of the

respiration sensor used under Condition 1 showed

that there have been many cases where, 0.5-1

seconds after the air struck the subjects’ cheeks, the

subjects inhaled strongly instead of their breathing

normally.

Fig. 3 shows the questionnaire results concerning

the impression of the contents under conditions 1

and 3. The vertical axis shows the average value of

the scores to which the subjects responded. No smell

stimulus or wind-touch stimulus was presented

under Condition 3, so it was natural that the

impressions of smell and wind were small, but the

realism was not high and, as a whole, the stimulation

was low. Meanwhile, it was revealed that when the

subjects sensed the smell and airflow under

Condition 1, feelings of realism and tension

increased. There were significant differences in

average scores among the conditions at significance

level 5% except for the feeling of tension.

3.5 Discussion

It was verified that when the feeling of wind

associated with steam was presented together with

the image where steam rose up from the pot, it

tended to cause a large inhaling behavior. According

to the questionnaire results, this is considered to be a

positive inhaling behavior desiring to effectively

perceive the smell. This behavior is also considered

to reflect the rise in realistic sensation and the sense

of immersion in the scene.

Figure 2: Comparison of smell perception frequency

between condition 1 and 2.

Figure 3: Results of the questionnaires.

4 EXPERIMENT OF

PRESENTATION OF FOOD

IMAGE WITH SMELL (2)

4.1 Purpose of Experiment

As a result of the questionnaire concerning the

experiment in the previous chapter, many subjects

answered that when a food image was attached with

an appropriate smell, it enhanced the realism. In

SIGMAP2014-InternationalConferenceonSignalProcessingandMultimediaApplications

264

order to complement these subjective results, the

author et al. attempted to objectively evaluate this

realistic sensation. If a person sees food when

hungry or sees his or her favorite food, the person

often will salivate. This is considered to be a

psychological reaction that occurs when old

episodes of food from a previous time are recalled

from the visual simulation, which is associated with

the real thing, causing a desire to eat it. It was

hypothesized that when an appropriate memory was

recalled by the presented stimuli, much salivation

occurred, and if it was not recalled, less salivation

occurred. In other words, it was assumed that a

realistic sensation could be objectively assessed by

measuring the amount of saliva. Various saliva

measurement methods have been developed

(Nakano A., et al., 2011). However, the method of

inserting a sensor into the mouth was not appropriate

for the purpose of this experiment. Recently, a

sophisticated cerebral blood flow measurement

device was developed, and it has been discovered

that salivation central nerve exists near temple by a

food intake experiment using this device (Sato H., et

al., 2011).

Therefore, in this research, the relationship

between multimodal stimulation and the activity of

the salivation central nerve was clarified by

measuring the change of cerebral blood flow near

the temple. It was expected that the more presented

stimuli there are, the higher the recognition level

becomes. Therefore, if the appropriate combinations

of image and smells were presented, the salivation

center would become more active than by images

alone. A verification experiment was conducted

based on these considerations.

4.2 Experimental Method

4.2.1 Multimodal Stimulation Presenting

Condition

A blower was installed behind the display panel in

the center of the display device, as shown in Fig.4.

When the blower was operated, a gas was

discharged through the screen and a feeling of wind

was presented to the subjects. A smell element was

vaporized as necessary, and the smell was

discharged through airflow. This condition applied

the results of Chapter 3.

Two images of grapefruit and curry rice being

served were used as food images. The grapefruit

scent of PalmTree was used for the smell of

grapefruit and the curry powder of S&B was used

for the smell of curry rice. The presentation

conditions were as follows: 1) only the image was

presented, 2) the image and smell were

simultaneously presented, and 3) the image and

smell were incongruously presented. “Incongruously”

means that the smell of food different from the one

being served in the presented image was ejected.

The presentation time was 30-60 seconds under each

condition. Six kinds of contents were presumed as a

set and two sets were presented at the subject

experiment. The 2nd set was for the purpose offered

to examine the effects of the presentation order.

Figure 4: Experiment that measures blood stream change

in salivation central nerve

4.2.2 Questionnaire

Before the cerebral flood flower is measured, it is

important to confirm if the food images with smell

are matched with the initial purpose. In other words,

it is desirable to previously evaluate the realistic

sensation and the sense of appetite of the presented

contents and discuss the results of questionnaire

corresponding to the changes of cerebral blood flow.

Therefore, in order to investigate the characteristics

of the contents an interval scale and free descriptive

FoodImagePresentationSystemthatDischargeSmellThroughScreenandPsychologicalEffect

265

questionnaire with five stages using six pairs of

impressive adjective. The subjects were questioned

mainly about food and saliva discharge.

4.2.3 Measurement of Cerebral Blood Flow

using NIRS

Near infrared oxygenation monitor NIRO-200 of

Hamamatsu Photonics Co., Ltd. was used for the

measurement of cerebral blood flow. The

measurement principle was based upon Near Infra-

Red Spectroscopy (NIRS) (Eda H., 2006). After a

near-ultraviolet light is radiated to skull and it

migrates for a predetermined distance within brain,

the reduced light is caught with a light receiving

sensor. If light absorbing substances such as

hemoglobin in the migration pathway, the light

attenuates by Beer-Lambert Law, so that the analysis

of the output from light sensor can convert the result

to blood flow. Two probes integrated with

irradiating part and light receiving sensor were

attached to both temples.

4.3 Experimental Results

4.3.1 Results of Questionnaire

The scores of answers to the question items about

the six kinds of contents above were summarized as

shown in Fig. 5. The subjects were 19 students (13

males and 6 females. In all evaluation items, the

Condition 2 discharging smells matching to food

images showed high scores then Condition 1 and

Condition 2. In comparison with Condition 2 and

Condition 3 showed that although Condition 2

showed high evaluation in “Just like smelling” and

“No unpleasant feeling”, Condition 3 show

evaluation in unpleasant feeling. The contents giving

unpleasant feeling showed lower scores in other

evaluation items. As the result of test of difference

in average value, risk rate was 1%, and null

hypothesis was rejected and the differences were

clarified in all items. It shows that an appropriate

smell enhances the value of contents, but an

inappropriate smell decreases the value of contents.

4.3.2 Change of Blood Flow in Saliva

Central Nerve and Discussion

Fig. 6 shows the experiment where the image of

curry rice was displayed from resting period and

then the image of curry rice with the smell of curry

was displayed, and at that time, the concentration

oxygenated hemoglobin at the left temple showed

the change. As it greatly increased and it showed and

it showed that the saliva center was active. However,

the output from the sensor differed by the way of

attaching sensor probes and individual head tissues.

Therefore, the output values of hemoglobin

oxygenated during watching the contents were not

simply compared, but the comparison was made

with ratio of the variation while contents were being

watched to the variation during the resting period

(⊿R: difference between peak values) as a feature

amount. In other words, in the case of Fig. 6, ⊿M/

⊿R was used as a feature amount when the image

of curry rice was displayed and ⊿Msf/⊿R was used

as a feature amount of the image with smell, and the

results under each condition were sorted out.

Figure 5: Subjective evaluation experiment results (19

Subjects).

Fig. 7 shows the results of the average ratio to

the six kinds of contents concerning five subjects

among 19 ones. From the result, the fluctuations of

cerebral blood flows were larger in all periods than

the resting period. When the contents of image and

smell were matched, the fluctuations were largest

SIGMAP2014-InternationalConferenceonSignalProcessingandMultimediaApplications

266

and it is presumed that saliva secretion volume was

also largest. On other hand, when they were not

matched, the saliva secretion volumes were

supposed to be less. There was clearly significant

difference in average values between Condition 1

and Condition 2 at significance level 5%, so that it

suggests that the contents with appropriate smell

makes saliva discharge more. The result of Fig. 7

can be interpreted as the same as the result of

questionnaire of Fig. 5.

Figure 6: A change in cerebral blood flow before and after

curry and rice image presentation.

Figure 7: Relation between stimulation presentation and

change of cerebral blood flow.

In Condition 2, the comparison between the

results of curry rice and of grapefruits showed a high

score for grapefruits. In order to investigate the

reason, an analysis concerning the scent and strength

of the smells was conducted. Fig. 8 shows the results.

In the comparison of the parts shown by Condition 2

in Fig. 8, there were good scores for both scents, and

one answer stated that the smell of curry rice was

strong. Therefore, the different strengths of the

smells created a difference between these smells.

Figure 8: Results of questionnaire concerning favor and

strength of the smell.

5 CONCLUSION

In order to evaluate the experiment used to enhance

the smell perception rate through KMMD that can

simultaneously present visual, smell and airflow

information and to objectively evaluate the realistic

sensation of the images along with smell, an

experiment was performed to infer the saliva

discharging volume and the following were clarified.

1) When the feeling of wind linked to the scene

of the pot from which steam rose was shown, the

inhaling behavior was induced in subjects, and when

a smell was discharged timely, the smell perception

rate enhanced.

2) When the smell matched the food image

contents, a realistic sensation was clarified, a sense

of appetite and saliva discharge were enhanced and

the changes of cerebral blood flow in the saliva

center increased. In this way the subjective

evaluation results were complemented with

objective data.

The author et al. will recruit more subjects for

the experiments using NIRS in order to enhance the

credibility of the experiment and further clarify the

psychological effect of the images with smell using

other psychological analytical devices, such as a

gaze detection device and pupil measuring device,

and also will discuss the application to digital

signage.

REFERENCES

Bembich S., Lanzara C., Clarici1 A., Demarini S., Tepper

B. J., Gasparini1 P., and Grasso D. L.,(2010).

FoodImagePresentationSystemthatDischargeSmellThroughScreenandPsychologicalEffect

267

Individual Differences in Prefrontal Cortex Activity

during Perception of Bitter Taste Using fNIRS

Methodology, Chem. Senses vol.35, Issue 9, pp.801–

812.

Burke, R. R. , (2009). Behavioral effects of digital

signage. In Journal of Advertising

Research,49(2),180-185.

Eda H., (2006). Near infrared spectroscopic measurement

of brain action and its limitations. In IEICE Tcchnical

Report, HIP2006-56, pp.63-66.

Kokan N, Sakai N, Doi K, Fujio H, Hasegawa S,

Tanimoto H, Nibu K., (2011). Near-infrared

spectroscopy of orbitofrontal cortex during odorant

stimulation, American Journal of Rhinology &

Allergy; May/Jun2011, Vol. 25 Issue 3, p163.

Matsukura H., Yoneda T., Ishida H. , (2012). Smelling

screen: Technique to present a virtual odor source at

an arbitrary position on a screen, Virtual Reality Short

Papers and Posters (VRW), 2012 IEEE, pp.127-128.

Nakamoto T., et al., (2012). Human Olfactory Displays

and Interfaces: Odor Sensing and Presentation. In IGI-

Global, ISBN13: 9781466625211.

Nakano A., Yamaguchi M., (2011). Evaluation of Human

Stress using Salivary Amylase. In Japanese Journal of

Biofeedback Research, 38(1), pp.3-9.

Sakaino A., (2008). The concept of sensitivity

communication using aroma and a plan of

demonstration experiment. In IEICE technical report.

Electronic information displays, 108(291), pp.53-57.

Sato H., et al., (2011). Application of near-infrared

spectroscopy to measurement of hemodynamic signals

accompany stimulated saliva secretion. In Journal of

Biomedical Optics, 16 (4), pp.047002.1-8.

Sawada E., Awaji T., Morishita K., Furukawa M., Aruga

T., Kimura H., Fujii T., Takeichi R., Shimizu N., Ida S.,

Sugimoto M., Inami M., (2008). A Wind

Communication Interface : BYU-BYU-View,

Transactions of the Virtual Reality Society of

Japan,vol.13,No.1, pp.375-383.

Tomono. A., and Tomono K., (2008). Display. In

WO2008/093721, Japan Patent No.5317706.

Tomono K., Katsuyama H., Tomono A., (2011). A Scent-

Emitting Display Panel and Its Psychological Effects. In

The Journal of The Institute of Image Information and

Television Engineers, Vol.65, No.10, pp.1411-1419.

Tomono K., et al., (2012). Image Presentation with Smell

for Digital Signage and The Effect on Eye Catching.

In Proceedings of SIGMAP 2012, No.34, pp.157-162.

SIGMAP2014-InternationalConferenceonSignalProcessingandMultimediaApplications

268