Recipe Enrichment: Knowledge Required for a Cooking Assistant

Nils Neumann and Sven Wachsmuth

Research Institute for Cognition and Robotics (CoR-Lab), Bielefeld University,

Universit

atsstr. 25, 33615 Bielefeld, Germany

Keywords:

Knowledge Representation and Reasoning, Knowledge-based Systems, Cognitive Assistance, Assisted

Living, Applications of AI.

Abstract:

The preparation of a meal consisting of multiple components with different timing and critical synchronization

points is a complex task. An automated system assisting a human in the preparation process needs to track the

progress state, prompt the next recipe steps, control kitchen devices, monitor the ﬁnal preparation time, and

deal with process deviations. This requires a detailed process representation including knowledge about states

with critical timing, control signals of devices, preparation steps and cooking times of ingredients, and neces-

sary user attention. At the same time, the system should be ﬂexible enough to allow the free combination of

component recipes and kitchen devices to support the preparation of complete menus independent of a speciﬁc

kitchen setup. To meet these requirements, we propose a method to automatically enrich simple component

recipes with process-speciﬁc information. The resulting detailed process description can be processed by stan-

dard scheduling algorithms to sequence the preparation steps of complex meals. The control information for

kitchen devices is already included in the process description, so that monitoring of the progress becomes

possible. The reasoning process is driven by so-called action templates that allow to decouple knowledge on

recipes, ingredients, and kitchen devices in seperate re-usable knowledge bases.

1 INTRODUCTION

Preparing a meal consisting of multiple components

is a task with a high cognitive workload, especially

for inexperienced human cooks. It requires planning,

coordination, multitasking as well as memorizing and

accessing knowledge about ingredients, cooking de-

vices and associated actions. We aim at the devel-

opment and implementation of an automated intelli-

gent assistance that supports humans in the cooking

process at their regular kitchen environment. In this

paper, we address the challenge of formalizing and

enriching component recipes so that process descrip-

tions for preparing complete meals can be automat-

ically generated. These can be utilized for monitor-

ing, control, and feedback loops during the human-

centered cooking process in order to signiﬁcantly re-

duce his or her cognitive workload. In order to close

the representational gap between regular ﬁxed, pre-

deﬁned recipes for complete meals towards a generic

process description of a human cooking process, we

need to model implicit knowledge that an experienced

cook would add by his or her “skilled practice, the

senses and memory” (Sutton, 2018). As a key con-

cept, we propose an action-centered representation

consisting of a composable template structure.

In a regular predeﬁned recipe, implicit knowl-

edge is already considered during its creation, such

as causal relationships between the sub-tasks, use of

the devices, and the attention resource required by the

cook. However, meals are typically composed of mul-

tiple components, with one sub-recipe for each com-

ponent, that partially interfere with each other. Thus,

recipes cannot be freely combined without solving

conﬂicts between device usage and task order. In or-

der to ﬁnd a generic approach that allows free com-

bination of component recipes, more information for

each sub-task are required, like its necessary attention

or logical connection between tasks. Having a com-

posable representation in place would allow to scale,

individualize, and adapt the cooking process to the

needs, preferences, and expertise of the user as well

as to the availability of the kitchen equipment.

In the cooking domain, several assistant systems

(An et al., 2017; Sato et al., 2014) exist following dif-

ferent approaches. Some cooking assistants like the

“KogniChef” (Neumann et al., 2017) model the skills

together with dependencies between the tasks and ad-

ditional information for device coordination inside the

recipe. This provides a good solution for cooking

822

Neumann, N. and Wachsmuth, S.

Recipe Enrichment: Knowledge Required for a Cooking Assistant.

DOI: 10.5220/0010250908220829

In Proceedings of the 13th International Conference on Agents and Artiﬁcial Intelligence (ICAART 2021) - Volume 2, pages 822-829

ISBN: 978-989-758-484-8

ﬁxed sequential recipes but does not allow the cook

to prepare component variations or different meals in

parallel. Another cooking assistant “Cooking navi”

(Hamada et al., 2005) is utilizing action units that

have dependencies between each other and uses de-

vices and the user as resources to plan the execution

order of the action units. A plan is provided over a

touch panel as the user interface, without taking the

feedback of kitchen devices into account.

The Assistive Kitchen (Beetz et al., 2008) and

RoboEarth (Tenorth et al., 2012), among others, pre-

pare a meal with a robot. Therefore, they use a recipe

that provides a set of steps, add missing information

and additional micro actions which contain motion

primitives that can be executed by the robot. These

systems provide an approach for task representation

using an ontology and multiple small steps but focus

on a different goal with other required information,

due to coordinating a robot and not a human.

The TAAABLE system, from (Badra et al., 2008),

focuses on providing cooking recipes with a case

based system on a semantic wiki. It receives requests

with constrains for the recipe like desired ingredients

and provides a text based solution for the recipe. It

also forms the basis for the Computer Cooking Con-

test (CCC) (Nauer and Wilson, 2015). Teams com-

pete to ﬁnd the best solution for given challenges.

2015 challenges like providing a sandwich recipe

or making a cocktail with speciﬁed wanted and un-

wanted ingredients were solved. The EVER project

(Bergmann et al., 2017) creates semantic workﬂows

based on textual sources, using similarity-based re-

trieval methods to ﬁnd the best matching workﬂow.

They also participated multiple times in the CCC.

While the TAAABLE system, EVER project and the

CCC generate or adapt recipes for humans, we focus

on enriching recipes for the execution with a cooking

assistance, that requires additional information about

the recipe steps.

In this paper, a recipe representation utilizing ac-

tion templates is described that closes the gap be-

tween a regular predeﬁned recipe and a composable

recipe for a cooking assistant supporting a human

cook. The action templates introduce a new abstrac-

tion layer, which captures device-dependent interac-

tion patterns and feedback loops between a human

user and the partially automated device control (com-

parable to the robot’s micro actions in the Assis-

tive Kitchen). They further allow a free combina-

tion of different component recipes considering the

required implicit knowledge, device setup and user

skills, which enables a high individualization and

ﬂexibility for the cook.

2 PROBLEM STATEMENT AND

SYSTEM DESCRIPTION

Regular predeﬁned recipes already encode ﬁxed im-

plicit knowledge about tasks, task dependencies, in-

gredients, actions, and kitchen devices. Either they

leave their combination with other recipes to the skills

of the cook or need to be completely rewritten for

any changing component or device. They do not

consider recipe variations or interference with other

recipes. Consequently, the sequence of tasks is de-

ﬁned from the perspective of the creator of the spe-

ciﬁc recipe and his or her assumptions about a stan-

dard cook. Enabling an automated cooking assistant

to support an individual human cook, while prepar-

ing multiple component recipes together as a meal re-

quires a higher ﬂexibility.

Therefore we propose an action-centered ap-

proach that explicitly represents the knowledge im-

plicitly coded before. Figure 1 shows the difference

between a simple component recipe (provided as an

input to the system) and a process description (pro-

vided as an output of the system). The method pro-

posed in this paper is able to automatically enrich

such a recipe utilizing generic knowledge bases.

The action-centered approach is embedded in a

complete system for assisting users in a cooking pro-

cess at home. The main components of this system

are shortly described in the following. In this paper,

we focus on the recipe enrichment method providing

the process description:

• Knowledge Representation: provides the re-

quired recipe information from three sources

(action-template ontology, ingredient representa-

tion, recipe representation) and combines these

into multiple component recipes with different

types of preparation, which are usable at the same

time by the cooking assistance.

• Scheduler: schedules multiple component

recipes together providing an executable se-

quence of tasks, based on the available devices

and the different types of preparation.

• Device Control: provides the available devices

for the planning component, monitors and exe-

cutes tasks for the devices.

• Monitoring: supervises and controls the execu-

tion of the recipe, handles input from the user in-

terface and device control, detects deviations from

the plan and starts re-planning if necessary.

• User Interface: provides information and inter-

acts with the user.

Recipe Enrichment: Knowledge Required for a Cooking Assistant

823

Figure 1: The upper part of the picture shows a regular recipe for the preparation of cooked potatoes. The lower part of

the picture shows the enriched recipe for cooked potato with the action-centered approach that enables the use by the kitchen

assistant. Here the ”cook potato” task is split into several smaller tasks. Each task includes the necessary resource information

for the execution (task color), the logical dependency between the task (arrows over the tasks), device control commands for

automatic device control and prompting information for the user when the state of the device changes.

2.1 Modeling Composable Recipes for

Assisted Cooking

Most regular (human-deﬁned) recipes are designed by

combining an average of ten actions together with an

average of nine ingredients (Yamakata et al., 2017;

Salvador et al., 2017). Only a limited number of

around 100 common action-verbs are used in a cook-

ing scenario, as shown in the EPIC-KITCHENS data

set (Damen et al., 2018). Therefore, we keep our

action-centered approach with a similar subdivision

of action types.

Our action-centered approach consists of three

different knowledge sources which are, then, com-

bined to create an executable component recipe by

utilizing the type of action as binding concept con-

necting them:

• Action-template Ontology: an action template is

a concept in the action-template ontology deﬁn-

ing the action types to realize the appropriate

tasks given by the recipe representation which

is enriched by the ingredients from the ingre-

dient ontology. Therefore the action templates

can include multiple action-options to realize a

task in a recipe, e.g. cook with steamer or

hob, each with previous implicit sub-actions and

their dependencies, necessary devices/utensils re-

quired for each action-option. It further deﬁnes

how the sub-actions are parameterized applying

the information from the ingredient representation

and recipes, as shown in Figure 1 for cook hob

with the parameterization of the potato ingredi-

ent. The action template instantiates the task from

the recipe with (multiple-) actions executable by

the cooking assistant based on the devices, cook

attention, devices commands, and device interac-

tion types.

• Ingredient Representation: an ingredient inside

the ingredient representation is a concept that de-

ﬁnes all executable action templates for the spe-

ciﬁc ingredient and their parameterization (dura-

tion etc.).

• Recipe Representation: a recipe in the recipe

representation contains the speciﬁc tasks which

must be carried out to prepare and cook the de-

ﬁned component recipe. Each task from a recipe

consists of a reference to an action template, a list

of ingredients with quantities and the logical de-

pendencies to other tasks. If no further parameters

are set, the ingredient information for the spec-

iﬁed action template is used to parameterize the

task.

Combining the component recipes selected by the

cook, as described in the recipe representation, cre-

ates the menu. Each step from a component recipe

is represented as a task with its logical dependen-

cies. These dependencies are described as connec-

tions to its predecessor and successor tasks, instead of

a chronological sequence as frequently found in regu-

lar recipes. This creates a higher ﬂexibility in combin-

ICAART 2021 - 13th International Conference on Agents and Artiﬁcial Intelligence

824

ing component recipes, but means that the enriched

process description – which combines all component

recipes – must explicitly include all information re-

quired for a later scheduling of the tasks. As a con-

sequence, each component recipe can be composed

individually making it easier to extend the knowledge

base.

Deriving Ingredient Information for Recipes. In-

side the ingredient representation all feasible types

of action are deﬁned for each ingredient and neces-

sary parameters are parameterized with default val-

ues independent of the recipe. These action types are

linked to the action templates and provide all infor-

mation for the execution by a cooking assistant, like

the abstract device interaction type, the necessary sub-

actions for the execution or scaling function for dif-

ferent ingredient quantities. For an instantiation in

the process description, each action-ingredient pair of

a task from the recipe is bound to an action type in

the ingredient representation and the task is enriched

with missing information from the ingredient repre-

sentation and the associated action template. Through

this distributed representation of knowledge, the re-

dundancy is reduced, while keeping the opportunity

to adjust parameter speciﬁcally for a recipe. As a con-

sequence, only very few information are required in a

component recipe, such that its creation in a machine-

readable format is a nearly effortless task. In this way,

the tasks in the component recipes just need recipe

speciﬁc information (e.g. divergent cooking times,

quantities of ingredients). All general knowledge is

taken from their default parameters. As an example,

duration times (relative to the quantity) are deﬁned

in the action templates (by using scaling functions).

This allows an automated scaling of different quanti-

ties in the recipe.

Deriving Sub-tasks for Monitoring, Prompting

and Device Control. Enabling a cooking assistant

to support a human cook during the handling of

kitchen devices requires a more detailed representa-

tion than typically encoded in recipes, as shown in

Figure 1. These information are coded in the action

templates and describe the execution of a task in a

more precise way. The action templates are parame-

terized with the information from the recipe utilizing

the task representation as an action-ingredient pair.

Combing the recipe and ingredient information (dura-

tion, temperature, dependency between tasks, ingredi-

ents and quantities) with the action templates that pro-

vide information like user attention, device parame-

ters, and additionally required actions, creates the de-

tailed actions that are used by the cooking assistant

for prompting and monitoring the cooking progress.

Splitting a task into multiple smaller actions pro-

vides advantages in terms of utilizing more precise

information of the cooking process and a more ex-

act planning. Considering the limited user attention,

e.g. cooking in a steamer has a changing attention

level during the execution. Adding and removing has

a high user attention, while cooking has no user atten-

tion. Having smaller actions also creates device con-

trol and feedback loops based on the combination of

device feedback and abstract device interaction types

as coded in action templates. These abstract seman-

tic information (e.g. Add, Remove, Execute, Inter-

act) together with other action parameters (e.g. dura-

tion, temperature, device mode) can be executed and

monitored by the device control component, which

receives the device’s feedback over a network inter-

face (e.g. state of the device (program, door contact

switch)). Therefore, the device interaction type from

the action template, combined with the device feed-

back, as instantiated in the process description is uti-

lized to automatically monitor the task progress, e.g.

while an action with an Add interaction type is run-

ning, opening and closing an oven, can be interpreted

as putting something inside. Using a limited set of

abstract interaction types reduces the number of cases

that can occur. This enables an automatic device con-

trol and progress detection, reducing the conﬁrmation

actions for the user and creating a more appropriate

feedback for the human cook.

Deriving Planning Options for Task Scheduling.

Preparing multiple component recipes as a meal re-

quires solving resource conﬂicts. While the necessary

information are available in the process description,

there is often more than one option to perform an ac-

tion with a speciﬁc ingredient considering the avail-

able devices, (e.g. boil noodle in a hob or steamer).

Therefore the action templates have a hierarchical

structure, that contains all action-options (e.g. differ-

ent devices/utensils) for the speciﬁc action type (e.g.

action template cook has action-options cook hob and

cook steamer). These possible options of preparation

can be limited in the ingredient or recipe representa-

tion by the speciﬁcation and parameterization of pos-

sible devices/utensils. In this way, the scheduling al-

gorithm is able to ﬁnd the best plan depending on the

set of chosen component recipes, enriched by the dif-

ferent knowledge sources, the possible action-options

and the available devices provided by the device con-

trol component.

The separation of tasks and action templates fur-

ther enables a personalization of the user actions, stor-

ing previous cooking performances in relation to the

Recipe Enrichment: Knowledge Required for a Cooking Assistant

825

action and ingredient. Also, their personal prefer-

ences, like preferred devices and recipe combinations,

can be considered in future planning processes.

Through the enrichment of the recipe, we know

the logical dependencies between tasks and actions,

the necessary kitchen devices and user attention for

each action. This deﬁnes a valid process descrip-

tion for computing the execution order of the actions

in the component recipes selected. Considering the

cook and the kitchen devices as resources, with the

user attention level as workload for the resources and

the individual recipes as projects, we can interpret

it as multi-mode resource-constrained multi project

scheduling problem. We solved the scheduling prob-

lem with the optaplanner tool (De Smet et al., 2016),

while considering rules that take the dependency be-

tween actions, the workload of the resources (device

and user), the maximum duration of the cooking pro-

cess and the synchronization of the end time for the

individual recipes into account.

2.2 Generating Process Descriptions

from Composable Recipes

In the following, we explain how the process de-

scription (e.g. prepare potatoes with meatballs and

cauliﬂower) is computed in the knowledge repre-

sentation from scalable and composable component

recipes, which is then passed to the scheduler. The

cooking assistant is implemented as a distributed sys-

tem, as described in the section 2, offering interfaces

via the MQTT network protocol using the interface

description language protocol buffers to serialize the

data.

At ﬁrst, the cook selects the composable compo-

nent recipes to be prepared together with the number

of servings. For each action in a selected component

recipe, the content information required for its coor-

dination in the cooking process are automatically de-

duced from the ingredients in the ingredient represen-

tation, if they are not speciﬁed in the recipe.

A recipe, as shown in Figure 2, has general infor-

mation (title, number of servings) and a list of linked

tasks. The number of servings allows to scale a recipe

consistently relative to the number of servings cho-

sen from the cook. The list of tasks includes unique

identiﬁers for all actions required for a recipe. The

action key in each task references to an action in the

appropriate ingredient representation, if an ingredient

is used in the step and the ingredient can be used with

the corresponding action template. Furthermore, each

task contains the ingredients, their quantity, and a list

of predecessor tasks using the unique identiﬁer from

the recipe task list. Optional parameters for each task

Figure 2: Recipe “Cooked Potatoes”, with two tasks

“cookpotato” and “peelpotato” that have a dependency,

where “peelpotato” must be ﬁnished before “cookpotato”

can start. The ingredient potato with 800g is used for 4 serv-

ings. The actions “cook” and “peel” from the recipe tasks

are used as references for the actions in the ingredient.

can be a speciﬁc duration if, e.g., the cooking time

is shorter than normal for an ingredient, images and

videos if they are speciﬁc for the recipe. They can

also contain, required and preferred devices, utensils

and additional action parameters if they are required

by the action template.

Figure 3: Ingredient “potato” with parameters for ac-

tions that can be performed with the ingredient. The task

“peelpotato” from the recipe in Figure 2, with action “peel”

and ingredient “potato” use the parameter from the “peel”

action in the ingredient “potato” for the parameterization of

the peel action template. The duration for the action “peel”

is relative to the quantity of potatoes in the recipe. The ac-

tion “cook” has parameters for the preparation options with

a steamer (17 min) or with a hob (25 min).

Missing non-recipe-dependent parameters are added

from the ingredient representation, as shown in Figure

3. This template contains basic information about the

ingredient, as well as parameter for each processable

action type. E.g. ”peelpotato” from the recipe task in

Figure 2 uses 800g potatoes for 4 servings. This in-

formation is combined with a duration of 2 min/100g

from the ”peel”-action in the ingredient representa-

tion and a scaling function from the action template

to calculate the duration with the used amount of the

ingredient. The scaling function from the action tem-

plate is necessary, since the scaling can differ depend-

ICAART 2021 - 13th International Conference on Agents and Artiﬁcial Intelligence

826

ing on the action (e.g. constant value for cooking, lin-

ear scaling for peeling).

Subsequently, the list of tasks, completed by the

ingredient representation, is expanded by applying the

action templates to the named appropriate tasks. The

action templates enrich the recipe and create the ac-

tions for the cooking assistant as shown in Figure 4.

Each action template may contain several actions as

pre-/post-actions, that split the recipe task in multiple

more ﬁne graded actions. They further deﬁne if pre-

/post-actions are parameterized with ﬁxed values or if

their parameters are passed through from the recipe.

Additionally, some action templates model several

options to carry out a task. E.g. the action template

“cook” can be performed using either a steamer or a

hob as shown in Figure 4. Both are modeled as sub-

action templates derived from the “cook” template.

All valid options that fulﬁll the constraints of avail-

able devices and recipe speciﬁcations are taken into

account for the subsequent scheduling task.

After connecting the tasks from the recipe with the

action-templates, the relation between the tasks im-

plies dependencies between ﬁrst and last (pre- or post-

) actions of the corresponding action templates as

shown in Figure 4, part 3. The user attention (depicted

for each action by the stick ﬁgure) is provided by

the action-templates and allows a parallel execution

of actions as long as their user attention together do

not exceed the maximum workload for the resource.

The action-templates further specify how the dura-

tion of the action scales with ingredient quantities de-

ﬁned in the recipe. A post action priority ensures the

immediate execution of time-critical tasks following

the action. Additionally, categorized types of (user-)

device interactions, like add, execute, remove, inter-

act, and preheat, allows an automated progress detec-

tion based on sensor data of the devices (e.g. contact

switches, temperature drops). This can be utillized

for generating feedback loops for the user in the de-

vice control component. For prompting and control,

they include information like device commands, pos-

sible cutting states, temperature, or utensils required

to expand the actions.

Utilizing the separated knowledge sources, indi-

vidual user proﬁles can also be maintained. If reg-

istered users have used the cooking assistant before,

their preferences for multiple action-options, their

preferred recipe combinations, and their preparation

times required are saved in relation to the action and

ingredient. Based on this information, recipe com-

binations can be recommended, cooking instructions

can be adjusted to user skills and preferences, and

preparation times for an action can be estimated more

precisely.

After enriching the recipe with necessary infor-

mation for the execution by a cooking assistant, the

recipe is planned by the scheduling component, as

shown in Figure 5. The scheduling process takes the

available resources (devices provided by the device

control component and cook provided by the user in-

terface) and dependencies between tasks into account

and generates an executable sequence of tasks with

the available devices. The planned sequence of tasks

is executed and supervised by the monitoring compo-

nent. Therefore, it uses feedback from the user inter-

face and the device control component. If the execu-

tion of the recipe deviates too much from the planned

recipe, a re-planning is started. If possible, the con-

ﬁrmation of the recipe task is automatically accom-

plished by the device control, else the conﬁrmation

is carried out by the cook via the device interface.

For the automatic task conﬁrmation the device con-

trol component uses the sensor input from the devices,

e.g. door contact switch, selected program, tempera-

ture, in combination with the device interaction types.

Using the device interaction types abstracts the action

independently of the devices, e.g. hob, oven, steamer,

and deﬁnes a ﬁxed amount of interaction possibilities.

The detection of the device interaction types may dif-

fer for different devices, e.g. ”Add something in the

steamer” can be detected by the door contact switch,

while ”Add something in a pot on the hob” requires

other device information due to the missing door con-

tact switch. By knowing which interaction type has to

be carried out, sensor information can be interpreted

in the appropriate context.

Due to the action-options with different devices,

a more ﬂexible planning for multiple component

recipes is possible. E.g. if the steamer is used for the

cauliﬂower, the potato can alternatively be prepared

with the hob. By the division of tasks into smaller

sub-tasks, as shown in Figure 5 for the cook potato

task, a more accurate time management for the cook

is made possible, which can enable better planning

options.

3 VALIDATION OF THE RECIPE

ENRICHMENT

The approach was validated in a prototypical kitchen

environment with the cooking assistant as described

in the section 2. The setup included real kitchen

devices which were automatically registered as re-

sources in the system. The knowledge base consisted

of a set of 3 carbohydrate side dishes (C), 3 veg-

etable side dishes (V) and 3 main components (M),

with multiple preparation possibilities, which were

Recipe Enrichment: Knowledge Required for a Cooking Assistant

827

Figure 4: The recipe tasks enriched with the ingredient parameter (1. blue), are combined with the corresponding action

templates (2. grey), creating the actions (2. green) with their pre-/post-actions. The actions are parameterized with the recipe

parameter (blue text) and the action template information (gray arrow, e.g. stick ﬁgure for user attention). The ﬁrst row of

each action shows the action type and used device/utensil. The second line the ingredient. While “peel potato” only has one

option and no pre-/post-actions, “cook potato” has two options with pre-/post-actions. 3. The dependencies from the recipe

are transferred onto the actions and the “peel potato” action on the left now points to both possible action-options, where the

scheduler have to choose one for the cooking process.

Figure 5: Planning example of one component recipe. Starting from the generated plan in Figure 4 part 3. The recipe is

planned depending on the available devices. In this example the kitchen only contains a steamer and no hob. Therefore the

possibility with the hob (Figure 4 part 3 upper option) can not be executed with the available setup and is removed. The

remaining tasks are planned, as shown, due to the utilization of resources (steamer and cook), and the dependency between

the tasks. During the execution of the recipe, the completion of the ﬁrst cooking step ”peel potato” (red 1) must be conﬁrmed

by the cook, since no devices are used. The steps ”add-/remove- steamer potato” (red 2/4) requires both the cook and the

steamer as resources, in this cases the task execution is detected by the device due to the interaction with the door and the next

step is started. The cooking step ”cook steamer potato” only needs the steamer as resource and requires no attention from

the cook. In this time the cook could perform other tasks from other component recipes if multiple component recipes are

planned together. Due to the feedback from the device the end of the task (red 3) is detected automatically.

ICAART 2021 - 13th International Conference on Agents and Artiﬁcial Intelligence

828

provided by a professional cook. These component

recipes were freely combined to valid recipes, with

one component from each set, considering the avail-

able devices (changing combinations of hob, steamer,

oven). The average number of tasks in a menu was

33.6 (C: 14.3, V: 4.6, M: 14.6). After the enrich-

ment of the recipes into a complete description of the

cooking process, the average task number changed to

44.3 (in case of multiple options the average num-

ber of tasks was counted). Multiple freely combined

recipes were prepared as system validation, both with

and without food, resulting in a functional cooking as-

sistance, in terms of planning, re-planning in case of

deviations, device control and automatic conﬁrmation

of user interactions from the devices.

4 CONCLUSION

While regular (human readable) recipes look well-

structured and simple, the cooking process itself fre-

quently includes many pitfalls and unwritten (but nec-

essary) small pre- and post-actions which determine

the success of cooking. Therefore, today’s available

cooking assistants are designed either for a very spe-

ciﬁc device or for predeﬁned hand-modeled recipes

which are completely decoupled from kitchen de-

vices. In comparison to other cooking assistants our

action templates enriches the recipe steps not only

with necessary information for visualization, plan-

ning, micro actions, and commands for device coor-

dination, but also with information about user-device

interaction and optional preparation alternatives. In

this paper, we propose action templates as a back-

bone for modeling component recipes that can be en-

riched and combined to full-blown descriptions of the

human-device interactions during a cooking process

and show its potential and beneﬁts.

REFERENCES

An, Y., Cao, Y., Chen, J., Ngo, C.-W., Jia, J., Luan, H., and

Chua, T.-S. (2017). Pic2dish: A customized cook-

ing assistant system. In Proceedings of the 25th ACM

International Conference on Multimedia, MM ’17,

page 1269–1273, New York, NY, USA. Association

for Computing Machinery.

Badra, F., Bendaoud, R., Bentebibel, R., Champin, P.-A.,

Cojan, J., Cordier, A., Despres, S., Jean-Daubias, S.,

Lieber, J., Meilender, T., Mille, A., Nauer, E., Napoli,

A., and Toussaint, Y. (2008). Taaable: Text mining,

ontology engineering, and hierarchical classiﬁcation

for textual case-based cooking.

Beetz, M., Stulp, F., Radig, B., Bandouch, J., Blodow, N.,

Dolha, M., Fedrizzi, A., Jain, D., Klank, U., Kresse, I.,

Maldonado, A., Marton, Z., Mosenlechner, L., Ruiz,

F., Rusu, R., and Tenorth, M. (2008). The assis-

tive kitchen — a demonstration scenario for cognitive

technical systems. pages 1 – 8.

Bergmann, R., Minor, M., M

uller, G., and Schumacher, P.

(2017). Project ever: Extraction and processing of

procedural experience knowledge in workﬂows. In

ICCBR (Workshops), pages 137–146.

Damen, D., Doughty, H., Farinella, G. M., Fidler, S.,

Furnari, A., Kazakos, E., Moltisanti, D., Munro, J.,

Perrett, T., Price, W., and Wray, M. (2018). Scaling

egocentric vision: The epic-kitchens dataset. In Euro-

pean Conference on Computer Vision (ECCV).

De Smet, G. et al. (2016). Optaplanner user guide. Red

Hat and the community. URL http://www. optaplanner.

org. OptaPlanner is an open source constraint satis-

faction solver in Java.

Hamada, R., Okabe, J., Ide, I., Satoh, S., Sakai, S., and

Tanaka, H. (2005). Cooking navi: assistant for daily

cooking in kitchen. In Proceedings of the 13th annual

ACM international conference on Multimedia, pages

371–374.

Nauer, E. and Wilson, D. C. (2015). Computer cooking

contest.

Neumann, A., Elbrechter, C., Pfeiffer-Leßmann, N.,

oiva, R., Carlmeyer, B., R

uther, S., Schade, M.,

Uckermann, A., Wachsmuth, S., and Ritter, H. J.

(2017). “kognichef”: A cognitive cooking assistant.

KI - K

unstliche Intelligenz, 31(3):273–281.

Salvador, A., Hynes, N., Aytar, Y., Mar

ın, J., Oﬂi, F., Weber,

I., and Torralba, A. (2017). Learning cross-modal em-

beddings for cooking recipes and food images. pages

3068–3076.

Sato, A., Watanabe, K., and Rekimoto, J. (2014). Mimi-

cook: A cooking assistant system with situated guid-

ance. pages 121–124.

Sutton, D. (2018). Cooking Skills, the Senses, and Memory:

The Fate of Practical Knowledge*, pages 88–109.

Tenorth, M., Perzylo, A. C., Lafrenz, R., and Beetz, M.

(2012). The RoboEarth language: Representing and

Exchanging Knowledge about Actions, Objects, and

Environments. In IEEE International Conference on

Robotics and Automation (ICRA), St. Paul, MN, USA.

Best Cognitive Robotics Paper Award.

Yamakata, Y., Maeta, H., Kadowaki, T., Sasada, T., Ima-

hori, S., and Mori, S. (2017). Cooking recipe search

by pairs of ingredient and action —word sequence

v.s. ﬂow-graph representation—. Transactions of The

Japanese Society for Artiﬁcial Intelligence, 32:71–79.

Recipe Enrichment: Knowledge Required for a Cooking Assistant

829