The Shopping Scout: A Framework For An Intelligent
Shopping Assistant
Matthias Kaiser and Christine Mueller
SAP Research Center Palo Alto LLC, 3475 Deer Creek Road, CA 94304, USA,
Abstract. The growing availability of mobile devices leads to a new area of
applications which give support to human users in situations they could
previously not benefit from. In this paper we would like to describe the concept
of proactive user interfaces, exemplified in the realization of a shopping scout.
By this we mean an application on a mobile computational device which can
act as a recommender system for an optimized shopping experience. The
shopping scout guides users in a supermarket to those places where products on
their shopping list are located. Additionally, the route planned to be pursued
during shopping can be modified according to user preferences with regards to
properties of certain items. By using capabilities of WIFI communication, we
are able to guide users on their way through a store and provide the possibility
to call for assistance.
1 Introduction
The main goal of our research consists of finding ways to conceptualize and realize
user interfaces for complex applications. Applications become more accessible and
usable by exposing users to a high degree of navigation-, selection- and decision-
support, by making interfaces available on diverse computational platforms, and by
considering a variety of user needs and preferences, [1, 5]
As a result of our endeavors in this area, we have developed a framework of
intelligent user interfaces which we will outline and briefly apply to a familiar
scenario: shopping in a supermarket.
The core of our approach is the ability of interfaces to intelligently reason about the
content which they are communicating to human users. This presupposes a domain
model including objects, relations and properties of things that are subject to the
human-computer discourse as well as actions and goals that can be carried out and
achieved [6, 7].
It is our goal to use the content awareness of the interface to automatically or semi-
automatically adapt the communication including the choice and configuration of
presentation modalities and formats according to user needs and preferences.
In section two describe the objective and benefits proactive user interfaces. In section
three we highlight what content proactive user interface provide to support users. In
Kaiser M. and Mueller C. (2005).
The Shopping Scout: A Framework For An Intelligent Shopping Assistant.
In Proceedings of the 1st International Workshop on Web Personalisation, Recommender Systems and Intelligent User Interfaces, pages 146-153
DOI: 10.5220/0001420801460153
Copyright
c
 SciTePress

section four we outline a common application scenario. Section five will describe this
scenario in the context of proactive user interfaces. We will conclude with a brief
discussion of future work in section six.
2 Objectives and Benefits of Proactive User Interfaces
In order to build highly accessible and usable human-computer interfaces we have to
face vital challenges in today's software development: Most powerful applications are
not appreciated if users cannot take advantage of their power in a reasonably easy
way [5].
Our work in the field of advanced user interfaces resulted in the framework of
intelligent content-driven interfaces which we call proactive user interfaces. Based on
knowledge about the domain, tasks and goals, proactive UI’s provide information
about the domain, help to perform tasks and to achieve goals [3, 6, 7].
With our approach we address the following objectives:
- Accessibility for users with special needs and preferences
- High usability for all users, whether novice or expert
- Consistent applicability and behavior on diverse hardware, from the desktop
to the smart phone
- Easy adaptability across interaction modalities, from GUI’s to voice to natural
language communication
The special focus in our research is on cognitive support for all users, not just for
people with disabilities [4]. We think this is valuable at least for the following two
reasons:
1. The cognitive support users need depends on their cognitive capabilities in
relation to the cognitive load imposed by the task or problem to be solved.
This means cognitive support is not only justified for disabled or specially
challenged users but even for a Nobel Prize Laureate given a complex
enough issue.
2. Computers have strengths partially complementary to those of humans. An
important point we want to make is to use computer devices to enhance
cognitive capabilities by providing complementary aid to the human problem
solver. An example of a more trivial kind would be using a pocket calculator
instead of mental arithmetic.
We think of cognitive support as timely help given by the computer with respect to
different stages in the problem solving process. This means that we view HCI as a
manifestation of a problem-solving process, in which human users try to solve a
147

problem by exploiting the capabilities of the software to optimize the problem solving
results [8].
The computer will be of greatest help for users if it
1. Supports exactly those stages of the problem solving process in which users
are currently in, and
2. Does not itself pose a problem, thus adding to the problem solving tasks
instead of reducing them.
3 How Proactive User Interfaces leverage Cognitive Support
Proactive interfaces comprise the following main functionalities to provide cognitive
support:
1. Situational Awareness
Proactive user interfaces have the capability of describing the users’ situations with
special focus on those objects, properties and relations which are relevant for
achievable goals. Realizing situation awareness directly satisfies one of the
components of accessibility. It is important and therefore explicitly stated here that
situational awareness does not necessarily mean a complete description of all
situational constituents. If the interface has the capability to do so only those
constituents are communicated which directly will or can influence the further actions
of users. Situations are described in terms of the domain lingua, i.e., in a way users
can understand without any application-specific technical knowledge.
2. Recommendation of feasible Goals
Based on situation descriptions, the next step is to explore feasible options. Relevant
options to be pursued in a domain depend on the users’ expertise and goals [12].
Users’ final goals are accomplished by achieving a number of sub-goals. Therefore,
our next feature of proactive user interfaces consists of the recommendation of sub-
goals that are the best next choice on the way of achieving the users’ final goals.
3. Guidance towards accepted Goals
Once an option in the form of a feasible goal based on the users’ current situation has
been chosen, the interface will provide step-by-step guidance to realize this goal [9].
This feature is a major part of cognitive support in the human problem-solving
process.
4. Generation of Instructions, Justifications, and Explanations
If an interface can actively provide cognitive support it should justify its selection of
relevant situational constituents in order to induce trust and believability in users.
148

Furthermore, the interface should be able to explain why a goal was recommended,
instruct how to achieve a goal and justify each step. The crucial idea is that the
knowledge provided by the cognitive supportive interface must relate to the current
knowledge of users to allow users to reconstruct the "thinking" of the supportive
interfaces.
Offering explanations during the human-computer interaction is of great value as a
just-in-time learning approach. While solving an actual problem, users are most
motivated to learn about solving this problem. As opposed to learning offline, giving
justifications and explanations dynamically constitutes a flexible form of help that can
adapt to individual needs and preferences. In contrast to other forms of help such as
hypertext documents or paper manuals, online explanations can be generated just in
time when the user needs information. This all helps increase the efficiency and
usability through the learning experience, thereby increasing user satisfaction.
Compare also [10, 11].
After the content to support users has been generated, the question is how to present
it. We use templates stored in template libraries. These templates are filled with the
content based on parameters describing special user needs and preferences, as well as
parameters describing device constraints and modalities. Innovative mapping
techniques are used to transform the content into the optimal presentation format on
different devices and modalities. Content generated for a desktop interface using
mouse, keyboard and a monitor can now easily be presented in a PDA interface using
stylus and a small display, or even to an interface using voice as input and output.
While using the devices they prefer, users can be supported based on their individual
needs and preferences.
4 The Application Scenario
We would now like to describe how a proactive user interface can improve the
shopping experience which can be quite challenging since a number of questions have
to be solved such as:
1. Where am I currently and where can I find the next item on my list?
2. What would be the best and shortest route to get all items?
3. How can I optimize my shopping so that I do not need to carry around
products I prefer to buy as late as possible, like frozen goods or heavy
things?
4. With all this hassle, how do I still remember what I wanted to buy?
In order to provide an intelligent user interface that can answer all of the questions
above, we have to represent several knowledge sources in the interface:
- Domain knowledge about products and product properties such as weight,
and storing temperature
- Location knowledge including store map, locations of products and customer
149

- Customer-specific knowledge including shopping list and preferences which
determine the shopping behavior as well as the preferred presentation format
- Device-specific knowledge about the device used by the customer
Shopping list and preference have to be entered manually by the costumers. The
interface allows customers to create new shopping lists, as well as reuse and modify
existing once. Alternatively, it recommends shopping items or adds items
automatically based on the customers’ previous shopping behaviors. When entering
preferences, templates and previous settings can be used. Preferences influence the
path taken to pick up products: e.g. sequential according to the shopping list, using the
shortest path through the supermarket, or by considering product properties such as
weight and temperature.
Domain knowledge and store map are dynamically loaded to the device when
entering the store. The location of customers is dynamically determined and
permanently updated using WIFI. Details on the technologies used for location
sensing will not be addressed in this paper. For further information please see [2].
Shopping list, customer preferences, store map including location and availability of
products, and location of customers are used to generate optimal routes through the
supermarket. Presentation-preferences as well as device-specific knowledge are used
to identify the best presentation format for the customers: e.g. using a map displayed
on the PDA screen or voice output.
While the algorithm of how to generate the optimal route is not our concern, we focus
on how proactive user interfaces can provide cognitive support of users. We have
identified five ways to support customers:
1. After generating the optimal route, the interface can recommend which item
to purchase next.
2. After approving the recommendation of the interface, the user gets
instruction on how to find the item.
3. The user can request justification of why certain items have been
recommended.
4. The interface provides the possibility to call customer service for further
inquiries.
5 In Dialogue with the Proactive User Interface
Let us consider Mary who intends to go shopping in a supermarket. She needs
different kinds of products: cosmetic articles, spaghetti, sugar, bread, some items from
the refrigerated section such as cheese, milk, and meat, some frozen goods like ice
cream and frozen pizza, as well as heavy beverages like sodas, beer, and wine. She
150

would like to get her shopping done in a rational way. Mary does not want to go back
and forth but rather pick up things along the shortest route possible through the store.
However, she has some preferences regarding when she wants to put certain goods
into her shopping cart:
- The light products first.
- The cooled and especially the frozen things last.
Mary uses a PDA which provides a supportive user interface that will guide her
through the supermarket. Before starting her shopping experience, Mary’s preferences
and shopping items need to be stored on the PDA. Since she does not wish to change
her defaults settings, the interface will use preferences and shopping list from Mary’s
previous shopping trip.
After preferences and shopping list are set, the interface is still missing the map of the
store and Mary’s position in the store. The map of the store and the product
knowledge base are automatically transferred to Mary’s PDA when entering the store.
Using WIFI, Mary’s position in the store is identified, allowing the interface to
describe her current situation. Based on shopping list, preferences, and location data
the system dynamically generates the optimal path for Mary. The interface then
proactively guides her through the store by giving directions according to Mary’s
movements.
The following example shows a possible interaction between Mary and the supportive
interface:
Interface: “Welcome Mary. You are now close to the
frozen products and the cosmetics you want to buy.”
Now follows the goal recommendation. On the knowledge the interface has it
recommends to get the cosmetics first.
Interface: “According to your preferences, I recommend
to buy the cosmetics first.”
The interface now enters the guidance phase and leads Mary to the cosmetic section.
Interface: “I will guide you to the cosmetic products.
Turn left and proceed to the next aisle. … Now turn
left again and you will enter the cosmetics section.”
151

Mary follows the instructions and upon arrival receives new directions:
Interface: “You have picked up the cosmetic products as
noted on your shopping list. You are close to the aisle
containing bread. I recommend going to the aisle with
bread.”
Mary accepts.
Interface: “Proceed to your left. … Turn right please.
… Now walk down the aisles. … Stop, the bread is right
in front of you.”
Mary follows the instructions and on arrival receives further directions. The interface
will next guide her to the spaghetti, to the sugar, then to the frozen section and finally
to the beverages, as specified in Mary’s preferences. When arriving at the beverage
section, Mary cannot reach her favorite red wine and approaches the interface.
Mary: “I need assistance.”
Interface: “I will inform customer service. An
assistant will be with you momentarily.”
While Mary waits, a message is send to the closest customer assistant who finds Mary
and offers his service. After receiving the red wine, Mary continues to collect her
items, proceeds to the cashier, and leaves the store.
6 Conclusion and Future Work
The goal of this paper was to outline and illustrate the core idea of proactive user
interfaces, which provide timely cognitive support in accordance with the phases of
problem solving in which users engage. We find this approach to be a promising
means to improve accessibility, usability and adaptability of interfaces for diverse
input/output devices and presentation styles. We stressed the fact that cognitive
support is valid for any kind of user rather than only to users with disabilities. Finally,
we showed how proactive user interfaces can be utilized in everyday scenarios where
they can help simplify human-computer interaction as well as assist with effective,
efficient and satisfactory achievement of complex goals. Our future work will be
targeted towards efficient implementation of proactive user interfaces, thorough
testing, and verification of their benefits in terms of accessibility and usability.
152

7 References
1. J. Fenn, A. Linden: Hype Cycle for Human-Computer Interaction, Strategic Analysis Report
R-19-9408, Gartner Research, May 2003.
2. G. Roussos, Location Sensing Technologies and Applications, TechWatch Report TSW 02-
08, The Joint Information Systems Committee, Nov. 2002. Available:
http://www.jisc.ac.uk/index.cfm?name=techwatch_report_0208.
3. R. Shank, R. Abelson.: Scripts, Plans, Goals and Understanding: an Inquiry into Human
Knowledge Structures, L. Erlbaum, Hillsdale, New Jersey, 1977.
4. Andrew Walenstein: Toward Abstract Patterns of Usefulness, In: Proceedings of the 14th
Annual Conference on Design, Specification, and Verification of Interactive Systems
(DSVIS'2002), 2002, pp. 8 – 17.
5. Mark A. Neerincx, Jasper Lindenberg: Support Concepts for Web Navigation: A Cognitive
Engineering Approach, In Proceedings of the 10th International Conference on World Wide
Web, 2001, pp. 119 – 128.
6. F. Baader, D. Calvanese, D. McGuinness, D. Nardi, P.F. Patel-Schneider eds.: The
Description Logic Handbook: Theory, Implementation and Applications, Cambridge
University Press, UK 2003.
7. F. Noy, Deborah L. McGuinness: Ontology Development 101: A Guide to Creating Your
First Ontology, Stanford University 2002, available:
http://protege.stanford.edu/publications/ontology_development/ontology101.pdf.
8. J.R. Anderson: Cognitive Psychology and Its Implications, Fifth Edition, Worth Publishing,
New York 2001.
9. J. Top, H. Ackerman: Tasks and Ontologies in Engineering Modeling, In: International
Journal of Human-Computer Studies, Vol. 41, No. 4, 1994, pp. 585-617.
10. Bradley J. Rhodes: The Wearable Remembrance Agent: A system for Augmented
Memory, In: Personal Technologies Journal Special Issue on Wearable Computing, Personal
Technologies 1(4) 1997, pp. 218-224.
11. Bradley J. Rhodes: Margin Notes Building a Contextually Aware Associative Memory, In:
The Proceedings of the International Conference on Intelligent User Interfaces (IUI '00),
January 9-12, 2000.
12. B. Sarwar, G. Karypis, J. Konstan, and J.Riedl: Analysis of Recommendation Algorithms
for e-Commerce, In: Proceedings of ACM e-Commerce 2000 Conference, 2000.
153