Authors:
Sven Hertling
1
;
Markus Schröder
1
;
Christian Jilek
1
and
Andreas Dengel
2
Affiliations:
1
German Research Center for Artificial Intelligence (DFKI) GmbH, Germany
;
2
German Research Center for Artificial Intelligence (DFKI) GmbH and Knowledge-Based Systems Group, Germany
Keyword(s):
Information Retrieval, GUI Automation, Accessibility Interface.
Related
Ontology
Subjects/Areas/Topics:
Applications
;
Artificial Intelligence
;
Computational Intelligence
;
Enterprise Information Systems
;
Evolutionary Computing
;
Human-Computer Interaction
;
Intelligent User Interfaces
;
Knowledge Discovery and Information Retrieval
;
Knowledge Engineering and Ontology Development
;
Knowledge-Based Systems
;
Machine Learning
;
Natural Language Processing
;
Pattern Recognition
;
Soft Computing
;
Symbolic Systems
Abstract:
In feature-rich software a wide range of functionality is spread across various menus, dialog windows, toolbars etc. Remembering where to find each feature is usually very hard, especially if it is not regularly used.
We therefore provide a GUI search engine which is universally applicable to a large number of applications. Besides giving an overview of related approaches, we describe three major problems we had to solve, which
are analyzing the GUI, understanding the users’ query and executing a suitable solution to find a desired UI element. Based on a user study we evaluated our approach and showed that it is particularly useful if a not
regularly used feature is searched for. We already identified much potential for further applications based on our approach.