iTree: Skill-building User-centered Clariﬁcation Consultation Interfaces

Martina Freiberg and Frank Puppe

Department of Artiﬁcial Intelligence and Applied Informatics, Institute of Computer Science

University of W

urzburg, Am Hubland, D-97074 W

urzburg, Germany

Keywords:

Knowledge-based Systems, Clariﬁcation Consultation, UI Design, Skill-building UI, Usability Evaluation.

Abstract:

Developing web-based, knowledge-based systems (wKBS) still challenges developers, mostly due to the in-

herent complexity of the overall task. The increased focus on knowledge-base development/evaluation and

consequent neglect of UI/interaction design and usability evaluation raises the need for a tailored wKBS de-

velopment tool, leveraging the overall task while speciﬁcally supporting the latter activities. As an example

for such a tool, we introduce the wKBS development tool ProKEt. With the help of that tool, we developed

the novel UI concept interactive clariﬁcation tree (iTree) with skill-building ability, that speciﬁcally is suitable

for clariﬁcation consultation systems. Also, we report a recent case study, where iTree was implemented for

knowledge-based clariﬁcation consultation in the legal domain.

1 INTRODUCTION

Despite increasing distribution in many domains,

web-based knowledge-based systems (wKBS) still

challenge developers: Development of appropriate

knowledge bases alone is an effortful task in terms of

time and money; thus, intentional UI/interaction de-

sign and usability evaluation activities remain rather

neglected. Yet, wKBS are often applied in critical or

specialized contexts—e.g., consultation in the medi-

cal or legal domain—where the chosen UI/interaction

style can contribute strongly to either the success or

the failure of the system. Thus, UI/interaction design

and usability evaluation should rather be a key factor

for wKBS development. This increases the need for

a tailored software tool that fosters experimentation

and evaluation of novel wKBS styles. We propose the

tailored software tool ProKEt, that supports efﬁcient

affordable,UI-/interaction design-focussed wKBS de-

velopment while at the same time seamlessly integrat-

ing usability evaluation functionality. To the best of

our knowledge there does not exist previous work to

date regarding similar tools.

Regarding consultation in contexts such as the le-

gal or medical domain, it often can be valuable to not

only have general consultation systems available that

derive one or more diagnoses based on the user input,

but additionally to have specialized clariﬁcation sys-

tems, for investigating only one distinct diagnosis—

potentially pre-selected by general consultation sys-

tems or by the users themselves. In this paper, we

introduce the interactive clariﬁcation tree (iTree) as

a novel, hierarchical clariﬁcation UI/interaction style

that we developed with the help of the tool ProKEt;

iTree thereby is particularly suitable for a mixed, di-

verse user population and additionally provides skill-

building ability. A ﬁrst study in the course of a current

project in the domain of legal consultation suggests

general beneﬁts of the proposed iTree UI style.

Related Work. With regards to general

KBS/wKBS development there exist various tailored

software tools—such as JavaDON (Tomic et al.,

2006), or KnowWE (Baumeister et al., 2011)—and

methodologies—e.g., MIKE (Angele et al., 1998), or

CommonKADS (Schreiber et al., 2001). However,

such approaches still mostly focus on the design

and evaluation of the knowledge base; in contrast,

we propose ProKEt as tailored wKBS development

tool that seamlessly couples efﬁcient, agile wKBS

development, creative experimentation regarding

KBS UI/interaction design, and semi-automated

usability evaluation activities. ProKEt can be further

seen as user-centered prototyping tool for wKBS—a

concept deﬁned by (Leichtenstern and Andr

e, 2010)

as an all-in-one tool solution for enabling efﬁcient,

effective and satisfactorily design, evaluation and

analysis of developed artifacts.

Probably due to the numerous beneﬁts of web-

based systems—e.g., availability, acceptance, or

maintainability—to date an increasing number of

knowledge-based/expert systems seems to be devel-

137

Freiberg M. and Puppe F..

iTree: Skill-building User-centered Clariﬁcation Consultation Interfaces.

DOI: 10.5220/0004093701370142

In Proceedings of the International Conference on Knowledge Engineering and Ontology Development (KEOD-2012), pages 137-142

ISBN: 978-989-8565-30-3

 2012 SCITEPRESS (Science and Technology Publications, Lda.)

oped for the web—i.e., integrated in websites or as

separate, complex web applications; recent examples

are (Patil et al., 2009) or (Rahimi et al., 2007). How-

ever, such wKBS apparently are being developed in a

rather ad hoc manner, not following systematic meth-

ods or processes, and not (re)using (neither provid-

ing) any patterns or best practices especially regard-

ing the UI/interaction design—probably due to a gen-

eral lack of scientiﬁc research in web-based expert

systems, cf. (Duan et al., 2005). Similarly, wKBS de-

velopers seem to be individuals, performing all tasks

required for developing and distributing a wKBS by

themselves. This further increases the need for a tai-

lored tool that not only renders overall wKBS devel-

opment an efﬁcient, pragmatic task, but equally im-

portant speciﬁcally supports design and experimen-

tation with web-based UI/interaction forms and their

usability evaluation.

Paper Structure. The rest of the paper is organized

as follows: In Section 2, we shortly introduce the tai-

lored wKBS development tool ProKEt. Afterwards,

we discuss iTree, a novel hierarchical UI concept for

knowledge-based clariﬁcation consultation systems

with skill-building ability in Section 3. We report on a

recent case study in Section 4, where the proposed UI

style was practically implemented for a wKBS in the

legal domain. We conclude with a short summary of

the presented research and an outlook to prospective

future work in Section 5.

2 ProKEt

ProKEt is a tailored, Prototyping and Knowledge

systems Engineering tool for web-based, knowledge-

based systems (wKBS), that additionally provides in-

tegrated support for various usability evaluation re-

lated activities. Thereby, ProKEt speciﬁcally sup-

ports web-based consultation and documentation sys-

tems, which can be developed equally well as (pure)

prototypical demo systems and as fully-ﬂedged sys-

tems for productive use. Thereby, extensible pro-

totyping is put into action, facilitating a nearly ef-

fortless transition from prototype to productive sys-

tem; for a more extensive introduction of the agile,

extensible prototyping and engineering process with

ProKEt, see (Freiberg et al., 2012). The main ap-

plication logic is implemented in Java. The result-

ing artifacts are Servlet-based web applications, us-

ing HTML, StringTemplate, and CSS for UI creation,

and JavaScript for interactivity. Regarding the knowl-

edge representation, an XML-based speciﬁcation is

used for the pure prototypes, which can be directly

cerated/edited with ProKEt itself. For productive sys-

tems, d3web (URL d3web, 2012) knowledge bases

are integrated and (mostly) replace the XML speci-

ﬁcation; the latter, however, can not directly be edited

with ProKEt, thus in that case an external d3web-

supporting tool such as KnowWE (Baumeister et al.,

2011) is required. Yet recently, we implemented a

mechanism to couple KnowWE and ProKEt artifacts,

thus drastically improving and easing the workﬂow of

UI/front-end development, knowledge base develop-

ment and their integration into a productive wKBS.

For supporting the straightforward evaluation of

its artifacts, ProKEt further allows for seamlessly in-

tegrating both qualitative and quantitative data col-

lection both for prototypes and productive wKBS;

this enables developers to assess the current develop-

ment state in a favorable way at any time by conduct-

ing manifold, potentially iterative, evaluations. For

qualitative data collection, ProKEt supports both the

integration of form-based questionnaires/surveys—

standards such as the SUS (Brooke, 1996) and the

NasaTLX (Hart, 2006) are supported out of the box,

but tailored own questionnaires can be added with no

effort—and of anytime feedback—mechanisms for

collecting free user feedback at any time during a

wKBS session. Regarding quantitative data, ProKEt

provides a tailored, mouse click and keyboard event

logging mechanism that records all relevant actions

during wKBS sessions. Based on that data, ProKEt

furthermore automatically can calculate a bunch of

known usability metrics—such as Success Rate, or

Average Task Time—proposed e.g. by (Constantine

and Lockwood, 1999), but it is equally well possi-

ble to just export qualitative and quantitative data into

a standard CSV format for further investigation with

external tools, e.g., standard spreadsheet calculation

or advanced statistical software. A more detailed in-

troduction of that usability extension of ProKEt can

be found in (Freiberg and Puppe, 2012).

3 iTree FOR CLARIFICATION

ProKEt particularly supports the development of

consultation- and documentation systems. A consul-

tation system thereby provides decision support in a

particular domain based on given user input, whereas

a documentation system contrastingly focusses on

supporting uniform, efﬁcient and high quality data en-

try. In this paper, we propose the interactive clariﬁca-

tion tree (iTree) UI style speciﬁcally for clariﬁcation

systems as a sub-class of consultation systems.

KEOD2012-InternationalConferenceonKnowledgeEngineeringandOntologyDevelopment

138

3.1 Clariﬁcation Consultation

As a subarea of classiﬁcation, clariﬁcation relates

to hypothesize-and-test as follows: Separate, gen-

eral multiplex consultation systems can be applied

ﬁrst for narrowing the complete set of potential di-

agnoses/hypotheses down to one or several most suit-

able elements (hypothesize step); each of those hy-

potheses can then be further investigated by a corre-

sponding clariﬁcation module (test step). As shortcut,

users could alternatively start directly with a clariﬁca-

tion system for a chosen hypothesis themselves.

3.2 iTree: Skill-building Clariﬁcation

We propose iTree as a novel UI with skill-building

ability that fosters an efﬁcient and usable user experi-

ence in the context of clariﬁcation systems.

Y N ? X

Core Issue to Clarify…

Y N ? X

Question 1 (Rating Core Issue)

Y N ? X

Question 2 (Rating Core Issue)

Y N ? X

Question 2.1 (Rating Question 2)

Y N ? X

Question 2.2 (Rating Question 2)

Y N ? X

Question 3 (Rating Core Issue)

Figure 1: Schematic drawing of the iTree UI style.

Figure 1 presents a schematic drawing of iTree

for clariﬁcation systems. The core issue to be rated

is presented as root element of the hierarchical tree

structure (Figure 1, Core Issue). Its rating is de-

rived from the ratings of any desired number of top-

level questions, placed directly underneath the core is-

sue (Figure 1, Quest.1, Quest.2, Quest.3). Questions

are a tailored form of yes/no questions with an ad-

ditional value Neutral/Uncertain; provided answers

further can be withdrawn/adapted at any time, indi-

cated by the X button. The current implementation—

a practical example of which is depicted in Fig-

ure 2—allows three possible abstract ratings for the

core issue as well as for all questions: Conﬁrmed,

uncertain/neutral and rejected, which correspond to

the answers Yes, ?, and No per default. Some do-

mains may require to swap that mapping for particu-

lar questions in favor of a more understandable ques-

tion wording. Figure 2 depicts an example: The core

issue is conﬁrmed (rating: yes) if the cancellation

was NOT prohibited due to time limitations; in that

case, the swapped yes/no mapping allows for reword-

ing the question as depicted, which is much clearer

than its negated alternative. In case the user can-

not answer a question directly, more detailed reﬁne-

ment questions—if available—can be retrieved for the

current element, represented by the D button in the

scheme and by the arrow in Figure 2; as example, the

second top-level question in Figure 2 contains two re-

ﬁning questions, which list in more detail the condi-

tions which conﬁrm/reject its parent question. Ques-

tion ratings are always propagated from inner levels

of the hierarchy up to the topmost question(s) by ei-

ther AND or OR connections. Let pn be a parent node

and cn

a child node of pn; for calculating the rating

of pn, the following rules apply:

AND nodes:

IF ∃cn

= no THEN pn = no

IF ∀cn

= yes THEN pn = yes

IF ∀cn

= rated AND ∃ cn

= neutral AND @cn

= no

T HEN pn = neutral

IF ∃cn

= unknown AND ∃ cn

= neutral AND@cn

= no

T HEN pn = unknown

OR nodes:

IF ∃cn

= yes T HEN pn = yes

IF ∀cn

= no T HEN pn = no

IF ∀cn

= rated AND ∃ cn

= neutral AND @cn

= yes

T HEN pn = neutral

IF ∃cn

= unknown AND ∃cn

= neutral AND @cn

= yes

T HEN pn = unknown

This means, e.g., that the core issue in Figure 2

is rated yes only if all of its children are rated yes, as

those are connected to the core issue by AND (second

rule above); likewise, cancellation prohibited due to

time limitation is rated yes as soon as one of its chil-

dren is rated yes (ﬁfth rule above) due to the OR con-

nection. One advantage of iTree is the suitability for

Was the working contract terminated effectually?

Core Issue

Is the cancellation prohibited due to time limitation?

Is the cancellation allowed despite time limitation?

Was the working contract a fixed-term contract?

Is the cancellation formally passable?

Yes No

Yes

Yes No

Yes

-?-

-?- X

XIf

And

Figure 2: Exemplary iTree Implementation.

a diverse user population—i.e., users with different

background and expertise might be able to proﬁt from

the same system. This is achieved by the possibility

to derive the solution rating both by answering more

abstract top-level questions (domain specialist level)

or by stepping into more reﬁned, elaborate questions

(less expertise required) in iTree. By the visual repre-

sentation of the knowledge base structure, moreover

a form of focus-and-context view is created: Not only

the currently active/processed question(s) are visible,

but also surrounding elements are indicated—limited

only by the display size. As the user thus can visually

trace the result of an answer by the distinct presen-

tation of the questions and their current state, that is

propagated all throughout the tree, the core issue rat-

ing becomes more transparent. The chosen visual rep-

resentation of the knowledge further supports users in

iTree:Skill-buildingUser-centeredClarificationConsultationInterfaces

139

AND

Core Issue

reject

confirm

neutral

(C)

(D)

(F)

(A)

(B)

reject

confirm

neutral

(G)

Details

Is the dismissal formally legal?

Is the dismissal legally correct regarding the contents?

Is the dismissal not prohibited due to timely limitations?

Is the dismissal not prohibited due to special laws?

Was the statutory period of notice adhered to?

(E)

(G)

reject

confirm

neutral

(G)

Details

Is the dismissal formally legal?

Is the dismissal legally correct regarding the contents?

Is the dismissal not prohibited due to timely limitations?

Is the dismissal not prohibited due to special laws?

Was the statutory period of notice adhered to?

Figure 3: JuriSearch clariﬁcation module as iTree (large) and oneQ UI (small). AND/OR rules for rating the (sub-)questions

(A) are visually represented; reversed question example (underneath B); dummy node example, only serving for rule connec-

tion modeling (above B); four simple buttons (B) for rating the questions (Y:yes—N:no—?:neutral—X/empty:retract), rating

highlighted by background color (C); core issue rating prominently displayed and updated continuosly (D); additional infor-

mation displayed in separate panel when mouse-overing question (E); anytime feedback/data collection features integrated

in UI header (F); clariﬁcation core component in oneQ style (G) always displays current active question with additional-

information panel, previously answered questions remain presented in a more condensed view.

gaining a thorough understanding of the investigated

core issue and the coherences between its clarifying

questions and the core issue itself. Thus, users acquire

additional knowledge by means of the system, yet are

also enabled to bring in their existing knowledge for

potentially shortening the clariﬁcation session or for

focussing on only those parts in detail that are rather

unclear. Together with optional, auxiliary informa-

tion that can be integrated for each of the elements

(not contained in the scheme—see e.g. the auxiliary

information panel in Figure 3, E), iTree speciﬁcally

can serve as a skill-building UI type.

4 CASE STUDY—JuriSearch

At the beginning of 2012, the JuriSearch project

was initiated as a cooperation between the univer-

sity of W

urzburg and the RenoStar corporation (partly

founded by the Free State of Bavaria). JuriSearch

aims at building a wKBS for the legal domain: The

target system is intended to integrate both a stan-

dard consultation (entrance) module—hypothesize—

and various clariﬁcation modules for each potential

core issue—test—as to provide encompassing advice

on various legal topics, such as the right of cance-

lation or the law of tenantry. Potential target users

are diverse, ranging from legal laymen—searching

for a basic understanding/estimation of their case to

(fresh) lawyers seeking for guidance regarding legal

(sub)domain(s) that are not exactly their special ﬁeld

of work. Those framing conditions provided a per-

fect opportunity to implement and evaluate the iTree

UI style. Therefore, a comparative study with iTree

and a more common, conversational UI style was con-

ducted; the latter was implemented as a one-question

UI style (oneQ). In contrast to the free, explorative

interaction with iTree, oneQ is based on the metaphor

of a conversation: The system always presents only

the one suitable next question at a time, thus imitat-

ing a strict dialogue between a user and the system.

Yet, both UI types are based on the exact same knowl-

edge base with their core difference being the presen-

tation of the questions: Hierarchical tree (iTree) vs.

single question (oneQ). Reﬁnement questions are as

well available in oneQ; yet there, the former current

question is folded and the ﬁrst of the reﬁnement level

questions is presented, thereby destroying much of

KEOD2012-InternationalConferenceonKnowledgeEngineeringandOntologyDevelopment

140

the ’contextual knowledge’ that iTree facilitates by al-

ways presenting all questions of the current hierarchy

level in addition to the surrounding, further structure

(limited by display size only).

Figure 3 presents the iTree implementation in Ju-

riSearch (A-F) as well as the alternative, conversa-

tional oneQ UI (G).

4.1 User Study—Framing Conditions

21 members from our department—all male, mostly

between 25 and 35 years—participated in the ﬁrst

study; as computer scientists, they all were versed in

general computer and web system usage, yet in most

cases had little to no experience regarding the spe-

ciﬁc wKBS types, and no experience regarding the

target domain at all. Two exemplary problem descrip-

tions from the domain of cancellation were created,

and participants were asked to solve one problem with

iTree and the other with oneQ; to avoid biased re-

sults due to the sequence of using the UIs, that se-

quence was altered between participants. The study

was conducted remotely: The test systems were de-

ployed on a speciﬁcally conﬁgured server—enabling

the integrated logging and feedback/questionnaire

mechanisms—and the participants were given all re-

quired instruction material per email.

4.2 User Study—Results & Discussion

The collected log-data revealed a general applicability

of the iTree concept for implementing a clariﬁcation

wKBS UI in the legal domain and speciﬁcally the fol-

lowing results: First, iTree exhibited an average task

time of 13m 38s±6m 49s in contrast to 10m 39s±5m

49s for oneQ (by a narrow margin statistically not

signiﬁcant on a one-sided unpaired t-test, p=0,068).

The higher task time of iTree could possibly be ex-

plained by its ability to provide intuitively for free,

extensive exploration of the system. Yet on the other

hand, task time should not be overrated at all, here;

the extent of usage of the test systems depended in

larger parts on a) the reading speed of the participants

regarding the questions and explanations, b) the us-

age conditions (during daily job routine vs. after end

of work) which, due to the setting of the remote study

could not be controlled strictly, and c) the potentially

already existing knowledge regarding the problem at

hand, in turn leading to highly subjective task time re-

sults between users. Regarding the success- and error

rate, a case was classiﬁed successful, if the correct

rating of the core issue was derived by the user with

the respective system type, and not successful if ei-

ther the wrong or no solution was found. For iTree,

the success rate was 42,86% and 38,1% for oneQ

(both: no statistical signiﬁcance on a one-sided bi-

nomial test with p=0,11 and p=0,16); along with sub-

jective user feedback, this clearly indicates the need

to rework the knowledge base contents/structure for

yielding better results. Furthermore, both anytime-

and questionnaire-based feedback were collected as

qualitative user data. The ﬁrst remarkable ﬁnding was

the fact, that iTree nearly concordantly was perceived

more intuitively usable, and that it thus further was

reported to be the preferred UI type by 81% of the

study participants, whereas oneQ only was preferred

by 14% and no preference was stated by 5%; this is

statistically signiﬁcant on a χ

test with p<0,05 and

with an anticipated distribution of 50% (iTree), 30%

(oneQ), and 20% (both equally). One possible expla-

nation might be the speciﬁc characteristics of the par-

ticipant population, that—as computer scientists—

might simply be used to tree representations and thus

perceived iTree as naturally more intuitive to use. Re-

garding further subjective (questionnaire) topics iTree

scored better all over; on a scale from 0 (worst) to

6 (best) the results were: Comprehensibility of the

system reactions 4.43±1.54 (iTree) vs. 2.76±1.45

(oneQ) or of the derived results 4.53±1.54 (iTree)

vs. 3.33±1.85 (oneQ), and the mediation of do-

main knowledge to the user 4.05±1.32 (iTree) vs.

2.95±1.72 (oneQ); those differences are all statisti-

cally signiﬁcant using an unpaired, one-sided t-test

with p≤0,05. Especially the latter value afﬁrmed

our assumption that iTree particularly evinces skill-

building abilities. Additional insights from anytime

feedback included: The wording of the questions

was perceived as incomprehensible/cumbersome in

11 cases (52%) due to often used duplicate negations

and legal specialist language, probably further aggra-

vated by the fact that the chosen participants were le-

gal laymen and thus not at all familiar with legal terms

and language; also, the hierarchical structure and rep-

resentation of the knowledge base—that followed the

legal subsumption logic—was perceived unfavorable.

In such a hierarchy/sequence, the questions most in-

teresting for legal laymen appear far down while at

the same time more abstract concepts are contained

at the upper levels; this led to (laymen) users hav-

ing difﬁculties to make sense of the concepts at the

top/beginning of the hierarchy/questioning sequence.

A solution to this issue might be a complete restruc-

turing of the knowledge base, so that the most rele-

vant questions and distinctions—from the users’ point

of view, e.g., typical reasons for dismissal, size of

company, etc.—also appear on rather top levels, sure

posing a difﬁcult trade off between legal correctness/-

schematic thinking and understandability; yet it ap-

iTree:Skill-buildingUser-centeredClarificationConsultationInterfaces

141

parently could greatly contribute to tailoring the UI

to the users in enabling them to bring in their own

perspective and knowledge in the dialog. Thus a fur-

ther reﬁnement of the knowledge base with regards to

a clear, easily understandable language and structure

turned out indispensable. Another interesting ﬁnding

was the fact, that in 4 (19%) cases, the real mean-

ing of the -?- button as an answer alternative was not

grasped; users rather expected the system to display

more elaborate explanations on the issue at hand or to

open up the next reﬁnement level of the questions in-

stead of receiving just a rating of the current question.

Similarly, the X/empty button—designated to clearing

a previously entered answer—was not intuitively un-

derstood in 3 (14%) cases.

5 CONCLUSIONS

In this paper, we claimed the importance of a care-

ful UI/interaction design for web-based, knowledge-

based systems. Regarding the consultation systems’

sub-class clariﬁcation systems, we suggested iTree as

novel UI/interaction style for increased efﬁciency and

usability. In a ﬁrst comparative user study from the

legal domain, an initial iTree prototype as well as

an alternative, one-question style prototype were im-

plemented using the prototyping and knowledge sys-

tems engineering tool ProKEt. The results suggest,

that iTree generally is a favorable UI style for clari-

ﬁcation systems, that supports free, explorative sys-

tem usage and thus provides skill-building potential

on the side of the users. Yet, the study also showed

the need to rework the knowledge base of the system,

regarding both the question wording as well as their

structuring. One assumption requiring further studies

is, that the legal iTree at its current state is satisfy-

ing for legal experts, whereas a restructured system

could be more appropriate for non-expert users. Ad-

ditionally, we plan on developing and evaluating sim-

ilar iTree systems for the medical domain. This raises

the requirement of more ﬁne-granular rating options,

e.g., by scoring rules. Finally, further experimentation

with potential UI enhancements is intended to help

improve the iTree concept; one such idea is the inte-

gration of an interactive system state preview that is

overlaid when mouse-overing the respective answer

option.

ACKNOWLEDGEMENTS

We thank the RenoStar (Großwallstadt, GER) corpo-

ration for valuable discussions and cooperation.

REFERENCES

Angele, J., Fensel, D., Landes, D., and Studer, R. (1998).

Developing Knowledge-Based Systems with MIKE.

Automated Software Engineering: An International

Journal, 5(4):389–418.

Baumeister, J., Reutelshoefer, J., and Puppe, F. (2011).

KnowWE: A Semantic Wiki for Knowledge Engineer-

ing. Applied Intelligence, 35(3):323–344.

Brooke, J. (1996). SUS: A quick and dirty usability scale.

In Jordan, P. W., Weerdmeester, B., Thomas, A., and

Mclelland, I. L., editors, Usability evaluation in in-

dustry. Taylor and Francis, London.

Constantine, L. L. and Lockwood, L. A. D. (1999). Soft-

ware for Use: A Practical Guide to the Models and

Methods of Usage-Centered Design. Addison-Wesley

Professional.

Duan, Y., Edwards, J. S., and Xu, M. X. (2005). Web-based

expert systems: beneﬁts and challenges. Information

& Management, 42:799–811.

Freiberg, M. and Puppe, F. (2012). Prototyping-based

Usability-oriented Engineering of Knowledge-based

Systems. In Proceedings of Mensch und Computer

2012 (to appear).

Freiberg, M., Strifﬂer, A., and Puppe, F. (2012). Extensible

prototyping for pragmatic engineering of knowledge-

based systems. Expert Systems with Applications,

39(11):10177 – 10190.

Hart, S. G. (2006). Nasa-Task Load Index (Nasa-TLX);

20 Years Later. In Human Factors and Ergonomics

Society Annual Meeting, volume 50.

Leichtenstern, K. and Andr

e, E. (2010). MoPeDT: features

and evaluation of a user-centred prototyping tool. In

Proceedings of the 2nd ACM SIGCHI Symposium

on Engineering Interactive Computing Systems, EICS

’10, pages 93–102, New York, NY, USA. ACM.

Patil, S. S., Dhandra, B. V., Angadi, U. B., Shankar, A. G.,

and Joshi, N. (2009). Web based expert system for di-

agnosis of micro nutrients’ deﬁciencies in crops. In

Proceedings of the World Congress on Engineering

and Computer Science, volume 1, pages 20–22.

Rahimi, S., Gandy, L., and Mogharreban, N. (2007).

A web-based high-performance multicriteria decision

support system for medical diagnosis: Research ar-

ticles. International Journal of Intelligent Systems,

22:1083–1099.

Schreiber, G., Akkermans, H., Anjewierden, A., de Hoog,

R., Shadbolt, N., de Velde, W. V., and Wielinga, B.

(2001). Knowledge Engineering and Management -

The CommonKADS Methodology. MIT Press, 2 edi-

tion.

Tomic, B., Jovanovic, J., and Devedzic, V. (2006).

JavaDON: an open-source expert system shell. Expert

Systems with Applications, 31(3):595 – 606.

URL d3web (2012). http://d3web.sourceforge.net/ (last

checked Apr. 15th, 2012).

KEOD2012-InternationalConferenceonKnowledgeEngineeringandOntologyDevelopment

142