STUDY FOR ESTABLISHING DESIGN GUIDELINES FOR

MANUALS USING AUGMENTED REALITY TECHNOLOGY

Verification and Expansion of the Basic Model Describing “Effective Complexity”

Miwa Nakanishi

Facuty of Engineering, Chiba University, 1-33 Yayoi, Inage, Chiba, Japan

Shun-ichro Tamamushi and Yusaku Okada

Faculty of Science and Technology, Keio University, 3-14-1, Hiyoshi, Kohoku, Yokohama, Japan

Keywords: Augmented Reality, Information design, Mathematical modeling, Task performance.

Abstract: Augmented reality (AR), a technology that enables users to see an overlay of digital information on the real

view, is expected to be applied more and more to human factor innovation. It has been suggested that a

manual using AR (AR manual) improves accuracy and efficiency in actual work situations. To make an AR

manual practical, hardware such as see-through display or retinal scanning display has been actively

developed. However, software, i.e., information provided by the AR manual, has not been sufficiently

examined. In a recent study, the authors built a mathematical model that describes the “effective

complexity” of an AR manual according to the complexity of the real view. In this study, the basic model is

verified by applying it to the AR manual for a realistic task. Furthermore, the applicability of the basic

model is examined by assuming two different situations where either accuracy or efficiency has high

priority. The objective of this study is to establish rough but practical guidelines for designing an AR

manual.

1 INTRODUCTION

In recent years, the applicability of augmented

reality (AR), a technology that enables

superimposition of the real view and digital

information (Wellne et al., 1993) to manuals used in

actual work situations, has been discussed. For

example, when a manual is available to a worker

through a see-through head-mounted display

(HMD), he/she can see it superimposed on his/her

real view. Such AR manuals are considered to

reduce human errors and enhance task efficiency,

because they allow workers to easily compare a real

object with related information (Azuma,

1997)(Feiner, 2002). Moreover, as HMD technology

is improving rapidly, the hardware for AR manuals

has become almost ready for practical use. However,

the software has not yet reached this stage, because

requirements for designing information provided by

HMDs are not sufficiently clear. Thus, it is

necessary to establish guidelines for designing AR

manuals, which will differ from those for paper-

based manuals.

In a previous study (Nakanishi et al., 2008), the

authors examined how workers’ performance

changed depending on the layout of information

given by an AR manual through an experiment in

which real-world conditions were generated by a

computer program and presented on a monitor. From

the results, the authors built a model that provided

the most effective design of AR manuals according

to the real-world conditions (described more

specifically in the next section). We have positioned

that model as the basic model, which will be

fundamental to the guidelines for designing AR

manuals.

As the next step, we verify and expand the basic

model through another experiment in which a task

was performed not under computer-generated

conditions, but under real conditions. The objective

of this study is to analyze the relationship between

the design of AR manuals and task performance in

Nakanishi M., Tamamushi S. and Okada Y. (2009).

STUDY FOR ESTABLISHING DESIGN GUIDELINES FOR MANUALS USING AUGMENTED REALITY TECHNOLOGY - Veriﬁcation and Expansion of

the Basic Model Describing “Effective Complexity”.

In Proceedings of the 11th International Conference on Enterprise Information Systems - Human-Computer Interaction, pages 21-26

DOI: 10.5220/0001856300210026

 SciTePress

conditions similar to actual work situations, and

suggest how to expand the basic model to practical

guidelines for designing AR manuals.

2 KEY IDEA OF THE BASIC

MODEL

In general, the real-world conditions in front of a

worker’s eyes cannot be controlled. Thus, we

attempt to clarify how AR manuals should be

designed when a condition of the real view is given.

For example, when excess information is

overlaid on an object in the real view, visibility may

decrease. On the other hand, when information is

overlaid on an object unit by unit, the worker is

required to switch the overlay repeatedly in a task

sequence.

In our previous study (Nakanishi and Okada,

2006), we performed an experiment in which

different real-world conditions were virtually

generated. We found that when the real view was

relatively uncomplicated, task performance was

affected more by switching images of the AR

manual than by seeing excess information.

Conversely, when the real view was complicated,

task performance was affected more by seeing

excess information than by switching images of the

AR manual. When these two factors were balanced,

task performance was the highest. Based on these

results, we built a model that describes the most

effective design of AR manuals according to real-

world conditions, as follows.

First, we considered two aspects of task

performance, “accuracy” (lack of errors) and

“efficiency” (speed). Assuming that both accuracy

and efficiency were equally necessary and important,

we defined “damage to task performance” (DP)

using both error rates (E) and unit operation time (T).

)T(S5.0)E(S5.0DP +=

(1)

S(E): Standardized E

S(T): Standardized T

Second, we quantified the conditions of visual

information based on the idea of “complexity.” In

general, the more crowded the items are, the more

complex the information looks. Thus, we defined

complexity (C) as the number of items to be

attended to (n) divided by their dispersion (M). M

was defined as the standard deviation of the distance

from each item to the center of them (d

: i = 1, 2,...,

n) divided by the mean of the distances (

), so that

C did not depend on measurement of d

∑

−−==

)1n/()dd(/dnM/nC

(2)

We examined the data obtained from the

experiment and found that the relationship between

the complexity of the real view (C

), complexity of

the AR manual (C

), and DP could be expressed by

the following equation.

4131

(2.57 10 8.71 10 )(6.63 10 8.76 10 )

1.00 10

(1 (2.57 10 8.71 10 )( ) 3.00)

6.22

−

−− −

×+× ×−×

−−

+− × + × −

(3)

Moreover, we suggested that when C

was

given, C

that minimized

could be determined

by the following equation.

min.

22.6

))1071.8C1057.2(1063.6

))1071.8C1057.2(1(1000.1

−

×+××

×+×−×

−−−

−−+

(4)

Equation (4) provides the “effective complexity”

of AR manuals according to the complexity of the

real view. Accordingly, it can be regarded as the

basic model, which describes effective design of AR

manuals using the number and dispersion of

information items.

3 METHOD

In this study, we applied an AR manual that was

designed according to the basic model, not to a

computer-generated task but to a realistic task, and

examined the practicability of the model by

evaluating task performance.

3.1 Experimental Task

A wiring task, in which a subject plugged many

lines into a panel, was chosen as the experimental

task, because it has been proven that task

performance improved when subjects used an AR

manual compared to when they used a paper-based

manual in the same task.

The fixed panel (280 mm × 300 mm) included

randomly arranged holes (r = 3.5 mm) to be plugged.

Figure 1 shows an example pattern of the panel. An

HMD displayed the AR manual corresponding to

each pattern. Figure 2 shows an image of the AR

manual. The AR manual indicated which colored

lines should be plugged into which holes on the

ICEIS 2009 - International Conference on Enterprise Information Systems

Figure 1: Example of panel pattern.

Figure 2: Image of AR manual. Figure 3: Superimposition. Figure 4: HMD.

Figure 5: Experimental environment.

panel. (Y: yellow, G: green, R: red, W: white, B:

blue).

A subject wearing the HMD faced the panel and

performed the task. The frames were drawn on both

the panel and the AR manual, so that he/she could

see them superimposed by adjusting his/her own

position and angle according to it (see Figures 3–5).

The subject’s task was to plug lines into all the

holes on the panel according to the AR manual. The

subjects were required to complete the task correctly

and quickly. Even if they recognized their own

errors, they were not allowed to correct them. A

subject started the task when the AR manual was

displayed on the HMD, and finished when all of the

holes on the panel were plugged.

3.2 Experimental Conditions

To set different conditions for the real view, five

panel patterns were prepared (Figures 6-1–6-5). The

AR manual was displayed in the following three

ways: “one-by-one indication” (each hole was

indicated one by one), “all-once indication” (all

holes were indicated at once), and “model-based

indication” (which holes were indicated at once was

determined according to the basic model).

Specifically, model-based indication was given as

follows. First, substituting each value of complexity

for each panel pattern (C

= {44.8, 96.3, 174.6,

271.4, 391.2}) in equation (4), the value of the

effective complexity of the AR manual (C

n = 24,

Figure 6-1: Pattern of panel (CR

= 44.8, n = 24).

Figure 6-2: Pattern of panel

(CR = 96.3, n = 40).

Figure 6-3: Pattern of panel

(CR = 174.6, n = 70).

Figure 6-4: Pattern of panel

(CR = 271.4, n=100)

Figure 6-5: Pattern of panel (CR

= 391.2, n=144).

Figure 7-1: One-by-one

indication (CR = 174.6).

Figure 7-2: Model-based

indication (CR = 174.6).

Figure 7-3: All-once

indication (CR = 174.6).

R W G

R B

Y W

G Y

R R

R W G

R B

Y W

G Y

R R



750mm

Subject

Panel

Line

45 deg.

  

R W G

R B

Y W

G Y

R R

WRY

GB B WR Y G

BWYG

RYWBGR

YWBGY

BRGWR

BW GY GY B WR

GY RB

WW R B G Y B

GW Y B R

WYBGGBY

BWYR

GBYG

STUDY FOR ESTABLISHING DESIGN GUIDELINES FOR MANUALS USING AUGMENTED REALITY

TECHNOLOGY - Verification and Expansion of the Basic Model Describing "Effective Complexity"

Table 1: Complexity of panels, effective complexity of AR manuals, and frequency of switching indication.

Number of holes Complexity of Panels Efective Complexity Frequency of Switching Indication

(n) (C

) (C

) (one-by-one) (model-based) (all-once)

24 44.8 33.3 23 2 0

40 93.6 31.3 39 3 0

70 174.6 27.9 69 5 0

100 271.4 22.8 99 8 0

144 391.2 14.2 143 14 0

follows. First, substituting each value of complexity

for each panel pattern (C

= {44.8, 96.3, 174.6,

271.4, 391.2}) in equation (4), the value of the

effective complexity of the AR manual (C

n = 24,

40, 70, 100, 144) was given for each panel pattern.

Second, the dispersion of the holes on each panel

pattern (M

n = 24, 40, 70, 100, 144) was given as

described in section 2. Third, substituting each M

equation (2), the number of holes to be

approximately indicated at once was determined for

each panel pattern. Table 1 shows the values of C

the “effective complexity” of the AR manual,

corresponding to each panel pattern. Moreover,

Figures 7-1–7-3 show example images of the one-

by-one, all-once, and model-based indication

corresponding to one of the panel patterns.

In this experiment, 15 conditions (5 real-world

conditions * 3 conditions of the AR manual) were

tested.

In the case of one-by-one or model-based

indication, the subject had to switch the image from

one hole to the next or from one part of the panel to

the next with a handy button. Table 1 shows how

many times the indication image was required to be

switched in each condition.

3.3 Experimental Settings

Eighteen students (age 21 to 25 years) with good

vision participated in the experiment. After they

repeated the procedure of the task in each condition

for training, they performed the task once in each

condition for data recording. Then the order of the

three indication patterns of the AR manual was

alternated within each condition of the real view.

The HMD was a retinal scanning device (NOMAD,

made by Microvision, Inc.). The transparency to the

real view was almost 100%. The image of the AR

manual was drawn in monochrome red.

During the task, subjects’ actions were recorded

with a digital video camera, and the time taken for

the task was automatically recorded. Moreover, after

the task, the panel with lines was compared with the

AR manual, and errors were noted.

4 RESULTS

To examine the task performance for each condition,

in particular, the condition in which model-based

indications were provided by the AR manual, we

analyzed the data in terms of accuracy and

efficiency.

4.1 Error Rate

The following errors were observed: omitting

plugging, plugging wrong-colored lines, and

plugging to wrong holes. The solid lines in Figures

8-1–8-5 show the error rates for each condition of

the real view (C

= {44.8, 96.3, 174.6, 271.4,

391.2}). In each chart, the left vertical axis is scaled

individually, in order to focus on how the error rates

changed according to the indication pattern of the

AR manual under the given condition of the real

view.

The error rate was high in the case of all-once

indication for any condition of the real view.

However, it tended to be low in the case of one-by-

one indication for most real-world conditions. In

addition, it was not always that the error rate became

low in the case of model-based indication.

4.2 Unit Operation Time

Plugging a line into a hole was defined as the unit

operation. The dotted lines in Figures 8-1–8-5 show

the unit operation time in each condition of the real

view (C

= {44.8, 96.3, 174.6, 271.4, 391.2}). In

each chart, the right vertical axis is scaled

individually, for the same reason as above.

The unit operation time tended to be short in the

case of model-based indication, however, it was

longer in the case of one-by-one indication that

required subjects to switch the image of the AR

manual, than in the other cases, in particular, when

the complexity of the real view was comparatively

low.

ICEIS 2009 - International Conference on Enterprise Information Systems

Figure 8-1: Error rate per

task and unit operation

time (CR=44.8, n=24).

Figure 8-2: Error rate per

task and unit operation

time (CR=96.3, n=40).

Figure 8-3: Error rate per

task and unit operation

time (CR=174.6, n=70).

Figure 8-4: Error rate per

task and unit operation

time (CR=271.4, n=100).

Figure 8-5: Error rate per task and unit operation time

(CR=391.2, n=144).

* In Figures 8-1 to 8-5, the left-hand axis corresponds to

error rate (%), expressed by the solid line, and the right-

hand axis corresponds to unit operation time (s), expressed

by the dotted line.

5 VERIFICATION & EXPANSION

OF THE BASIC MODEL

As described in section 2, the basic model was built

under the assumption that accuracy and efficiency

are equally significant for evaluating task

performance. However, in actual situations, there are

cases in which workers absolutely should not make

errors even if it takes time to do so, and cases in

which they have to complete a task within a limited

time, wherein a few errors are permitted.

Figure 9-1: Damage to

task performance

(CR=44.8, n=24).

Figure 9-2: Damage to tas

erformance (CR=96.3,

n=40).

Figure 9-3: Damage to

task performance

(CR=174.6, n=70).

Figure 9-4: Damage to tas

erformance (CR=271.4,

n=100).

Figure 9-5: Damage to task performance (CR=391.2,

n=144).

** In Figure 9-1 to 9-5, the solid line shows DP(w = 0.5),

the dotted line shows DP(w = 0.2), and the dashed line

shows DP(w = 0.8). The horizontal axis indicates the

complexity of the AR manual. Accordingly, in any chart,

the middle plot indicates the value of DP in the case of

model-based indication.

In this section, first the applicability of the basic

model is checked under the assumption that

accuracy and efficiency are equally weighted.

Second, the applicability of the basic model is

discussed under the assumption that either accuracy

or efficiency is more heavily weighted.

5.1 Verification of the Basic Model

According to the basic model, DP can be calculated

for each condition by substituting E and T data in

3.0

3.5

4.0

0.000

0.005

0.010

0.015

one‐by‐one model‐based all‐once

Errorrate(%)

Unitoperationtime(sec)

3.5

4.0

4.5

0.000

0.005

0.010

one‐by‐one model‐based all‐once

Errorrate(%)

Unitoperationtime(sec)

4.0

4.5

0.000

0.005

0.010

one‐by‐one model‐based all‐once

Err orrate(%)

Unitoperationtime(sec)

4.0

4.1

4.2

0.000

0.005

0.010

one‐by‐one model‐based all‐once

Errorrate(%)

Unitoperationtime(sec)

4.0

4.5

5.0

0.000

0.005

0.010

one‐by‐one model‐based all‐once

Err orrate(%)

Unitoperationtime(sec)

‐1.0

‐0.5

0.0

0.5

1.0

0 1020304050

DP(0.5) DP(0.2) DP(0.8)

‐1.0

‐0.5

0.0

0.5

1.0

050100

DP(0.5) DP(0.2) DP(0.8)

‐1.0

‐0.5

0.0

0.5

1.0

0100200

DP(0.5) DP(0.2) DP(0.8)

‐1.0

‐0.5

0.0

0.5

1.0

0 100 200 300

DP(0.5) DP(0.2) DP(0.8)

‐1.0

‐0.5

0.0

0.5

1.0

0 100 200 300 400

DP(0.5) DP(0.2) DP(0.8)

STUDY FOR ESTABLISHING DESIGN GUIDELINES FOR MANUALS USING AUGMENTED REALITY

TECHNOLOGY - Verification and Expansion of the Basic Model Describing "Effective Complexity"

equation (1) (see section 2). The solid lines in

Figures 9-1–9-5 show the values of DP in this case.

DP is minimized in the case of model-based

indication for any condition of the real view,

indicating that the AR manual designed according to

the basic model enhances task performance. This

suggests that the basic model describes effective

design of an AR manual not under computer-

generated conditions, but also under the real

conditions, if it is assumed that accuracy and

efficiency are equally important for the situation.

5.2 Expanding the Applicability of the

Basic Model

Assuming cases where either accuracy or efficiency

is more heavily weighted, damage to performance

(DP (w)) is redefined as follows.

)T(S)w1()E(wS)w(DP −+=

(5)

In the discussion below, two different cases are

simulated.

Case 1) Efficiency is weighted more heavily than

accuracy (w = 0.2).

)T(S8.0)E(S2.0)w(DP +=

(6)

Substituting the data of E and T in equation (6),

DP(0.2) in each condition of the real view (C

{44.8, 96.3, 174.6, 271.4, 391.2}) is calculated, as

shown by the dotted lines in Figures 9-1–9-5.

Like DP(0.5), which is expressed by the solid

line, DP(0.2) is lowest in the case of model-based

indication for any condition of the real view. This

indicates that the basic model is also applicable to

cases where efficiency has higher priority than

accuracy.

Case 2) Accuracy is weighted more heavily than

efficiency (w = 0.8).

)T(S2.0)E(S8.0)w(DP +=

(7)

Substituting the data of E and T in equation (7),

DP(0.8) for each condition of the real view (C

{44.8, 96.3, 174.6, 271.4, 391.2}) is calculated, as

shown by the dashed lines in Figures 9-1–9-5.

In some real-world conditions (C

= 44.8 and

271.4), DP(0.8) is minimized in cases other than

model-based indication. However, DP(0.8) tends to

be low in the case of one-by-one indication for any

condition of the real view. This suggests that one-

by-one indication should be used in situations where

accuracy has higher priority than efficiency.

In summary, the effectiveness of the basic model

depends on whether accuracy or efficiency is more

important in a particular situation. In fact, it is

difficult to quantitatively estimate the weight of each

in real situations. However, we roughly recommend

that AR manuals designed according to the basic

model should be used in most situations, but AR

manuals should provide information unit by unit in

situations where errors have to be strictly avoided.

6 CONCLUSIONS

In this study, we applied the basic model not to

computer-generated conditions, but to realistic

conditions and verified its effectiveness. Further, we

examined the applicability of the basic model to

different situations.

Essentially, both accuracy and efficiency are

important in actual situations, and it is not

appropriate to determine the weight of each.

However, AR manuals are expected to be widely

used. Thus, even rough guidelines considering

different situations in designing such a manual will

be helpful. In future studies, we will validate the

model, and demonstrate feasibility for actual field

use.

ACKNOWLEDGEMENTS

This work was supported in part by the Ministry of

Education, Science, Sports and Culture, a Grant-in-

Aid for Young Scientists (B), 20710130.

REFERENCES

P.Wellner, W.Mackay, R.Gold, 1993.Computer-Augmented

Environments: Back to the Real World,

Communications of the ACM, 36(7), 24-26.

R.T.Azuma, 1997. A Survey of Augmented Reality,

Teleoperators and Virtual Environments, 6(4), 355-

385.

S.K.Feiner, 2002, Augmented Reality: A New Way of

Seeing: Computer scientists are developing systems

that can enhance and enrich a user's view of the world,

Scientific American.

M.Nakanishi, T.Akasaka, Y.Okada, 2008. Modeling

Approach to “Effective Augmentation” for Designing

Manuals with Augmented Reality: Proceedings of the

2nd International Conference on AHFE, on CD-ROM.

M.Nakanishi, Y.Okada, 2006. Practicability of using

active guidance with retinal scanning display in

sequential operations. Proceedings of IEA 2006

Triennial Congress, on CD-ROM.

ICEIS 2009 - International Conference on Enterprise Information Systems