Spatial Positions of Operator’s Finger and Operation Device Inﬂuencing

Sense of Direct Manipulation and Operation Performance

Kazuhisa Miwa

, Hojun Choi

, Mizuki Hirata

and Tomomi Shimizu

Graduate School of Informatics, Nagoya University, Nagoya, 4648601, Japan

Advanced Development Div., Tokai Rika Co., Ltd., Toyota, Oguchi-cho, Niwa-gun, Aichi, 4800195, Japan

Keywords:

Indirect Manipulation, Interface, Sense of Agency.

Abstract:

When operating an interface using an input device (such as a mouse or trackpad,) one’s ﬁngers (referred to

as the “Operating Subject”), indirectly operate a target device through a pointer displayed on the interface

(referred to as the “Operation Media”). Our experiment investigated the effects of the spatial positions of

the Operating Subject and Operation Media on the sense of direct manipulation and operation performance.

The results showed that the sense of direct manipulation increased when the Operation Media was placed

diagonally toward the left than on the front, and the operation performance was higher when the Operating

Subject was placed on the right side of the body than on the front (for right-handed individuals).

1 INTRODUCTION

1.1 User Representation

When operating an interface using an input device

(such as a mouse or trackpad), one’s hands or ﬁngers

[hereafter referred to as “Operating Subject” (OS)]

indirectly operate a target device, such as a naviga-

tion system in an automobile or an operation panel

of an in-car device, through a pointer displayed on

the interface [hereafter referred to as “Operation Me-

dia” (OM)]. Figure 1 shows the relationship among

the OS, OM, and manipulated target device.

Figure 1: Relationship among the ﬁnger as an operating

subject, a pointer as an operation media, and the target de-

vice for operation.

Manipulating the target device through such an

OM is in contrast to manipulating a device like a

tablet, which are manipulated by directly touching the

device.

The OM acts as a substitute for our own bodies. In

indirect manipulation, we use this substitute to touch

and interact with the target device. This substitute is

called “user representation,” and various types of in-

direct operations have been discussed (Seinfeld et al.,

2021). Seinfeld et al. called the space that the OS

touches the “input space,” and the space that the OM

touches the “output space.” The output space is the

operating surface or interface of the target device that

is manipulated by the operator.

The concept of user representation is applied to

understanding interactions with various artifacts. The

example discussed at the beginning of this study was

an arrow-shaped mouse’s cursor on a desktop; how-

ever, for on-screen avatars (Marcos et al., 2010), body

parts such as arms displayed in a VR space (Rautaray

and Agrawal, 2015), robots, and even drones can be

included as examples of user representation.

1.2 Tacit Knowledge

In considering the issue of indirect manipulation, it

is useful to refer to the concept of tacit knowledge as

described by Polanyi (Polanyi, 1958).

Generally, tacit knowledge is deﬁned as the

knowledge that is used empirically but cannot be ex-

plained in words. Tacit knowledge is typically related

to physical actions. Consider, for example, riding a

bicycle, which requires the integrated coordination of

Miwa, K., Choi, H., Hirata, M. and Shimizu, T.

Spatial Positions of Operator’s Finger and Operation Device Inﬂuencing Sense of Direct Manipulation and Operation Performance.

DOI: 10.5220/0011724500003417

In Proceedings of the 18th Inter national Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications (VISIGRAPP 2023) - Volume 2: HUCAPP, pages

229-235

ISBN: 978-989-758-634-7; ISSN: 2184-4321

 2023 by SCITEPRESS – Science and Technology Publications, Lda. Under CC license (CC BY-NC-ND 4.0)

229

multiple muscles. However, upon acquiring this phys-

ical knowledge, one can perform it unconsciously and

without any difﬁculties. Moreover, once one learns

how to ride, one never forgets. A characteristic of

tacit knowledge is that it is difﬁcult to verbalize, de-

spite being easily operationalized. It is almost impos-

sible to explain verbally and teach others how to ride

a bicycle.

One of the most important concepts in tacit knowl-

edge is the proximal and distal terms.

For instance, while walking down a street with a

cane (see Figure 2), if an obstacle hits the tip of the

cane, vibration is transmitted to the palm of the hand

that is holding it. The vibration felt by the hand is

the proximal term and the feeling of detecting the ob-

stacle at the end of the cane is the distal term. We

sense the vibration of the hand as the proximal term,

but what we perceive is the obstacle at the end of the

cane. We do not know the vibration, but we know that

there is an obstacle.

Figure 2: Proximal and distal terms.

In tacit knowledge, this phenomenon is referred to

as the tacitization of the proximal term or the forma-

tion of perceptions of the distal term using the proxi-

mal term as a cue.

It is reasonable to say that the solution to various

issues in indirect manipulation can be summed up as

the formation of tacit knowledge, as described so far.

In indirect manipulation, the proximal term emerges

in the operating space that the OS directly touches

(the input space), and the distal term emerges in the

space that the OM touches (the output space), such as

the operating surface and the interface of the operat-

ing device.

If the proximal term is not tacitly known, it is im-

possible to obtain a good operating experience. The

key to improving the experience of direct operation is

to make the proximal term tacitly known and form an

awareness of the distal term.

1.3 Sense of Agency

In direct manipulation, the body touches the target de-

vice directly. In contrast, in indirect manipulation, the

target device is manipulated through an OM. One of

the problems that can occur with indirect manipula-

tion in such a situation is the separation of the OS and

OM. In this case, the OM is manipulated, instead of

the target device; the manipulated OM interacts with

the target device. In this case, the sense of directly

manipulating the target device, that is, the direct ma-

nipulation sensation, is missing.

An important concept related to the sense of direct

manipulation is the “sense of agency” (SoA), which is

the feeling that one is the operating principal of a cer-

tain movement, controls the target, and feels respon-

sible for the result of the movement (Limerick et al.,

2014; Moore, 2016). In this study, SoA was used as

an evaluation index for a good sense of operation in

indirect manipulation.

SoA measurement can be divided into two meth-

ods: subjective methods rated by the operator, and

objective methods. For the former, answering ques-

tions “How much your actions caused that event?”

(Sato and Yasuda, 2005; Aarts et al., 2006) and “How

much control you had over the actions?” (Kalckert

and Ehrsson, 2012; Braun et al., 2014) in a question-

naire is the gold standard method. For the latter, a

method to measure the degree of Intentional Binding

using an interface called the Libet Clock is often used

(Haggard et al., 2002; Limerick et al., 2014). A high

SoA score indicates that the proximal term is known

tacitly.

1.4 Spatial Relation Between OS and

In indirect manipulation, it is necessary to focus on

the spatial positional relationship between the opera-

tor and the OS (which is a part of one’s body), and

between the operator and the OM (which touches the

target device). For example, if the target device in

front of the operator is manipulated with their hand

facing the direction of the device (in front of their

body), the spatial orientations of the OS and OM will

be coincident. By contrast, when the hand is placed

on the right side of the body and the device placed

on the operator’s left side is operated by the hand, the

spatial orientations of the OS and OM will not coin-

cide.

Researchers examined the inﬂuence of the spatial

alignment of the OS’ positions (in this case, the ﬁn-

ger) and the tablet as the target device on the sense of

operation in a VR experimental environment (Feucht-

HUCAPP 2023 - 7th International Conference on Human Computer Interaction Theory and Applications

230

ner and M

uller, 2018). When the tablet was located

above the front of the operator, the operator was re-

quired to raise their arm to reach the tablet, which

aligned the OS and OM. However, with long-term use

of the tablet, the physical fatigue of the arm increased.

The participant gradually lowered their arm to oper-

ate the tablet in a stable position, resulting in a spatial

mismatch between the OS and OM; however, the pa-

tient still reported no signiﬁcant reduction in the sense

of operation.

The length of the operator’s arm is the distance

between the operator and the OS. However, when the

target device is manipulated by the hand displayed in

a VR space, the length of the arm on the VR is the dis-

tance between the operator and the OM. The effect of

this congruence on the feeling of manipulating the tar-

get device has been discussed (Feuchtner and M

uller,

2017). In this experiment, the operator manipulated

the target object in a VR space with an arm that was

signiﬁcantly extended in length. Even in this case,

the participants reported that it was possible to have a

sense of physical ownership of the long extended arm

as one’s own arm.

1.5 Motion-Capture Device

Recently, motion-capture devices have attracted at-

tention as input devices (Vlasic et al., 2007). Input

using motion-capture devices has many advantages

from the user’s perspective, including the fact that it

does not require a physical manipulation device, the

input location is not restricted by physical space, and

various input tokens can be generated by free body

movements. They hold great promise as input devices

in the future.

In input using a motion-capture device, the oper-

ator can manipulate the target device by taking their

hand to any location in space and pointing to the hol-

low. When a motion-capture device is used to ma-

nipulate a target device placed at various locations in

space, the spatial positioning relationship between the

operator and the OS, and between the operator and the

OM becomes diverse.

1.6 Research Questions

Based on the above, this study aimed to answer the

following two research questions:

Research Question 1:

In indirect manipulation using a motion-capture in-

put device, how would the operator’s sense of direct

manipulation of the target device be affected by the

spatial arrangement of the OS (in this case, a ﬁnger)

and OM (in this case, a pointer displayed on the in-

terface of the target device)? In this study, the spatial

placement of the OM was manipulated by placement

of the target device.

Research Question 2:

How would the operational performances of the target

device be affected by the spatial arrangement of the

OS and OM?

Figure 3: Framework of experimental design.

Our main aim was to investigate whether there is

an interaction between the effects of OS and OM spa-

tial placement on the sense of direct manipulation and

operational performances. For example, do the effects

on the sense of direct manipulation when the target

device is placed in the front or the front-left of the

body (OM factor) differ when manipulation is per-

formed in the space in the front or on the right side

of the body (OS factor)?

The experimental framework is shown in

Figure 3.

2 EXPERIMENT

2.1 Participants

A total of 16 right-handed adults were recruited from

the public.

2.2 Apparatus

In this study, a motion-capture device (Figure 4) was

developed as an input device. This device could mea-

sure a participant’s ﬁnger movements by attaching

markers to the tip and root of the index ﬁnger of the

right hand.

A display monitor was used as the target device.

An overview of the experimental apparatus is shown

in Figure 5.

Spatial Positions of Operator’s Finger and Operation Device Inﬂuencing Sense of Direct Manipulation and Operation Performance

231

Figure 4: Motion capture device used in experiment.

Figure 5: Scene of experimental setting.

Participants performed a pointing task by manipu-

lating the pointer on the monitor through index ﬁnger

movements performed in midair.

In this experiment, the participant’s ﬁnger corre-

sponded to the OS, the monitor on which the point-

ing task was displayed corresponded to the target de-

vice, and the pointer displayed on the monitor corre-

sponded to the OM.

2.3 Experimental Design

2.3.1 Independent Variables

The three independent variables in this experiment

were the input device position factor, the monitor di-

rection factor, and the monitor distance factor, all of

which were within-participant factors (Table 1).

Input Device Position Factor

The spatial position of the OS was manipulated by

moving the casing of the motion-capture device. This

was the input device position factor, and was com-

posed of the front and right conditions.

Speciﬁcally, in one condition, the motion-capture

device was positioned in front of the participant, and

diagonally to the right of the participant in the other

condition.

Monitor Direction Factor

The spatial position of the OM was manipulated by

moving the position of the monitor as the target de-

vice. This was the monitor direction factor, and was

composed of the front and left conditions.

Speciﬁcally, the monitor was placed either in front

of the participant or a little to their left . In the latter

condition, the monitor was rotated 55 degrees to the

left from the participant’s frontal direction.

Monitor Distance Factor

The distance between the participant and monitor was

manipulated. This was the monitor distance factor,

and was composed of the near and far conditions.

Speciﬁcally, in the near condition, the monitor

was placed 50 cm from the participant; in the far con-

dition, the monitor was placed 100 cm from the par-

ticipant.

2.3.2 Independent Variables

Operation Performance

For each trial, task performance was measured in

terms of the number of errors that indicated the num-

ber of incorrect inputs, completion time (ms), and

pointer travel distance (mm).

SoA

As a subjective measure, participants’ SoA was mea-

sured. Speciﬁcally, participants were asked “How

well did you feel you had control over pointing?” and

rated on a seven-point scale from “not at all” to “very

much.”

2.4 Task

The task used in this experiment was to input math-

ematical expressions (such as 21-51=), using a cal-

culator that appeared at the top of the monitor screen.

The calculator was operated by a pointer that was ma-

nipulated by the participant’s right-hand movements.

Figure 6 shows the pointer for inputting digits for the

calculator.

The participants were instructed to enter mathe-

matical expressions as quickly and accurately as pos-

sible. When a digit was entered, its color changed

from blue to orange and a sound was played to pro-

vide feedback to the participants. If the digit was en-

tered correctly, it was displayed below the formula; if

an incorrect digit was entered, it was not displayed.

HUCAPP 2023 - 7th International Conference on Human Computer Interaction Theory and Applications

232

Table 1: Experimental design.

Figure 6: Pointer displayed on the monitor as target device.

2.5 Procedures

The participants ﬁrst performed practice trials. In

each of the eight conditions, that is, the 2 × 2 × 2 con-

ditions of the three experimental factors, they entered

free formulas for one minute, and conﬁrmed their op-

erations before moving on to the production trials.

The production trial consisted of task and evalua-

tion phases. For each condition, the evaluation phase

was conducted after the task phase was completed.

Twenty-ﬁve trials were conducted for each condition

during the task phase. The order of the conditions was

counterbalanced by using the Latin square method.

During the evaluation phase, the participants re-

sponded to a paper-based questionnaire for measuring

the SoA.

3 RESULTS

3.1 SoA

For sense of agency, a three-factor within-participant

analysis of variance (ANOVA) (2: input device posi-

tion factor × 2: monitor direction factor × 2: mon-

itor distance factor) revealed a marginal signiﬁcance

in the main effect of the monitor direction factor (see

Figure 7). Speciﬁcally, the SoA tended to be rated

higher in the left condition than in the front condition

(F(1,15) = 4.158, p = .060).

Figure 7: Result for sense of agency evaluation. Error bars

show standard errors.

Neither the main effects of the input device posi-

tion factor nor the monitor distance factor was signif-

icant.

None of the second-order interaction or the three

ﬁrst-order interactions were signiﬁcant.

3.2 Operation Performances

Number of Errors

For the number of errors, the same ANOVA revealed

a signiﬁcant interaction between the input device po-

sition factor and the monitor distance factor (F(1,15)

= 9.401, p = .008). Figure 8 shows the result for the

number of errors. Simple main effect tests showed

that there were signiﬁcantly fewer errors in the near

condition than in the far condition of the monitor dis-

tance factor when the input device was placed in front

of the participants (t(15) = 9.775, p = .007), and also

showed fewer errors in the right condition than in

Spatial Positions of Operator’s Finger and Operation Device Inﬂuencing Sense of Direct Manipulation and Operation Performance

233

the front condition of the input device position fac-

tor when the target device was placed far from the

participants (t(15) = 10.757, p = .005).

Figure 8: Result for number of errors. Error bars show stan-

dard errors.

None of the three main effects were signiﬁcant.

A second-order interaction did not reach signiﬁ-

cance, and none of the ﬁrst-order interactions, other

than the above interaction, reached signiﬁcance.

Completion Time

For the completion time, the same ANOVA showed

that none of the main effects, the ﬁrst-order, or the

second-order interactions were signiﬁcant.

Pointer Travel Distance

For the pointer travel distance, the same ANOVA

showed that the main effect of the monitor distance

factor was signiﬁcant (Figure 9). Speciﬁcally, in the

far condition, the operating distance was signiﬁcantly

shorter in the near condition (F(1,15) = 11.137, p =

.005).

Figure 9: Result for pointer traveled distance. Error bars

show standard errors.

Neither the main effects of the input device posi-

tion factor nor the monitor direction factor was signif-

icant.

None of the second-order interaction or the three

ﬁrst-order interactions were signiﬁcant.

4 DISCUSSION AND

CONCLUSIONS

The SoA (or the sense of direct manipulation) was

slightly higher when the target device was placed at

a left oblique angle than at the front of the partici-

pants. Notably, this effect did not depend on whether

the participants pointed at the front or to the right of

their bodies because we did not detect the interaction

effect of the monitor direction and the input device

position factors.

These results suggest that the improvement in the

sense of direct manipulation does not depend on the

consistency of the orientation of the OS and OM; in-

stead, assigning the OM to the front-left, regardless

of the spatial arrangement of the OS, has a positive

effect on the sense of direct manipulation.

A series of perceptual psychology studies have

shown that humans prefer viewing objects from an

angle rather than from the front (Blanz et al., 1999;

Niimi and Yokosawa, 2009). This is called the three-

quarter view and is often considered the canonical

view. The advantage of the three-quarter view, which

has been conﬁrmed in the perceptual psychology do-

main, is thought to be brought about by the appear-

ance of objects. Although the situation in those stud-

ies is different from that in the present experiment, the

nature of the underlying human cognitive architecture

may explain the experimental results.

There was no interaction effect of the spatial po-

sitions of the OS and OM in determining the sense

of direct manipulation; however, for operational per-

formance, the interaction effect of the input device

position factor and the monitor distance factor was

detected. Speciﬁcally, when the input device was

placed in front of the participant (when the participant

stretched their hand in front of themselves to perform

the input), a decrease in performance was observed

when the monitor as the target device was placed far-

ther away. No such performance reduction was ob-

served when the input device was placed at the right

of the participant (when they performed the input on

their right side).

This result indicates the advantage of perform-

ing input operations on the right side of the body for

right-handed participants. This advantage may be be-

cause of the constraints of the body structure, wherein

the right hand rests at the right side of the body. In

the current experiment, we identiﬁed only this phe-

nomenon. From a cognitive science perspective, we

need to explain why these advantages emerge.

Finally, from the current limited experiment, we

drew the following hypotheses for two design princi-

ples for indirect manipulation devices when the oper-

HUCAPP 2023 - 7th International Conference on Human Computer Interaction Theory and Applications

234

ation is performed by pointing with a ﬁnger:

• When an operator is right-handed, the input oper-

ation should be made on the right side of the their

body (maybe on the left side of the body for left-

handed).

• The target device should be placed on the left front

of the operator. We do not need the consistence

of direction of the target device and the space in

which the ﬁnger operates.

REFERENCES

Aarts, H., Wegner, D. M., and Dijksterhuis, A. (2006). On

the feeling of doing: Dysphoria and the implicit mod-

ulation of authorship ascription. Behaviour Research

and Therapy, 44(11):1621–1627.

Blanz, V., Tarr, M. J., and B

ulthoff, H. H. (1999). What ob-

ject attributes determine canonical views? Perception,

28(5):575–599.

Braun, N., Thorne, J. D., Hildebrandt, H., and Debener, S.

(2014). Interplay of agency and ownership: the in-

tentional binding and rubber hand illusion paradigm

combined. PloS one, 9(11):e111967.

Feuchtner, T. and M

uller, J. (2017). Extending the body for

interaction with reality. In Proceedings of the 2017

CHI Conference on Human Factors in Computing Sys-

tems, pages 5145–5157.

Feuchtner, T. and M

uller, J. (2018). Ownershift: Facili-

tating overhead interaction in virtual reality with an

ownership-preserving hand space shift. In Proceed-

ings of the 31st Annual ACM Symposium on User In-

terface Software and Technology, pages 31–43.

Haggard, P., Clark, S., and Kalogeras, J. (2002). Voluntary

action and conscious awareness. Nature neuroscience,

5(4):382–385.

Kalckert, A. and Ehrsson, H. H. (2012). Moving a rubber

hand that feels like your own: a dissociation of own-

ership and agency. Frontiers in human neuroscience,

6:40.

Limerick, H., Coyle, D., and Moore, J. W. (2014). The

experience of agency in human-computer interactions:

a review. Frontiers in human neuroscience, 8:643.

Marcos, S., G

omez-Garc

ıa-Bermejo, J., and Zalama, E.

(2010). A realistic, virtual head for human–computer

interaction. Interacting with Computers, 22(3):176–

192.

Moore, J. W. (2016). What is the sense of agency and why

does it matter? Frontiers in psychology, 7:1272.

Niimi, R. and Yokosawa, K. (2009). Three-quarter views

are subjectively good because object orientation is un-

certain. Psychonomic bulletin & review, 16(2):289–

294.

Polanyi, M. (1958). Personal knowledge: Towards a Post-

Critical Philosophy. University of Chicago Press.

Rautaray, S. S. and Agrawal, A. (2015). Vision based hand

gesture recognition for human computer interaction: a

survey. Artiﬁcial intelligence review, 43(1):1–54.

Sato, A. and Yasuda, A. (2005). Illusion of sense of self-

agency: discrepancy between the predicted and actual

sensory consequences of actions modulates the sense

of self-agency, but not the sense of self-ownership.

Cognition, 94(3):241–255.

Seinfeld, S., Feuchtner, T., Maselli, A., and M

uller, J.

(2021). User representations in human-computer

interaction. Human–Computer Interaction, 36(5-

6):400–438.

Vlasic, D., Adelsberger, R., Vannucci, G., Barnwell, J.,

Gross, M., Matusik, W., and Popovi

c, J. (2007). Prac-

tical motion capture in everyday surroundings. ACM

transactions on graphics (TOG), 26(3):35–es.

Spatial Positions of Operator’s Finger and Operation Device Inﬂuencing Sense of Direct Manipulation and Operation Performance

235