Immersive Previous Experience in VR for Sports Performance

Enhancement

Dan Mikami, Mariko Isogawa, Kosuke Takahashi, Hideaki Takada and Akira Kojima

NTT Media Intelligence Laboratories, 1-1 Hikarino-oka, Yokosuka, Kanagawa, Japan

1 OBJECTIVE

Scouting opponent players or teams before games on

the basis of video has become generalized. In re-

cent years, video-based-scouting opportunities have

increased because of the popularization of video shar-

ing services such as YouTube. Watching videos of

opponent players or teams has two main advantages.

One is that it enables the viewer to analyze tenden-

cies the players or teams show in their play. This

information can then be used to help devise strate-

gies to use against the players or teams. The other

is that it can provide pre-experience of a sort. In

many sports, players having unique forms have signif-

icant advantages. In baseball, for example, left-hand

sidearm pitchers are comparatively rare and so batters

are likely to have problems confronting their deliver-

ies. This paper focuses on video-based scouting as a

tool for preparatory training.

In the context of ICT (information and communi-

cation technology), much research has been done on

ways to provide immersive experience (Ochi et al.,

2014). Three ways that have been developed merit

particular attention. The ﬁrst is the use of three-

dimensional displays (or projectors). The second is a

cave automatic virtual environment, better known by

the acronym CAVE. It provides immersive experience

by projecting videos to walls surrounding a user. The

third is head mounted displays (HMDs). Recent de-

velopments in wide ﬁeld-of-view HMDs with which

head movements are tracked, such as Oculus, have

made it relatively easy to provide immersive experi-

ence.

We believe that this kind of immersive high reality

virtual experience enhances the effect of previous ex-

perience and can help users to signiﬁcantly improve

their own performance in practice. In this paper, we

mainly focus on motions a player performs in hitting

back an oncoming ball in sports such as baseball, and

volleyball. We believe that these motions are particu-

larly applicable in preparatory training.

2 RELATED WORK

In this section we review systems that have been de-

veloped and trials that have been conducted in the area

of providing immersive experience in sports through

the use of virtual reality (VR).

The Nissan PlayStation GT (Gran Turismo)

Academy is a trial program for discovering and devel-

oping professional car racing drivers. It uses immer-

sive VR experience for discovering drivers. Academy

members are selected in a TV game titled “Gran Tur-

ismo”, and so far 16 members have become profes-

sional drivers through their participation in the pro-

gram. However, it is still unknown whether the VR

experience actually enhances performance in actual

competitions.

Chua et al. (Chua et al., 2003) proposed an im-

mersiveVR system for motor learning. In this system,

a trainee practices tai chi while wearing motion cap-

ture markers and a head mounted display (HMD). The

HMD enables teacher and trainee motions to be ob-

served. Although the system doesn’t aim at providing

realistic previous experience, it provides an opportu-

nity to examine the effects of immersive experience

on motor learning.

3 POTENTIAL IMMERSIVE VR

SYSTEM

To develop an immersive VR system, capturing or

synthesizing omnidirectional videos of players and

displaying them should be considered. This section

shows possible methods of doing this and describes

these advantages and disadvantages.

3.1 Capture Methods

3D Camera: The easiest way to capture a 3D

video is to use a 3D camera, which has two lenses

corresponding to the left and right eyes. Many

commercial, easy-to-use 3D cameras are already

Mikami, D., Isogawa, M., Takahashi, K., Takada, H. and Kojima, A..

Immersive Previous Experience in VR for Sports Performance Enhancement.

Server

Receiver

View direction

change

View position

change

Figure 1: View direction and view position change.

Server

Receiver

Real camera image is

imported for player region

Background and ball are

rendered by CG

Figure 2: Combination of CG and video.

commercially available.

Omnidirectional Camera: Cameras that can capture

omnidirectional views havealso been released and are

commercially available. They enable users to obtain

arbitrary viewing angle directions from the camera’s

view point. The ease with which they capture images

and provide immersive experience due to their wide

ﬁeld of view is a signiﬁcant advantage. The advan-

tage of having an omnidirectionalvideo is particularly

relevant in cases where the viewing angle changes

markedly as one hit a ball back, as shown in the vol-

leyball example in Fig. 1.

A number of research groups and companies have

released omnidirectional stereo video synthesis sys-

tems (Zhu, 2001) that use sets of multiple cameras.

There are also systems that use a single 3D camera,

but there is one major difﬁculty with them: it is very

difﬁcult to change the view position. This is because

these systems capture videos from the point where

the camera is placed.

Combination of 3D Modeling and CG: To support

changes in view positions and angles, one possible so-

lution is using CG (computer graphics) as a basis for

3D modeling of a sports ﬁeld. Because the shapes and

sizes of sports ﬁelds are determined by regulations,

3D modeling them becomes relatively easy. In ad-

dition, recent developments in computer vision tech-

niques have made it possible to ascertain ball trajec-

tories in some sports. For example, the Hawk-Eye

system shows ball trajectories in tennis and the “track

man” system does the same for baseball.

An example for volleyball is shown in Fig. 2.

In this example, to generate receiver’s view, only the

server is rendered by using captured video, and the

rest of the scene, i.e., the background and playing

ﬁeld, is rendered by using CG.

3.2 Presentation Method

To provide immersive experience, we examine the

following three representation methods, which are

listed in Table 1.

3D Display: In this decade, many 3D movies and

their DVDs have been released. This has made it

possible for people to get 3D display devices without

difﬁculty. The most popular devices of this sort

are 3D TV sets and 3D projectors. Systems using

these devices are the easiest way for users to enjoy

immersive experience; all that is needed is a pair of

3D glasses.

CAVE: The aforementioned cave automatic virtual

environment (CAVE) is known as a representative

method for providing immersive virtual reality

experience. This environment usually comprises a

small room whose walls are rear-projected screens.

When users go into the CAVE, they feel as if they are

standing on the place created by projected images.

Since the projected images change depending on

the usersf viewpoint, in the common CAVE system

it is necessary to capture users’ view position and

rotation. Thus, CAVE is an extensive and expensive

solution.

HMD: Recent years have seen head mounted dis-

plays (HMDs) become increasingly popular. These

displays intuitively control view positions and angles

by tracking head position and rotation. The Oculus

Rift DK2 supports head tracking by means of a sup-

plemental tracking camera that has a tracking range

from 0.4 to 2.5 meters for horizontal angles within 74

degrees and vertical angles within 54 degrees.

A well knowneasier solution is Google cardboard,

which uses a smartphone and a gyrometer to detect

view angles. Although it does not support view posi-

tion changes, it still provides a nice intuitive immer-

sive experience.

4 IMPLEMENTATION

We implemented two combinations from among the

potential solutions described above. The ﬁrst one is

capturing 3D video by using a 3D camera and dis-

playing the video on a 3D display (3D camera + 3D

display). The second one is synthesizing a whole 3D

model of a sports ﬁeld and showing view dependent

images from the model on a HMD (3D CG + HMD).

Detailed analysis is needed for these combinations,

but in this paper we only describe intuitive ﬁndings

Table 1: Possible representation method for immersive experience; (*) when omnidirectional stereo camera is used, binocular

disparity images can be obtained.

View change

Binocular disparity Translation Rotation Correctness

3D camera X × × X

Omnidirectional Camera * × × X

3D Modeling & CG X X X ×

Screen

Shown in life-size

Figure 3: 3D camera + 3D display.

we found from implementing them. For implementa-

tion purposes, we used the situation in which a user

hits back a pitched baseball.

4.1 3D Camera + 3D Display

Figure 3 shows an example image of this implemen-

tation. The pitcher, who is displayed life-size on a 3D

screen, pitches the ball to the batter, who is wearing

3D glasses. We believe that having to wear 3D glasses

only should not be unduly annoying to users.

The most important advantage of this setting is

ease of preparation. This is suitable for situations in

which a user doesn’t move much, such as catching a

ball coming toward his face.

However, situations in which the viewing angle

changes signiﬁcantly when a ball is hit back, such

as in the volleyball example in Fig.1, or when hit-

ting a ball thrown by a pitcher, are difﬁcult to repro-

duce virtually. If the 3D camera focuses on a server

or a pitcher, the video at hand will be difﬁcult to re-

produce. However, if the 3D camera is located so

that it will correctly reproduce the players’ hands, the

appearance of initial motions (service and pitching)

becomes invisible. This setting makes it difﬁcult to

achieve accuracy in the area around the pitcher and

the area near home plate from the batter’s view simul-

taneously.

4.2 CG Model and HMD

Figure 4 shows an example of practice in a VR envi-

ronment comprising a 3D CG and an Oculus HMD.

The baseball stadium was modeled based on baseball

rules, and the trajectory of the ball was manually ob-

tained. Both were rendered by CG. In addition, the

area around the pitcher was overlaid from video.

Head Mount Display

(HMD)

Tracking camera captures

position & rotation of HMD

Figure 4: User wearing HMD for immersive experience.

Because this setting involves a model of a whole

baseball ﬁeld, arbitrary view positions and orienta-

tion can be generated. Therefore, the areas around

the pitcher area and home plate from the batter’s view

can be accurately reproduced simultaneously. In the

trials we conducted, the users got a very high sense

of reality that they were standing in the batter’s box,

and many of them actually lurched back from the ball

when they sensed it was heading toward their face.

There are two points we should consider. One is

that the users’ motions are restricted. This is because

the HMD, which is connected to a PC with an HDMI

cable, is relatively large and heavier than 3D glasses.

The other is that the position of the pitcher is re-

stricted. Generally speaking, it is difﬁcult to render a

player, who is a non-rigid object, by CG. Therefore,

we used an actual video to overlay the area around

the pitcher. This is a suitable approach because the

pitcher stands well away from the batter and the dis-

tance between them does not vary.

5 CONCLUSION

In our study we focused on providing pre-experience

for enhancing sports performance. This paper re-

viewed potential solutions for capturing, rendering,

and displaying 3D video, from which we imple-

mented two systems. The implementations were quite

limited in scope, but they gave us the strong feeling

that the use of omnidirectional video is an important

means for providing pre-experience. In future work

we will analyze the current implementations and oth-

ers through objective evaluations for perception of 3D

space, and through subjective evaluations for effects

on performance improvement.

REFERENCES

Chua, P. T., , Crivella, R., Daly, B., Hu, N., Schaaf, R.,

Ventura, D., Camill, T., Hodgins, J., and Pausch, R.

(2003). Training for physical tasks in virtual environ-

ments: Tai chi. In IEEE VR.

Ochi, D., Kunita, Y., Fujii, K., Kojima, A., Iwaki, S.,

and Hirose, J. (2014). Hmd viewing spherical video

streaming system. In Proc. of ACMMM.

Zhu, Z. (2001). Omnidirectional stereo vision. In Proc. of

the workshop on Omnidirectional Vision.