REMOTE RENDERING OF COMPUTER GAMES

Peter Eisert and Philipp Fechteler

Fraunhofer Institute for Telecommunications, Einsteinufer 37, D-10587 Berlin, Germany

Keywords:

Remote gaming, graphics streaming, 3D coding, networking.

Abstract:

In this paper, we present two techniques for streaming the output of computer games to an end device

for remote gaming in a local area network. We exploit the streaming methods in the European project

Games@Large, which aims at creating a networked game platform for home and hotel environments. A

local PC based server executes a computer game and streams the graphical and audio output to local devices

in the rooms, such that the users can play everywhere in the network. Dependent on the target resolution of

the end device, different types of streaming are addressed. For small displays, the graphical output is captured

and encoded as a video stream. For high resolution devices, the graphics commands of the game are captured,

encoded, and streamed to the client. Since games require signiﬁcant feedback from the user, special care has

to be taken to achieve these constraints for very low delays.

1 INTRODUCTION

Computer games are a dynamic and rapidly grow-

ing market. With the enormous technical develop-

ment of 3D computer graphics performance of nowa-

days home computers and gaming devices, computer

games provide a broad range of different scenarios

from highly realistic action games, strategic and ed-

ucational simulations to multiplayer games or virtual

environments like Second Life. Games are no longer

a particular domain of kids but are played by peo-

ple of all ages. Games offer also leisure time activity

at home, for guests in hotels, and visitors in Internet

Cafes.

Modern games, however, pose high demands on

graphics performance and CPU power which is usu-

ally only available for high end computers and game

consoles. Other devices such as set top boxes or hand-

held devices usually lack the power of executing a

game with high quality graphical output. For ubiqui-

tous gaming in a home environment, a hotel, or a cafe,

however, it would be beneﬁcial to run games also on

devices of that kind. This would avoid placing a noisy

workstation in the living room, or costly computers in

each room of a hotel. This problem could be solved

by executing the game on a central server and stream-

ing the graphics output to a local end device like a low

cost set top box. Besides the ability to play games ev-

erywhere in the entire network such a scenario could

also beneﬁt from load balancing when running multi-

ple games simultaneously.

In this paper, we present ﬁrst investigations for

streaming games’ output over a local area network.

We consider two different approaches: video stream-

ing of an already rendered frame of the game and the

streaming of graphics commands and local rendering

on the end device. Both approaches are part of the

European project Games@Large (Tzruya et al., 2006)

that develops a system for remote gaming in home and

hotel environments. The paper is organized as fol-

lows. First, we present the architecture of the gaming

system. Then, we investigate the usage of video cod-

ing techniques for streaming graphical content and il-

lustrate the differences for game scenarios. In Section

4 we ﬁnally depict our system for the streaming of

graphics commands, which can be exploited for trans-

mission of high resolution content.

2 ARCHITECTURE OF THE

SYSTEM

The Games@Large system targets at providing a plat-

form for the remote gaming in home, hotel, and other

local environments. The architecture of the system is

438

Eisert P. and Fechteler P. (2007).

REMOTE RENDERING OF COMPUTER GAMES.

In Proceedings of the Second International Conference on Signal Processing and Multimedia Applications, pages 428-433

DOI: 10.5220/0002143204280433

 SciTePress

Figure 1: System architecture.

depicted in Fig. 1. The core of the system is a PC that

executes the game. No special game adaptations are

necessary, but any commercial game can be played.

The user just selects the desired game from a web site.

In order to avoid a local installation of the game, it

runs in a virtual machine. An image of the game en-

vironment is downloaded from a provider. Since im-

portant parts of the data are transmitted ﬁrst, the game

can be started before the download is completed.

Games usually require severe constraints on com-

putational power and graphics capabilities. Since

such high-end PCs are not available in each room

of a household or a hotel, video, audio, and graph-

ics streaming over local networks is used to enable

ubiquitous gaming. The output of the game, executed

on the server, is grabbed and sent to different end de-

vices. These can be smaller PCs or laptops, but also

much cheaper set top boxes or handheld devices. De-

pendent on the capabilities of the end device, differ-

ent streaming techniques are used. For devices with

small displays like handhelds or PDAs, which usually

do not have hardware graphics support, video stream-

ing is applied, whereas devices having a graphics card

are supplied directly with the graphics commands and

render the graphics of the game locally. The two ap-

proaches of streaming game content to the end de-

vices is described in more detail in the following two

sections.

3 VIDEO STREAMING OF

SYNTHETIC CONTENT

One solution to stream the visual output of the game

to the end device is the use of video streaming tech-

niques as shown in Fig. 2. Here, the server renders the

computer graphics scene, the framebuffer is captured,

eventually downsampled to match the target resolu-

tion, and the current image is encoded using a stan-

dard video codec (Stegmaier et al., 2002). This solu-

tion has the advantage, that also end devices with no

hardware graphics capabilities can be supported. De-

coding video is usually computationally not very de-

manding and can be performed even on small devices

like PDAs or mobile phones. Also, the bit-rate for

streaming the graphics output is rather predictable and

not fully inﬂuenced by the complexity of the graph-

ics scene. On the other hand, encoding a video leads

to high computational load at the server which has

to be shared with the execution of the game. Espe-

cially if multiple games run in parallel on the server,

video encoding might be less applicable and graphics

streaming could be the better choice. Therefore, we

intent to exploit video streaming technologies only to

support devices with small displays, where video en-

coding is less demanding. Other devices like PCs or

set top boxes are connected using graphics streaming

as described in Section 4.

3.1 Video Coding Performance

Figure 3: Games used for analyzing the different codecs for

streaming synthetic games content.

For the analysis of video coding techniques for

the use of streaming game output to end devices,

we have performed some experiments using MPEG-

4 (MPEG-4, 1999) and H.264 (MPEG-4 AVC, 2003)

codecs. The output of different games as shown in

Fig. 3 was grabbed and encoded. Fig. 4 depicts the

rate-distortion plot for different games using H.264

at 4CIF resolution and 30 fps. It can be seen that

bit-rates vary between the scenes dependent on the

amount of motion and explosions of the game.

The computational complexity, however, for en-

coding 4CIF with H.264 is rather demanding. For

our implementation, frame rates of only 9 fps could

be achieved on a 2 GHz Pentium 4. The complexity

and performance of different codecs is illustrated in

Fig. 5. The upper curves with the best coding perfor-

mance refer all to different settings of H.264, the two

lower ones to MPEG-4. However, the lower quality

results also in a lower complexity, and frame rates of

80 fps and 110 fps, respectively, can be achieved for

our MPEG-4 implementations.

REMOTE RENDERING OF COMPUTER GAMES

439

Figure 2: Video streaming from the gaming server.

0 2 4 6 8 10 12 14

rd3 menu

just cause drive

just cause menu

just cause shoot

just cause swim

rd3 accident

rd3 drive

Figure 4: Rate-distortion plot for encoding different game

output with H.264 in 4CIF resolution, 30 fps. The curves

show a large variability of bit-rates between different

scenes.

0 2 4 6 8 10

h264

mpeg4

h264

Figure 5: Rate-distortion plot for encoding a game sequence

at 4CIF resolution, 30fps with different proﬁles of MPEG-4

and H.264.

Streaming games’ output, however, is somewhat

different than streaming real video. First, the syn-

thetic content is usually free of noise and has different

statistics compared to real video. Also the require-

ment on extremely low delay are much higher in order

to enable interactive gaming. On the other hand, ad-

ditional information about the scene is available from

the render context, like camera data or motion and

depth information. We exploit that information to re-

duce the complexity of the H.264 encoding in order to

combine high coding efﬁciency with lower encoding

complexity.

4 GRAPHICS STREAMING

The second approach exploited for streaming the

game’s output is to directly transmit the graphics

commands to the end device and render the image

there (Buck et al., 2000; Humphreys et al., 2002; Ig-

nasiak et al., 2005). For that purpose, all calls of the

OpenGL or DirectX library are intercepted, encoded

and streamed. In this framework, encoding is much

less demanding and independent from the image res-

olution. Therefore, high resolution output can be cre-

ated and parallel game execution on the server is en-

abled. Also, the graphics card of the server has not

to be shared among the games, since rendering is per-

formed at the remote client. On the other hand, bit-

rates are less predictable and high peaks of data-rate

are expected, especially for scene changes, where a

lot of textures and geometries have to be loaded to the

graphics card. Also, hardware support for rendering

is required at the end device and an adaptation of the

game’s output to the capabilities of the end device is

necessary, which means that not all games can be sup-

ported in this mode. In the next section, some statis-

tical analysis on issues related to graphics streaming

is presented as well as some description of the current

graphics streaming implementation.

5 STREAMING OF OPENGL

GAMES

In this paper, we concentrate on OpenGL games in

a Linux environment. Examples of supported games

are shown in Fig. 7. Besides direct usage of OpenGL

commands, we also support the SDL (Simple Direct-

Media Layer) library, which is often used for game

programming. All calls to the OpenGL and SDL li-

brary are intercepted as well as the glXSwapBuffers

SIGMAP 2007 - International Conference on Signal Processing and Multimedia Applications

440

Figure 6: Graphics streaming from the gaming server.

and SDL

GL SwapBuffers command in order to de-

termine if the frame is ready for display. In order to

support the dynamic loading of graphics commands

and OpenGL extensions, also the functions glXGet-

ProcAddressARB and SDL GL GetProccAddress are

replaced by new versions.

All the graphics commands are streamed to the

client and rendered there. Fig. 8 shows the bit-rate

(measured in bits per frame) for the raw commands

for different game scenarios. It can be seen that there

is a high variability of bit-rates. Especially at scene

changes, new textures and display lists have to be

streamed, leading to extreme peaks of bit-rate. In

order to equalize them, intelligent caching and pre-

fetching of textures are required by analyzing the

game structure in advance and using side information

of the games’ provider. Also encoding of textures, ge-

ometries (M

uller et al., 2006) and graphics commands

is added.

5.1 Commands with Feedback

Another issue of streaming graphics, which is rather

problematic, are commands that require a feedback

from the graphics card. These can be requests for ca-

pabilities of the graphics card, but also current state

information like the actual projection or modelview

matrix. Fig. 9 depicts the temporal change of the

number of graphics commands that require some re-

turn values from the graphics card. The curves show,

that for some frames more that 1000 commands ask

for feedback (especially at scene changes), but also

during normal game play, several commands request-

ing feedback are issued. Average and maximum val-

ues of bit-rate and commands are also illustrate in Ta-

ble 1. Since we do not know, what the game does

with the return values, we have to wait in our stream-

ing environment until the answer has been returned.

This could introduce enormous round trip delays in

the client-server structure and makes interactive gam-

ing impossible.

Table 1: Command statistics and bit-rate for the three games

gltron, penguin racer, and scorched3d. All numbers are

measured per frame.

game gltron penguin scorched

avg. bit-rate 51 kbit 261 kbit 227 kbit

max bit-rate 2.1 Mbit 26.6 Mbit 3.6 Mbit

avg. no cmds 3300 14700 13000

max no cmds 37000 41000 106000

avg. no return 4.1 1.5 23.2

max no return 47 1410 1380

5.2 Local Simulation of OpenGL State

In order to avoid waiting for the client returning re-

sults from the graphics card, we simulate the current

state of the graphics card at the server and can there-

fore reply directly without sending the command to

the client. We distinguish between different types of

commands:

• Static values

• Return values that can be predicted

• Values simulated locally at server.

All static properties, which usually refer to the capa-

bilities of the graphics card, are initially tested at the

client and sent to the gaming server. This includes

information about maximum texture size, number of

display lists, etc. Also, the support of OpenGL ex-

tensions can be initially tested once. After the static

information has been received at the server, all re-

quests for this data can then be answered locally with-

out sending the command to the client.

Other commands like glGetError can be predicted

and dummy return values can be provided. Assuming,

e.g., that no error has occurred, can avoid the round

trip time for such commands. This might not work for

all games, but did not lead to problems for the games

analyzed here.

However, most of the graphics states change reg-

ularly and require a special treatment. In our imple-

mentation, we simulate this state in software in par-

REMOTE RENDERING OF COMPUTER GAMES

441

Figure 7: Three different OpenGL games used for the statistical analysis and experiments.

0 500 1000 1500 2000 2500 3000 3500 4000 4500 5000

frame

bits / frame

0 200 400 600 800 1000 1200

frame

bits / frame

0 50 100 150 200 250

frame

bits / frame

Figure 8: Bit-rate in bits per frame of the uncompressed graphics stream.

0 500 1000 1500 2000 2500 3000 3500 4000 4500 5000

frame

# OpenGL commands with feedback

gltron

0 200 400 600 800 1000 1200

500

1000

1500

frame

# OpenGL commands with feedback

Penguin Racer

0 50 100 150 200 250

200

400

600

800

1000

1200

1400

frame

# OpenGL commands with feedback

Scorched 3D

Figure 9: Number of graphics commands that require a feedback from the graphics card.

allel to the graphics card. All commands that affect

and change the state in the graphics card, initiate the

corresponding changes also at the state simulated at

the server. One example is the request for the cur-

rent ModelView or Projection matrix. These matrices

are updated locally each time a command like glRo-

tate, glTranslate, glScale, glLoadMatrix, glPushMa-

trix, etc. is called. Since these operations do not

occur that often, the overhead for doing that in soft-

ware is rather moderate. However, the round trip de-

lay caused by the game asking for the current matrix

can be avoided. Similarly, the other graphics states

are simulated such that no single command requiring

feedback is executed during normal play of the ana-

lyzed games.

5.3 Streaming of Graphics Commands

All graphics commands which need to be send to the

client, are encoded, packetized and streamed to the

end device. Encoding is currently rather simple and

byte oriented. Colors, texture coordinates, and nor-

mals can for example be represented by short code-

words and need not be represented by multiple ﬂoats

or doubles. Also textures are encoded using some

transform coding based on the H.264 interger trans-

form. Currently, more efﬁcient coding for textures,

geometries, and graphics commands are under inves-

tigation. The command itself is represented by a to-

ken of 1 or 2 bytes length. However, some groups of

commands like, e.g., glTexCoord, glNormal, glVer-

tex that occur often jointly are represented by a single

SIGMAP 2007 - International Conference on Signal Processing and Multimedia Applications

442

token in order to reduce the overhead.

The commands are then packetized for transmis-

sion. For that purpose, it is assured that a command

with all its arguments will reside in the same packet,

except for commands that contain pointers to large

memory areas. Once a packet is full, it is sent to

the client. Currently, we use a reliable TCP connec-

tion which is not optimal for minimizing delay. How-

ever, in contrast to the video streaming approach, that

can start with encoding of the video ﬁrst if the frame

buffer has been rendered completely, we can start with

the transmission of packets bevor the current frame is

processed completely minimizing the delay by almost

one frame.

5.4 Experimental Results

The proposed streaming system has been tested with

different OpenGL games like scorched3d, gltron,

penguinracer, OpenArena, neverball and the Flight-

Gear simulator. When streaming all graphics com-

mands directly without any compression and graph-

ics state simulation, the games are not interactively

playable, since delay is much too high. With our

proposed system, for locally simulating the graphics

card’s state, the number of commands during normal

game play requiring a feedback could be reduced to

zero. This leads to a signiﬁcant reduction in delay

and enables the ability to play the tested games in a

local area network. Graphics command compression

was enabled but in the current version rather moder-

ate. For the arguments of the commands, an average

compression of about 30 % was achieved. The size

of the target window can, however, be varied inter-

actively (by manipulating the glViewport command)

and need not be the same as the game’s resolution.

For the considered games, interactive frame rates of

up to 30 fps for the simpler games was achieved. For

more demanding games like OpenArena, delays are

relatively visible. Here, we work on reducing the bit-

rate by more efﬁcient compression methods and thus

reducing the delay for enhanced gaming.

6 CONCLUSIONS

We have presented in our paper a system for the re-

mote gaming in local networks. The architecture

of the proposed system is targeted for an execution

of commercial games in a virtual environment and

ubiquitous gaming due to different streaming tech-

niques. Both, video streaming and transmission of

graphics commands are investigated. First analysis

shows the applicability of the approaches for different

end devices. In order to reduce delay for the graphics

streaming, a simple real-time compression of graph-

ics commands and a local simulation of the graphics

state has been implemented. This resulted in a sig-

niﬁcant reduction of delay which is a prerequisite for

interactive gaming.

REFERENCES

Buck, I., Humphreys, G., and Hanrahan, P. (2000). Track-

ing graphics state for networked rendering. In Proc.

SIGGRAPH/EUROGRAPHICS Workshop On Graph-

ics Hardware.

Humphreys, G., Houston, M., Ng, R., Frank, R., Ah-

ern, S., Kirchner, P. D., and Klosowski, J. T. (2002).

Chromium: a stream-processing framework for inter-

active rendering on clusters. In Proc. International

Conference on Computer Graphics and Interactive

Techniques.

Ignasiak, K., Morgos, M., and Skarbek, W. (2005).

Synthetic-natural camera for distributed immersive

environments. In Proc. Workshop on Image Analysis

for Multimedia Interactive Services (WIAMIS), Mon-

treux, Switzerland.

MPEG-4 (1999). ISO/IEC 14496-2: Coding of audio-visual

objects - Part 2: Visual, (MPEG-4 visual).

MPEG-4 AVC (2003). Advanced video coding for generic

audiovisual services, ITU-T Rec. H.264 and ISO/IEC

14496-10 AVC.

uller, K., Smolic, A., Kautzner, M., Eisert, P., and Wie-

gand, T. (2006). Rate-distortion-optimized predictive

compression of dynamic 3D mesh sequences. Signal

Processing: Image Communication, 21(9):818–828.

Stegmaier, S., Magall

on, M., and Ertl, T. (2002). A generic

solution for hardware-accelerated remote visualiza-

tion. In Proceedings of the Symposium on Data Vi-

sualisation 2002.

Tzruya, Y., Shani, A., Bellotti, F., and Jurgelionis, A.

(2006). Games@large - a new platform for ubiquitous

gaming and multimedia. In Proceedings of BBEurope,

Geneva, Switzerland.

REMOTE RENDERING OF COMPUTER GAMES

443