COMPONENT APPROACH TO DISTRIBUTED MULTISCALE

SIMULATIONS

Katarzyna Rycerz

Institute of Computer Science, AGH, al. Mickiewicza 30, 30-059 Krak

ow, Poland

Academic Computer Centre – CYFRONET, Nawojki 11, 30-950 Krak

ow, Poland

Marian Bubak

Institute of Computer Science, AGH, al. Mickiewicza 30, 30-059 Krak

ow, Poland

Faculty of Science, Informatics Institute, University of Amsterdam, Science Park 107, 1098 XG Amsterdam, The Netherlands

Keywords:

Multiscale simulations, HLA, components, Distributed simulations, Grid computing.

Abstract:

This paper presents a component approach suitable for distributed multiscale simulations. We describe the

integration of the High Level Architecture (HLA) distributed simulation standard with modern computing

frameworks: component technologies and Grid infrastructures, which enable users working on distributed

simulations to easily exchange, the simulation models available to them. We discuss the design challenges

involved in developing an HLA component which can be steered by a user during simulation runtime. More

speciﬁcally, we present a solution which supports concurrent control of two different layers: the Grid (ex-

ploited by the user to steer the component). and the HLA. This functionality is presented on a sample multi-

scale stellar system simulation.

1 INTRODUCTION

Multiscale simulations are of great importance for

complex system modeling. Examples of such simu-

lations include e.g. blood ﬂow simulations (Caiazzo

et al., 2010), solid tumor models (Hirsch et al., 2009),

stellar system simulations (Portegies Zwart et al.,

2008) or computational chemistry simulations (Suter

et al., 2009) .

The main scientiﬁc objective of this work is to de-

termine how to exploit large-scale computing tech-

nologies in order to facilitate the creation and exe-

cution of complex simulations consisting of modules

with different time scales. We focus on the require-

ments of composability, interoperability and reuse –

here the main challenge is to support easy creation

and interconnection of simulation modules by multi-

ple users, for instance when one user creates modules

while another one connects them in different ways.

Current solutions are usually restricted to speciﬁc

models or simulation methods. However, partial so-

lutions standardizing complex connections between

simulation models already exist. In our work we have

chosen to apply the High Level Architecture (HLA)

High Level Architecture speciﬁcation – IEEE 1516

as it enables plugging in and unplugging various sim-

ulations from a single simulation system. HLA func-

tionality is useful for multiscale applications, espe-

cially with these with different time scale, as it pro-

vides ability to connect simulation modules with dif-

ferent time management mechanisms. Additionally,

as a means of composability support, HLA introduces

a uniform way of describing events and objects being

exchanged between modules.

In our previous work (Rycerz et al., 2008) we

have proposed a component approach to the HLA

standard solutions and integrate it with the Grid tech-

nology. The main advantage of this approach over the

raw HLA is that it facilitates the development of simu-

lation systems from components prepared by external

users. In our solution, the behavior of each simula-

tion module, wrapped as a component, is deﬁned by

an external user who has the ability to steer (from out-

side) the functionality of HLA used by his/her com-

ponent. In order to enable collaboration and resource

sharing, we rely on the Grid technology; more specif-

ically on the H2O environment which is capable of

hosting HLA components (Kurzyniec et al., 2003).

In this paper we focus on the design aspects of

such components, namely on support for concur-

rent control of two different layers: the Grid layer

122

Rycerz K. and Bubak M..

COMPONENT APPROACH TO DISTRIBUTED MULTISCALE SIMULATIONS.

DOI: 10.5220/0003592701220127

In Proceedings of 1st International Conference on Simulation and Modeling Methodologies, Technologies and Applications (SIMULTECH-2011), pages

122-127

ISBN: 978-989-8425-78-2

 2011 SCITEPRESS (Science and Technology Publications, Lda.)

exploited by the user to steer the component and

the HLA layer which supports multiscale-speciﬁc

requirements, such as advanced time management.

Basing on the Active Object pattern (Lavender and

Schmidt, 1996) we have introduced a scheduler deal-

ing with user’s external requests comming from a

Grid layer to a component.

This paper is organized as follows: in Section 2

we outline the available Grid and component tech-

nologies and related work for multiscale simulations.

In Section 3 we describe the HLA component model.

Section 4 presents an experiment involving a sample

multiscale simulation of a dense stellar system. Con-

clusions and information on future work can be found

in Section 5.

2 RELATED WORK

The requirements mentioned above are addressed by

numerous partial solutions, including tools for multi-

scale simulation development as well as component

and Grid approaches supporting composability and

reusability.

The Multiscale Multiphysics Scientiﬁc Environ-

ment (MUSE) (Portegies Zwart et al., 2008) is a

software environment for astrophysical applications

where different simulation models of stars systems

are incorporated into a single framework and a script-

ing approach is used to couple individual models.

The Multiscale Coupling Library and Environment

(MUSCLE)(Hegewald et al., 2008) provides a soft-

ware framework for developing simulations accord-

ing to the complex automata theory (Hoekstra et al.,

2007). It has been applied in coronary artery in-stent

restenosis simulations (Caiazzo et al., 2010).

Requirements related to reusability and compos-

ability of existing models can be met by apply-

ing modern IT solutions such as component and

Grid technologies. Examples of the component ap-

proach include the Service Component Architecture

(SCA)

where business functionality is provided as

a series of services assembled together to create so-

lutions that serve a particular business need, and

the Common Component Architecture (CCA) (Arm-

strong et al., 2006) (with implementations such as

MOCCA (Malawski et al., 2006)), used in high-

performance computing where scientiﬁc components

are directly connected by their uses and provides

ports.

G. Barber. Service compo-

nent architecture home, 2007. http://

osoa.org/display/Main/Service+Component+Architecture+Home

The Grid technology (Foster et al., 2002) is ori-

ented towards joining geographically distributed com-

munities of scientists working on similar problems.

Currently, one of the most important Grid middleware

platforms is the Globus Toolkit (Foster et al., 2002)

together with the WS-Resource Framework (WSRF)

which repurposes the classical Web Service interface

to provide stateful WS-Resources

. The Grid also

can, for example, be constructed from pure Web ser-

vices and Grid Services (Foster et al., 2002) or by

using the H2O resource sharing platform (Kurzyniec

et al., 2003). Examples of production grids include

European e-infrastructures such as EGI

or DEISA

To the best of our knowledge, none of the exist-

ing multiscale computing tools or component mod-

els supports advanced time synchronization between

simulation modules. Thus, we propose to apply the

component approach to HLA simulations. Addition-

ally, as our goal is to support the exchange of simu-

lation modules between scientists regardless of their

actual geographical location, we have decided to use

the Grid, opting for the H2O framework (Kurzyniec

et al., 2003) which is - at the same time - portable

(Java-based), secure, scalable and lightweight.

3 HLA COMPONENT MODEL

FOR MULTISCALE

SIMULATIONS

A C

uses port provides port

connect

join resign

HLA RTI

time and data management

Figure 1: Comparison of two component models: (a) direct

connections in CCA by provides and uses ports (b) compo-

nents joining/resigning from the tuple space in HLA.

The features of the presented component model cor-

respond to multiscale simulation requirements. Typ-

ically, such simulations require a signiﬁcant amount

WSRF home page: http://www.globus.org/wsrf/

European Grid Infrastructure http://www.egi.eu/

Distributed European Infrastructure for Supercomput-

ing Applications http://www.deisa.eu/

COMPONENT APPROACH TO DISTRIBUTED MULTISCALE SIMULATIONS

123

of time to complete, producing partial results during

execution. They also require support

for advanced types of connections between com-

ponents, including time and data management. Addi-

tionally, as many simulation kernels exist in the form

of legacy code, the component developer should be

able to reuse such code in the most convenient way.

To fulﬁll these requirements we proposed a HLA

Component model (Rycerz et al., 2008). In this pa-

per we focus on the design aspects of HLA compo-

nents, namely on the support for concurrent control

useful for steering of a component during the simula-

tion runtime. The steering include joing and risigning

from a simulation, changing a time policy (useful for

joining modules of different time scale) and altering

components’ connections for exchanging data. Our

previous work simply exploited concurrency control

mechanisms provided by the HLA RTI, in this paper

we present comparison to alternative solution based

on Active Object pattern.

The basic idea of the component model is shown

in Fig. 1. Components can use advanced HLA RTI

time and data management to which they can plug

in via join/resign mechanisms. In comparison, the

well-known CCA model (Armstrong et al., 2006) in-

troduces direct connections between components by

joining their provides and uses ports. The two key fea-

tures which distinguish the proposed approach from

existing ones can be described as follows:

• Support for time-consuming simulations: setup

and steering of components is possible not only

prior to actual execution (as in most component

models) but also at runtime.

• The functionality provided by HLA that can be

used by a component is available by external in-

terfaces of the component – this facilitates the

development of simulation systems from compo-

nents provided by external users and allows to

fully exploit the mechanisms of collaboration and

resource sharing.

Supporting advanced steering of components dur-

ing runtime raises many issues. One of the most im-

portant problems is how to effect appropriate trans-

fer of control across many layers: (1) requests from

the Grid layer external to the component (joing and

risigning from the simulation, changing time and data

management) (2) simulation code layer and (3) HLA

RTI layer. The component should be able to efﬁ-

ciently process external requests for changing simula-

tion state in the HLA RTI layer and, at the same time,

deal with the actual simulation which may commu-

nicate with other simulation components via its RTI

layer.

The solution presented in (Rycerz et al., 2008)

exploited concurrency control mechanisms provided

by the HLA RTI. External requests were processed in

a synchronous fashion as they arrived, and were syn-

chronized with internal simulation calls to HLA RTI

within the HLA RTI itself (Fig. 2a). The advantage of

this solution was that, apart from abort requests, ex-

ternal request handling did not require modiﬁcations

in legacy simulation kernels. However, the solution

also required appropriate exception handling in HLA

itself and (as already remarked) the simulations ex-

hibited a tendency to behave nondeterministically: we

observed deadlocks and starvation of external or in-

ternal HLA requests which signiﬁcantly slowed down

execution.

In the current solution, presented in Fig.2b, we

have decided to apply ideas derived from the Ac-

tive Object pattern which separates request invoca-

tion from execution. We introduce a scheduler which

deals with user external requests comming from the

Grid layer and stores them in a queue. Once called

from the simulation loop, the scheduler dispatches

contents of the queue without interfering with actual

simulation execution. This proposed solution requires

only small modiﬁcations to the legacy simulation ker-

nel (single routine); in return the developer acquires

full control over external requests. Our solution ex-

hibits deterministic execution patterns with no dead-

lock or starvation. The actual overhead of the sched-

uler is small, as presented in Tab. of Section 4. A

detailed comparison of both component design ap-

proaches is shown in Tab. 1.

4 EXPERIMENTS INVOLVING

SIMULATION OF A DENSE

STELLAR SYSTEM

Currently, the implemented prototype of the Com-

poHLA system includes the CompoHLA components

and a Java client that allows the user to manipulate

these components. The implementation is based on

the H2O technology.

The functionality of the CompoHLA system is

presented with the example of a multiscale dense stel-

lar system simulation. We have decided to build com-

ponents from two MUSE modules: evolution (macro

scale) and dynamics (meso scale) that run concur-

rently. In this experiment we use HLA time manage-

ment features when one simulation controls the tim-

ing of the other. The dynamics module (called con-

strained) needs to receive an update from evolution

(called regulating) before it passes a predetermined

checkpoint. Data containing mass, radius, 3D posi-

SIMULTECH 2011 - 1st International Conference on Simulation and Modeling Methodologies, Technologies and

Applications

124

external

requests

(join,

resign,

set time

policy, etc)

(join,

resign,

set time

policy, etc)

H2O

external

requests

H2O

HLAHLA

Component

HLA

HLA RTI HLA RTI

a) b)

User code User code

CompoHLA library

Component Component

HLAComponent

Scheduler

Figure 2: Architectural comparison of HLA components: (a) external and user simulation requests processed concurrently by

CompoHLA and the HLA Runtime Infrastructure (RTI) library, (b) external requests queued by a scheduler which processes

them following a call from user code.

Table 1: Comparison of HLA component design approaches.

Using mechanisms of HLA RTI concur-

rent access control

Explicit scheduler

transparent to the developer requires calling a single routine in a sim-

ulation loop

synchronous mode - requests processed as

soon as possible

asynchronous mode - invocation sepa-

rated from execution

requests processed at any point in the sim-

ulation loop

requests processed when the scheduler is

called from the simulation loop

dependent on implementation of concur-

rency control in HLA RTI

independent of the HLA implementation

behavior

concurrency difﬁcult to handle effectively -

e.g starvation of requests causing overhead

in simulation execution

concurrency easy to handle; low sched-

uler overhead

tions and velocity of stars is exchanged between evo-

lution and dynamics. The HLA components are asked

to join the simulations (both), set the time-regulating

policy and publish data (evolution component), set

the time-constrained policy and subscribe to data (dy-

namics component). After the initial 10 simulation

steps, both components were asked to unset their time

policy, resign from the simulation and halt. Our im-

plementation uses H2O v2.1 and HLA CERTI im-

plementation v3.3.2. Experiments were performed

on different grid sites of Dutch Grid DAS3

. The

The Distributed ASCI Supercomputer 3 web page

actual setup of the HLA Component experiment is

shown in Fig.3. The component client was run on

the Leiden Grid node, the dynamics component was

run on the Delft Grid node, the evolution component

was run at UvA, Amsterdam while the HLA con-

trol process was run at Vrije Universiteit, Amster-

dam. All Grid nodes shared a similar architecture

(dual AMD Opteron computing nodes, 2.4 GHz, 4GB

RAM). DAS 3 employs a novel internal wide-area in-

terconnect (StarPlane) based on light paths between

its Grid sites.

http://www.cs.vu.nl/das3

COMPONENT APPROACH TO DISTRIBUTED MULTISCALE SIMULATIONS

125

Dynamics

Component

@Delft

Evolution

Component

@Amsterdam

set regulation policy

HLA RTI

Client

@ Leiden

set constrained policy

start/stop

join/resign

start/stop

publish

join/resign

Figure 3: Setup of HLA Components experiment. The client sends requests to two components with differing scales, respon-

sible for simulating the dynamics and evolution of a dense stellar system.

Table 2: Timing of actions in HLA Components.

External requests time, millisec σ

Join dyn. 10 1

Join evol. 9 0.8

Publish evol 6 0.4

Subscribe dyn 8 0.5

Set time policy evol. 4 0.4

Set time policy dyn. 9 0.1

Start dyn. 7 0.4

Start evol. 4 0.1

Unset time policy dyn. 14 1

Unset time policy evol. 9.5 0.5

Resign evol. 5.6 0.9

Resign dyn. 8 0.1

Stop dyn. 9 0.5

Stop evol. 5 0.1

Actions during time, millisec σ

simulations

dynamics calculations 17.6 0.08

evolution calculations 0.004 0.0001

synch dyn with evol 0.007 0.0001

synch evol with dyn. 0.04 0.001

scheduler (both) 0.6 0.05

total sim time (evol) 1.1 0.05

total sim time (dyn) 18.3 0.08

The simulation model, wrapped by the dynamics

component, was more computationally intensive: in

our experiment the total execution time of the dynam-

ics simulation was 18.3 seconds while the evolution

simulation took 1.1 seconds. Detailed results are pre-

sented in Tab.2 which lists the execution time of two

different types of actions in the dynamics component.

We have calculated average values from 10 runs; σ

indicates standard deviation. One group contains the

timing of external requests for the HLA component

(from the client perspective). As can be seen, this is

on the order of several milliseconds. The requests are

called asynchronously, so that during the call they are

only scheduled in the queue. The other group contains

actions taken during the simulation run: dispatch-

ing requests from the queue by the scheduler, actual

computations and synchronization with the evolution

component by means of HLA RTI. The results of the

experiment show that time of external requests pro-

cessing and later dispatching by a scheduler is consid-

erably shorter than calculation time of computation-

aly intensive (dynamics) component and that the com-

ponent layer does not introduce signiﬁcant overhead.

Performance tests have also shown that the overhead

of HLA–based distribution (especially its repeating

part which involves synchronization between multi-

scale elements) is small and that HLA can be success-

fully exploited in multiscale simulations.

5 SUMMARY AND FUTURE

WORK

In this paper we have presented a new design of

HLA component model which enables the user to

dynamically compose/decompose distributed simula-

tions from multiscale elements residing on the Grid.

The presented approach differs from raw HLA where

all decisions about actual interactions are set during

implementation of simulation modules. In our solu-

tion, the multiscale application can be build from ex-

isting components by controlling HLA mechanisms

from outside and joining of components is based on

HLA functionality. Therefore, the components can

communicate by exchanging data with time stamps

using HLA (data management facility). To join com-

ponents of different time scale one can set specyﬁc

time policy of particular component (time manage-

ment facility).

The additional feature of our solution is that com-

ponent steering can also be done during actual sim-

SIMULTECH 2011 - 1st International Conference on Simulation and Modeling Methodologies, Technologies and

Applications

126

ulation runtime. For that, we have used Active Ob-

ject pattern with a scheduler that separates invocation

of external requests from their execution. Therefore,

we avoid interference between user request process-

ing and running simulation. The features of that solu-

tion have been comparated with the approach of sim-

ply using concurrency control mechanisms provided

by the HLA RTI.

The functionality of the prototype is presented

with the example of a multiscale dense stellar system

simulation – the MUSE environment (Portegies Zwart

et al., 2008). The results of the experiment compare

time execution of (1) actual computation, (2) the user

external request processing, (3) dispatching requests

by scheduler and (3) modules synchronization using

HLA communication. The results show that the intro-

duced component layer does not introduce signiﬁcant

overhead in comparison to computation time.

The HLA Components described in this paper

are designed to facilitate composability of simulation

models by means of HLA mechanisms accessible and

steerable from the Grid layer. However, to fully ex-

ploit their composability potential, the use of such

components should be part of a larger system which

supports development and execution of complex mul-

tiscale simulations by applying the presented HLA

component model. Therefore, we are working on in-

tegration of HLA Components with the GridSpace

Virtual Laboratory

ACKNOWLEDGEMENTS

The authors wish to thank Alfons Hoekstra and Jan

Hegewald for discussions on MUSCLE, Simon Porte-

gies Zwart for valuable discussions on MUSE and

our colleagues from DICE team for input concern-

ing GridSpace. The authors are also very grateful to

Piotr Nowakowski for his suggestions. The research

presented in this paper was partially supported by the

European Union in the EFS PO KL Pr. IV Activity

4.1 Subactivity 4.1.1 project UDA-POKL.04.01.01-

00-367/08-00 ”Improvement of the Didactic Potential

of the AGH University of Science and Technology –

Human Assets” and also by MAPPER project – grant

agreement no 261507, 7FP UE.

REFERENCES

Armstrong, R., Kumfert, G., McInnes, L. C., et al. (2006).

The CCA component model for high-performance sci-

http://dice.cyfronet.pl

entiﬁc computing. Concurrency and Computation :

Practice and Experience, 18(2):215–229.

Caiazzo, A., Falcone, J.-L., Evans, D., et al. (2010). Com-

plex Automata models for tissue growth in a stented

artery. In Proceedings of European Consortium For

Mathematics In Industry, ECMI08, Mathematics in

Industry. Springer. in press.

Foster, I., Kesselman, C., Nick, J., et al. (2002). The

Physiology of the Grid: An Open Grid Services Ar-

chitecture for Distributed Systems Integration. Open

Grid Service Infrastructure WG, Global Grid Forum.

http://www.globus.org/alliance/publications/papers.php.

Hegewald, J., Krafczyk, M., T

olke, J., et al. (2008). An

Agent-Based Coupling Platform for Complex Au-

tomata. In ICCS ’08: Proceedings of the 8th interna-

tional conference on Computational Science, Part II,

pages 227–233, Berlin, Heidelberg. Springer-Verlag.

Hirsch, S., Szczerba, D., Lloyd, B., et al. (2009). A

Mechano-Chemical Model of a Solid Tumor for Ther-

apy Outcome Predictions. In ICCS ’09: Proceed-

ings of the 9th International Conference on Compu-

tational Science, pages 715–724, Berlin, Heidelberg.

Springer-Verlag.

Hoekstra, A., Lorenz, E., Falcone, J.-L., et al. (2007). To-

ward a Complex Automata Formalism for Multi-Scale

Modeling. International Journal for Multiscale Com-

putational Engineering, 5(6):491–502.

Kurzyniec, D. et al. (2003). Towards Self-Organizing

Distributed Computing Frameworks: The H2O Ap-

proach. Parallel Processing Letters, 13(2):273–290.

Lavender, R. G. and Schmidt, D. C. (1996). Active Object :

an Object Behavioral Pattern for Concurrent Program-

ming. In Pattern Languages of programs design 2,

pages 483–499.

Malawski, M., Bubak, M., Placek, M., et al. (2006). Exper-

iments with distributed component computing across

grid boundaries. In Proceedings of the HPC-GECO

/ CompFrame workshop in conjunction with HPDC

2006, Paris, France.

Portegies Zwart, S., Mcmillan, S., Nuall

ain, B. O., et al.

(2008). A Multiphysics and Multiscale Software En-

vironment for Modeling Astrophysical Systems. In

ICCS ’08: Proceedings of the 8th international con-

ference on Computational Science, Part II, pages 207–

216, Berlin, Heidelberg. Springer-Verlag.

Rycerz, K., Bubak, M., and Sloot, P. M. (2008). HLA Com-

ponent Based Environment For Distributed Multiscale

Simulations . In Priol, T. and Vanneschi, M., edi-

tors, From Grids to Service and Pervasive Computing,

pages 229–239, Berlin, Heidelberg. Springer-Verlag.

Suter, J. L., Anderson, R. L., Christopher Greenwell, H.,

et al. (2009). Recent advances in large-scale atomistic

and coarse-grained molecular dynamics simulation of

clay minerals. J. Mater. Chem., 19:2482–2493.

COMPONENT APPROACH TO DISTRIBUTED MULTISCALE SIMULATIONS

127