Architecture of a Multi-domain Processing and Storage Engine

Johannes Luong, Dirk Habich, Thomas Kissinger and Wolfgang Lehner

Database Technology Group, Technische Universit

at Dresden, 01062 Dresden, Germany

Keywords:

Database Architectures, Query Languages, Data Types, Code Generation.

Abstract:

In today’s data-driven world, economy and research depend on the analysis of empirical datasets to guide de-

cision making. These applications often encompass a rich variety of data types and special purpose processing

models. We believe, the database system of the future will integrate ﬂexible processing and storage of a vari-

ety of data types in a scalable and integrated end-to-end solution. In this paper, we propose a database system

architecture that is designed from the core to support these goals. In the discussion we will especially focus on

the multi-domain programming concept of the proposed architecture that exploits domain speciﬁc knowledge

to guide compiler based optimization.

1 INTRODUCTION

Recent developments in the available amounts of data

as well as the number and scope of typical use cases,

have created the need for scalable data management

systems with a focus on complex analytical process-

ing. In many cases, applications process multiple

types of data such as tables, matrices, graphs or un-

structured data (Abadi et al., 2014). In addition, a

greater variety of people is involved in the overall pro-

cess of generating, processing, and consuming data.

Therefore, usability becomes a major challenge for

data management.

In this article, we propose an architecture for a

data management system that combines scalable and

efﬁcient processing of a variety of data types with an

easy to use but ﬂexible programming model. To em-

phasize our architecture’s support for multiple data-

types and related programming models, we will call

it the multi-domain architecture throughout the arti-

cle.

The multi-domain architecture (ﬁgure 1) com-

prises three layers : the programming interface layer,

the translation and optimization layer, and the stor-

age and processing layer. The storage and process-

ing layer at the bottom of the architecture deﬁnes an

efﬁcient and scaleable data management API that op-

erates on a group of physical data formats. The API

provides a set of operators for each physical format

and an operator orchestration language that is used to

compose larger workloads. Most of the operators ap-

ply a user deﬁned function (UDF) to stored data ob-

jects according to a predeﬁned scalable and efﬁcient

data access pattern. Operators that process data using

a UDF are called processing operators.

In our approach, the key to efﬁcient processing

is an extensible compiler framework for domain spe-

ciﬁc languages. This compiler framework forms the

translation and optimization layer at the middle of the

architecture. It accepts user deﬁned data processing

programs as input and translates them into optimized

physical workloads, which can be executed by the

storage and processing layer eventually. Access to

statistics and meta data allows the compiler to apply

advanced optimizations similar to query optimization

in traditional DBMS.

At the top of the architecture, the programming in-

terface layer provides a multi-domain programming

language for user deﬁned processing tasks. The lan-

guage consists of domain speciﬁc elements such as

SQL clauses

or linear algebra statements, low-level

operators that are mirrored from the storage and pro-

cessing layer, and a slim procedural core language.

Additional syntactic elements can be added to the lan-

guage by extending the translation and optimization

layer with additional compiler components.

Our architecture allows users to write programs

that compose different data domains at the level of in-

dividual statements. The compilation framework uses

domain speciﬁc as well as general rules to optimize

these programs and translates them into executable

workloads. Finally, the storage and processing layer

To ﬁt with our model, SQL syntax has to be adapted

slightly.

Luong, J., Habich, D., Kissinger, T. and Lehner, W.

Architecture of a Multi-domain Processing and Storage Engine.

DOI: 10.5220/0006009301890194

In Proceedings of the 5th International Conference on Data Management Technologies and Applications (DATA 2016), pages 189-194

ISBN: 978-989-758-193-9

189

Storage &

Processing

Operator Orchestration Language

Row

Format

Column

Format

Matrix

Format

Graph

Format

Translation &

Optimization

Row Ops

Extension

Column Ops

Extension

Matrix Ops

Extension

Graph Ops

Extension

SQL

Extension

MapReduce

Extension

Linear Alg.

Extension

Programming

Interface

Multi-domain Language

• Procedural core language

• Format specific operators

• Embedded processing DSLs

Figure 1: The three layers of the multi-domain architecture.

provides a set of optimized physical data formats and

processing operators that implement the concrete ex-

ecution environment of the architecture.

In the following sections we are going to discuss

the multi-domain architecture in detail. In section 2,

we will examine the individual layers and their inter-

faces. In section 3, we provide a brief review of re-

lated work and in the last section we will discuss and

summarize our approach and give an outlook on fu-

ture work.

2 THE MULTI-DOMAIN

ARCHITECTURE

The multi-domain architecture comprises three main

layers. The programming interface layer contains the

system’s multi-domain programming language, the

translation and optimization layer provides the com-

piler framework and extensions, and the storage and

processing layer consists of physical data formats,

low-level processing operators, and the operator or-

chestration language.

2.1 Programming Interface

Users deﬁne data processing workloads, using the

multi-domain programming language deﬁned in the

programming interface layer. The language consists

of a slim procedural core and a set of domain and

format speciﬁc language extensions which add their

own operators and data types. The procedural core

deﬁnes primitive types and provides statement com-

position and scopes (blocks), variables, and a set of

standard control ﬂow structures for conditional (if,

Listing 1: SQL language extension example.

1 d ef q u e r y ( ) {

v a l c u s t o m e r s =

3 SELECT( ” c name ” , ” c k e y ” )

.FROM( ” c u s t o m e r ” )

5 .WHERE( ” c a c c t b a l ” < 0 . 0 )

7 v a l n a t i o n s =

SELECT( ” n name ” , ” n key ” )

9 .FROM( ” n a t i o n ” )

11 l o g (

SELECT( ” c name ” , ” n name ” )

13 .FROM( c u st o me r s , n a t i o n s )

.WHERE( ” c k e y ” == ” n ke y ” ) )

15 }

else) and repeated (while, for-each) execution.

The core language is used as a general means for op-

erator orchestration and for the deﬁnition of user de-

ﬁned functions (UDF).

Language extensions add new operators and types

to the core language. Listing 1 deﬁnes a program that

uses elements of the relational language extension to

deﬁne and compose simple SQL-like queries. The

extension provides statements such as SELECT, FROM,

and WHERE and adds additional deﬁnitions for stan-

dard operators as < and == that are used when needed.

Listing 2 demonstrates the composition of state-

ments from multiple language extensions. The ﬁrst

six lines use the relational extension to select three

columns of the relation objects and store them in C.

Lines 7-14 use operators and types of the linear alge-

bra extension (*, sum, Mat etc.) to create a rotation

matrix R and to multiply R with a transposed copy of

C. Lines 16-24 use the ColMap operator of the matrix

format extension to apply a UDF to each column vec-

DATA 2016 - 5th International Conference on Data Management Technologies and Applications

190

Listing 2: Multi-domain program.

1 v a l C =

SELECT( ” x ” , ” y ” , ” z ” )

3 .FROM( ” o b j e c t s ” )

. ORDER BY( ” i d ” )

v a l a = 0 . 8

7 v a l R = Mat ( 3 , 3 ) (

c o s ( a ) , 0 , s i n ( a ) ,

9 0 , 1 , 0 ,

− s i n ( a ) , 0 , c o s ( a )

11 )

13 v a l Cr = R ∗ t r a n s p o s e ( C )

15 v a l Crn = ColMap ( Cr ) {

( c o l ) =>

17 v a l l e n g t h =

s q r t ( c o l ( 0 ) ∗ c o l ( 0 )

19 + c o l ( 1 ) ∗ c o l ( 1 )

+ c o l ( 2 ) ∗ c o l ( 2 ) )

r e t u r n Mat ( 3 , 1 ) (

23 c o l ( 0 ) / l e n g t h ,

c o l ( 1 ) / l e n g t h ,

25 c o l ( 2 ) / l e n g t h

)

27 }

tor of Cr.

The transpose operation in line 12 is deﬁned on

matrices and therefore implies a format transforma-

tion of C which is initially stored in a table format. Al-

though ColMap on line 16 is not deﬁned in the linear

algebra extension, but instead in the matrix format ex-

tension, it does not require a transformation because

both extensions operate on the same physical format.

In our current model, format transformations are in-

serted implicitly by the compilation framework and

do not require user intervention.

2.2 Translation and Optimization

The efﬁciency and performance of the multi-domain

architecture is predicated on the ability of the transla-

tion and optimization layer to compile multi-domain

programs into efﬁcient operator workloads. Domain

speciﬁc optimizations, efﬁcient use of format trans-

formations, and the generation of code that limits run-

time dynamism, are the compiler’s main techniques to

achieve this goal.

The general compilation process consists of three

steps: i) derive an abstract, tree-shaped intermediate

representation (IR) from the input program, ii) apply

optimization rules to the IR, iii) traverse the IR tree

and invoke code generation rules that match the en-

countered node types.

Mul

M1 M2

(a) (M1 * M2) * v

Mul

M1 Mul

M2 v

(b) M1 * (M2 * v)

Figure 2: Simple Tree IR optimization.

When a program is transformed into the tree IR,

all operator and function calls, as well as their argu-

ments, are turned into IR nodes. Each IR node has

a type that identiﬁes the kind of operator, function

or object that the node represents. Figure 2 shows

two simpliﬁed tree IR representations of the linear

algebra expression M1 ∗ M2 ∗ v. Tree 2(a) is the re-

sult of the default left to right evaluation order of the

matrix multiplication operator. Code generation for

that version would perform the matrix-matrix mul-

tiplication ﬁrst (res0 ← M1 ∗ M2) and the matrix-

vector multiplication second (res1 ← res0 ∗ v). On

the other hand, linear-algebra domain knowledge tells

us i) that vector-matrix multiplications are in general

much cheaper than matrix-matrix multiplications, ii)

that the result of a matrix-vector multiplication is an-

other vector, and iii) that matrix multiplications are

associative. In summary, domain knowledge tells us

to perform the matrix-vector multiplication ﬁrst (tree

2(b)), in order to replace the expensive matrix-matrix

multiplication with a cheaper matrix-vector multipli-

cation.

Compilers of general purpose languages do not in-

corporate domain knowledge for special purpose data

structures such as matrices, relations, or graphs. They

are therefore unable to perform domain speciﬁc op-

timizations as the one we just described. The com-

piler framework of the multi-domain architecture on

the other hand, is explicitly designed to support a spe-

ciﬁc set of data processing domains and therefore in-

cludes the types and operations of these domains as

IR nodes. Once these IR nodes are available, pattern

matching can be used to deﬁne the matrix multiplica-

tion optimization and similar rules.

Transformations that require larger parts of the

tree as context can be implemented using dedicated

IR tree traversals. This technique is especially impor-

tant during the translation of abstract domain speciﬁc

operations such as SELECT, FROM, WHERE into con-

crete operations of the storage and processing layer.

The transition from abstract domain speciﬁc to con-

crete physical operations is carried out, once all do-

main speciﬁc optimizations have been applied.

At that point, a lowering traversal replaces do-

main nodes with operator nodes, in a process that of-

Architecture of a Multi-domain Processing and Storage Engine

191

ten involves the merging of multiple domain nodes

into a single operator node or vice versa the splitting

of one domain node into several operator nodes. For

example, relational selection and projection nodes can

often be merged into a single table scan node, but a

single join node requires multiple scan nodes. When

all domain speciﬁc IR nodes have been translated into

physical operator nodes, additional physical transfor-

mations can be carried out. Eventually, the IR tree

reaches its ﬁnal form and a last traversal invokes code

generation rules for each node to generate the exe-

cutable storage and processing layer workload.

2.2.1 Physical Format Transformations

One aspect that has been left open so far, is physi-

cal format transformation. In listing 2 of section 2.1,

the linear algebra operation transpose is applied to

a relation C. The relation consists of three numeri-

cal columns X, Y, Z. Therefore there is a semantically

sound transformation of the relation into a 1x3 ma-

trix representation. On a technical level, there are two

ways to make this call legal in the statically typed

multi-domain language: i) there is an implicit type

conversion from relations to matrices and ii) there are

multiple overloaded versions of transpose, one of

which accepts a relation as its input type and possibly

also returns a relation as its output.

Option i) implies the existence of dedicated trans-

formation operations that can be represented explic-

itly in the IR tree. This approach has the advantage

that a single set of transformation operations can be

reused wherever a type mismatch occurs. The trans-

formation operations can even be targeted in opti-

mization rules and, depending on the implementa-

tion, can be potentially merged with other operators.

One disadvantage of this approach is that possibly ex-

pensive format transformations are carried out inde-

pendent of the cost of subsequent operations. Some-

times, the performance penalty of an operation that is

overloaded for a sub-optimal physical representation

might be much smaller than the added transformation

overhead.

Option ii) is to overload domain speciﬁc opera-

tions for multiple physical formats, accepting the de-

graded performance of some of these implementa-

tions. This approach is beneﬁcial in cases where the

cost of a format transformation outweighs its beneﬁt.

The largest disadvantage of this approach is the neces-

sity to provide multiple implementations for domain

operations.

The ideal solution will probably combine both ap-

proaches into a single solution. For example, the com-

piler could use a cost model to decide whether to use

a format transformation or an overloaded version of

an operator. In general, the topic of format transfor-

mations is still an open research question of the archi-

tecture and we count on future work to ﬁnd the best

and most practical approach.

2.3 Storage and Processing

The storage and processing layer at the bottom of the

multi-domain architecture deﬁnes the physical data

management API. All higher-level domain speciﬁc

constructs have to be mapped onto this API eventu-

ally. The data management API consists of a set of

operators that take a number of arguments and an op-

erator orchestration language that is used to compose

individual operators into larger workloads.

The ﬁrst argument of an operator always identi-

ﬁes the data container on which the operator is to be

applied. A data container stores the physical data of

a single logical data object in a speciﬁc data format.

For example, a RowContainer stores a relation in row

format and a SparseMatrixContainer stores a ma-

trix in a sparse format.

Some operators, such as InsertColumnRecord,

perform a simple predeﬁned function on their data

container. However, the largest class of operators

does not implement a ﬁxed functionality, but instead

applies a user deﬁned function (UDF) to a data con-

tainer. We call these operators processing operators.

Each processing operator implements a well-known,

reusable, and efﬁcient data access pattern on its un-

derlying physical format. This is the most important

property of processing operators and their main ben-

eﬁt compared to classical data retrieval operators that

do not combine data access and processing.

The UnorderedScan(Container, UDF(Record))

operator, that is deﬁned for both relational formats,

is a typical example of a processing operator. The

operator applies its UDF to batches of records of the

target container in arbitrary order. The UDF can not

make assumptions regarding which speciﬁc records or

what number of records will be included in a batch.

Typical use cases of the UnorderedScan operator are

therefore limited to the context of a single record, as

the relational projection and selection operations.

The data access pattern of the UnorderedScan op-

erator can be summarized as: unordered access to in-

dividual records. Knowledge of this access pattern

enables a scaleable data-parallel operator implemen-

tation where UDFs are applied concurrently to mul-

tiple batches of records. Other operators provide dif-

ferent guarantees to their UDFs, but the guiding prin-

ciple is always to enable some form of scaleable data

parallel processing.

The ability to achieve implicit data parallelism

DATA 2016 - 5th International Conference on Data Management Technologies and Applications

192

is the main beneﬁt of known data access patterns,

but not the only one. Another beneﬁt is the possi-

bility to fuse multiple equivalent operator calls into

a single one. Two UnorderedScan operations on

the same input container can be fused into a sin-

gle UnorderedScan by simply concatenating the two

UDFs.

The remaining component of the storage and pro-

cessing layer is the operator composition language

which is used to combine operators into larger pro-

grams. In our current model we use a procedural lan-

guage for this task. This is a straightforward choice as

it enables trivial compilation of the multi-domain core

language of the programming interface layer. If this

choice should turn out to be problematic, we might

switch to a more restricted model in the future.

3 RELATED WORK

In the following, we provide a brief discussion of re-

lated work on multi-domain processing and genera-

tive programming.

3.1 Multi-domain Processing

The need for multi-domain data management systems

has been widely recognized. In this article, we pro-

pose to tightly integrate multiple storage formats and

programing models into a single system. An alter-

native approach that has been discussed lately is the

integration of several DBMS behind a data manage-

ment middleware layer.

The BigDAWG polystore system (Duggan et al.,

2015) provides an example for this approach. The

authors hide multiple ”of the shelf” DBMS behind

a central management layer, which deﬁnes a uniﬁed

querying interface for all attached systems. The man-

agement layer accepts multi domain queries, splits

these queries into parts that can be processed by one

of the attached DBMS and sends the partial queries

to the respective engines. The management layer also

handles cases, where one DBMS depends on data that

is currently stored in another DBMS and initiates the

necessary data transfers.

In contrast to the multi-domain architecture, Big-

DAWG does not need to reimplement data manage-

ment functionality and can instead reuse proven so-

lutions. What is more, BigDAWG can incorporate

DBMS that vary widely in important systems char-

acteristics. BigDAWG could for example attach a re-

lational in-memory DBMS and a classical ﬁle based

DBMS at the same time and trade off processing

speed versus durability guarantees on a per table ba-

sis.

On the other hand, data exchange between DBMS

is a rather expensive operation as it depends on net-

work communication. Frequent format changes are

therefore much more feasible in the integrated single

system approach, proposed by the multi-domain ar-

chitecture. Furthermore, BigDAWG implies the ad-

ministration of multiple separate DBMS, which in-

creases the management cost compared to our inte-

grated approach.

To summarize, the middleware approach provides

greater ﬂexibility with regard to non-functional prop-

erties but incurs a higher price for format transfor-

mations. In addition, administration of the system is

more complex.

3.2 Generative Programming

Many big data applications repeatedly execute the

same lines of code for millions or billions of data el-

ements. Even expensive optimization becomes viable

in that environment as their cost is amortized over

time. This realization has sparked interest in runtime

code compilation and compiler based optimizations.

These two techniques trade additional one time com-

pilation overhead for very efﬁcient code that saves a

couple of instructions for every data element.

Beckmann et al. introduce an embedded DSL

for C++ that can be compiled, optimized, and exe-

cuted at runtime of a host program (Beckmann et al.,

2004). They use the DSL to write image manipula-

tion kernels that get optimized for speciﬁc transfor-

mation matrices and generate code that signiﬁcantly

outperforms standard solutions, given large enough

image sizes. The primary efﬁciency gains of the gen-

erated code are based on removed indirections and

runtime checks that are unavoidable in more general

solutions. Newburn et al. follow a very similar ap-

proach and provide an embedded C++ DSL that is

speciﬁcally targetted at data parallelism in multi-core

systems (Newburn et al., 2011). They use code gener-

ation to specialize performance critical code passages

to the speciﬁc runtime environment of their programs.

Another group (Alexandrov et al., 2015) uses code

generation, domain speciﬁc languages, and speciﬁc

data access patterns to derive implicit parallelism in

an approach that is very similar to the one described in

this article. On the other hand they are not concerned

with multi-domain integration and do not optimize for

specialized data formats.

Architecture of a Multi-domain Processing and Storage Engine

193

4 SUMMARY AND FUTURE

WORK

In this article we have proposed a programming

model and system architecture for a data management

system that integrates multiple specialized data do-

mains and programming models into an efﬁcient and

ﬂexible end to end solution.

In the language layer, at the top of our architec-

ture, we extend a ﬂexible host language with a set of

embedded domain speciﬁc languages that cover dif-

ferent use cases.

Underneath the language layer, we use a compiler

framework to translate abstract multi-domain pro-

grams into efﬁcient physical workloads. The frame-

work incorporates domain knowledge which makes it

possible to apply powerful domain speciﬁc optimiza-

tion rules. In addition, the compiler generates format

transformation code to enable the composition of op-

erations that are deﬁned on different physical repre-

sentations.

At the bottom of the architecture, we propose to

use a storage and processing engine that supports

multiple optimized physical data formats. Actual data

access is implemented by a set of processing opera-

tors that apply user deﬁned functions to data objects

according to predeﬁned data access patterns. These

patterns enable data parallel operator implementa-

tions and advanced optimizations such as operator fu-

sion.

We are currently in the process of developing a

prototypical implementation of the proposed architec-

ture. In order to fully focus on the multi domain as-

pects of our approach, we reuse previous work as ex-

tensively as possible possible. Most importantly, we

use the in-memory data store ERIS (Kissinger et al.,

2014) as the starting point for our multi format data

management engine. ERIS uses data parallel process-

ing operators to achieve vertical scaleability on large

shared memory multi-processor machines. In addi-

tion, ERIS already provides column and row formats

for the storage of relations and is therefore designed

to support multiple physical storage formats. We plan

to extend ERIS and add an additional matrix format to

support the important use case of relational and linear

algebra integration.

The compilation layer uses the DSL compilation

framework LMS (Rompf and Odersky, 2010). LMS

uses Scala as host language for embedded DSLs and

provides the generation of a tree IR with custom node

types. Further, LMS deﬁnes a ﬂexible component

based extension mechanism, which allows the inte-

gration of new DSL elements, IR nodes, optimization

rules, and code generation.

We propose the multi-domain architecture as our

vision for a data management system that provides

tight and efﬁcient integration of multiple processing

domains. In future work, we want to evaluate the im-

pact of frequent format transformations on the perfor-

mance of the system and expect to ﬁnd speciﬁc multi-

domain optimizations to mitigate these effects.

ACKNOWLEDGEMENT

This work is partly funded by the German Fed-

eral Ministry of Education and Research (BMBF)

in VAVID project under grant 01IS14005 as well as

by the German Research Foundation (DFG) in the

Collaborative Research Center 912 Highly Adaptive

Energy-Efﬁcient Computing.

REFERENCES

Abadi, D. et al. (2014). The beckman report on database

research. ACM SIGMOD Record, 43(3):61–70.

Alexandrov, A., Kunft, A., Katsifodimos, A., Sch

uler, F.,

Thamsen, L., Kao, O., Herb, T., and Markl, V. (2015).

Implicit parallelism through deep language embed-

ding. In Proceedings of the 2015 ACM SIGMOD

International Conference on Management of Data,

pages 47–61. ACM.

Beckmann, O., Houghton, A., Mellor, M., and Kelly, P. H.

(2004). Runtime code generation in c++ as a foun-

dation for domain-speciﬁc optimisation. In Domain-

Speciﬁc Program Generation, pages 291–306.

Duggan, J., Elmore, A. J., Stonebraker, M., Balazinska, M.,

Howe, B., Kepner, J., Madden, S., Maier, D., Mattson,

T., and Zdonik, S. (2015). The bigdawg polystore sys-

tem. ACM SIGMOD Record, 44(2):11–16.

Kissinger, T., Kiefer, T., Schlegel, B., Habich, D., Molka,

D., and Lehner, W. (2014). ERIS: A numa-aware in-

memory storage engine for analytical workload. In

ADMS Workshop at VLDB, pages 74–85.

Newburn, C. J., So, B., Liu, Z., McCool, M., Ghuloum,

A., Toit, S. D., Wang, Z. G., Du, Z. H., Chen, Y.,

Wu, G., et al. (2011). Intel’s array building blocks:

A retargetable, dynamic compiler and embedded lan-

guage. In Code generation and optimization (CGO),

2011 9th annual IEEE/ACM international symposium

on, pages 224–235. IEEE.

Rompf, T. and Odersky, M. (2010). Lightweight modular

staging: a pragmatic approach to runtime code gen-

eration and compiled dsls. In Acm Sigplan Notices,

volume 46, pages 127–136. ACM.

DATA 2016 - 5th International Conference on Data Management Technologies and Applications

194