PANGAEA

An ICSU World Data Center as a Networked Publication and Library System for

Geoscientiﬁc Data

Michael Diepenbroek, Uwe Schindler

MARUM, Universitt Bremen, Leobener Str, D-28359 Bremen, Germany

Hannes Grobe

Alfred Wegener Institute for Polar and Marine Research, Am Handelshafen 12, D-27570 Bremerhaven, Germany

Keywords:

Digital libraries, world data center, data publisher, data portals.

Abstract:

Since 1992 PANGAEA



serves as an archive for all types of geoscientiﬁc and environmental data. From the

beginning the PANGAEA group started initiatives and aimed at an organisation structure which – beyond the

technical structure and operation of the system – would help to improve the quality and general availability of

scientiﬁc data. Project data management is done since 1996. 2001 the ICSU World Data Center for Marine

Environmental Sciences (WDC-MARE) was founded and since 2003 – together with other German WDC

– the group was working on the development of data publications as a new publication type. To achieve

interoperability with other data centers and portals the system was adapted to global information standards.

PANGAEA



has implemented a number of community speciﬁc data portals. 2007 – under the coordination of

the PANGAEA



group – an initiative for networking all WDC was started. On the long range ISCU supports

plans to develop the WDC system into a global network of publishers and open access libraries for scientiﬁc

data.

1 INTRODUCTION

Data centers were created with the motivation to as-

sure the long-term availability of scientiﬁc data. The

Geophysical Year 1957 had been the starting point for

the foundation of the system of World Data Centers

(WDC), a number of globally distributed data cen-

ters, which were supposed to archive and distribute

the geophysical data produced in that and the fol-

lowing years. Since then, the WDC system, which

is related to the International Council for Scientiﬁc

Unions (ICSU), has been extended to more than 50

data centers covering all ﬁelds of geosciences. More

recently, ICSU expects the system to go through a

major revision process. The exponentially increas-

ing data volumes and the development of the Internet

led to many new data managing and archiving sys-

tems. One of them was the Publishing Network for

Geoscientiﬁc and Environmental Data PANGAEA

1

(Diepenbroek et al., 2002), implemented in 1992. In

2001 the PANGAEA group founded the World Data

Publishing Network for Geoscientiﬁc and Environmen-

tal Data. http://www.pangaea.de

Center for Marine Environmental Sciences (WDC-

MARE)

From the beginning PANGAEA



was conceived

as a system that could cope with a wide spectrum of

observational data. The heterogeneity and dynamics

of the geosciences (including biology) required a ﬂex-

ible system for the acquisition, processing and archiv-

ing of the various data.

Nevertheless, already in the ﬁrst phase of imple-

mentation it became clear that an efﬁcient technical

system is a necessary prerequisite, but cannot solve

the principal problems of data quality and availability.

Following the principle of open access (ESF, 2000;

President of the Max Planck Society, 2003; OECD,

2004) scientiﬁc primary data are – besides publica-

tions – the second important result that must be long-

term available in a re-usable state. A few decades

ago it was still usual to publish primary data directly

within a publication. Due to increasing data volumes

and the transition to electronic publishing this practice

was left. Scientiﬁc publishers allow for storing pri-

World Data Center for Marine Environmental Sciences.

http://www.wdc-mare.org

149

Diepenbroek M., Schindler U. and Grobe H. (2008).

PANGAEA - An ICSU World Data Center as a Networked Publication and Library System for Geoscientiﬁc Data.

In Proceedings of the Fourth International Conference on Web Information Systems and Technologies, pages 149-154

DOI: 10.5220/0001518401490154

 SciTePress

mary data as electronic assets. Nevertheless, archiv-

ing is not compliant to any standards or unique struc-

tures and is excluded from peer review, hence, can

also not be seen as a template for a general solution

of the problem. In contrast, many data centers includ-

ing a good part of the ICSU WDC, are well prepared

in a technical sense, although, archiving mostly does

not comply to global standards either. The separation

of scientiﬁc publications from the underlying primary

data can be seen as a severe structural problem in the

empirical sciences. It hampers not only the evaluation

of a publication but also re-usage of results.

There are no authorized and authenticated places

for the long-term storage of scientiﬁc data, no cross-

referencing between scientiﬁc publications and pos-

sibly archived data and no or only rudimentary net-

working between data centers. Needed are global li-

brary structures and systems for the publication of sci-

entiﬁc data. In this context the ISCU WDC play an

active role. The German WDC-Climate, WDC Re-

mote Sensing, and WDC-MARE together with the

GeoForschungsZentrum Potsdam and the Technical

Library in Hannover have implemented a practical

system for the publication of scientiﬁc data (Schindler

et al., 2005). In this connection WDC-MARE with

the information system PANGAEA



and its editorial

system already can be seen as a reference for a pub-

lication and library system for scientiﬁc data. In ad-

dition, due to its interoperability, PANGAEA



is net-

worked with various other data centers, libraries, por-

tals, and services. In the following it will be described

in more detail.

2 FROM DATA AQUISITION TO

PUBLICATION

WDC-MARE / PANGAEA



is operated as a perma-

nent facility by the Center for Marine Environmen-

tal Sciences (MARUM) of the University Bremen and

the Alfred Wegener Institute for Polar and Marine Re-

search (AWI) in Bremerhaven. 4 scientists are re-

sponsible for the organisation and development of the

system. A team of 8-10 scientists take care of the

data management services, which are supplied on an

international level since 1996. Until 10/2007, PAN-

GAEA



was and is partner in more than 60 European

to international projects covering all ﬁelds of environ-

mental sciences. The budget amounts approximately

1.2 Mio Euro per year for personnel, hard-, and soft-

ware. Third party funds are about 70% of the total

budget.

3 AQUISITION, QUALITY

ASSURANCE, EDITORIAL AND

ARCHIVING

The acquisition of scientiﬁc data is a time consuming

problem. Based on own estimates only a few percent

of the globally produced scientiﬁc data are generally

available and even less is long-term archived in ade-

quate data centers. Seldom, data are spontaneously

handed out to a data center. For scientiﬁc institu-

tions there is – since several years – an obligation for

long-term storage of data. Likewise, many projects

and programs are conﬁgured with corresponding con-

straints. Agreements in such contexts facilitate data

acquisition, however, cannot solve the problem com-

pletely.

On the other hand data management as a funded

component of scientiﬁc projects has proven rather

efﬁcient. For EU projects addressing environmen-

tal research data management is an important eval-

uation criterion. For projects like e.g. CAR-

BOOCEAN

,which aim at improved quantiﬁcations

of CO

balances in the marine environment, a high

availability of quality assured data is a necessary pre-

requisite for the success of the project. In general,

large scale or complex scientiﬁc approaches in Global

Change research are based on the results and data of

many smaller projects.

The PANGAEA



group is supplying project data

management since more than 10 years. This is the

most important source for new data to be archived,

mostly because of the proximity with the scientists. In

addition, project data management considerably con-

tributes to the operational costs of PANGAEA



. This

creates capacities, which enables the group to realize

also not funded projects as e.g. the ﬁnal global har-

monisation, archiving, and publication of data from

the IGBP project Joint Global Ocean Flux Studies

(JGOFS) (Sieger et al., 2005).

Quality assurance is an indispensable part of data

management. Essential in this respect is not the data

quality itself, but assessment of the data quality. Im-

portant are completeness und correctness of data de-

scriptions (metadata) and compliance with existing

content standards as ISO19115 (Kresse and Fadaie,

2004) or DIF

. At the minimum the metadata have to

answer the question: Who has measured what, when,

where, and how? In addition, PANGAEA



regularly

checks the validity of used methods and whether the

precision of data values corresponds with the meth-

CARBOOCEAN. http://www.carboocean.org/

Directory Interchange Format.

http://gcmd.nasa.gov/User/difguide/

WEBIST 2008 - International Conference on Web Information Systems and Technologies

150

ods used. Outliers are identiﬁed and ﬂagged. The

data producer (principal investors or institution) take

the responsibility for the actual quality of the data.

Editing and archiving of data sets varies with data

types and data centers. Practically, there are neither

archiving standards nor are there editorial systems,

which could be generally used. Common is usage

of relational data bases, which guarantees at least a

certain consistency of metadata. At present, almost

Water

Sediment

Ice

Corals

Atmosphere

unclassified

Figure 1: Contents of PANGAEA



. (9/2007): Data for

≈ 30 000 parameters (e.g.: sediment & ice proﬁles, seismic

proﬁles, atmospheric proﬁles, ocean geochemistry, min-

eral distributions, geological maps, plankton & ﬁsh, sea

ﬂoor pictures and ﬁlms), data sets: 570 676, data items:

1 834 869 117.

600 000 data sets with nearly 2 billion observations

(numerical, text, or binary data items) are available.

The data are related to about 30 000 different mea-

surement types (parameter), more than 10 000 princi-

pal investigators (PI), about 6 000 scientiﬁc publica-

tions, and more than 300 000 sample locations. The

yearly increase is more than 10% of the total inven-

tory (see ﬁgure 1).

In PANGAEA



data and metadata are systemati-

cally recorded through an editorial system. The sys-

tem contributes signiﬁcantly to the efﬁciency of data

curation. For smaller data centers with relatively spe-

cialized data contents such a system might be dis-

pensable. PANGAEA



, however, was conceived as

a large scale system to handle various types of data.

On the server side the challenge of managing

the heterogeneous and dynamic data of environmen-

tal and geosciences was met through a ﬂexible data

model, which reﬂects the information processing

steps in the earth science ﬁelds and can handle any

related analytical data. The basic technical structure

corresponds to three tiered client/server architecture

with a number of clients and middleware components

controlling the information ﬂow and quality. A re-

lational database management system (RDBMS) is

used for information storage. Physical backups are

regularly stored in different locations, thus protecting

the data inventory from loss. Figure 2 shows the sim-

pliﬁed setup of PANGAEA



. Mass data, like geo-

physical data or binary objects, as e.g. pictures and

ﬁlms, are stored on hard disk arrays from where they

eventually migrate into related tape silos. All data

are replicated on a frequent base into a data ware-

house (Sybase IQ). This enables high-performance

retrievals of any space/time or keyword constrained

section of the data inventory. The compiled metadata

are part of the search results. The web-based clients

include a simple search engine and an IQ interface

which will be productive by the end of 2007.

Sybase

ASE

Middleware Webserver

Editorial

system

PANGAEA

engine

Harddisk

+ tape (silo)

RDB

Sybase

RDB

interface

…

Figure 2: Technical setup of PANGAEA



4 DATA PUBLICATION

Within the last three years the PANGAEA



group to-

gether with the WDC-RSAT, WDC-Climate, and the

German Technical Library (TIB) has developed and

prototypically implemented a concept for the publica-

tion of scientiﬁc data (Schindler et al., 2005; Klump

et al., 2006). The project – funded by the German

Science Foundation (DFG) – investigated general re-

quirements for this new publication type:

• The formal structure of the publication, that is,

which describing elements are mandatory, which

are optional, how should they be conﬁgured,

which data formats and standards are useful?

• The granularity of data sets to be published.

• The development of ”peer review” like procedures

for quality assurance.

• The requirement for data centers with respect to

long-term archiving and persistent referencing of

archived data, e.g. through “Digital Object Iden-

tiﬁer” (DOI). Certiﬁcation of data centers is – be-

sides own experiences – essentially based on the

OAIS reference model (NASA Consultative Com-

mittee for Space Data Systems, 2002) and results

from the German BMBF project NESTOR

NESTOR. http://www.langzeitarchivierung.de/

PANGAEA - An ICSU World Data Center as a Networked Publication and Library System for Geoscientific Data

151

The results were used as guidelines in the partici-

pating facilities to adapt organisational and technical

structures, in particular to develop editorial schemes

for the import and curation of data. Such schemes

were prototypically realized in all data centers. They

are, however, more or less – depending on the facil-

ity – integrated into the technical environment and the

scientiﬁc process. In this respect the above mentioned

problem of granularity is crucial. A principal prob-

lem is that data centers traditionally treat their data

archives as a continuously extendible and updatable

data space which does not allow for a subdivision

into static data entities. With data publishing, how-

ever, persistence and version control of data entities

are needed. So far, the WDC in the data publication

project have agreed on a simple model which differ-

entiates between archived or accessible data entities

and citable data entities: Archived data entities can

be citable or may be comprised to citable data enti-

ties in a second step. Citable data entities represent

the interface between data archive and scientiﬁc liter-

ature. They allow for cross-referencing data publica-

tions and traditional scientiﬁc publications.

Legacy data are a further problem. For WDC-

MARE / PANGAEA



a signiﬁcant effort is needed

to replenish the whole data inventory in a way to get

citable data sets in the end. Each data set needs con-

sultation with the original PI(s) or further scientists

from the corresponding research ﬁeld and eventually

manual changes on the metadata. The current work

led to ﬁrst trials of a “peer review” for scientiﬁc data.

All data sets are annotated with a Digital Object

Identiﬁer (DOI) and are registered at the DOI reg-

istry for scientiﬁc data at the Technical Library in

Hannover (TIB), which have a corresponding con-

tract with the International DOI Foundation (IDF)

Citable data sets are subsequently recorded in the li-

brary catalogue of the TIB

. Both, DOI registration

and migration into the library catalogue, are auto-

mated routines.

Since more than 10 years PANGAEA



uses a

client/server system for the import of new data and the

curational works. The system minimizes the manual

work for the data curators and can be used globally.

The development of the system into an editorial sys-

tem is an iterative process in which system managers,

data curators, and partners of the data publication

project are participating. Besides numerous adapta-

tions it was necessary to include a chronological se-

quence into the editorial process. Newly imported

data sets are not registered immediately but with a

time-lag of 28 days, allowing for further changes on

International DOI Foundation. http://www.doi.org

TIBORDER. http://www.tib-hannover.de/en/

or replacement of data sets. After expiry of the time-

limit data sets are registered and might be ﬂagged as

citable. Except for some minor metadata elements

data sets are subsequently static. The DOI registration

was harmoniously integrated into the existing infras-

tructure.

Overall, the necessary conversion to a publi-

cation system has been completed. The editorial

effort – except for the increased communica-

tion – has stayed about the same. This is an

important aspect with respect to the running

operational costs. Examples of citable data

sets are e.g.: doi:10.1594/PANGAEA.472287,

doi:10.1594/PANGAEA.472492,

doi:10.1594/PANGAEA.370797

5 STANDARDS, NETWORKING

AND PORTALS

Networking of data producers, archives, and con-

sumers, compliant to Global Spatial Data Infrastruc-

tures

(Nebert, 2004) is a necessary prerequisite for

geospatial one stop shops and large scale data compi-

lations. It is a vision, more recently described in the

10-year implementation plan of a Global Earth Ob-

serving System of Systems (GEOSS) (Battrick, 2005)

of the Group on Earth Observations (GEO)

which –

on the ministerial level – the ﬁrst time supplies an ef-

ﬁcient framework for networking geospatial service

suppliers and users. Due to lacking resources, how-

ever, GEOSS is highly dependant on existing capac-

ities and activities. Therefore, on the meeting of the

WDC directors in 2007 it was decided to start an ini-

tiative for networking WDC. This is not only a useful

contribution to GEOSS but can also be seen as a ﬁrst

step towards the creation of a global network of data

libraries. The WDC system so far is a unique consor-

tium of data centers that supply free and unrestricted

online access on their data holdings. The WDC net-

working initiative is coordinated by PANGAEA



During the last 5 years the group has worked system-

atically on the networking capabilities of the PAN-

GAEA



system and by now supplies a variety of dif-

ferent, generally available services, all conform to

global geospatial standards (ISO, OGC und W3C).

The metadata for each data set are ’marshalled’ from

the relational database into a XML blob. The corre-

sponding scheme is proprietary. It comprises all in-

Global Spatial Data Infrastructures (GSDI).

http://www.gsdi.org/

Group on Earth Observations.

http://www.earthobservations.org/

WEBIST 2008 - International Conference on Web Information Systems and Technologies

152

data management &

longterm archiving

RDB

catalo

ues

PANGAEA

ISO19xxx

STD-DOI

XSL

Index

Dublin

Core

rotocols

marshalle

(

SOAP/WSD

Frontends /

ortals

PangaVista

+GE + UNM

WFS

(

OGC

)

OGC

catalogue

service

OAI-PMH

ISO690

GeoPortal.

Bund®

TIB National

Library

(SOAP/WSDL)

catalo

ues

DOI registry

DIF

Dublin

Core

harveste

Google

Scientific

Commons

HGF Fedora

harveste

GCMD

EUR-OCEANS

CARBOOCEAN

IODP

Darwin

Core

DiGIR

Darwin

Core

ISO19xxx

DIF

OBIS

GBIF

harveste

D-GRID

Figure 3: Metadata infrastructure for PANGAEA



. Grey shaded parts belong into the domain of PANGAEA



. Portals with

a red outline were implemented by the PANGAEA



. group.

formation needed for mapping (per XSLT) the meta-

data on to the various content standards as ISO19115

or the Directory Interchange Format (DIF), Important

protocols are the OGC Catalogue Service (CS-W)

and the Open Archives Initiatives Protocol for Meta-

data Harvesting (OAI-PMH) (Van de Sompel et al.,

2004). The latter is relatively simple to be imple-

mented and is widely used in the library world. An

overview supplies Figure 3. Because of the dynamics

of IT developments PANGAEA



deliberately builds

on an internal architecture that can cope with different

or new standards.

In addition, the PANGAEA



group has imple-

mented a number of community and project speciﬁc

metadata portals. The portal framework is generic

and based on the components harvester, indexer with

search engine (Apache Lucene

) and corresponding

API (Schindler and Diepenbroek, 2008)

. Exam-

ples are the portal for the International Ocean Drilling

OGC Catalogue Service.

http://www.opengeospatial.org/standards/cat

Apache Lucene (Hatcher and Gospodnetic, 2004).

http:// lucene.apache.org/java/docs/

PANGAEA Framework for Metadata Portals

(panFMP). http://www.panFMP.org/

Program

and for the EU projects EUR-OCEANS

and CARBOOCEAN

. A precondition for these por-

tals is that participants not only supply metadata cat-

alogues, but also enable direct and open access to the

corresponding data entities.

An even higher level of networking was reached

with the participation in the German Community

GRID C3

. In this project PANGAEA



supplies its

portal framework and contributes to the data GRID

with observational data served by the data warehouse.

Nevertheless, GRID projects are still restricted to spe-

cial data types and workﬂows. For general and simple

to be implemented architectures more development is

needed. A special problem with heterogeneous data

as supplied by PANGAEA



is the availability of stan-

dardized vocabularies for the control of applications.

Corresponding concepts are supplied by ISO19109

and ISO19110 (Kresse and Fadaie, 2004). Practical

progress can be expected through the European ini-

Scientiﬁc Earth Drilling Information Service - SEDIS

(Miville et al., 2006). http://sedis.wdc-mare.org/

EUR-OCEANS data portal. http://dataportal.eur-

oceans.eu/

CARBOOCEAN data portal.

http://dataportal.carboocean.org/

Collaborative Climate Community Data and Process-

ing Grid. http://www.c3grid.de/

PANGAEA - An ICSU World Data Center as a Networked Publication and Library System for Geoscientific Data

153

tiative INSPIRE

(The European Parliament, 2007).

This, however, must be regarded as a long-term task.

6 CONCLUSIONS

With its long-term and secured archiving structure,

the highly efﬁcient editorial system, and the exten-

sive interoperability with other data centers and por-

tals, PANGAEA



has developed into an exemplary

publication and library system for scientiﬁc data. The

approach for publication of scientiﬁc data developed

within the German WDC consortium and realized

within PANGAEA



, is way beyond the usual inter-

linking of scientiﬁc publications with related data as

e.g. practiced within the Human Genome Commu-

nity. It allows for self-contained data publications.

Each data publication is provided with a meaningful

citation and a persistent identiﬁer (DOI) und thus en-

ables reliable references. The citability gives a strong

motivation for scientists to publish their data. It is a

bottom-up approach which on the long range will im-

prove data quality and availability.

The concept met with wide response from data

producers. Nevertheless, it might take years for this

new publication type to be generally accepted. First

talks with ISI Thompson have indicated that data pub-

lications might be recognized for the citation index.

The reference systems, developed within the German

WDC, need to be extrapolated. With the network-

ing initiative of ICSU WDC a ﬁrst step is done in

the direction of a global library consortium for scien-

tiﬁc data. Such a network would be trans-disciplinary

and has the advantage that all data are available with-

out any restriction according to the open access rules.

However, a sustainable framework is needed on the

one hand to guarantee long-term availability of sci-

entiﬁc data and on the other hand to foster the work

in the data centers in the direction of standards for

processing, archiving, and publication of data as well

as interoperability of data centers. The revision of

ICSU WDC will support such a framework. Nev-

ertheless, long-term operation requires further safe-

guarding through national or international contracts.

A memorandum of understanding could be a good

starting point.

REFERENCES

Battrick, B. (2005). Global Earth Observation System of

Systems (GEOSS) 10-Year Implementation Plan Ref-

erence Document. ESA Publications Division.

INSPIRE. http://www.ec-gis.org/inspire/

Diepenbroek, M., Grobe, H., Reinke, M., Schindler, U.,

Schlitzer, R., Sieger, R., and Wefer, G. (2002).

PANGAEA–an information system for environmental

sciences. Computers & Geosciences, 28(10):1201–

1210.

ESF (2000). Good scientiﬁc practice in research and schol-

arship.

Hatcher, E. and Gospodnetic, O. (2004). Lucene in Action.

Manning Publications.

Klump, J., Bertelmann, R., Brase, J., Diepenbroek, M.,

Grobe, H., Hck, H., Lautenschlager, M., Schindler,

U., Sens, I., and Wchter, J. (2006). Data publication

in the open access initiative. Data Science Journal,

5:79–83.

Kresse, W. and Fadaie, K. (2004). ISO Standards for Geo-

graphic Information. Springer, Heidelberg.

Miville, B., Soeding, E., and Larsen, H. C. (2006). Sci-

entiﬁc Earth Drilling Information Service for the In-

tegrated Ocean Drilling Program. Geophysical Re-

search Abstracts, 8:05486.

NASA Consultative Committee for Space Data Systems

(2002). Reference Model for an Open Archival In-

formation System (OAIS).

Nebert, D. D., editor (2004). The SDI Cookbook, Version

2.0. Global Spatial Data Infrastructure Association,

Technical Working Group Chair.

OECD (2004). Science, Technology and Innovation for the

21st Century. In Meeting of the OECD Committee

for Scientiﬁc and Technological Policy at Ministerial

Level, 29-30 January 2004 - Final Communique.

President of the Max Planck Society (2003). Berlin Decla-

ration on Open Access to Knowledge in the Sciences

and Humanities.

Schindler, U., Brase, J., and Diepenbroek, M. (2005). Web-

services Infrastructure for the Registration of Scien-

tiﬁc Primary Data. In Rauber, A., Christodoulakis,

S., and Tjoa, A. M., editors, Research and Advanced

Technology for Digital Libraries, volume 3652 of

Lecture Notes in Computer Science, pages 128–138.

Springer.

Schindler, U. and Diepenbroek, M. (2008). Generic XML-

based Framework for Metadata Portals. Computers &

Geosciences. Submitted.

Sieger, R., Grobe, H., Diepenbroek, M., Schindler, U., and

Schlitzer, R., editors (2005). International Collection

of JGOFS – Volume 2: Integrated Data Sets (1989-

2003). Number 0003 in WDC-MARE Reports. WDC-

MARE.

The European Parliament (2007). DIRECTIVE 2007/.../EC

Of the European Parliament and of The Council of es-

tablishing an Infrastructure for Spatial Information in

the European Community (INSPIRE). Directive not

yet ofﬁcially released.

Van de Sompel, H., Nelson, M., Lagoze, C., and Warner,

S. (2004). Resource Harvesting within the OAI-PMH

Framework. D-Lib Magazine, 10(12).

WEBIST 2008 - International Conference on Web Information Systems and Technologies

154