SWS CHALLENGE

First Year Overview

Charles Petrie

∗

, Holger Lausen

†

and Michal Zaremba

†

∗

Computer Science Dept., Gates Building, Stanford, CA 94305-9020, USA

†

DERI Innsbruck, University of Innsbruck, Austria

Keywords:

SWS, Semantic Web, Web Services, Challenge, Methodologies.

Abstract:

This paper provides an overview of the results of the ﬁrst year of the Semantic Web Services Challenge.

Other papers in this same conference proceedings provide detail of participant contributions as well as lessons

learned. This paper is an introduction to the Challenge purpose, organization, methodology, problems, and

solutions.

1 SWS CHALLENGE MISSION

AND ORGANIZATION

Service-Oriented Computing is one of the most

promising software engineering trends for future dis-

tributed systems. Pushed by major industry play-

ers and supported by many standardization efforts,

Web services are a prominent implementation of the

service-oriented paradigm. They promise to foster

reuse and to ease the implementation of loosely cou-

pled distributed applications.

Though Web services are appealing especially

in the area of enterprise application integration, the

idea of service-oriented computing and the envisioned

availability of thousands of services can not be fully

leveraged as long as service oriented applications

must be created and maintained manually. Semantic

technology may help here, by lifting service-oriented

applications to a new level of adaptability and robust-

ness. By using semantic annotations to describe ser-

vices and resources, the tasks of service discovery, se-

lection, negotiation, and binding could be automated.

Currently there are many different approaches to

semantic Web service descriptions and many frame-

works built around them, yet a common understand-

ing, evaluation scheme, and testbed to compare and

classify these frameworks in terms of their abilities

and shortcomings is still missing.

The purpose of the ongoing Semantic Web Service

Challenge (www.sws-challenge.org) is precisely to

develop this common understanding of various tech-

nologies intended to facilitate the automation of medi-

ation, choreography and discovery for Web Services

using semantic annotations. This explores trade-offs

among existing approaches, reveals the strengths and

weaknesses of the proposed approaches as well as

which aspects of the problem space are not yet cov-

ered.

The series of three workshop in the ﬁrst year of

the SWS Challenge has provided a forum for discus-

sion based on a common application. The Challenge

focuses on the use of semantic annotations: partici-

pants are provided with semantics in the form of nat-

ural language text that they can formalize and use in

their technologies. Being a challenge rather than a

contest, workshop participants mutually evaluate and

learn from each others’ approaches.

The Challenge has participating groups from in-

dustry and academia developing software components

and/or intelligent agents able to automate mediation,

choreography and discovery processes between Web

services. All approaches and participants are invited.

Though “Semantic” is in the title of this Challenge,

we invite non-semantic approaches and evenly evalu-

ate all submissions in our methodology.

407

Petrie C., Lausen H. and Zaremba M. (2007).

SWS CHALLENGE - First Year Overview.

In Proceedings of the Ninth International Conference on Enterprise Information Systems, pages 407-412

 SciTePress

2 SWS CHALLENGE

METHODOLOGY

At the SWS workshops, the approaches were pre-

sented and demonstrated, but also the code was jointly

reviewed. The common application helped develop-

ing a profound mutual understanding of each other’s

technology and the collaborative discussion of the

proﬁles of the various approaches. The participants

developed an evaluation scheme that classiﬁes the

functionality and the agility offered by the various ap-

proaches, and applied it to the participating technolo-

gies.

The SWS Challenge is an evaluation of function-

ality rather than performance. We are not interested in

how fast a particular piece of code works. Certainly

algorithms for analyzing web services are relevant,

but algorithms can be analyzed in terms of complexity

apart from code. We are interested not in the speed of

the code but in programmer productivity. For a given

change in the emerging era of Service-Oriented Ar-

chitectures, how hard will it be for programmers to

make changes in an increasingly ﬂexible and dynamic

IT environment? This Challenge seeks to understand

the advantages and tradeoffs, wrt. this question, of

various programming approaches.

There is no “winner” in these challenges, though

one can look at the results of each workshop and see

which team has so far solved the most problems with

what level of difﬁculty. This Challenge is intended

to be an objective certiﬁcation of approaches to the

problems of semantic technologies, with an emphasis

on industrial problems in order to make the technolo-

gies relevant.

Therefore the SWS Challenge is taking a soft-

ware engineering approach to evaluating Semantic

Web Services.

The working hypothesis of the se-

mantic technology community is that a semantic ap-

proach will allow a given change to be made with

less difﬁculty than with traditional coding techniques.

This is essentially a software engineering claim. Thus

we allow “all comers” to participate. If it develops

that a particular coding technique manages the prob-

lem changes of the challenge scenarios better than a

semantic approach, this will also be be valuable infor-

mation for the semantic community.

In the Challenge methodology, teams automati-

cally validate their solution to problems by having

their system send correct messages to the web ser-

vices in the SWS Challenge infrastructure. At the

workshops, teams present papers about their approach

“It’s the Programming, Stupid”, IEEE Internet Com-

puting, May-June, 2006.

http://www-cdr.stanford.edu/ petrie/online/peer2peer/w306.pdf

with claims about the ease of changing from one prob-

lem to another. Then we peer-review these claims and

agree upon an evaluation of the approach, as well as

certifying the technology problem level.

The problems are speciﬁed in English, other than

the WSDL descriptions associated with the test ser-

vices. We challenge the participating teams to de-

velop their own semantic annotation formalisms that

are sufﬁcient to solve the problems. Additionally, we

analyse the general difﬁculty in moving from solving

one problem level or sub-level to another.

This is why we use standards such as RosettaNet

and WSDL and make our scenarios at least similar

to industrial problems. It is also why we insist that

submissions actually solve the problems by sending

correct messages to the Challenge web services for

each scenario. It is easy to make claims in academic

papers that such-and-such a problem has been solved.

It turns out to be much more difﬁcult in practice, as

our teams have discovered, to make such approaches

actually work. Our slogan is “no participation with-

out invocation”. In order to be evaluated on a prob-

lem level, the submission must have demonstrated the

correct exchange of messages with the corresponding

Challenge web services.

3 SWS CHALLENGE PROBLEM

SCENARIOS

The problems being solved by the teams are busi-

ness scenarios divided into major problem levels with

sub-problem variations. The ﬁrst major problem level

consists of developing a mediator that allows a hy-

pothetical company to have its legacy web services

to conform to a RosettaNet purchase order (PO) stan-

dard that is being used by a customer. We then change

the web services, the protocol, and the order in con-

secutive variations.

In particular, we have presented two broad areas

of problem scenarios:

• The mediation scenario problems concern making

a legacy order management system interoperable

with external systems that use a simpliﬁed version

of the RosettaNet PIP3A4 speciﬁcations

• The discovery scenario problems concern the dy-

namical discovery, selection, binding, and invoca-

tion of the most appropriate shipment service for

a set of given shipment requests.

Subsequent problem levels involve increasingly

difﬁcult web service discovery and composition sce-

http://www.rosettanet.org/PIP3A4

Figure 1: Mediation Scenario.

narios. We have not yet developed all of the scenar-

ios, but in the end, the teams should process the or-

der from the company, mediating the PO process, or-

der the right parts from the suppliers, the suppliers

should ship the parts, and the company should ship

to the customer the completed order, with associated

“paperwork”.

4 THE MEDIATION SCENARIO:

PROCESS AND DATA

MEDIATION

For this challenge we focus on the very basic scenario

of purchasing goods using a simpliﬁed version of the

RosettaNet PIP3A4 speciﬁcations.

There are three main components taking part in

the scenario, as depicted in Figure 1:

• Company Blue, which is a customer (service re-

quester) ordering products,

• Mediator which is a piece of technology provid-

ing automatic or semi-automatic mediation for the

Moon company

• Legacy System of the Moon Company

While the external interfaces must follow the

RosettaNet speciﬁcation, internally Moon uses a pro-

priety legacy system in which data model and mes-

sage exchange patterns differ from those of Roset-

taNet. Participants shall basically enable Moon to

”talk RosettaNet” and implement the Purchase Order

receiving role part of the interaction described in the

RosettaNet PIP 3A4.

Both the Moon legacy systems and the customer

Web services (Blue) are provided by the challenge or-

ganizers as technical infrastructure running at DERI

in Innsbruck and accessible online, and can not in

principle be altered (although their description may

be semantically enriched). The sketch of the media-

tor of Fig. 1shall be implemented by the participants.

In the mediation scenario, Moon uses two back-

end systems to manage its order processing, namely a

Customer Relationship Management system (CRM)

and an Order Management System (OMS). The chal-

lenge provides access to both systems through public

Web services described using WSDL. The scenario

describes how Moon has signed agreements to ex-

change purchase order messages with its client com-

pany called Blue using the RosettaNet PIP 3A4 spec-

iﬁcation.

In order to address integration of Blue and Moon

companies, the participating groups are encouraged

to use SemanticWeb service technology to facilitate

conversation between all systems, to mediate between

the PIP 3A4 and the XML schema used by Moon, as

well as to ensure that the message exchange between

all parties is correctly choreographed. In particular,

• Data mediation is involved in mapping the Blue

RosettaNet PIP 3A4 message to the messages of

the Moon back-end systems.

• Process mediation is involved in mapping of

message exchanges deﬁned by the RosettaNet PIP

3A4 process to those deﬁned in the WSDL of the

Moon back-end systems.

• Conversations between the systems including

data and process mediation operate on semantic

descriptions of messages, thus requiring the trans-

formation from messages used by existing sys-

tems to the ontological level.

Infrastructure. The existing systems are Moon’s

back-end applications including CRM and OMS sys-

tems as well as Blue’s RosettaNet system (gateway).

Each system communicates using different formats,

e.g. Blues RosettaNet gateway communicates ac-

cording to the RosettaNet PIP 3A4 message (Pur-

chase Order), whereas communication with the CRM

and OMS systems is more proprietary - speciﬁed in

their WSDL descriptions. Detail descriptions of these

WSDL interfaces can be found at SWS-Challenge

web site.

Challenge Levels. The workshop organizers pro-

vide a set of challenge problems. These build upon

the initial mediation problem, which we call Level 0.

On top of this various levels are added, each corre-

sponding to a general kind of problem, and each with

sublevels of complexity. For the mediation scenario:

Level 0: Mediation Scenario (static)

Evaluation: is determined automatically by the

sandbox system - when the solution is able to pass

the basic conversation.

Level 1: Mediation Scenario (adapting to changes in

systems), with the following distinction:

Level 1a: Data Mediation: signatures are

changed.

Level 1b: Process Mediation: more than one

item appears in the order.

Evaluation: determined by peer review at the

SWS workshops.

The changes involved in the levels 1a and 1b were

not published from the beginning, but made available

to the participating groups only once they solved the

previous levels.

5 THE DISCOVERY SCENARIO

The discovery scenario is a more visionary scenario

compared to the mediation scenario in the sense that

it cannot be solved with current syntactic techniques

alone. This fact is also reﬂected by the smaller num-

ber of solutions submitted to this scenario.

The scenario deﬁnes ﬁve shipping services (de-

scribed via their WSDL and a natural language doc-

umentations) and presents a set of increasingly com-

plex shipping requests. Given a request, a suitable

shipper needs to be discovered and invoked. Thus,

participants have to create (semantic) descriptions for

the available shippers and the given shipping requests

such that the discovery and invocation task can be per-

formed by an automated autonomous agent.

Recently the shipping scenario has been extended

but here we describe the scenario in the version

that has been evaluated at the Third SWS-Challenge

Workshop in Athens (November 2006).

Shipping services are characterized by the follow-

ing properties:

• Operation range: Shippers operate worldwide or

in a set of listed countries or continents.

• Package limitations: Shippers deﬁne maximum

bounds on the dimensions and the weight of pack-

ages. Additionally the notion of a dimensional

weight is used: Packages with a low weight,

but big dimensions need to use the dimensional

weight (computed from the dimensions of the

package) instead of the actual weight.

• Price: Four shippers statically specify the price

as rules how to compute the price of a package

depending on shipping location and package di-

mensions or weight. One shipper requires to dy-

namically call a webservice endpoint to gather

the current price providing the same information.

Thus for goals specifying an upper price limit for

the shipping operation, this service could not be

A description of the extensions released since can be

found in the “SWS Challenge: Status, Perspectives, and

Lessons Learned so far” paper in these same conference

proceedings.

discovered by exploiting static descriptions alone,

but required dynamic negotiation during the dis-

covery process.

• Package collection: Shippers offer collection of

packages and deﬁne various constraints on the

minimum or maximum advance notice for collec-

tion or the total length of the collection interval.

(This aspect was not covered by the set of goals

evaluated in Athens.)

The predeﬁned requests specify a required ship-

ping operation, characterized by concrete pickup and

delivery addresses as well as concrete package di-

mensions and weight. The more complex goals ad-

ditionally specify a maximum price for the shipping

operation. During discovery, participants have to ﬁl-

ter unsuitable shippers, automatically choose a suit-

able one and invoke it by calling the corresponding

web service endpoint. Since the shipper do not use

a common XML-Schema for their messages, partici-

pants also have to deal with issues of data mediation

to create the properly formatted messages.

One advanced goal requested the sending of two

packages instead of one. Since none of the shippers

support multiple packages, this goal has to be mapped

to multiple invocations of the same or different ship-

pers.

6 OVERVIEW OF

CONTRIBUTIONS

In the ﬁrst year of the SWS Challenge, we have had

three workshops with six (6) teams participating. The

other papers in this conference proceedings describes

the contributions of these teams in more detail. Here

we give an overview that will help the reader with

an overall perspective. In each case, we provide the

name of the primary contract for the approach(s).

• DIANE (Universit

at Jena)

Contact Person: Ulrich K

uster, Institute for

Computer Science, Friedrich-Schiller-Universit

Jena, 07743 Jena (Germany), Tel: +49 3641

946433, Ulrich.Kuester@uni-jena.de

DIANE is a method for automated service match-

making, selection, binding and invocation and

was used in the discovery scenarios.

• WebML/Webratio (Politecnico di Mi-

lano/CeFRIEL)

Contact Person: Federico Michele Facca,

Politecnico di Milano, Dip. di Elettronica e

Informazione, 20133 Milano (Italy), Tel: +39 02

2399 9623, facca@elet.polimi.it

and

• jABC/jETI (Universit

at Dortmund/Universit

Potsdam)

Contact Person: Christian Kubczak, Universit

Dortmund, Chair of Programming Systems,

Dortmund (Germany), Tel: +49 231 755 7757,

christian.kubczak@cs.uni-dortmund.de

These two teams took a software engineering ap-

proach to the mediation scenario.

• WSMX (DERI)

Contact Person: Maciej Zaremba, Digi-

tal Enterprise Research Institute (DERI -

http://www.deri.ie), National University of

Ireland, Galway, (maciej.zaremba@deri.org)

This team solved both mediation and discovery

problems using WSMX.

• Swashup (IBM)

Contact Person: Max Maximilien, IBM Almaden

Research Center, San Jose (CA), tel: +1 408

9272124, maxim@us.ibm.com

The “Swashup” approach uses Ruby on Rails for

a purely engineering approach to solving the ﬁrst

level of the mediation problem.

• METEOR-S (Wright State University)

Contact Person: Karthik Gomadam, Kno.e.sis

Center (http://knoesis.org), Dept. of Computer

Science & Engineering, Wright State University,

Dayton, OH, karthik.gomadam@gmail.com (with

collaborators at LSDIS lab, University of Georgia,

Athens, GA).

This team’s approach uses SAWSDL + Planning

+ Data Mediation Web service to solve the medi-

ation problem.

The precise results of all of the teams as of the

Athens, George Workshop can be found in the SWS

Challenge Wiki

. It should be noted here that the

three teams of Politecnico di Milano/Cefriel, DERI,

and Jena currently have solved the most problems in

both the mediation and discovery scenarios. The IBM

approach was near to solving the ﬁrst level of the me-

diation problem but had not done so as of February

2007. We expect a solution at the Innsbruck work-

shop in June of 2007. Other participants are invited to

contribute to this workshop and another planned for

November at Stanford University.

Finally we would emphasize that the SWS Chal-

lenge is open, both to participation and to the submis-

sion of new scenario problems. Some of the scenarios

http://sws-challenge.org/wiki/index.php/Workshop

Athens

can be “stand alone” and others will reﬁne and extend

the “Blue Moon” customer/company/supplier/shipper

scenario. Eventually, this scenario should include the

company fulﬁlling the customer order using a supply

chain composed of the best suppliers and shippers for

the speciﬁc customer order.

Our mission is to supply not only a large use-

ful “sandbox” for testing semantic web service ap-

proaches, but also a de facto standard for certifying

such technologies, as well as furthering an academic

understanding of the beneﬁts and trade-offs of these

approaches. The papers in this session are the ﬁrst

results from this understanding as they include com-

parisons of approaches.