Towards Conformance Testing of REST-based Web Services

Luigi Lo Iacono and Hoai Viet Nguyen

Cologne University of Applied Sciences, Cologne, Germany

Keywords:

REST, HTTP, URI, Conformance Testing, Web Services, SOA, Cloud.

Abstract:

Despite the lack of standardisation for building REST-ful HTTP applications, the deployment of REST-based

Web Services has attracted an increased interest. This gap causes, however, an ambiguous interpretation of

REST and induces the design and implementation of REST-based systems following proprietary approaches

instead of clear and agreed upon deﬁnitions. Issues arising from these shortcomings have an inﬂuence on

service properties such as the loose coupling of REST-based services via a unitary service contract and the

automatic generation of code. To overcome such limitations, at least two prerequisites are required: the avail-

ability of speciﬁcations for implementing REST-based services and auxiliaries for auditing the compliance of

those services with such speciﬁcations.

This paper introduces an approach for conformance testing of REST-based Web Services. This appears con-

ﬂicting at the ﬁrst glance, since there are no speciﬁcations available for implementing REST by, e.g., the

prevalent technology set HTTP/URI to test against. Still, by providing a conformance test tool and leaning

it on the current practice, the exploration of service properties is enabled. Moreover, the real demand for

standardisation gets explorable by such an approach. First investigations conducted with the developed con-

formance test system targeting major Cloud-based storage services expose inconsistencies in many respects

which emphasizes the necessity for further research and standardisation.

1 INTRODUCTION

Building distributed systems based on the abstraction

of software services has been and still is an impor-

tant paradigm. First dominated by SOAP (Simple

Object Access Protocol) (Gudgin et al., 2007) and

its accompanying technology stack, the architectural

style REST (REpresentational State Transfer) (Field-

ing, 2000) has gained traction as an alternative ap-

proach for designing service systems. REST targets

the scalability of application interaction, the unifor-

mity of interfaces and the independent evolvement of

components and intermediates with the intention to

reduce latency, enable security and provide long-lived

services.

Besides the availability of a set of constraints de-

ﬁned in the dissertation of Fielding, there still does

neither exist a deﬁnition nor a clear understanding of

how to apply this architectural style to technical in-

stantiations. As a consequence, REST is often enough

mistaken as being a standard composed of its under-

lying foundations: URI (Berners-Lee et al., 2005) and

HTTP (Fielding et al., 1999). The source for this dif-

fuse view on the REST concept lies in the fact that the

two aforementioned standards are currently the only

notable technology choice for implementing REST-

based service systems.

Issues arising from these shortcomings in stan-

dardisation have, e.g., an inﬂuence on the fu-

sion of REST and SOA (Service-Oriented Architec-

ture) (Gorski et al., 2014b). Founding a SOA-based

system on a REST architecture is challenging due to

missing service properties including the loose cou-

pling of REST-based services. This is because of the

lack of service contracts, which makes, moreover, the

automatic generation of client-side code infeasible. In

Cloud and Utility Computing this lack of standardisa-

tion causes cross-service incompatibilities promoting

vendor lock-in and hindering the simultaneous adop-

tion of multiple distinct Clouds in a user-intended

manner.

Approaching these issues requires at least two de-

termining factors: (1) the availability of speciﬁcations

for applying the REST principles to technical founda-

tions (in a SOA fashion) and (2) conformance test-

ing instruments for auditing the compliance of REST-

based services to these technical environments. This

paper approaches these gaps by introducing a mean

for the conformance testing of REST-based Web Ser-

vices focussing on the dominating implementation

217

Lo Iacono L. and Nguyen H..

Towards Conformance Testing of REST-based Web Services.

DOI: 10.5220/0005412202170227

In Proceedings of the 11th International Conference on Web Information Systems and Technologies (WEBIST-2015), pages 217-227

ISBN: 978-989-758-106-9

 2015 SCITEPRESS (Science and Technology Publications, Lda.)

base: HTTP/URI. This appears conﬂicting at the ﬁrst

sight, since there are no speciﬁcations available yet

to test against. Still, by providing a conformance test

tool and leaning it on the state of the art, the explo-

ration of service properties such as the uniformness of

requests and responses is enabled. In practice, how-

ever, a real demand for standardisation might actu-

ally not be present. With the proposed conformance

test methodology and tooling this matter can be ex-

plored empirically. To prove the viability of this ap-

proach REST services offered by major Cloud stor-

age providers are analysed. Other available research

in this domain provides equivalent insights while fo-

cussing on REST frameworks instead of deployed real

world services (Gorski et al., 2014b). Note, that the

present topic has not been an active research area yet

and that due to the speciﬁc nature of REST there is not

much proper related work available so far. Accord-

ing studies in other Web Services technology stacks

such as SOAP, e.g., do not share fundamental ser-

vice properties including the Uniform Interface of

REST-based services. Moreover, the available work

in the SOAP-based Web Services arena is focussing

on SOAP speciﬁcs such as the interoperable code gen-

eration from an WSDL service contract (Elia et al.,

2014). Henceforth, an adoption and consideration in

the REST domain is not feasible.

The rest of the paper is organised as follows. In

Section 2 a speciﬁcation for mapping the REST ar-

chitectural principles and constraints to HTTP/URI is

deduced from the current practice and related work.

Following up in Section 3, a test methodology is intro-

duced, which provides a framework for conformance

testing of REST-based services. Available test tools

are then analysed according to their capabilities of

performing the required tests given in Section 4. The

analysis of the state of the art reveals a lack of an

adequate REST test tooling. This results in the de-

velopment of an own approach which is described

in Section 5. First ﬁndings obtained from investiga-

tions performed with the introduced methodology and

the developed tooling against four major Cloud-based

storage services, expose inconsistencies and hetero-

geneities in many respects. This is a violation of

the Uniform Interface constraint. More details about

these issues are discussed in Section 6. The paper

concludes in Section 7 with a summary of the main

contributions and ﬁndings as well as an outlook on

future work.

2 SPECIFICATION OF

REST-BASED WEB SERVICES

To test a REST-based service according to its confor-

mance to a particular standard, such a standard needs

to be present in the ﬁrst place. Regrettably, this is

not the case for the mapping of the REST architec-

tural components and principles to any particular set

of technologies. Thus, such a mapping speciﬁcation

needs to be derived in order to lay the ground for the

goals of this paper. Based on the least common de-

nominator obtained from the current practice and re-

cent empirical investigations (Gorski et al., 2014b),

the speciﬁcation for this paper is deﬁned as an instan-

tiation of the REST principles based on the URI spec-

iﬁcation and HTTP while taking additional clariﬁca-

tions of HTTP semantics and contents (Fielding and

Reschke, 2014) into account (see Table 1). URIs are

denoted in the URI Template (Gregorio et al., 2012)

notation. For the sake of readability but without the

loss of generality the following speciﬁcations will fo-

cus on the core request and response components only

and will leave aspects such as caching, versioning, se-

curity, streaming and response header ﬁelds for error

treatment out of consideration.

A POST request creates a resource on the ser-

vice side. The URI in the request denotes the set in

which a resource is to be added. In order to do so, a

complete representation of such a resource needs to

be sent in the request payload. This requires accord-

ing meta data in the request header to signal the type

and length of the contained resource representation.

In case the processing of a POST request has been

completed successfully, the response contains the sta-

tus code 201 (Created), meaning that the resource has

been created. Such a response must contain a Loca-

tion header in addition, which provides the identiﬁer

of the created resource in form of an URI. The re-

sponse payload is empty. To notify the client on spe-

cial conditions regarding the request processing, sev-

eral other status codes are deﬁned. The status code

202 (Accepted) must be used, when the request itself

can be processed successfully but the creation of the

resource is so time-consuming that the response needs

to be returned by the service before ﬁnalizing the re-

source instantiation. The other status codes have all

to do with issues in the request. If the addressed re-

source set does not exist, 404 (Not Found) is returned.

In case the speciﬁed content type is not supported,

415 (Unsupported Media Type) is passed back. The

status code 411 (Length Required) is used, if the con-

tent length is missing. When the POST method is not

deﬁned for the resource set but is invoked anyhow,

then the 405 (Method Not Allowed) status code is re-

WEBIST2015-11thInternationalConferenceonWebInformationSystemsandTechnologies

218

Table 1: Speciﬁcation of REST to HTTP/URI mapping.

HTTP

Method

Resource

Identiﬁer

Request

Headers

Request Payload Response Codes Response

Headers

Response Payload

POST /{resources} Content-Type

Content-Length

Resource representation

(complete)

201 Created

202 Accepted

400 Bad Request

404 Not Found

405 Method Not Allowed

411 Length Required

413 Payload Too Large

415 Unsupported Media Type

501 Not Implemented

505 HTTP Version Not Supported

Location Empty

GET /{resources}

/{resources}/{id}

Accept Empty 200 OK

400 Bad Request

404 Not Found

405 Method Not Allowed

406 Not Acceptable

501 Not Implemented

505 HTTP Version Not Supported

Content-Type

Content-Length

Link

Resource representation

(complete)

PUT /{resources}/{id} Content-Type

Content-Length

Resource representation

(complete)

201 Created

202 Accepted

204 No Content

400 Bad Request

404 Not Found

405 Method Not Allowed

411 Length Required

413 Payload Too Large

415 Unsupported Media Type

501 Not Implemented

505 HTTP Version Not Supported

None Empty

PATCH /{resources}/{id} Content-Type

Content-Length

Resource representation

(partial)

202 Accepted

204 No Content

400 Bad Request

404 Not Found

405 Method Not Allowed

411 Length Required

413 Payload Too Large

415 Unsupported Media Type

501 Not Implemented

505 HTTP Version Not Supported

None Empty

DELETE /{resources}/{id} None Empty 202 Accepted

204 No Content

400 Bad Request

404 Not Found

405 Method Not Allowed

409 Conﬂict

501 Not Implemented

505 HTTP Version Not Supported

None Empty

OPTIONS /*

/{resources}

/{resources}/{id}

None Empty 204 No Content

400 Bad Request

404 Not Found

405 Method Not Allowed

501 Not Implemented

505 HTTP Version Not Supported

Allow Empty

HEAD /{resources}

/{resources}/{id}

Accept Empty 200 OK

400 Bad Request

404 Not Found

405 Method Not Allowed

406 Not Acceptable

501 Not Implemented

505 HTTP Version Not Supported

Content-Type

Content-Length

Link

Empty

turned. With the status code 413 (Payload Too Large)

the service signals that an oversized request payload

has been received. This is, e.g., the case when the re-

quest message is larger than the capabilities available

to the service. Any other syntactical or semantical er-

ror in the request is repulsed by the status code 400

(Bad Request).

A GET request accesses a particular resource in

a certain representation. The URIs for accessing re-

sources differ in regards to whether a single instance

or all instances of a resource are to be accessed. To

declare the desired resource representations, the re-

quest must carry an Accept header. The payload of

the request is empty, since no data is transferred to the

service. If the resource is available in the requested

representation, the service delivers it to the client in

the response payload and sets the status code 200

(OK) for the according response. The content type

and length are also stated in the response, but these

entries are contained in the response header. Further-

more, a GET response can contain a Link header ref-

erencing another resource providing additional meta

data of the requested resource (Nottingham, 2010).

To signal error conditions the response contains the

status code 404 (Not Found) in case the requested re-

source does not exist, 406 (Not Acceptable) if the re-

quested resource representation is not provided, 405

(Method Not Allowed) when the GET method is not

deﬁned for the resource and 400 (Bad Request) in

cases in which any other syntactical or semantical er-

TowardsConformanceTestingofREST-basedWebServices

219

ror is contained in the request.

A PUT request updates a particular resource. In

case the addressed resource does not exist and the ser-

vice allows for resource creation via PUT, it will be

created. Hence, in some circumstances a PUT request

can act like a creation request. A PUT always trans-

fers a complete resource representation with the re-

quest to the service. The resource identiﬁer addresses

a concrete resource. In case the resource exists it is

updated by the provided payload and the according

response code is set to 204 (No Content). In case

a PUT operation addresses a non-existing resource

identiﬁer, the resource enclosed in the request will

be created and the server must return the response

code 201 (Created). If the service only grants the

PUT method for update operations and does not of-

fer the functionality of resource creation, it must re-

sponse with the code 405 (Method Not Allowed). In

all other cases and especially the possible error con-

ditions the behaviour corresponds to the one of the

POST method.

A PATCH (Dusseault and Snell, 2010) request up-

dates a given resource virtually as a PUT request. In

contrast to PUT a PATCH request, however, speciﬁes

only those data ﬁelds that need to be changed and

does henceforth not provide the complete resource

representation with the request payload. Thus, with a

PATCH request a resource cannot be created but efﬁ-

ciently updated, since only the required changes need

to be send to the service. PATCH responses follow the

same status code policies as deﬁned for PUT, leaving

the codes related to resource creation apart.

A DELETE request erases a particular resource

from the service side. No further meta data is re-

quired in the request and the response headers. The

status code 204 (No Content) expresses the successful

deletion of the targeted resource speciﬁed in the corre-

sponding request. If the addressed resource does not

exist, this is signalled back to the client by the status

code 404 (Not Found). In case the DELETE request is

issued although it is not implemented, the status code

405 (Method Not Allowed) is returned. In some cir-

cumstances the erasing of a resource is not allowed,

due to reasons lying in the business logic and the cur-

rent resource state. In such a case, the service must

return the status code 409 (Conﬂict). As for any other

request, syntactical and semantical errors are denoted

by the status code 400 (Bad Request).

An OPTIONS request is the basis for gathering

information on the provided methods regarding a par-

ticular resource or set of resources. The request is

addressed towards a resource identiﬁer with no partic-

ular header and an empty payload. The correspond-

ing response discloses the offered actions in the Allow

header. The occurring error conditions are treated and

reported by the response codes as discussed for the

other request types.

A HEAD request is used to retrieve status infor-

mation and meta data for a certain resource. It be-

haves exactly as the GET method with the only dif-

ference that the response payload is empty and does

henceforth not contain a resource representation.

If a client performs a request with an unfamiliar

feature such as an unknown method, the service must

reply with the status code 501 (Not Implemented) in-

forming the counterpart that this functionality is not

implemented. Last but not least, requests contain-

ing an unsupported HTTP version number must be

rejected by the status code 505 (HTTP Version Not

Supported).

3 TEST METHODOLOGY

With this mapping speciﬁcation and by utilising the

REST constrains, an ideal conformance testing of

REST-based services is an automatic task which re-

quires very little preparation and input. The core of

the proposed methodology is based on a catalogue of

standard tests allowing for a structured audit of REST-

based services (see Figure 1).

REST Service

REST Conformance

Test Tool

Resource

Identifiers

Autogenerated Tests

Reports

Service Contract

Standard

Tests

Specific

Tests

Figure 1: REST conformance test methodology.

As initial input, the conformance test tool requires

the base resource identiﬁer of the service to be tested

only. By issuing a status request to this resource iden-

tiﬁer, the service returns a response including the re-

source identiﬁer referring to the service contract. In

a REST architecture, a service contract is an informa-

tion entity describing the Uniform Interface provided

by the according resource. Basically, the Uniform In-

terface is set of instructions each composed of three

main elements: the action (method), the resource

identiﬁer syntax and the supported resource represen-

tations (media types) (Erl et al., 2013). Others dis-

closures could also be delineations of non-functional

meta data including authentication and authorization

policies. By a subsequent read request the test tool

retrieves the denoted service contract. The represen-

WEBIST2015-11thInternationalConferenceonWebInformationSystemsandTechnologies

220

tation of the service contract can be of any format

suitable to describe the data structure of the targeted

resource, such as realised by RAML (RESTful API

Modeling Language) (Sarid et al., 2014).

The retrieved service contract is then the en-

try point for generating a series of service invoca-

tions. Due to the interface uniformity, a standard-

ised catalogue of test cases can be deﬁned for REST-

based services, containing a comprehensive combina-

tion of constructive and destructive tests, with which

any REST-based service can be tested exhaustively.

Such a standardised test catalogue for REST-based

Web Services has been deﬁned for the purpose of

this paper, which is based on the speciﬁcation laid

in Section 2 and the test cases utilised in (Gorski

et al., 2014b) for the empirical investigations of the

SOA-readiness of REST frameworks for developing

Web-based systems. The test cases of the standard-

ised catalogue will not be described in detailed here,

since they correspond with the 87 test cases presented

in (Gorski et al., 2014b). The reader is henceforth re-

ferred to the original source for further reference. The

ID scheme used for naming each test case in a unique

manner—the ﬁrst two letters of the HTTP verb fol-

lowed by a dot and a sequence number—has been

maintained in this paper in order to support cross-

reading. If required, this standard set of tests can

be accomplished by speciﬁc tests addressing unique

aspects of a particular service environment and im-

plementation. By this approach, the proposed con-

formance testing can be adopted to any REST-based

Web Service equally straightforward. The marginal

adoptions required for the empirical evaluation of ma-

jor Cloud-based storage services are described in Sec-

tion 6.

Each performed test run is logged and the results

are stored in a report. The report entries contain the

raw communication as well as a judgement on the

conformity of the targeted service in respect to the

deﬁned speciﬁcation. The conformance is veriﬁed

by executing all tests and examining the obtained re-

sponses. For each of the 87 standard test requests a

corresponding expected response has been deﬁned. If

a test request to a service returns a response which

differs from the expected one, a candidate for a con-

formance breach has been found.

4 STATE OF THE ART

From the introduced REST conformance test ap-

proach the requirements for according tools got ap-

parent and can henceforth be recorded as follows:

R1: A REST test tool must be able to manage

a set of test cases for the uniform interface

mapped to a certain service protocol—at the

moment mainly HTTP.

R2: In order to be able to express a comprehen-

sive set of constructive and destructive tests,

many of the header entries including the re-

quest line, the Content-Type, the Content-

Length and the Accept header need to be ad-

justable in a ﬂexible manner.

R3: This is equally true for the resource repre-

sentations. Here, various types of represen-

tations must be supported and modiﬁable ac-

cording to the test cases’ needs.

R4: A REST test client requires to perform the

service inspection steps described in Sec-

tion 3 to retrieve the service contract. The

only required input for the test runs is a set of

resource identiﬁers from which the test client

constructs everything on its own.

Available REST test client software for

HTTP/URI based instantiations such as Advanced

REST Client (Psztyc, 2014), Postman (Postdot

Technologies, 2014) and the REST testing features

of SoapUI (SmartBear Software, 2014) do not fulﬁl

all the demanded requirements. Such tools enable the

creation and maintenance of multiple test requests,

but lack in functionalities regarding the ﬂexible

adjustment of header ﬁelds, the request line, or

resource representations and do not include any pro-

cedure to obtain the service contract automatically.

Consequently, test cases implemented based on these

tools need to be conﬁgured manually, which comes

at a high cost due to the many distinct parameters to

provide and set. Thus, for the purpose of this paper,

an own development has been undertaken that fulﬁls

the described set of requirements.

5 IMPLEMENTATION

The developed conformance test system is shown in

Figure 2. The underlying idea is to contribute a soft-

ware for managing conformance test projects target-

ing REST-based Web Services. To investigate the

compliance of a service, a project needs to be cre-

ated, ﬁrst. Then a service auditor can launch test runs

which execute all test cases of a deﬁned catalogue.

The tool itself is designed according to the architec-

tural style REST. Via a REST-based API clients are

able to create, read, update and delete test, project,

run and case resources. A browser-based client imple-

ments this API and provides a user interface to create

projects, set up and run tests and access the generated

reports.

TowardsConformanceTestingofREST-basedWebServices

221

REST

Interface

REST Conformance Test Tool

Standard

Test Catalog

Request

...

REST

Interface

REST Service

Response

...

Evaluation Report

(HTML, JSON, XML, Text)

Client

Request

(Test Creation)

Target Service URI

Schema

Representations

Actions

Authentication

Evaluation

projects

cases

runs

tests

Specific

Test Catalog

Request

Generator

Figure 2: Architecture of the developed REST conformance test tool.

As introduced by the test methodology in Sec-

tion 3, the test tool integrates a catalogue of standard

tests at its core. Test projects can beneﬁt from the

standardised catalogue of tests, which can be adapted

and performed on any interface speciﬁed by an ac-

cording resource identiﬁer. As mentioned in Sec-

tion 3, the tests contained in the standard tests cat-

alogue are based on the 87 tests deﬁned in (Gorski

et al., 2014b). Additional domain- and application-

speciﬁc tests can be added as required. When execut-

ing a test suite, the test run is generated out of the stan-

dardised as well as the project speciﬁc tests. When a

test run is completed, the client can retrieve an eval-

uation report in various resource representations such

as in HTML, XML, plain text or JSON advising the

service examiner about which test case has passed the

conformance audit and which not.

The developed system fulﬁls all requirements ex-

cept the one regarding the discovery and retrieval of

the service contract (R4). An implementation of this

feature is currently not feasible due to it being still

a research and development topic (Verborgh et al.,

2011; Bennara et al., 2014; Amundsen, 2014). The

support in operational systems is henceforth lacking.

Thus, the developed conformance test tool integrates

mechanisms to manually enter information on the ser-

vice contract as an interim work around. This in-

cludes the schema of the tested resources in distinct

representation formats as well as the HTTP methods

to test. The latter controls the amount of test cases

contained in the various catalogues to be issued in a

test run. As an additional set of parameters, the imple-

mented test system enables to specify access creden-

tials for the requests requiring authentication. This is

especially true for the analysed Cloud storage services

which deploy a custom HTTP authentication mecha-

nism each. Besides the implementation of the require-

ments R1 and R3 the tool fulﬁls also the requirement

R2, as it allows customizing any single header en-

try including the Content-Length header and the Host

header as well as the request line. R2 is crucial re-

quirement to enable destructive testing.

6 CONFORMANCE TESTING OF

CLOUD-BASED STORAGE

SERVICES

The intention of the present paper is to contribute a

methodology for the conformance testing of REST-

based services, although a required technical mapping

of the REST principles to a concrete technology stack

is lacking. Still, going the other way round enables

to explore the effective need for such a standardisa-

tion. The introduced methodology together with the

implemented test tool have been used to analyse var-

ious REST-based Web Services on their conformity

with the speciﬁcation given in Section 2, but also on

their consistent understanding of REST and the uni-

form realisation of REST services. The investiga-

tions have been focused on Cloud-based ﬁle storage

services including Amazon S3

, Google Cloud Stor-

age

, Microsoft Azure Storage

and HP Helion Public

Cloud Object Storage

, also because they are under-

stood to be particularly mature. The results obtained

by exploring their REST-based service interfaces are

discussed subsequently.

https://aws.amazon.com/s3/

https://cloud.google.com/products/cloud-storage/

https://docs.hpcloud.com/object-storage/

https://azure.microsoft.com/en-us/services/storage/

WEBIST2015-11thInternationalConferenceonWebInformationSystemsandTechnologies

222

6.1 Service Tests

As one core foundation of the conformance test me-

thodology, the standardised catalogue of test cases

forms the common ground for evaluating the compli-

ance of a particular REST-based Web Service in re-

spect to the introduced speciﬁcation. A series of con-

structive and destructive requests containing a com-

bination of all sorts of valid and invalid request set-

tings are used to observe the behaviour of a REST-

based Web Service. The test cases are grouped

by the HTTP methods GET, HEAD, POST, PUT,

PATCH, DELETE and OPTIONS including one non-

existing method named “EVIL”. Required adaptations

to the tests contained in the standard catalogue have

been minimal. The test case “Non-erasable resource”

(DE.2) in the DELETE request group has been dis-

abled. This test tries to erase a resource which must

not be deleted according to the business logic and the

particular resource’s state. As the explored Cloud-

based ﬁle storage services do not provide an option to

write-protect a ﬁle stored in the Cloud. The providers

promote only an Access Control List (ACL) which

allows deﬁning write permission for a distinct set of

users, but do not offer a feature to protect a resource

against a general deletion. Thus, trying to remove a

resource by a unauthorized user will result in a per-

mission error and not a delete rejection due to busi-

ness logic constraints. Thus, the intention of test case

DE.2 is not being covered by the Cloud storage ser-

vices, leading to it being left out.

The test case “Flawed content length” (PU.5) has

been extended with one new test run, which checks

the behaviour of the service in the presents of a dis-

crepancy in the content length value being in this test

run actually smaller than the de facto payload size.

The goal is to observe how many bytes are stored by

the service if any. A last test speciﬁc to storage ser-

vices has been added, which tests the behaviour when

exceeding the maximum allowed ﬁle size. This test

has been denoted as “Content-Length exceeding the

allowed payload size” (PU.10). It performs a ﬁle up-

load with a ﬁle size going beyond the limit set by the

storage provider. No further tests have been identiﬁed

as missing or speciﬁc to the underlying services.

6.2 Test Results

The obtained results from testing the four REST-

based Cloud storage services with the introduced tool

reveal that none of the services is in conformance

with the speciﬁcation introduced in Section 2. The

broad range of inconsistencies amongst the services

and between each service and the speciﬁcation can

be observed in many respects (see Table 2 in the Ap-

pendix).

Although all services provide an equivalent be-

haviour in case of the HEAD, GET, PUT and

DELETE methods, where the status codes of the

GET and HEAD results are even uniform, still, the

picture for the remaining methods looks more di-

verse. All tested services provide, e.g., a distinct

interpretation of the POST method. Amazon of-

fers the POST method as an upload function for

clients which do not support the PUT method—such

as Web browsers. This induces the requirement for

the client to encode the payload as “multipart/form-

data” (Amazon, 2014). Due to this missing precondi-

tion almost all status codes obtained from the POST

tests are 412 (Precondition Failed), since the POST

requests contain a resource representation in “appli-

cation/json” or “application/xml” instead of the ex-

pected “multipart/form-data”. Google provides the

POST method for different functionalities. Clients

can create access control rules, add new storage area,

compose ﬁles, copy ﬁles and create new ﬁles or meta-

data. If the service consumer intends to create new

ﬁles via POST, it has to use a different URL (Google,

2014) than deﬁned by the speciﬁcation in Section 2.

HP serves POST as an operation to add meta data

to accounts, storage areas and ﬁles. This is realised

via custom HTTP headers while the payload of the

POST request remains empty. Also, the HP ser-

vices do not permit uploading ﬁles with the POST

method (Hewlett-Packard, 2014). For a ﬁle upload

into a storage area a PUT request needs to be issued

instead. Likewise, Microsoft does not advertise any

utilization of the POST method. Any upload or up-

date operation is triggered by the PUT method, even

when a client wants to append meta data to storage

areas (Microsoft, 2014).

Other inconsistencies can be observed by focusing

on the status codes contained in the response mes-

sages. One example of this divergence is the non-

uniform treatment of regular PUT requests by all eval-

uated services (see PU.1 and PU.2). Amazon and

Google reply with a status code of 200 (OK) whether

a stored resource is updated or a new resource is cre-

ated. HP and Microsoft, on the contrary, reply with

201 (Created) for both cases: the creation of a new ﬁle

or the update of an existing ﬁle. Other discrepancies

can, e.g., be found in the answers to a PUT request

providing a wrong resource identiﬁer (PU.6). Ama-

zon and Google behave in the same way and react by

sending back a status code of 400 (Bad Request). HP

interprets the resource identiﬁer as intention to create

a new storage area and hence returns 202 (Accepted).

Microsoft responds with 404 (Not Found) in order to

TowardsConformanceTestingofREST-basedWebServices

223

signal that this resource is not available.

The biggest discrepancy can be illustrated by

OP.7, where all providers reply with diverse response

codes. In this destructive test, a wrong HTTP version

is provided in the request line. Amazon responds with

the expected code 505 (HTTP Version Not Supported)

according to the deﬁned speciﬁcation (see Listing 1

and Table 2).

∗∗∗ OPTIONS −− OP.7: Unknown Protocol Version ∗∗∗

Request:

OPTIONS /rssblobs/blob HTTP/1.2

Accept: application/json

Connection: Close

User−Agent: REAL SOA Security, REST−CTT

Host: s3−eu−west−1.amazonaws.com

[...]

Response:

HTTP/1.1 505 HTTP Version Not Supported

Server: AmazonS3

Connection: close

[...]

Listing 1: OPTIONS request with an unknown protocol

version (Amazon).

HP returns a 501 (Not Implemented) and indicates

in the response payload that the method may not be

implemented although the OPTIONS method is sup-

ported (see Listing 2 and Table 2).

∗∗∗ OPTIONS −− OP.7: Unknown Protocol Version ∗∗∗

Request:

OPTIONS /v1/11876826348381/blobs/blob HTTP/1.2

Accept: application/json

Connection: Close

User−Agent: REAL SOA Security, REST−CTT

Host: region−a.geo−1.objects.hpcloudsvc.com

[...]

Response:

HTTP/1.0 501 Not Implemented

Content−Length: 28

Content−Type: text/html

[...]

This method may not be used.

Listing 2: OPTIONS request with an unknown protocol

version (HP).

In case of Microsoft and Google, both services do

not conform to the HTTP speciﬁcation in relation to

this destructive test case. These services ignore the

unfamiliar protocol and still process the request or re-

ply with another error code. Listings 3 and 4 demon-

strate this with the request/response messages logged

during the test execution of the Google and Microsoft

storage services. The latter one even replies with a

self-deﬁned reason phrase instead of the standardised

reason for the status code 400 which is Bad Request

(see Listing 4).

∗∗∗ OPTIONS −− OP.7: Unknown Protocol Version ∗∗∗

Request:

OPTIONS /rssblobs/blob HTTP/1.2

Accept: application/json

Connection: Close

User−Agent: REAL SOA Security, REST−CTT

Host: storage.googleapis.com

[...]

Response:

HTTP/1.1 200 OK

Connection: close

Content−Length: 0

Content−Type: text/html; charset=UTF−8

[...]

Listing 3: OPTIONS request with an unknown protocol

version (Google).

∗∗∗ OPTIONS OP.7 Unknown Protocol Version ∗∗∗

Request:

OPTIONS /blobs/blob HTTP/1.2

Accept: application/json

Connection: Close

User−Agent: REAL SOA Security, REST−CTT

Host: realsoasecurity.blob.core.windows.net

[...]

Response:

HTTP/1.1 400 A required CORS header is not present.

Content−Length: 293

Content−Type: application/xml

[...]

Listing 4: OPTIONS request with an unknown protocol

version (Microsoft).

None of the analysed services supports the

PATCH (Dusseault and Snell, 2010) method. A rea-

son might be that a partial update of ﬁles is not re-

garded as necessary, although, partial updates to over-

write certain blocks within a ﬁle or append data to

an existing ﬁle might be sensible operations which

are actually in parts available via the PUT method in-

stead.

Besides this incoherent behaviour of the services,

the evaluation reveals a critical behaviour of all ser-

vices in cases in which a request contains a content

length value that is larger than the actual payload size.

Such a destructive request confuses the service, keep-

ing it waiting for the anticipated missing bytes that

will never arrive and causing the TCP connection to

remain open (see Table 2). This weakness could po-

tentially lead to DoS (denial of service) vulnerabili-

ties.

Other interesting evaluation results expose that the

services do not issue an error message if a request

contains a mismatch in the speciﬁed content type and

the actual payload format. All audited storage ser-

vices store a JSON payload, e.g., as XML, when

the Content-Type header contains the value “applica-

tion/xml”. As a consequence, if the ﬁle will be down-

loaded it embodies this mismatch too.

Another observation is that the services do not

care about the media type value within the Accept

header and do not check if the media type in the Ac-

cept header matches with the inquired resource. In

other words, if clients want to access an object as

JSON and denote this by setting the value “applica-

tion/json” in the Accept header, but the addressed re-

source is an XML document, the server ignores the

WEBIST2015-11thInternationalConferenceonWebInformationSystemsandTechnologies

224

Accept header and sends back the XML ﬁle without

further notice.

One of the two added tests, which sets the

Content-Length header to a smaller value than the ac-

tual body size (PU.5, test case 59), exposes a different

behaviour between the evaluated services. Microsoft,

HP and Google store the amount of bytes denoted by

the Content-Length header value and do not consider

any further bytes possibly contained in the payload.

Thus, a request carrying the text string “HelloWorld”

as its payload and specifying a Content-Length of ﬁve

will result in storing the ﬁrst ﬁve bytes of the supplied

string, i.e. “Hello”. Due to the signature protection

scheme enforced by Amazon for their services, such

a mismatch leads here to an error. The signature vali-

dation procedure is not documented, but with the test

tool it could be deduced, that the content length value

is used to control the amount of data that is hashed

during signature veriﬁcation. Any mismatch will lead

to a negative veriﬁcation, which is signalled by 400

(Bad Request).

The second added test uploads an oversized ﬁle

to the storage service which goes beyond the maxi-

mum allowed ﬁle size (PU.10, test case 60). This test

revealed a homogeneous behaviour between Amazon

and Microsoft according the request payload process-

ing. According to the obtained results, the two ser-

vices inspect the Content-Length header in order to

ﬁnd out whether a request contains an outsized pay-

load or not. However, both providers differ by the

returned status codes. Microsoft returns a speciﬁca-

tion compliant status code 413 (Request Entity Too

Large) while Amazon responds with a non-compliant

status code 400 (Bad Request). HP does not consider

the value of Content-Length header. Instead, it counts

the bytes on the TCP stream until the threshold of the

allowed maximum ﬁle size is reached. Google does

not state any ofﬁcial information about the maximum

ﬁle size rendering this test not feasible.

Amongst the discussed ﬁndings are many short-

comings at the service access and service communi-

cation layer, but also issues in each of the services’

business logic. This emphasizes the beneﬁts of the

proposed conformance testing. It further highlights

the current state in REST-based service practice and

the impact of missing standardisations. The hetero-

geneity of all services relating to status codes and

meta data headers are not in line with the idea of a

Uniform Interface which is a crucial constrain of the

Fielding’s architectural style. This violation cumbers

automatic code generation, loose coupling of services

and interoperability. Still, in comparison to (Gorski

et al., 2014b) the analysed Cloud storage services are

much more coherent with the REST-ful HTTP speciﬁ-

cation deﬁned for the purpose of this paper than what

is produced by REST frameworks. Thus, an experi-

enced developer but unversed in respect to REST will

not be able to produce services which are aligned with

the REST principles.

7 CONCLUSIONS AND

OUTLOOK

Having more reliable standards in terms of technical

speciﬁcations which deﬁne clear mappings between

the general REST principles and concrete technolog-

ical instantiations is vital in order to address the ex-

plored issues. Standardisation ambitions such as the

proposed REST-ful HTTP speciﬁcation introduced in

Section 2 would foster approaching advanced require-

ments with REST and would leverage the interoper-

ability of REST-based services. Service properties

known from SOA such as the discoverability, and

the loose coupling of services are otherwise not im-

plementable across platforms and systems. The lat-

ter property requires to a certain extent the automatic

generation of code, which is a lacking feature in the

REST domain so far, requiring, e.g., the development

of REST client code from scratch without any mean-

ingful tool support. Furthermore, this inhomogene-

ity even affects Cloud Computing concepts such as

Elastic Computing (Dustdar et al., 2011), where soft-

ware systems are adjustable automatically, depending

on required resources, quality of service parameters

and operational costs.

From these arguments, the demand for a more

elaborated standardisations gets evident. In practice,

however, a real need for adequate standards might

still not be given for some reasons. By means of

the contributed conformance testing methodology and

tooling a ﬁeld study has been conducted to deter-

mine the impact of this lack of standardisation on real

world services. The analysis of distinct REST-based

Web Services from four major Cloud-based storage

providers made it apparent that the tested services dif-

fer in many aspects amongst each other and do so also

in respect to the deﬁned speciﬁcation. This inhomo-

geneous picture emphasises the consequences result-

ing from the diffuse understanding of REST and its

technical instantiations.

The introduced method for performing the con-

formance testing of REST-based services would have

a broad inﬂuence on the overall development of

REST. Not only it would provide a mechanism to in-

crease the comprehension of REST and its mapping to

HTTP/URI, it would also increase the interoperability

of REST components implemented and operated on

TowardsConformanceTestingofREST-basedWebServices

225

heterogeneous platforms. The latter ultimately paves

the way for features such as automatic code genera-

tion and the loose coupling of REST-based services.

Many other aspects will beneﬁt from a stringent

standardisation and an accompanying enhancement in

REST service testing. Security is one such facet. A

stable and reliable foundation for building REST ser-

vices is the basis for a deep integration of security

means into REST services, which is a required prereq-

uisite in order to provide an answer to the challenging

security demands in cross-organizational distributed

services systems (Gorski et al., 2014a).

This speciﬁcation does not constitute the ulti-

mate standard for building REST-based Web Ser-

vices. Some deﬁnitions might be arguable or need to

be discussed in more detail by future standardisation

activities. Some aspects have even been left out of

scope such as security, caching, versioning, stream-

ing and error treatment. Nevertheless, such kind of

technical documentation is a crucial basic assistance

for software developers and architects to design inter-

operable REST-based service applications.

REFERENCES

Amazon (2014). Authenticating Requests in Browser-

Based Uploads Using POST (AWS Signature Version

4). http://docs.aws.amazon.com/AmazonS3/latest/

API/sigv4-UsingHTTPPOST.html.

Amundsen, M. (2014). Hold Your Nose vs. Follow Your

Nose, Observations on the state of service description

on the Web. In 5th International Workshop on Web

APIs and RESTful Design (WS-REST).

Bennara, M., Mrissa, M., and Amghar, Y. (2014). An Ap-

proach for Composing RESTful Linked Services on

the Web. In 5th International Workshop on Web APIs

and RESTful Design (WS-REST).

Berners-Lee, T., Fielding, R., and Masinter, L. (2005). Uni-

form Resource Identiﬁer (URI): Generic Syntax. RFC

3986, IETF. http://www.ietf.org/rfc/rfc3986.txt.

Dusseault, L. and Snell, J. (2010). PATCH Method for

HTTP. RFC 5789, IETF. https://tools.ietf.org/ htm-

l/rfc5789.

Dustdar, S., Guo, Y., Satzger, B., and Truong, H.-L. (2011).

Principles of elastic processes. IEEE Internet Com-

puting, 15(5).

Elia, I. A., Laranjeiro, N., and Vieira, M. (2014). A Field

Perspective on the Interoperability of Web Services.

In 11th IEEE International Conference on Services

Computing (SCC).

Erl, T., Carlyle, B., Pautasso, C., and Balasubramanian, R.

(2013). SOA with REST - Principles, Patterns and

Constraints for Building Enterprise Solutions with

REST. Pearson Education.

Fielding, R. (2000). Architectural Styles and the De-

sign of Network-based Software Architectures. PhD

thesis, University of California, Irvine. https://

www.ics.uci.edu/∼ﬁelding/pubs/dissertation/top.htm.

Fielding, R., Gettys, J., Mogul, J., Frystyk, H., Masinter,

L., Leach, P., and Berners-Lee, T. (1999). Hyper-

text Transfer Protocol – HTTP/1.1. RFC 2616, IETF.

http://www.ietf.org/rfc/rfc2616.txt.

Fielding, R. and Reschke, J. (2014). Hypertext Transfer

Protocol (HTTP/1.1): Semantics and Content. RFC

7231, IETF. http://tools.ietf.org/html/rfc7231.

Google (2014). Google Cloud Storage - API Ref-

erence. https://developers.google.com/storage/docs/

json api/v1/.

Gorski, P. L., Lo Iacono, L., Nguyen, H. V., and Torkian,

D. B. (2014a). Service Security Revisited. In 11th

IEEE International Conference on Services Comput-

ing (SCC).

Gorski, P. L., Lo Iacono, L., Nguyen, H. V., and Torkian,

D. B. (2014b). SOA-Readiness of REST. In 3rd

European Conference on Service-Oriented and Cloud

Computing (ESOCC).

Gregorio, J., Fielding, R., Hadley, M., Nottingham, M., and

Orchard, D. (2012). URI Template. RFC 6570, IETF.

http://tools.ietf.org/html/rfc6570.

Gudgin, M., Hadley, M., Mendelsohn, N., Moreau, J.-

J., Nielsen, H. F., Karmarkar, A., and Lafon, Y.

(2007). SOAP Version 1.2 Part 1: Messaging Frame-

work (Second Edition). Recommendation, W3C.

http://www.w3.org/TR/soap12-part1/.

Hewlett-Packard (2014). HP Helion Public Cloud Object

Storage API Speciﬁcation. https://docs.hpcloud.com/

api/object-storage#4.RESTAPISpeciﬁcations.

Microsoft (2014). Microsoft Developer Network - Blob

Service REST API. http://msdn.microsoft.com/

en-us/library/dd135733.aspx.

Nottingham, M. (2010). Web Linking. RFC 5988, IETF.

https://tools.ietf.org/html/rfc5988.

Postdot Technologies (2014). Postman. http://

www.getpostman.com/.

Psztyc, P. (2014). Advanced REST Client. http://

chromerestclient.appspot.com/.

Sarid, U., Hervery, M., Lazarov, I., Rexer, P., Harnon, J.,

Lane, K., Musser, J., Gullotta, T., and Choudhary, S.

(2014). RAML Version 0.8: RESTful API Modeling

Language. Speciﬁcation. http://raml.org/spec.html.

SmartBear Software (2014). SoapUI. http://

www.soapui.org/REST-Testing/getting-started.html.

Verborgh, R., Steiner, T., Van Deursen, D., De Roo, J.,

Van de Walle, R., and Gabarr

o Vall

es, J. (2011). De-

scription and Interaction of RESTful Services for Au-

tomatic Discovery and Execution. In Proceedings of

the FTRA 2011 International Workshop on Advanced

Future Multimedia Services (AFMS).

WEBIST2015-11thInternationalConferenceonWebInformationSystemsandTechnologies

226

APPENDIX

Table 2: Obtained status codes by Cloud-based storage services for test cases.

# Test identiﬁer and description Amazon Google HP Microsoft

1 PO.1 Content-Type application/json 412 400 204 400

2 PO.1 Content-Type application/xml 412 400 204 400

3 PO.2 Unsupported Content-Type 412 400 204 400

4 PO.3 Content-Type and payload mismatch 412 400 204 400

5 PO.3 No Content-Type but with payload 412 400 204 400

6 PO.4 Content-Length bigger than payload size 412 No Response No Response 400

7 PO.4 Content-Length as String 400 400 400 400

8 PO.4 No Content-Length 412 411 204 411

9 PO.5 Wrong action on resource 405 400 404 405

10 PO.5 Not existing resource 412 400 204 405

11 PO.6 Malformed application/json 412 400 204 400

12 PO.6 Malformed application/xml 412 400 204 400

13 PO.7 Wellformed application/json, unprocessible content 412 400 204 400

14 PO.7 Wellformed application/xml, unprocessible content 412 400 204 400

15 PO.8 Unknown protocol version 505 400 501 400

16 OP.1 Ping * 400 200 200 400

17 OP.2 Regular 400 200 200 400

18 OP.2 Regular with resource id 400 200 200 400

19 OP.3 Accept application/json 400 200 200 400

20 OP.3 Accept application/xml 400 200 200 400

21 OP.4 Unsupported media type in accept header 400 200 200 400

22 OP.5 Wrong resource identiﬁer 400 200 200 400

23 OP.5 Not existing resource 400 200 200 400

24 OP.6 Containing content 400 200 200 400

25 OP.7 Unknown protocol version 505 200 501 400

26 HE.1 Accept application/json 200 200 200 200

27 HE.1 Accept application/xml 200 200 200 200

28 HE.2 Unsupported media type 200 200 200 200

29 HE.3 Wrong resource identiﬁer 404 404 404 404

30 HE.3 Not existing resource 404 404 404 404

31 HE.4 Containing content 200 400 200 200

32 HE.5 No Accept header 200 200 200 200

33 HE.6 Unknown protocol version 505 200 501 200

34 GE.1 Accept application/json 200 200 200 200

35 GE.1 Accept application/xml 200 200 200 200

36 GE.2 Unsupported media type 200 200 200 200

37 GE.3 Wrong resource identiﬁer 404 404 404 404

38 GE.3 Not existing resource 404 404 404 404

39 GE.4 Containing content 200 400 200 200

40 GE.5 No Accept header 200 200 200 200

41 GE.6 Unknown protocol version 505 200 501 200

42 PU.1 Content-Type application/json 200 200 201 201

43 PU.1 Content-Type application/xml 200 200 201 201

44 PU.2 Unsupported Content-Type 200 200 201 201

45 PU.3 Partial update with Content-Type application/json 200 200 201 201

46 PU.3 Partial update with Content-Type application/xml 200 200 201 201

47 PU.4 Content-Type and payload mismatch 200 200 201 201

48 PU.4 No Content-Type but with payload 200 200 201 201

49 PU.5 Content-Length bigger than payload size No Response No Response No Response No Response

50 PU.5 Content-Length as String 400 400 400 400

51 PU.5 No Content-Length 411 411 411 411

52 PU.6 Wrong resource identiﬁer 400 400 202 404

53 PU.6 Not existing resource 200 200 201 201

54 PU.7 Malformed application/json 200 200 201 201

55 PU.7 Malformed application/xml 200 200 201 201

56 PU.8 Wellformed application/json, unprocessible content 200 200 201 201

57 PU.8 Wellformed application/xml, unprocessible content 200 200 201 201

58 PU.9 Unknown protocol version 505 200 501 201

59 PU.5 Content-Length smaller than payload size 400 200 201 201

60 PU.10 Content-Length exceeding the allowed payload size 400 ? 413 413

61 PA.1 Content-Type application/json 405 405 501 400

62 PA.1 Content-Type application/xml 405 405 501 400

63 PA.2 Unsupported Content-Type 405 405 501 400

64 PA.3 Complete update with Content-Type application/json 405 405 501 400

65 PA.3 Complete update with Content-Type application/xml 405 405 501 400

66 PA.4 Content-Type and payload mismatch 405 405 501 400

67 PA.4 No Content-Type but with payload 405 405 501 400

68 PA.5 Wrong Content-Length 405 No Response 501 400

69 PA.5 Content-Length as String 400 400 501 400

70 PA.5 No Content-Length 405 405 501 400

71 PA.6 Wrong resource identiﬁer 405 405 501 400

72 PA.6 Not existing resource 405 405 501 400

73 PA.7 Malformed application/json 405 405 501 400

74 PA.7 Malformed application/xml 405 405 501 400

75 PA.8 Wellformed application/json, unprocessible content 405 405 501 400

76 PA.8 Wellformed application/xml, unprocessible content 405 405 501 400

77 PA.9 Unknown protocol version 505 405 501 400

78 DE.1 Regular 204 204 204 202

79 DE.3 All resources 409 409 409 400

80 DE.4 Not existing resource 204 404 404 404

81 DE.5 Containing content 204 400 204 202

82 DE.6 Unknown protocol version 505 204 501 202

83 EV.1 Accept application/json 405 502 501 400

84 EV.1 Accept application/xml 405 502 501 400

85 EV.2 Unsupported media type in accept header 405 502 501 400

86 EV.3 Wrong resource identiﬁer 405 502 501 400

87 EV.4 Containg content 405 502 501 400

88 EV.5 Unknown protocol version 505 502 501 400

Speciﬁcation

201

415

400

411

405

404

400

505

200

415

404

400

505

200

406

404

400

200

505

200

406

404

400

200

505

204

415

400

411

404

400

505

400

413

204

415

204

400

411

404

400

505

204

405

404

400

505

501

TowardsConformanceTestingofREST-basedWebServices

227