product, secure deployment and operation, and many
more. Not all approaches are generally equivalent
regarding efficiency or usefulness, and each
development team should analyse the best return of
investment for each; a good source of wisdom of the
crowd in this area is the Building Security In
Maturity model (BSIMM, 2013), a survey intended
to collect and document possible activities that
companies are applying to secure their products.
1.1 The Nature of Security Evaluation
Security evaluation of products is a challenging
discipline that requires a fundamentally different
mind-set from functional testing. While functional
testing simply consists of the verification of well-
defined requirements and the goal is usually to
statistically decrease the number of bugs, security
testing involves finding evidence of abnormal
operation in non-obvious borderline cases: it is not
about how the system should work (aligned to the
various use cases), but rather about how it should
not (considering misuse and abuse cases).
Since during security testing we are looking for
possible behaviour outside of the specified
functionality, in theory we should check “anything
else”, which is obviously an infinite and hard-to-
define set. This is why – due to the bounded
resources – one should first prioritise by doing a risk
analysis. To do that, solid expertise in security is
needed, complemented with the ability to think as
the attackers would do, and be aware not only of the
applicable methodologies, techniques and tools, but
also of the trends among the attackers.
1.2 Security Evaluation Methodologies
Various methodologies exist that aim at security
evaluation of products – some of them general, some
of them domain-specific.
One of the best-known schemes for certification
of products from a security point of view is
Common Criteria, applying the Common
Methodology for Information Technology Security
Evaluation (CEM, 2012). The methodology defines
roles and the evaluation process itself, distinguishing
between evaluating a Protection Profile (describing
generic criteria for certain product family) or an
actual product as the Target of Evaluation (ToE). A
Common Criteria evaluation, however, needs
enormous resources, which are often not in line with
the evaluated product.
Microsoft has gone a long way in the last decade
to secure its products, in parallel defining a
methodology for secure software development.
Microsoft Secure Development Lifecycle (MS SDL,
http://www.microsoft.com/security/sdl/default.aspx)
is a process covering all steps of software product
development; however, there is only a weak focus
on testing, since its Verification step includes only
the practices of applying dynamic analysis, fuzz
testing and the review of the attack surface.
One of the most referred sources regarding
security in the Web application domain is the
OWASP – Open Web Application Security Project
(http://www.owasp.org). One of its sub-projects is
the Application Security Verification Standard
(ASVS, 2013), which defines a methodology for
assessing the security level of products, governed by
a guidance and with a commercial approach through
providing requirements for project contracts. ASVS
defines four levels of verification: 0-Cursory, 1-
Opportunistic, 2-Standard and 3-Advanced, along
with verification requirements for each; though parts
of the methodology are stated quite generally, its
focus is still solely on Web applications.
Open Source Security Testing Methodology
Manual (OSTMM, 2010) provides a quite general
approach, and is a continuously developing, peer-
reviewed – open community developed –
methodology based on verified facts. The overall
process is, however, at some points quite vague, and
some find it hard to translate the presented general
approach to actual evaluation steps and procedures.
Some of the above-described methodologies are
too domain-specific, yet some are incomplete
considering the whole evaluation process or are
quite vague in describing certain steps. In the
following Chapter we will introduce our
comprehensive, practice oriented approach that has a
proven track record in evaluating the security of
various IT products.
2 MEFORMA OVERVIEW
MEFORMA is a security evaluation methodology
designed to be customer-oriented, meaning that the
evaluations are being accomplished on a project
basis using up resources fixed in advance, and the
outcomes not only provide a passed-or-failed result
like most of the certification schemes, but by the
recommendations given, the development groups
also receive valuable support on how to correct the
found problems.
Aligned to the usual terminology, ToE denotes
the system being evaluated, and we have two simple
PECCS2014-InternationalConferenceonPervasiveandEmbeddedComputingandCommunicationSystems
268