![](bg9.png)
considered experiment this can be translated into the
H
1E f f
, H
1Size
and H
1APFD
defined in Section 5. Ac-
cording to (Juzgado and Moreno, 2001; Wohlin et al.,
2012), we can apply the Paired T-Test to formally ver-
ify the Null Hypothesis with the confidence level of
95%. This choice was a natural consequence of the
type of design adopted, i.e., the paired comparison.
Therefore, following the standard best practices, we
can accept a probability of 5% of committing a Type-
1-Error (Juzgado and Moreno, 2001; Wohlin et al.,
2012), i.e., the Null Hypothesis is rejected if the com-
puted p-value is less or equal to 0,05 (alpha = 0.05).
7 CONCLUSIONS
In this paper we presented a family of controlled ex-
periments in the context of AC testing. The idea was
to define a set of standardized guidelines for correctly
and systematically performing the testing process in
order to avoid errors and improve the effectiveness of
the validation. The proposal relies on a characteriza-
tion of the first three steps of the experiment process
(i.e., Scoping, Planning and Operation) by leverag-
ing the Goal-Question-Metric template. Thus, we de-
tailed the activities necessary for performing the first
three steps of the experiment process (i.e., Scoping,
Planning and Operation). The example of the testing
of the Sun PDP engine is taken as a reference for bet-
ter explaining the three phases.
It was out of the scope of the paper providing the
complete list of testing goals or the realization of all
the possible testing frameworks. The example pro-
vided in the paper wanted to highlight the peculiar-
ity of the Controlled Experiments and the potentiality
they represent for the testing activity.
As a future work we intent to provide other im-
plementations of the Controlled Experiments for dif-
ferent testing purposes, so as to demonstrate its flex-
ibility and adaptability. We want also to apply the
proposed Controlled Experiment in real environments
so as to collect testing results and perform statistical
analysis.
ACKNOWLEDGEMENTS
This work is partially supported by CyberSec4Europe
Grant agreement ID: 830929.
REFERENCES
Basili, V. R., Caldiera, G., and Rombach, H. D. (1994). The
goal question metric approach. In Encyclopedia of
Software Engineering. Wiley.
Basili, V. R. and Rombach, H. D. (1988). The tame
project: towards improvement-oriented software envi-
ronments. IEEE Transactions on Software Engineer-
ing, 14(6):758–773.
Bertolino, A., Daoudagh, S., Lonetti, F., and Marchetti, E.
(2018). An automated model-based test oracle for ac-
cess control systems. In Proceedings of the 13th In-
ternational Workshop on Automation of Software Test,
AST ’18, pages 2–8, New York, NY, USA. ACM.
Bertolino, A., Daoudagh, S., Lonetti, F., Marchetti, E., Mar-
tinelli, F., and Mori, P. (2014a). Testing of polpa-
based usage control systems. Software Quality Jour-
nal, 22(2):241–271.
Bertolino, A., Daoudagh, S., Lonetti, F., Marchetti, E., and
Schilders, L. (2013). Automated testing of extensible
access control markup language-based access control
systems. IET Software, 7(4):203–212.
Bertolino, A., Le Traon, Y., Lonetti, F., Marchetti, E., and
Mouelhi, T. (2014b). Coverage-based test cases se-
lection for xacml policies. In Proceedings of ICST
Workshops, pages 12–21.
Bertolino, A., Lonetti, F., and Marchetti, E. (2010). Sys-
tematic XACML Request Generation for Testing Pur-
poses. In Proc. of 36th EUROMICRO Conference
on Software Engineering and Advanced Applications
(SEAA), pages 3 –11.
Calabr
`
o, A., Lonetti, F., and Marchetti, E. (2017). Access
control policy coverage assessment through monitor-
ing. In Proc. of TELERISE, pages 373–383.
Daoudagh, S., El Kateb, D., Lonetti, F., Marchetti, E., and
Mouelhi, T. (2015). A toolchain for model-based de-
sign and testing of access control systems. In Proc.of
MODELSWARD, pages 411–418. IEEE.
Daoudagh, S., Lonetti, F., and Marchetti, E. (2019a). A
framework for the validation of access control sys-
tems. In Saracino, A. and Mori, P., editors, Proceed-
ings of the 2nd International Workshop on Emerging
Technologies for Authorization and Authentication.
Daoudagh, S., Lonetti, F., and Marchetti, E. (2019b).
XACMET: XACML modeling & testing an auto-
mated model-based testing solution for access control
systems. Software Quality Journal.
Daoudagh, S., Lonetti, F., and Marchetti, E. (2020). As-
sessing testing strategies for access control systems:
A controlled experiment. In Proceedings of ICISSP
2020, Valletta, Malta, February 25-27, 2020.
Harumi, T. et al. (2016). D-mbtdd: An approach for reusing
test artefacts in evolving systems. In 2016 46th Annual
IEEE/IFIP International Conference on Dependable
Systems and Networks Workshops (DSN-W). IEEE.
Hu, V. C., Kuhn, R., and Yaga, D. (2017). Verification and
test methods for access control policies/models. NIST
Special Publication, 800:192.
Islam, S. and Falcarin, P. (2011). Measuring security
requirements for software security. In 2011 IEEE
Defining Controlled Experiments Inside the Access Control Environment
175