Overall, the successful completion of the descri-
bed evaluation steps in the context of FutureTrust de-
monstrates the practical applicability of the proposed
evaluation scheme. As the FutureTrust project is still
ongoing, the overall evaluation process is not yet fi-
nished. However, most process steps (including the
most challenging ones like the definition of evalua-
tion criteria) as defined by the proposed scheme have
already been applied successfully in practice.
6 CONCLUSIONS AND FUTURE
WORK
Its successful application within a concrete internati-
onal research project shows that the proposed evalu-
ation scheme meets its goals. By relying on a hierar-
chical evaluation-criteria model, the scheme enables
an in-depth evaluation of specific pilots, while still
guaranteeing comparability of obtained evaluation re-
sults. Ultimately, the proposed scheme yields more
valuable evaluation results, from which all involved
stakeholders can benefit in the end. Especially fun-
ding bodies can take advantage of more homogenous
evaluation results, which supports them in steering re-
search activities into the right directions, and in com-
plying with relevant IT agendas.
Lessons learned from applying the proposed eva-
luation scheme to the research project FutureTrust are
currently used to apply final minor improvements to
the scheme. For the future, we also plan to apply the
scheme to other projects, to further test its project-
independent applicability. At the same time, we aim
to extend the applicability of the proposed scheme.
While its current focus lies on pilot-based projects
from the public sector, we plan to make it applicable
to other project types as well.
ACKNOWLEDGEMENTS
This work has been supported by the FutureTrust pro-
ject (N.700542) funded under the programme H2020-
EU.3.7. - Secure societies - Protecting freedom and
security of Europe and its citizens (2013/743/EU of
2013-12-03).
REFERENCES
Asosheh, A., Nalchigar, S., and Jamporazmey, M. (2010).
Information technology project evaluation: An inte-
grated data envelopment analysis and balanced sco-
recard approach. Expert Systems with Applications,
37(8):5931 – 5938.
Eilat, H., Golany, B., and Shtub, A. (2008). R&d project
evaluation: An integrated dea and balanced scorecard
approach. Omega, 36(5):895 – 912.
European Commission (2018a). Digital Agenda for Europe.
https://eur-lex.europa.eu/legal-content/EN/TXT/
HTML/?uri=LEGISSUM:si0016&from=DE. Online;
accessed 05 July 2018.
European Commission (2018b). H2020-EU.3.7. - Secure
societies - Protecting freedom and security of Europe
and its citizens. https://cordis.europa.eu/programme/
rcn/664463 en.html. Online; accessed 05 July 2018.
European Commission (2018c). Shaping the Digital Single
Market. https://ec.europa.eu/digital-single-market/en/
policies/shaping-digital-single-market. Online; acces-
sed 05 July 2018.
European Union (2018). Regulation (EU) No 910/2014
of the European Parliament and of the Coun-
cil of 23 July 2014 on electronic identifica-
tion and trust services for electronic transacti-
ons in the internal market and repealing Di-
rective 1999/93/EC. https://eur-lex.europa.eu/legal-
content/EN/TXT/?uri=CELEX:32014R0910. Online;
accessed 05 July 2018.
Khan, Z., Ludlow, D., and Caceres, S. (2013). Evaluating a
collaborative it based research and development pro-
ject. Evaluation and Program Planning, 40:27 – 41.
Knall, T., Tauber, A., Zefferer, T., Zwattendorfer, B., Axf-
jord, A., and Bjarnason, H. (2011). Secure and
privacy-preserving cross-border authentication: the
stork pilot ”saferchat”. In Andersen, K. N., editor,
Proceedings of the Conference on Electronic Govern-
ment and the Information Systems Perspective (EGO-
VIS 2011), LNCS 6866, pages 94 – 106. Springer.
Leitold, H. and Posch, R. (2012). STORK - Technical Ap-
proach and Privacy book title: Digital Enlightenment
Yearbook 2012, pages 289 – 306. Bus, J. , Crompton,
M. , Hildebrandt, M. , Metakides, G.
Rath, C., Roth, S., Bratko, H., and Zefferer, T. (2015).
Encryption-based second authentication factor solu-
tions for qualified server-side signature creation. In
Francesconi, A. K. E., editor, Electronic Govern-
ment and the Information Systems Perspective, vo-
lume 9265 of Lecture Notes in Computer Science, pa-
ges 71 – 85. Springer.
Suzic, B. and Reiter, A. (2016). Towards secure collabora-
tion in federated cloud environments. In 2016 11th In-
ternational Conference on Availability, Reliability and
Security (ARES), pages 750–759.
Tauber, A., Zwattendorfer, B., and Zefferer, T. (2011).
Stork: Pilot 4 towards cross-border electronic deli-
very. In Verlag, T., editor, Electronic Government and
Electronic Participation - Joint Proceedings of On-
going Research and Projects of IFIP EGOV and ePart
2011, pages 295 – 301.
A Hierarchical Evaluation Scheme for Pilot-based Research Projects
363