![](bg8.png)
7 CONCLUSIONS AND FUTURE
WORK
In this paper, we have established a methodology for
the collaborative construction and evaluation of BP
repositories. For this purpose, we used a BP
searching mechanism applying graded measures to
determine the relevance degree of the retrieved
elements. Consequently, this allowed the
demonstration of the usefulness of the responses and
their relationship to queries submitted by users.
These responses serve as the most appropriate
responses for evaluating and comparing searching
mechanisms that use the same repository.
The collaborative evaluation allows judges to
have an overview of the relevance judgments issued
by each judge on elements retrieved in the results
list. As a result, judges can compare the concordance
or discordance in the relevance judgment issued for
an evaluated item and thus corroborate or change
their assessment.
The data shows that there are some differences in
the points of view of the evaluators. While most
experts considered the items ordered at the top of the
result list (1, 2, 3, 4) as relevant or very relevant, a
minority (10%) of these were considered as not
relevant or irrelevant. This is because the latter took
into account only one part of the evaluation process
(linguistic or structural), or simply because the
comparison between the BP query and each one of
these results was performed superficially, which
may have been due to fatigue as a result of the huge
number of evaluations performed.
The application methodology proposed serves as
the basis for the generation of stable evaluations of
BP repositories, which are thus more maintainable
and reusable. In addition, as a secondary
contribution, the BP repository that was used in our
evaluation can be seen as an open access repository
that will be shared, expanded with new models BP,
and can be used in future researches by any actor
interested in the area of BP management.
As a future work, it is aimed to expand the
evaluation methodology by manually creating
groups or families of BPs with those BPs considered
as truly relevant in each one of the queries. This
allows group representation of thematic topics or
structural patterns of the BPs within the repository.
ACKNOWLEDGEMENTS
The authors would like to acknowledge the
collaboration of the judges belonging to the Institute
of Informatics and of the Business Management
School of the Federal University of Rio Grande do
Sul (Brazil) and the Department of Telematics of the
University of Cauca (Colombia). They also
acknowledge the Institute of Informatics for sharing
their infrastructure. Finally, we would like to state
that this research was partially supported by CAPES
and CNPq, Brazil.
REFERENCES
Becker, M. and R. Laue (2012). "A comparative survey of
business process similarity measures." Computers in
Industry 63(2): 148-167.
Chris J. Turner, A. T., Jorn Mehnen (2010). "A Genetic
Programming Approach to Business Process Mining."
Dijkman, R., M. Dumas, et al. (2011). "Similarity of
business process models: Metrics and evaluation."
Information Systems 36(2): 498-516.
Dijkman, R., M. Dumas, et al. (2011). "Similarity of
business process models: Metrics and evaluation."
Information Systems 36: 498-516.
Koschmider, A., T. Hornung, et al. (2011). "Re-
commendation-based editor for business process
modeling" Data & Knowledge Engineering 70: 483-503.
Kunze, C. R. a. M. (2013). "An Extensible Platform for
Process Model Search and Evaluation." Business
Process Management Demos 2013: Beijing, China.
Kunze, M. and M. Weske (2012). "An Open Process
Model Library." Business Process Management
Workshops, BPM 2011 International Workshops
Clermont-Ferrand, France, August 29, 2011 Revised
Selected Papers, Part II: 26-38.
Küster, U. and B. König-Ries (2009). "Relevance
Judgments for Web Services Retrieval - A
Methodology and Test Collection for SWS Discovery
Evaluation." 2009 Seventh IEEE European
Conference on Web Services: 17-26.
La Rosa, M., H. a. Reijers, et al. (2011). "APROMORE:
An advanced process model repository." Expert
Systems with Applications 38: 7029-7040.
Ordoñez, C. F., Juan Carlos Corrales, Carlos Cobos
(2013). "Multimodal Model for Business Process
Search." Submitted to : Information-Sciences.
Rosa, L., H. M. Arthur, et al. (2010). "QUT Digital
Repository: http://eprints.qut.edu.au/ This is the author
version published as:."
Škrinjar, R. and P. Trkman (2012). "Increasing process
orientation with business process management:
Critical practices’" International Journal of Infor-
mation Management.
Tsetsos, V., C. Anagnostopoulos, et al. (2006). "On the
Evaluation of Semantic Web Service Matchmaking
Systems." 2006 European Conference on Web
Services (ECOWS'06): 255-264.
Yan, Z., R. Dijkman, et al. (2012). "Business process
model repositories – Framework and survey."
Information and Software Technology 54: 380-395.
ICEIS2014-16thInternationalConferenceonEnterpriseInformationSystems
318