A Research Agenda for Mobile Systems Evaluation
Tamara Högler
CyberForum e.V., Haid-und-Neu-Str. 18, 76131 Karlsruhe, Germany
Keywords: Research Agenda, Mobile Systems, Evaluation, Information and Communication Systems.
Abstract: The present work shows the necessity of an economic evaluation model that is based on singularities of
mobile systems and that takes into account the interdependencies of their individual components. A
motivation for this approach is not only given by the continuing discussion on the economic efficiency of
mobile systems, but also by the fact that appropriate methodologies for comprehensive evaluation still do
not exist. Starting point for a research agenda is the definition of the term mobile system, followed by the
explanation of the single components and singularities of such systems. The findings of the present work
motivate the development of a generic model for economic evaluation. By defining the research agenda we
provide guidance for constructing such a model.
1 INTRODUCTION
For nearly three decades, the debate about the cost-
effectiveness of information and communication
systems (ICS) is consistently resurrected. In the late
80´s Solow stated, that the effects of computers can
be seen everywhere – except in the productivity
statistics (Solow, 1987). Also Loveman had no
doubt that “IT capital had little, if any, marginal
impact on output or labor productivity, whereas all
the other inputs into production -including non-IT
capital – had significant positive impact on out-put
and labor productivity” (Loveman, 1994). The
current state of scientific knowledge though presents
opposite results: The productivity paradox does not
exist in praxis – it is caused by the lack of
appropriate methodologies for the economic analysis
of ICS (see e.g. Brynjolfsson, Hitt and S. Yang,
2002).
Information and communication technologies
(ICT) are often implemented in order to conduct
businesses as efficient as possible; the quantitatively
definable monetary effects are considered the most
important objectives when implementing such a
system. For the economic evaluation mostly
methodologies are applied that focus exactly on
these effects. This approach may easily fudge the
results due to the fact that the full benefits of the
technologies are insufficiently reflected – and thus
leads to the assumption that a productivity paradox
exists.
Literature study shows that there is still a lack of
appropriate evaluation methodologies (see e.g.
Högler, 2012; Ashurst, Doherty and Peppard 2008).
Especially integrative and qualitative effects of the
systems are mostly not considered in the calculation
(Pietsch, 1999) – one of the main benefits of ICS.
This is even more important for mobile systems that
represent a special "mobile" form of ICS. It can be
assumed that mobile systems face the same
difficulties concerning the economic evaluation as
stationary ICS. Additionally they are affected by
challenges that result from the fact that mobile
technologies are mostly used during “mobile
actions” like travelling or walking. Literature on the
effectiveness of mobile systems is scarce; therefore,
in this position paper, we explore the domain of
mobile systems from an evaluation angle.
The term “mobile system” is defined in section
2, and section 3 describes the single components of
such systems. The unique singularities of mobile
systems are elaborated in section 4. Section 5 will
outline economic evaluation of mobile systems
based on their singularities and main characteristics;
for this we define a number of research questions.
We finish our paper with conclusions.
2 HOW TO DEFINE MOBILE
SYSTEMS
Although often used synonymously in the literature,
454
Högler T..
A Research Agenda for Mobile Systems Evaluation.
DOI: 10.5220/0004972504540459
In Proceedings of the 16th International Conference on Enterprise Information Systems (ICEIS-2014), pages 454-459
ISBN: 978-989-758-028-4
Copyright
c
2014 SCITEPRESS (Science and Technology Publications, Lda.)
this work strictly distinguishes between ICT and
ICS. Starting point for the elaboration of a definition
is systems theory, an approach that focuses on
entities and that postulates that the system itself
comes into existence by the relationships among the
system elements and the resulting interactions. This
approach was chosen due to the fact that for an
economical evaluation based on singularities of
mobile systems the relationship of entities or
components respectively is of key importance. The
analysis of structures, reactions and functions of the
entities allows certain predictions about the expected
system behaviour, whereas it does not focus on a
separate consideration of each element (see also
Boulding, 1956 and Bertalanffy, 1976).
Having the above given considerations in mind,
it becomes clear that in contrast to ICT, the term ICS
has to include also system-elements of human nature
(human beings) besides technologies that support
information exchange and communication (so called
“technical components”) and thus take into account
the relationships between the single components,
properties as well as their behaviour (so called
system parameters).
System parameters are variables, whose values
characterize the behaviour of the system with a
given structure (DIN, 1995). Since the behaviour of
a system and therefore its cost-effectiveness are
influenced by their interaction or controlling of
system parameters, they play an important role for
the evaluation of mobile systems. System parameters
with the largest influence on a system are
characterized in the present work as “success
factors”. The term success factor is used in the
literature to characterize the cause of success.
(Corsten, 2000) defines success factors as "[...]
factors that have a significant impact on the potential
success of a strategic business area. While on the
one hand it is emphasized that the individual factors
[....] depend on the industrial sector in which the
company operates, on the other hand a hypothesis is
supported that there are so-called basic factors [....],
that matter for the success or failure over all
industries, i.e. it is assumed that the structure of the
factors´ system is relatively constant, while the
weighting of the individual factors is subjected to
frequent changes”. In dependence on (Rockart,
1979) and relating mobile systems, the current work
defines critical success factors as technical as well as
human system parameters that have a significant
impact on the economics of the mobile system.
Relationships between the single components are
represented by the processes that take place between
them whereas the structure and organization of the
ICS symbolize the characteristics of the elements.
When reflecting on the given definition of the term
ICS it becomes clear, that the human component is
of key importance when discussing the profitability
of ICS. Users are involved in all processes, they are
using the technical components and determine the
success factors for the (economic) efficiency of the
whole system.
Within this research work, mobile systems are
chosen as subject of investigation. Mobile systems
can be regarded as ICS that are extended by mobile
aspects. They exist in different forms and have a
multiplicity of characteristics, aiming at integrating
people, mobile processes and workstations into
internal, mostly stationary corporate and enterprise-
wide process chains and thus to overcome their
spatial separation and accompanying information
losses.
Basing on the general systems theory and
following the above given socio-technical definition
of the term ICS, the present work defines a mobile
system as a set of mobile technology and human
(system) elements, which are inherently related (see
also the discussion of (Goos and Zimmermann,
2005) concerning the term system). They form an
entity due to their interactions that is task-related
and that executes corresponding business processes.
Mobile systems, as a unit, distinguish themselves in
this respect from the surrounding environment by
the relations between their components and the
effects that take place between the single
components. In the following sections the
components as well as the singularities of a mobile
system are discussed.
3 COMPONENTS OF MOBILE
SYSTEMS
A mobile system consists of two types of
components: Technical and non-technical
components. The distinction of the components is
important because the economy of mobile business
processes is not only affected by technical, but
particularly by the human system components
(users).
Technical components comprise mobile
hardware (e.g. Smartphones and Tablets),
appropriate applications as well as mobile operating
systems and middleware (if necessary). They include
also wireless communication technologies like LTE,
UMTS and WLAN (see figure 1). The particular
potential of mobile technologies lies primarily in the
AResearchAgendaforMobileSystemsEvaluation
455
possibility of reorganizing processes and thus in the
exhaustion of the value added that is facilitated by
mobile technologies. Especially mobile ICT
contributes to the efficient support of processes by
bridging spatial and temporal distances (Schiller,
2000). Figure 1 shows technical as well as non-
technical components of mobile systems. Security
issues play an important role for mobile processes
and thus can influence significantly the economic
efficiency of such systems. The importance of
security in the field of wireless and Internet-based
systems is a key research topic of the project Be
Wiser – Building Enterprises: Wireless and Internet
Security in European Regions, funded by the
European Commission under the 7
th
Framework
Programme (project reference FP7-REGIONS-2012-
2013-1). It is expected that main findings of this
project will be included in the authors further
research.
Figure 1: Components of a mobile system.
Non-technical 'components' are users who use
the technical components in order to perform their
tasks. These can be for example employees who
accomplish their jobs in a mobile way (means: not at
a stationary workplace) either within or outside the
enterprise (e.g. sales people, customer service and
maintenance engineers). Human-Computer-
Interactions (HCI) is a well investigated research
field, thus this work refers to books and proceedings
like “Human-Computer Interaction – INTERACT”
(Gross et al. 2009), “Digital Technology Research”
(Price et al., 2013) and articles within the IEEE-
journal “Pervasive Computing”. They explain the
human-computer-interactions in detail. The effects
of HCI on economic efficiency is still an open
research topic and will be elaborated by the author to
a later point.
Mobile processes differs from stationary
processes primarily by its spatial distribution and the
mobility of the persons who are involved in the
process. Mobile processes can be characterized as
follows: At their beginning it is often not exactly
known where and when they will take place – many
uncertainties can influence these processes and thus
make them at least partly unpredictable. Example:
Even if a sales man has an appointment with a
customer at a defined location and at a defined time,
there is still an uncertainty that he will not be able to
attend this meeting: A strike, storm, breakdown or
even an accident can foil his plans. Thus, following
these explanations, distributed processes with a
determined distribution structure are not mobile
processes (Köhler and Gruhn, 2004).
The evaluation of a mobile system has to take
into account all interdependencies between the
single components. In order to do so, e.g. following
questions have to be answered: “How do technical
components like mobile devices, applications and
data transfer affect each other? And how can the
most important component of the mobile system –
the human being – be affected by the technical
components or by the surroundings when proceeding
tasks? How can the singularities of different working
profiles (e.g. maintenance engineer, businessman)
and their experience with mobile technologies be
considered within the holistic profitability analysis?”
In order to be able to answer these questions, it is
necessary to understand the singularities of mobile
systems that will be discussed in the following
section.
4 SINGULARITIES OF MOBILE
SYSTEMS
In an investigation we identified the following
singularities, which are supported by argumentation
as we present them. Note that not only literature
study but also practical experience and observation
have contributed to the identification of the
singularities.
ICEIS2014-16thInternationalConferenceonEnterpriseInformationSystems
456
Mobile systems have many different forms and a
multiplicity of characteristics and singularities in
comparison with stationary ICS. These findings
motivate the development of a profitability analysis
that takes into account the singularities of mobile
systems and that takes the human component as
central hub and pivotal point when evaluating such a
system.
The aim of mobile systems is to integrate mobile
processes and workstations with internal, mostly
stationary corporate and enterprise-wide process
chains and thus to overcome their spatial separation
and accompanying information losses. By the
ubiquitous access to relevant information, mobile
technologies promise an increased efficiency of
business processes and enable new ways of working.
At the same time, mobile systems face many
challenges and hurdles stationary ICS are not
confronted with, like security issues or the absence
of data networks. The following paragraphs will
enlighten the most apparent singularities of mobile
systems.
Mobile systems can be easily distinguished by
stationary ICS due to their singularities. Starting
with the technical components of a mobile system, it
becomes clear that in contrast to desktop computers
mobile devices are continuously transported. This in
turn requires a minimum weight and a small size of
the devices with maximum robustness.
According to (Schach et al., 2007, Lonthoff and
Ortner, 2007; Högler, 2014) mobile devices face
many restrictions despite intensive research and
technological progress of the past years. In contrast
to ICT, mobile devices have – due to the low battery
capacity – only a limited power supply and are
seldom plugged in local area networks. This fact
requires increased energy efficiency of mobile
devices and corresponding applications and stable
wireless Internet connection. With decreasing size of
the devices, also the computing capacity becomes
lower. In conjunction with inefficient main storage
mobile devices have lower information processing
capacities compared to the capacities of stationary
ICT. This fact must be taken into account when
developing mobile applications, which have to cope
with the mentioned restrictions of mobile devices
(see also Kornmeier, 2009). Also the input options
of mobile devices offer only restricted possibilities:
Most keyboards are missing or incomplete and in
many cases unhandy, virtual keyboards still do not
offer the same usability as standard ones.
Additionally, in many cases the worker has not both
hands free, which imposes additional usability
requirements on the keyboards and the input
methods respectively. Especially as regards to the
writing speed, this kind of keyboards will not
achieve the comfort and usability of traditional ones.
The output options hinder the usage of mobile
devices due to the relatively small displays, which
have limited facilities for the presentation of
contents. Thus, special applications which take into
account peculiarities of mobile devices are
specifically designed and developed.
The distraction caused by the surroundings
depicts also a singularity of mobile systems. In
contrast to stationary working places, a mobile
worker is distracted by his surroundings, e.g. by
noise, incidents and weather. For example, mobile
devices are hardly usable in rain or dusty areas, also
ambient light is a real challenge: Images and texts
are less visible than in closed rooms and thus
exhaust the eyes of the users, although automatic
recognition of ambient light and adjustment of the
backlight is available for most devices.
Reliable data transmission is an unsolved field,
too. Transmission problems can be caused by
fluctuating bandwidth or insufficient network
coverage and thus can hinder continuous work with
mobile devices (Gerpott and Kornmeier, 2004;
Princen and Schreurs, 2010). Slow or interrupted
connections represent disruptive factors and may not
only reduce the quality of service, but also affect the
efficiency of work: The accessibility of required data
everywhere and anytime is of key importance in
order to reach the maximum possible efficiency of
mobile systems (Högler, 2014).
The relatively broad variety and fast
enhancements of operating systems are still regarded
as a challenge for the employment of mobile
devices. Many mobile applications run only under
one operating system and thus can cause
synchronization problems. Additionally, the
integration of applications into existing systems and
their interoperability is not resolved satisfactorily in
many cases. With the widespread adoption of cloud-
based solutions these problems should become less
important within the next few years.
Compared to stationary computers, data security
in mobile applications and devices is low – although
a broad variety of security mechanisms already
exists. The main reason for this security problem is
not technology, but the user of mobile devices who
bypass security mechanisms for convenience or
ignorance. As mobile devices are lost or stolen much
more frequently than their stationary counterparts
(Frolick and Chen, 2004; Gluschke, 2001; Day et al.
2000) and as many users log into unsecure wireless
networks without taking into account all the risks
AResearchAgendaforMobileSystemsEvaluation
457
they are facing, the security issue is not yet solved in
the area of mobile technologies satisfactorily.
The last paragraphs have shown the most evident
singularities of mobile systems. When evaluating
such a system, it is necessary to take into account all
these restrictions and particularities – they may not
only affect the work but also the economic
efficiency of mobile systems. For example, major
security problems can decrease the monetarily
quantifiable advantages of mobile systems. In order
to benefit from the full potentials mobile systems
bear, it is important to take into account not only the
depicted singularities, but also to approve the human
component as the most influencing (success) factor
on the economic efficiency. This finding motivates
the development of a research agenda that covers
evaluation of mobile systems and that is based on
the singularities of mobile systems.
5 A RESEARCH AGENDA FOR
ECONOMIC EVALUATION OF
MOBILE SYSTEMS
In order to plan research steps for mobile systems
evaluation, based on the above considerations, we
are now able to identify a number of research paths.
We believe that once answers to related research
questions are provided mobile systems productivity
can be monitored and improved. Results pave a path
in 1) creating more insight into the productivity of
mobile ICT, 2) identifying possible areas of
improvement for existing mobile systems, 3)
managing running mobile system implementation
projects, and 4) evaluating mobile system
implementations. We identify the following research
areas:
- Generic identification and further validation of
components of mobile systems and their
relations as suggested in our figure 1;
- Further confirmation, detailing and identification
of singularities as touched upon in our section 3;
- Determination of success factors from
singularities, system components behaviour and
interdependencies between components;
- Construction of a model or models for mobile
systems evaluation taking into account success
factors, components and interdependencies of
components, leveraging system theory;
- Validation, case studies and more related to
constructed mobile systems evaluation models.
These areas can be stepwisely addressed in
further maturing the knowledge base of research on
mobile system evaluation.
6 SUMMARY AND
RECOMMENDATIONS
The present work has shown the motivation for and
necessity of economic evaluation models that
address singularities of mobile systems. The current
state of scientific knowledge has shown that the
productivity paradox does not really exist in praxis.
In fact, it is caused by the lack of appropriate
methodologies for the economic analysis of ICS.
Appropriate evaluation methodologies for ICS are
still missing. None of existing methodologies takes
into account all components, their interdependencies
and singularities of these kinds of systems (section
1). This is even more important for mobile systems
that represent a special "mobile" form of ICS.
The present work takes the systems theory as
starting point for the development of a definition for
mobile systems (section 2). The reason for this
approach lies in the fact, that systems theory focuses
on entities and that postulates that the system itself
comes into existence by the relationships among the
system elements and the resulting interactions – the
basis for the development of an economic evaluation
model that is based on singularities of mobile
systems. Section 3 has presented the single
components of a mobile system and thus rounded off
section 2.
Mobile systems, as seen in section 4, are affected
by challenges stationary systems do not face. Their
singularities result from the fact that mobile
technologies are mostly used when the employee is
working apart from a stationary workplace. It is
these singularities that – in addition to the human
component of a mobile system – decide about the
success and thus about the economic efficiency of a
mobile system. When evaluating such a system, it is
of key importance to take into account all
singularities and to analyse the interdependencies of
the single system components; our section 5
provides a stepwise research agenda for this.
ACKNOWLEDGEMENTS
First and foremost, I would like to thank Prof. Dr.
Johan Versendaal as my supervisor for his most
support and sharing his knowledge. He kindly read
ICEIS2014-16thInternationalConferenceonEnterpriseInformationSystems
458
my paper and offered detailed valuable comments
and advices on the topics addressed. Second, I
would like to thank Gunther Schäfer – without his
support and patience this work would not have been
possible.
REFERENCES
Ashurst, C. A.; Doherty, N. F.; Peppard, J. P. (2008):
Improving the Impact of IT Development Projects:
The Benefits Realization Capability Model. In:
European Journal of Information Systems 17 (2008) 4,
p. 352-370.
Bertalanffy, L. v. (1976): General System Theory. George
Braziller Inc., New York.
Boulding, K. (1956): General Systems Theory – The Ske-
leton of Science. In: Management Science 2: 197-208.
Brynjolfsson, E.; Hitt, L.; and S. Yang (2002): Intangible
Assets: How the Interaction of Computers and
Organizational Structure affects Stock Market
Valuations. In: Brookings Papers on Economic
Activity: Macroeconomics (1), p. 137-199.
Corsten, H. (2000): Lexikon der Betriebswirtschaftslehre.
4th Edition. Oldenbourg. Munich.
Day, R., Daly, J., Sheedy, T. and Christiansen, C. (2000):
Widening Your Secure eBusiness to Wireless.
Framingham: IDC. http://www.bandwidthco.com
/whitepapers/wirelessforensics/security/Widening%20
Your%20Secure%20ebusiness%20to%20Wireless.pdf
DIN 1995: OENORM DIN 19226-1:1995-07-01:
Leittechnik – Regelungstechnik und
Steuerungstechnik – Allgemeine Grundbegriffe.
Beuth, Berlin.
Frolick, M. N. and Chen, L.-D. (2004): Assessing m-
Commerce Opportunities. Information Systems
Management, 21 (2), 53-61.
Gerpott, T.J. and Kornmeier, K. (2004): Electronic
Security (E-Security). In: Sjurts, I. (Ed.): Gabler
Lexikon Medienwirtschaft. Gabler. Wiesbaden.
Goos, G. and Zimmermann, W. (2005): Vorlesungen über
Informatik, Band 1: Grundlagen und funktionales
Programmieren. 4th Edition, Springer, Berlin.
Gluschke, G. (2001): Sicherheit in mobilen Systemen. In:
Rossbach, G. (Ed.).: Mobile Internet. dpunkt.
Heidelberg.
Gross, T., Gulliksen, J., Kotzé, P., Oestreicher, L.,
Palanque, P., Prates, R.O., Winckler, M. (Eds.) (2009):
Human-Computer Interaction - INTERACT 2009.
Proceedings of the 12
th
IFIP TC 13 International
Conference, Uppsala, Sweden.
Högler, T. (2012): Framework for a Holistic Evaluation of
ICT. In: Proceedings of the 25th Bled eConference
Graduate Student Consortium. http://bledconference
.org/public/conferences/1/Bled2012GSC-Proceedings.pdf
Högler, T. (2014): Success Factors and the Holistic
Approach for Profitability Analysis of Mobile
Systems. Submitted for conference publication at the
Twenty Second European Conference on Information
Systems, Tel Aviv.
Köhler, A. and Gruhn, V. (2004): Mobile Process
Landscaping am Beispiel von Vertriebsprozessen in
der Assekuranz. In: Pousttchi, K.; Turowski, K. (ed.):
Mobile Economy: Transaktionen, Prozesse,
Anwendungen und Dienste. Proceedings of the 4.
Workshop Mobile Commerce. Köllen, Bonn.
Kornmeier, K. (2009): Determinanten der
Endkundenakzeptanz mobilkommunikationsbasierter
Zahlungssysteme: Eine theoretische und empirische
Analyse. PhD thesis, published at: http://
duepublico.uni-duisburg-essen.de/servlets/Derivate
Servlet/Derivate-21559/Dissertation_Kornmeier.pdf.
Lonthoff, J. and Ortner, E. (2007): Klassifikations- und
Lösungsansätze für Web Services im mobilen Umfeld.
In: König-Ries, B., Lehner, F., Malaka, R. and Türker,
C. (Ed.): Proceedings of the MMS 2007 – Mobilität
und Mobile Informationssysteme, p 73-84. Lecture
Notes in Informatics. Aachen.
Loveman, G. (1994): An Assessment of Productivity
Impact of Information Technologies. In: Allen, T. and
M. Scott Morton: Information Technology and the
Corporation of the1990‘s. Oxford. New York.
Pietsch, T. (1999): Bewertung von Informations- und
Kommunikationssystemen – Ein Vergleich
betriebswirtschaftlicher Verfahren. Erich Schmidt,
Berlin.
Price, S.; Jewitt, C.; Brown, B. (2013): Digital Technology
Research. London. Sage Publications.
Princen, T.; Schreurs, J. (2010): Secured Blackberry Used
in E-Learning. In: Auer, M.; Schreurs, J. (Ed.)
Academic and corporate e-learning in a global context.
International Association of Online Engineering.
Hasselt.
Rockart, J.F. (1979): Chief Executives Define Their Own
Data Needs. In: Harvard Business Review 57 (1979)
2, p. 81-93.
Solow, R. (1987): We‘d Better Watch Out. In: New York
Times Book Review, July 12.
Schach, R., Scherer, R., Menzel, K. et al. (2007): Mobile
Computing im Bauwesen: Konzepte, Anwendungen,
Potenziale. Expert Verlag. Renningen.
Schiller, J. (2000): Mobilkommunikation. Techniken für
das allgegenwärtige Internet. Addison-Wesley, Munic.
AResearchAgendaforMobileSystemsEvaluation
459
