Review and Analysis of Cloud Computing Quality of

Experience

Fash Safdari and Victor Chang

School of Computing, Creative Technologies and Engineering, Leeds Metropolitan University,

Headinley, Leeds LS6 3QR, U.K.

Abstract. Cloud computing is gaining growing interest from industry, organi-

zations and the research community. The technology promises many ad-

vantages in terms of cost saving by allowing organizations and users to remote-

ly gain access to a huge pool of data storage and processing power. However,

migrating services to cloud computing introduce new challenges in perfor-

mance and service quality. Quality of service (QoS) is and has been used as a

means of monitoring cloud service performance. However, QoS is based on

network parameters which do not necessarily reflect users’ quality of experi-

ence. This paper discusses cloud computing quality of experience as perceived

by the person using the cloud-based applications and services. We also review a

selected current contribution to the quality of experience (QoE). We have ex-

pert reviews undertaken to identify six key performance indicators (PKIs).

Based on these six KPIs, we sent our surveys, collected feedback and analyze

data to confirm that QoE measurement can meet organizational goals, user sat-

isfaction and stakeholders’ requirements.

1 Introduction

Cloud computing is a new phenomenon in information system and communication

technology and has gained a lot of interests from industry, organizations and the re-

search community. Moving services to cloud mean that applications that used to run

on users’ PCs will reside in data centers distributed across different continents. Cloud

computing promises economic benefits. However, migrating services to cloud com-

puting introduce new challenges in service performance, quality and reliability; tradi-

tionally service level agreements are used as a means of guaranteeing the delivery of

acceptable service performance level. Existing QoS metrics and service level agree-

ments (SLAs) offered by cloud service providers do not include anything about the

performance of service from user point of view and are not adequate to meet users'

quality of service expectations [16]. Organizations are unlikely to adopt cloud com-

puting if users’ experience is not as good as it was before moving to cloud or perfor-

mance does not meet users' perceived quality of experience and performance expecta-

tions [8]. However, due to subjective nature of users’ quality of experience, measur-

ing, capturing and quantifying QoE remains an open research question. In this paper,

we review selected current contributions to QoE. The remainder of this paper is struc-

Safdari F. and Chang V..

Review and Analysis of Cloud Computing Quality of Experience.

DOI: 10.5220/0004982800830088

In Proceedings of the International Workshop on Emerging Software as a Service and Analytics (ESaaSA-2014), pages 83-88

ISBN: 978-989-758-026-0

 2014 SCITEPRESS (Science and Technology Publications, Lda.)

tured as follows: Section 2 describes QoE. Section 3 provides an overview of the

related work on QoE. Section 4 presents the results and discussions from the expert

review and the survey. Section 5 concludes this paper.

2 Quality of Experience (QoE)

Many definitions for QoE have been proposed in literature. In [15] QoE is referred to

as ‘the basic character or nature of direct personal participation or observation’.

Descriptions in [6] states QoE as ‘a concept comprising all elements of a sub-

scriber's perception of the network and performance relative to expectations’. Synthe-

sis in [17] gives the following description for QoE: ‘QoE is a multi-dimensional con-

struct of perceptions and behaviors of a user, which represents his/her emotional,

cognitive, and behavioural responses, both subjective and objective, while using a

system.’. ITU-T [9] defines QoE as ‘overall acceptability of an application or service,

as perceived subjectively by the end user’. There is no a universally acceptable defi-

nition of QoE [15], QoE is outlined in various literatures as a subjective measure.

To model and measure QoE appropriate QoE metrics need to be identified. While

network performance parameters and QoS metrics are well-defined, in contrast QoE

metrics are not well understood and remain an open research question. A large vol-

ume of publications has been dedicated to the study of QoE. In section 3, we review a

selected list and summarize their findings and proposals.

3 Summary of Existing QoE Studies

Casa et al., [2] studied QoE in remote virtual desktop services. Authors outlined a

three layer QoE evaluation framework, consisting of user layer, application layer, and

network layer. Network layer deals with QoS parameters such as bandwidth, applica-

tion layer includes response time, screen compression rate, and colour parameters.

User layer is based on user feedback using Mean Opinion score (MOS). They per-

formed subjective experiment using 52 participants. Their experiment was focused on

users’ overall experience and round trip time. Their main finding is the correlation

between QoE and application QoS requirement. Interactive applications are more

delay sensitive, high RTT degrades user quality of experience. This indicates a direct

correlation between network bandwidth and QoE, low bandwidth has negative impact

on user experience.

Chen et al., [5] argue that obtaining user feedback after they used an application is

not the most appropriate mechanism. They proposed a QoE capture framework called

OneClick where users click a particular button indicating they are not satisfied with

the quality of an application. They assert that dissatisfaction refers to all QoE dimen-

sions and QoE is application specific as applications have different QoS require-

ments. Authors in [5] investigated QoE a group of 52 users using ‘The Box’, a Drop-

box-like application in a controlled lab environment. They observed a correlation

between poor user QoE and low network bandwidth. [8] state that QoE has the poten-

tial to become the new guiding standard for cloud computing quality management.

Authors argued that understanding and managing QoE of cloud services involves a

multidisciplinary view including network, user, and business aspects. Technical fac-

tors that influence cloud computing QoE vary from application to application. QoE-

based classification of application based on level of interactivity, usage domain, and

service complexity is presented.

A QoE management framework was proposed [12], where five steps are outlined:

Service classification provides service contents; Main quality parameters extraction

selects key quality metrics; user feedback collection, QoE analysis predicts reasons

affecting service quality and Performance upgrade. The framework uses simple real-

time oneclick [5] feedback mechanism. They performed an experiments using a 100-

second video streaming, only two network parameters, delay per packet, and packet

loss were measured. Their findings indicate increase in packet delay and packet loss

result in poor QoE. On the other hand, it is not easy to measure QoE in a unified form

because many factors affecting QoE [14]. Authors affirmed that taking into account

all factors could result in a complex QoE metrics; simple QoE metrics are more prac-

tical. An in-service feedback QoE framework is introduced, which runs over a multi-

agent environment, and abstract functions of different agents were discussed.

Kist and Brodie [15] did a case study using two groups of seven and eleven stu-

dents to identifying the effect of QoS on QoE. Their findings indicate that flow (level

of interruption to concentrate on a task), achievability (sense of completing the task),

and extra time needed to complete the task were directly affected by QoS. Their find-

ings are rather generic as there is no mapping of QoE and different QoS metrics.

Statements in [16] assert that QoE will play a major role in the success or failure of

cloud service. The authors argue that understanding QoE requires a multi-disciplinary

view of network, user and business aspects. A conceptual view of cloud services QoE

landscape is proposed, which consists of three main branches: user profile, applica-

tion, and network QoS.

In table 1 we summarized a list of factors identified by different authors that could

have an impact on QoE.

Table 1. Summary of authors’ views on factors affecting QoE.

User demographics

User personality

User interface

Device usability

Application charac-

teristics

Volume of data

Response time

Delay

Jitter

Bandwidth

Packet loss

CPU

Consumptions

Service reliability

[2]

 

[3]



[6]



 

[8]

 



 

[9]



 

  

[13]



   



[15]



 

[16]



[17]



 

   

4 Results from the Expert Review and Survey

QoE is an area that requires the support of results from surveys and quantitative anal-

ysis demonstrated by Chang [3], who identifies six key performance indicators (KPI)

for IT success and explains how these six KPIs can be used in measuring the success

of IT project delivery. The demonstrated examples by [3] are relevant to QoE, in

which expert reviews have been involved. Based on the extension of the work, we

have presented the summary of the importance of six KPIs as follows.

 Relevance for usability for Cloud QoE adoption is important with the support of

real use cases.

 Achieving good performance is an essential for Cloud QoE adoption.

 Security concern is a main reason for some organizations not to adopt Cloud

Computing which researchers describe challenges and issues to be improved.

 Data accuracy is important to compute accurate results so that organizations have

a higher trust and confidence in Cloud QoE adoption.

 Service and data portability are highly relevant to Cloud QoE adoption that re-

searchers demonstrate their usefulness for QoE.

 Scalability is a core characteristic for Cloud and the ability to scale up and down

resources promptly for different demands is essential for Cloud QoE adoption.

These six KPIs are essential for the development of QoE adoption and measurement.

While feedback from experts are important, the feedback from users (in this case,

students) are essential for the development of QoE adoption and measurement. Ques-

tions are based on these six PKIs. Each participant was asked their opinions and rated

the importance of these six KPIs. In this case, we identified that students are likely to

use or get involved in QoE at some stage, which explains why students take part in

this study. As a result, we sent out surveys and collected feedback from students

about their feedback about Cloud QoE adoption.

Analysis of variance (ANOVA) provides statistical test of whether means of sev-

eral groups are equal and can generalize t-test or F-test for two or more groups [4].

ANOVA can be used when these two cohorts have close means and each group has

two sets of data. Table 6 shows the ANOVA analysis with F-test for cohort group

one, where “grp_with_Cloud” is the shorter version of “group_with_Cloud”. Cohort

group one has F-value = 2.78 and Prob > F is 0.1148 (the smaller, the better). The R-

squared value is 0.7875 (the higher, the better) and the root mean square error (MSE)

is 2.74874 (the lower, the better). These key results confirm that the statistical analy-

sis agrees that the use of Cloud QoE adoption help students in their motivation for

learning. Statistical results confirm a good extent of accuracy in analysis by having

high R-squared values; low root mean square values and high F-test values. These

results confirm that we can achieve the stakeholders’ requirements in the first stage of

QoE investigation. The p-value is below 0.1 and is considered a good result. To en-

sure smooth delivery, p-value should be as small as possible. In this case, variations

in p-value have been explained by the variations before and after using Cloud QoE

adoption. Based on direct feedback from some participants, Cloud QoE adoption can

enhance learning and understanding of the subject involved.

Table 2. ANOVA Analysis for the Student Cohort Group.

5 Conclusion and Future Work

This research has focused on the literature on the current state of QoE of cloud com-

puting. The assessment of cloud computing QoE cannot be based only on datacenters

performance efficiency, SLA, and network QoS parameters due to the fact many

different intermediate networks and providers are involved in providing users access

to cloud services. Many factors affect QoE, having a framework that capture all indi-

vidual factors could result in a complex QoE management model. It is more practical

and efficient to capture and model users’ overall acceptability of quality of service in

real-time. Individual applications and services have different QoS requirements. It is

important to monitor network parameters and QoE for individual application and

identify parameters and factors that influence QoE for a given application. This pro-

vides a fundamental start towards developing a model understanding correlation be-

tween QoE and network performance metrics. Apart from the consolidation of litera-

ture, we present that the use of expert reviews and the survey analysis help shake up

the research for QoE adoption. We confirm that the six KPIs essential and statistical

results for the proposed QoE framework to ensure that stakeholders, users and stu-

dents are well engaged in the development work. Their feedback will be useful for a

successful delivery of QoE implementation to ensure that adoption and measurement

can meet the organizational goals, user satisfaction and stakeholders’ requirements.

We also identify more user groups that require QoE, collect their feedback and ana-

lyze our collected data.

References

1. Armbrust, M. Fox, A. Griffith, R. Joseph, A. D. Katz, R. H. Konwinski, A. Lee, G. Patter-

son, D. Rabkin, A. Stoica, I. Zaharia, M. ‘Above the Clouds: A Berkeley View of Cloud

Computing’, Technical Report, No. UCB/EECS-2009-28, UC Berkeley, (2009).

2. Casas, P. Seufert, M. Egger, S and Schatz, R. ‘Quality of Exeperience in Remote Virtual

Desktop Services’, IFIP/IEEE IM2013 Workshop: 1st International Workshop on Quality

of Experience Centric Management (2013).

Number of obs = 30 R-squared = 0.7875

Root MSE = 2.74874 Adj R-squared = 0.5042

Source Partial SS df MS F Prob > F

------------+----------------------------------------------------

Model 168 8 21 2.78 0.0943

grp_with_Cloud 168 8 21 2.78 0.0943

Residual 45.3333333 6 7.55555556

------------+----------------------------------------------------

Total 213.333333 14 15.2380952

3. Chang, V., ‘Business Intelligence in the Cloud, Future Generation Computer Systems, DOI:

http://www.sciencedirect.com/science/article/pii/S0167739X13002926, (2014).

4. Chang, V. and Wills, G., ‘A University of Greenwich Case Study of Cloud Computing –

Education as a Service’, E-Logistics and E-Supply Chain Management: Applications for

Evolving Business, IGI Global, April, (2013).

5. Chen, K. Tu, C, and Xiao, W. ‘OneClick: A Framework for Measuring Network Quality of

Experience’, IEEE, (2009).

6. Empirix. ‘Assuring QoE on Next Generation Networks, Empirix white paper’, (2001).

7. Fischer, H. Casas, P. Suette, S and Schatz, R. ‘First Look at Quality of Experience in Per-

sonal Cloud Storage Services’, IEEE International Conference on 1stt International Work-

shop on Mobile Cloud networking and Services, (2013).

8. Hobfeld, T. Schatz, R. Varela, M. and Timmerer, C. ‘Challenges of QoE Management for

Cloud Applications’, IEEE Communications Magazine, (2012).

9. ITU-T. ‘Vocabulary and effects of transmission parameter on customer opinion of trans-

mission quality’, (2006),

10. Laghari, K. Crespi, N. Molina, B. ‘QoE aware Service Delivery in Distributed Environ-

ment’, IEE Computer society (2011).

11. Jarschel, M. Schlosser, D. Scheuring, S and Hoßfeld, T. ‘An Evaluation of QoE in Cloud

Gaming Based on Subjective Tests', IEEE computer society., (2011).

12. Jie, Z and Kim, H, 'User Feedback Oriented Quality of Experience

Management Framework', China communications., (2013).

13. Khalil, R. Crespi, N. Molina, B. and Palau,C. ‘QoE aware Service Delivery in Distributed

Environment’, 2011 workshop of International Conference on Advanced Infromation Net-

working and Applications’, (2011).

14. Kim. H. Lee, K. and Zhang, J. 'In-Service Feedback QoE Framework', 2010 Third Interna-

tional Conference on Communication Theory, Reliability, and Quality of Service, (2010).

15. Kilkki, K. ‘Quality of Experience in Communications Ecosystem’, Journal of Universal

Computer Science, vol. 14, no. 5 (2008).

16. Varela, M. and Sankala, J, ‘ Understandin Quality Issues in the Cloud’, IEEE compting,

(2010).

17. Wu, w. Arefin, A. Rivas, R. Nahrstedt, K. Sheppard, Z. Yang, Z.: ‘Quality of Experience in

Distributed Interactive Multimedia Environments: Toward a Theoretical Framework’, Pro-

ceeding of ACM internamtional conference on Multimedia, (2009).