The Web-based Subjective Quality Assessment of an Adaptive Image

Compression Plug-in

Maria Laura Mele

1, 2

, Damon Millar

and Christiaan Erik Rijnders

Department of Philosophy, Social & Human Sciences and Education, University of Perugia, Perugia, Italy

COGISEN Engineering Company, Rome, Italy

Keywords: Adaptive Systems, Image Compression Methods, Subjective Quality Assessment.

Abstract: Images are a key element for conveying information about visual systems. However, image-based

representation and communication require large information bandwidth. Image compression is currently the

leading methodology for reducing bandwidth/load problems thus improving User Experience. Synthetic

objective metrics are often used to assess the quality of image compression models, but they often do not

reliably predict subjective ratings. This work shows the end-users’ quality evaluation of a new compression

plug-in fully compliant with all on-going image formats. The subjective quality assessment of jpeg pictures

compressed by the plug-in followed a new Web-based Single Stimulus Continuous Quality Scale method,

whose validity and reliability have been described in a previously published study. The results of this study

show that pictures compressed by the proposed adaptive image compression plug-in have a 55%

compression gain compared to jpeg images compressed by Facebook Mobile, with no loss in perceived

image quality.

1 INTRODUCTION

Images are often used as a representation method to

convey information in Web-based systems. Visual

content is cognitively processed in a non-

propositional way, that is to say, no decoding is

necessary to process the depicted data complexity

and its inner relations, since images keep the

perceptual structure of what they represent

(Sternberg and Sternberg 2015).

Given the importance of using images for Web-

based information and communication, reducing the

bandwidth usage due to a massive quantity of

images is a goal for practitioners, especially for

smartphones and other mobile devices. As Jakob

Nielsen highlights when describing his Law of

Internet Bandwidth, “bandwidth will remain the

gating factor in the experienced quality of using the

Internet medium” (Nielsen 1998).

Lossy image compression is still the primary

solution for reducing bandwidth and saving storage

space (Vidhya et al. 2016). The State of the Art

shows many new compression algorithms, which are

effective in reducing the size of the original

representation with no loss in image quality (Sarode

et al. 2016). Nevertheless, the jpeg file format,

which was released in 1991, is still the predominant

image format, because shifting into different file

formats files can result in compatibility issues with

systems. Indeed, many new compression algorithms

require format shifting and end-users might

encounter compatibility problems with their

software or devices.

Another problem with compression methods is

that their optimizations are driven by synthetic

objective quality evaluation metrics, which are often

not efficiently predictive of subjective evaluations.

Even though synthetic objective metrics are fast,

repeatable and do not have high costs, they are not

always able to reliably predict subjective image

quality ratings assigned by human participants. The

reason for this is that these objective metrics are

derived from subjective quality datasets, which are

subsets by definition. Instead, the proper method is

to be driven by the subjective quality evaluation

tests (Winkler et al. 2012).

Subjective quality evaluation is still a key

process in image or video compression methods

because low perceived quality contributes directly to

a poor user experience (Pedram et al. 2014).

This paper describes the subjective quality

assessment of a compression plug-in developed by

an engineering company called Cogisen

(www.cogisen.com). The compression method is

based on a new visual saliency algorithm, which, for

Mele M., Millar D. and Rijnders C.

The Web-based Subjective Quality Assessment of an Adaptive Image Compression Plug-in.

DOI: 10.5220/0006226401330137

In Proceedings of the 12th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications (VISIGRAPP 2017), pages 133-137

ISBN: 978-989-758-229-5

133

each image, calculates a map of the perception

threshold beyond which users might perceive any

reduction in quality. The system has been developed

to be compatible with any kind of image

compression models and flexible to all current image

formats. The system also has a minimal impact on

mobile processor usage.

The subjective evaluation of the Cogisen

compression plug-in followed an image quality

assessment method, which adapts the Single

Stimulus Continuous Quality Scale method

described by the ITU-R BT.500-8 recommendations

(ITU Telecom 2002) to a Web-based procedure

using an online testing tool and a crowdsourcing

Web platform for recruiting participants. Both

validity and reliability of the Web-based procedure

have been investigated in a previous study, which

compares the results obtained using the Web-based

method to data obtained with laboratory methods

(Mele et al. 2016).

This paper describes the subjective quality

evaluation of images compressed by two methods:

Facebook Mobile’s jpeg compression algorithm

(which represents the optimized maximum

compression level of commonly used jpeg

compression), and The Cogisen plug-in, which

reduces information in non-salient parts of jpeg

images.

2 SUBJECTIVE QUALITY

EVALUATION OF THE IMAGE

COMPRESSION PLUG-IN

2.1 Method

The subjective quality assessment procedure used in

this study was validated using a data set of images

whose quality has been previously ranked by

participants (Mele et al. 2016). We compared the

Cogisen’s compression algorithm with the Facebook

Mobile’s one because Facebook is a public service

by definition and its compression model complies

with the State of the Art.

2.2 Material

The stimuli used for this study were obtained from a

selection of six high quality reference pictures

(800x800 pixels) provided by the Colourlab Image

Database: Image Quality (CIDIQ) (Liu et al. 2014).

Six of 23 high-quality pictures were selected to

represent a wide range of photographic subjects

(Figure 1). The selected stimuli include the

following pictures: one outdoor panorama, one

indoor panorama, one man-made object picture, one

picture with distinct foreground/background

configurations, one picture without any main

specific object of interest, one close-up shot.

The reference pictures were compressed by two

compression methods:

1. The Facebook Mobile compression

algorithm.

2. The Cogisen plug-in, according to the

following compression gains over

Facebook Mobile: 25%, 35%, 45%, 55%.

Four distorted pictures (not belonging to

the experimental database) were placed

twice into the test in order to control

subjects’ attention.

The test consisted of a total of 49 trials presented

in a randomized order, in order to avoid that two

identical pictures are presented consecutively.

2.3 Procedure

Participants were asked about visual acuity, contrast

sensitivity, colour vision, light condition and prior

experience with video display systems or devices, by

a questionnaire shown before the test. Participants

were also asked to check the physical dimensions of

their display and to regulate it to the maximum

brightness. If participants met the preliminary

requirements (no vision impairments, only desktop

devices, no devices less than 13-inches wide,

maximum brightness on) test instructions were

displayed.

Each image was presented for 7 seconds, then a

1-100 integer quality scale, numerically marked and

divided in three parts by the “Bad”, “Fair”, and

“Excellent” labels, was displayed for at least 3

seconds. Subjects were asked to assess the quality of

each picture by dragging the slider on the quality

scale. This Single Stimulus Continuous Quality

Scale (SSCQS) evaluation method follows the ITU-

R BT.500-8 recommendations (ITU Telecom 2002).

The test has been developed by using an online

survey software tool, i.e., SurveyGizmo

(www.surveygizmo.com).

To prevent errors due to subjects’ fatigue and

loss of attention, the testing session lasted a

maximum 20 minutes. The subjective quality

assessment consisted in one single session with

participants recruited by means of a crowdsourcing

platform for psychological research called Prolific

Academic (www.prolific.ac). All participants were

first time users of such a quality assessment

procedure, and they were remunerated with a £1.25

payment.

HUCAPP 2017 - International Conference on Human Computer Interaction Theory and Applications

134

2.4 Subjects

Sixty-three subjects (mean age= 32.6 years old,

50.7% male, 100% English speakers), eight expert

users (mean age= 34.87, 87.5% male) and 28 non-

expert viewers (mean age= 32.3, 45.4% male) were

recruited. All the tests were completed in a single

session on March 10, 2016. Expertise was classified

according to the participants' employment as

reported in a preliminary questionnaire.

Since five outliers were excluded after the

descriptive analysis, the screened subjective

database included the scores provided by a total of

58 subjects, mean age= 37.2 years old, 48.3% male,

44.8% in-door with natural lights; 55.2% indoor

with artificial lights.

2.5 Results

The basic data analysis included:

 Mean opinion score (MOS): Opinion scores

were integers in the range 1-100;

 Difference mean opinion score (DMOS):

The raw opinion scores were converted to

quality difference scores:

= r

ref(j) – r

where r

is the raw score for the i-th subject and j-th

image, and r

ref(j) denotes the raw quality score

assigned by the i-th subject to the reference image

corresponding to the j-th distorted image (Sheikh et

al. 2006). Difference Mean Opinion Scores (DMOS)

were therefore obtained by calculating the difference

between the Mean Opinion Scores (MOS) (range 1-

100) assigned to high quality reference images and

those assigned to the compressed images (Table 1).

Table 1: Subjects’ Mean Opinion Scores and Difference

Mean Opinion Scores assigned to both Cogisen

compressed pictures and Facebook Mobile compressed

pictures.

COG

25% gain

COG

35% gain

COG

45% gain

COG

55% gain

FB Mobile

MOS

76.96 77.68 74.83 72.62 74.30

DMOS

-2.54 -3.27 -0.41 1.79 0.11

The Pearson linear correlation between the

DMOS assigned to the pictures compressed by the

plug-in and the jpeg pictures show high correlation

coefficients, which means that the perceived quality

of the Cogisen pictures is greatly correlated with the

perceived quality of Facebook Mobile pictures

(Table 2).

Table 2: Correlations between scores assigned to

Facebook Mobile compressed pictures and those

compressed by the Cogisen Plug-in. The star “*” marks

the mean differences that are significant at the 0.01 level.

Two stars “**” mark the mean differences that are

significant at the 0.05 level.

DMOS

COG

25% gain

COG

35% gain

COG

45% gain

COG

55% gain

FB Mobile

Pearson

Corr.

0.343*

0.262*

0.497* 0.404

Sig. (p value)

0.008 0.047 0.000 0.002

It was investigated whether the participants’

performance has been affected by (1) their expertise

with video display systems or devices (expert, non

expert) (2) the lighting condition of the setting

(natural light, artificial light; indoor, outdoor), and

(3) the order in which pictures are shown in the

testing sequence (first half of the test, second half of

the test).

1. Expertise. The one-way ANOVA shows no

effect of expertise on difference mean

opinion scores (F(1,57)= 2.332; p > 0.05).

2. Lighting condition. The one-way ANOVA

shows no significant difference in the

DMOS assigned in two different lighting

conditions (indoor with natural lights,

indoor with artificial lights), (F(1,57)=

2.386; p > 0.05).

3. Order effect. Multiple linear regression

analysis showed that the order of the

stimuli into the test (first half, second half)

was not able to predict the subjects’

answers (R²= 0.041, F(1,57)= 2.386, p >

0.05; β= 0.202, p>0.05).

The effect of compression method (Facebook

Mobile, Cogisen) on subjects’ performance was

investigated. The repeated measures ANOVA shows

an overall significant difference in DMOS assigned

to stimuli compressed by the Cogisen plug-in

compared to those assigned to jpeg pictures

compressed by the Facebook Mobile application

The Web-based Subjective Quality Assessment of an Adaptive Image Compression Plug-in

135

(Multivariate test; Wilks’ Lambda F(4,54)= 7.635;

p=0.000).

Pairwise comparisons among compression levels

and Facebook Mobile compression (adjustment for

multiple comparisons: Bonferroni) show significant

difference between both 25% and 35% compressed

pictures and Facebook Mobile pictures. No

difference between both 45% and 55% compressed

pictures and jpeg images was found (Table 3).

Table 3: Pairwise Comparisons. Table shows the mean

difference between the DMOS assigned to the Facebook

Mobile’s picture values and the DMOS assigned to the

Cogisen’s compressed pictures. Positive mean difference

values denote higher mean opinion scores assigned to

Cogisen pictures compared to Facebook pictures. A p

value >0.05 denotes that the mean difference is not

significant. The star “*” marks the mean differences that

are significant at the 0.01 level.

(I)

Compression

(J)

Compression

Mean

Diff. (I-J)

Std. Error

Sig.

(p value)

FB Mobile COG 25%

2.664* 0.719 0.005

FB Mobile COG 35%

3.397* 0.794 0.001

FB Mobile COG 45%

0.534 0.571 1.000

FB Mobile

COG 55%

-1.681 0.736 0.261

2.6 Discussion

The main results show that the difference mean

opinion scores assigned to both 25% and 35%

Cogisen compressed pictures were significantly

higher than those assigned to jpeg stimuli. These

results mean that Cogisen’s compression method is

able to reduce image file size in a way that better

manages the information that affects perceived

quality. It confirms that Cogisen’s adaptive image

compression model is more effective than currently

common image compression methods in preserving

the most salient aspects of images, with a 35% file

size gain over jpeg images compressed by Facebook

Mobile while also maintaining a higher perceived

image quality.

No difference between the perceived quality

scores assigned to both 45% and 55% Cogisen

compressed pictures and those assigned to Facebook

Mobile pictures means that the Cogisen plug-in

achieves similar results than the Facebook Mobile

compression algorithm with a 55% gain over it.

The authors of this paper are working on further

studies focusing on the design and the assessment of

the Cogisen plug-in for video compression

applications.

3 CONCLUSIONS

This work investigated the subjective quality

perception of images compressed by the Cogisen

plug-in, which can be integrated into the

compression settings of mobile and desktop

applications. Sixty-three participants assessed the

perceived quality of jpeg pictures compressed by the

Facebook Mobile application and by the Cogisen

compression plug-in.

The Single Stimulus Continuous Quality Scale

method was used to compare the quality score. The

quality scores assigned to compressed pictures were

compared to those assigned to high quality reference

pictures, which were randomly shown during the test

(as recommended the ITU suggestions). The

presentation used a Web-based administration

procedure validated in a previous study (Mele et al.

2016). The results obtained in this study show that

the compression plug-in does not significantly affect

the subjective perceived quality of previously jpeg

compressed pictures up to a gain of 55% file size

reduction.

REFERENCES

ITU Telecom. 2002. Standardization Sector of ITU.

Methodology for the Subjective Assessment of the

Quality of Television Pictures, Recommendation ITU-

R BT. 500 11. ITU Recommendation, I. T. U. R. B. T.:

500 11.

Liu, X., Pedersen, M., Hardeberg, J.Y., 2014. CID:IQ - A

New Image Quality Database. In Proceedings of the

International Conference on Image and Signal

Processing 2014 (ICISP 2014), June 30-July 2,

Cherbourg, Normandy, France.

Mele, M.L., Millar, D., Rijnders, C.E., 2016. Validating a

Quality Perception Model for Image Compression:

The Subjective Evaluation of the Cogisen’s Image

Compression Plug-in. In Proceedings of the 18th

International Conference on Human-Computer

Interaction (HCI 2016). Human-Computer Interaction:

Interaction Technologies. Springer International

Publishing.

Nielsen, J., 1998. Nielsen’s law of internet bandwidth.

Retrieved April 4, 2016 from http://www.useit.com/

alertbox/980405.html

Mohammadi, P., Ebrahimi-Moghadam, A., & Shirani, S.,

2014. Subjective and objective quality assessment of

image: A survey. arXiv preprint arXiv:1406.7799.

Sarode, M. C. A., & Patil, S. V., 2016. A Review on

Image Compression Techniques.

Sheikh, H.R., Sabir, M.F., Bovik, A.C., 2006. A Statistical

Evaluation of Recent Full Reference Quality

Assessment Algorithms, IEEE Transactions on Image

Processing, vol. 15, no. 11, pp. 3440-3451.

HUCAPP 2017 - International Conference on Human Computer Interaction Theory and Applications

136

Sternberg, R., & Sternberg, K., 2016. Cognitive

psychology. Nelson Education.

Vidhya, K., Karthikeyan, G., Divakar, P., & Ezhumalai,

S., 2016. A Review of lossless and lossy image

compression techniques.

Winkler, S. 2012. Analysis of public image and video

databases for quality assessment. Selected Topics in

Signal Processing, IEEE Journal of, 6(6), 616-625.

The Web-based Subjective Quality Assessment of an Adaptive Image Compression Plug-in

137