Accessibility of e-Government Websites in Italy: The User Experience

of People with Disabilities

Maria Claudia Buzzi

, Marina Buzzi

, Giuseppe Della Penna

Barbara Leporini

and Francesca Ricci

CNR-IIT, via Moruzzi 1, 56124 Pisa, Italy

Department of Information Engineering, University of L'Aquila, L’Aquila, Italy

CNR-ISTI, via Moruzzi 1, 56124 Pisa, Italy

Keywords: Human-Computer Interaction, Web Interfaces, Accessibility, People with Disabilities, e-Government.

Abstract: Public Administration services must be accessible for anyone, including people with disabilities who interact

via assistive technology. In 2016, the European Union published Directive 2016/2102 with the aim of making

such services more accessible to any citizen, regardless of its abilities. This paper investigates the accessibility

of e-Government services in Italy from the point of view of people with disabilities: seventy-six users

participated in an online survey, and the collected answers have been further refined through semi-structured

interviews. Results have been compared with a previous study, showing that the number of services has

increased but no substantial improvement in terms of accessibility has been recorded. Simplified interaction

and increased efficiency are still lacking, even if global user satisfaction seems to have slightly improved.

1 INTRODUCTION

E-Government is the use of technology to enhance the

access to and delivery of government services to

benefit citizens, business partners and employees

(Silcock, 2001). Thanks to the Internet and ICT, e-

Government services have been spreading out

worldwide, allowing citizens to access data and

information or perform online services 24 hours a day

in an efficient and cost-effective way, saving time and

reducing burden of the Public Administration (PA)

offices. More delivering e-Government services,

more transparency is reached, reducing corruption,

especially in developing countries (Alam et al.,

2023).

According to the EU Strategy for the rights of

persons with disabilities 2021-2030 (EU, 2021), to

guarantee an equal access e-Government services

must be accessible to all, including people with

disabilities. Applying accessibility (to guarantee

access to all) and usability (to make the interaction

https://orcid.org/0000-0001-7818-0601

https://orcid.org/0000-0003-1725-9433

https://orcid.org/0000-0003-2327-9393

https://orcid.org/0000-0003-2469-9648

easy and satisfactory) guidelines from early design

improves service effectiveness and efficiency, thus

increasing also user satisfaction. Although public

administrations gained an increased awareness on

usability and accessibility, also due to legal

obligations, some issues can still obstacle e-

Government services access by everyone despite

numerous guidelines have been available in the

literature for many years. Automatic tools can be

applied for testing web pages and services for

machine-detectable accessibility and usability issues;

however, according to W3C Web Accessibility

Initiative (WAI), they cannot catch severe

accessibility problems, so knowledgeable human

evaluation is also required

(W3C, 2022). Directly

involving and understanding the point of view of

people with disabilities can shed a light on interaction

issues of e-Government services.

This study investigates accessibility of PA

electronic services as perceived by people with

disabilities. An online questionnaire with 18 items

Claudia Buzzi, M., Buzzi, M., Della Penna, G., Leporini, B. and Ricci, F.

Accessibility of e-Government Websites in Italy: The User Experience of People with Disabilities.

DOI: 10.5220/0012177400003584

Paper published under CC license (CC BY-NC-ND 4.0)

In Proceedings of the 19th International Conference on Web Information Systems and Technologies (WEBIST 2023), pages 59-70

ISBN: 978-989-758-672-9; ISSN: 2184-3252

has been filled in by 76 citizens with disabilities.

Semi-structured interviews were performed with a

small set of participants to understand the

encountered problems more in depth. Results have

been discussed compared with previous studies.

The paper is organized as follows: Section 2

introduces the related work, Section 3 presents the

study methodology, the results of the questionnaire

and insights from the semi-structured interviews. In

Section 4 a discussion with a comparison with

previous studies is presented. Conclusion and future

work are described in Section 5.

2 RELATED WORK

Technology may play a crucial role in promoting the

participation and inclusion of people with disabilities

in society (Ferri and Favalli, 2018). Accessibility of

websites and services is important for everyone, but

it is crucial for people with disabilities, since their

interaction may require more time and cognitive

effort and may encounter more obstacles (Yesilada &

Harper, 2019). If not designed with accessibility in

mind, the Internet may increase the marginalization

of people with disability in society (Jaeger, 2022).

Different kind of barriers due to competences and

skills, geographic position and resources, income,

education and special needs, create the so called

“digital divide” and without urgent countermeasures,

in the future this gap could even increase (Seljan et

al., 2020). Several countries have published laws to

favor and guarantee equal rights for all to access

digital resources, web sites and services. In 2016, the

European Union published the Directive (EU)

2016/2102 on the accessibility of the websites and

mobile applications of public sector bodies. Such

requirements are mostly based on accessibility web

standards defined by the World Wide Web

Consortium (W3C), the Web Content Accessibility

Guidelines (WCAG), that evolved for over 25 years

to better incorporate usability into accessibility

guidelines (W3C, 2023).

E-Government accessibility is a widely

investigated topics, due to the importance to ensure

equal opportunities for all citizens. Attentions to

accessibility seems also bound to the transparency of

the public administrations. Alcaraz-Quiles et al.

investigated relations between transparency,

accessibility and usability of Spanish Regional

Government websites, showing that the transparency

of analyzed websites is inversely related to

accessibility since without visibility retrieving

information can require a considerable time (Alcaraz-

Quiles et al., 2018).

The assessment of the degree of accessibility of e-

Government services has been a subject of

investigations all over the world. Galvez and

Youngblood (2016) analysed more than one hundred

national and local e-government website sites in

Rhode, using a combination of code inspection,

heuristic evaluation, and automated analysis. Results

suggest that best-practice templates may be helpful in

improving usability, accessibility, and mobile

readiness, while it is critical for designers to receive

training and for governments to monitor Web sites

compliance with standards.

Kous et al. (2021) performed an accessibility

evaluation (compliance with Standard EN 301549) of

189 Slovenian municipalities’ websites using an

automatic tool (Achecker). The results of the

statistical analysis showed that the website home

pages in 2018 (after the adoption of European

Standard EN 301 549) have significantly improved

compared to 2017.

Paul (2022) presented the evaluation of the

accessibility of Indian e-government websites using a

sample of 65 websites of various ministries based on

the WCAG 2.1 standard, founding the majority of e-

government websites do not meet Level A

conformance with WCAG 2.1.

Al-Sakran and Alsudairi (2021) investigated how

well the Saudi mobile e-government websites comply

with usability standards and accessibility guidelines

recommended in the WCAG. Websites assessments

were conducted using manual evaluation and

complemented by different automated analysis tools.

The findings revealed usability and accessibility

problems that affect the performance of government

websites. Several recommendations for improving

the usability and accessibility of websites in Saudi

Arabia were also suggested.

Agrawal et al. (2022) investigated the usability,

accessibility, and mobile readiness of Indian

government websites. The analysis covered 164

government websites delivering e-services. The

evaluation of websites compliance with the WCAG

2.0 was done on various quality parameters using

automatic online tools. The results indicate that many

websites had low usability, most of the website does

not follow WCAG 2.0 guidelines, while none was

usable and fully accessible on mobile devices.

Siquiera et al. (2022) investigated whether public

prosecutors' websites are following web accessibility

guidelines. The authors evaluated the websites of

each of the 27 states of Brazil using the WCAG 2.0.

WEBIST 2023 - 19th International Conference on Web Information Systems and Technologies

Results indicated that the websites violated between

16 and 33 different success criteria out of 61 criteria.

Very recently, Lynn et al. (2023) examined the

web accessibility of local authority websites in the

Republic of Ireland specifically referring the Web

Accessibility Directive, Results confirmed that most

local government websites examined present

significant accessibility issues.

Similar studies were performed also in Italy.

Barricelli et al (2018) exploited automatic tools for

assessing the degree of accessibility of PA web sites

in Italy, showing that on more than 8000 websites of

the Italian municipalities only the 1% was fully

conform to the Italian legislation that required PA to

implement a subset of the WCAG guidelines. This

issue was confirmed by several successive studies.

Buzzi et al. (2019) presented a study that sheds light

on Italian public administration (PA) accessibility via

an online survey answered by 68 people with

disabilities from Tuscany, accessing PA services in

Italy. Results from the sample highlighted the need to

improve service accessibility and usability, and the

request for increasing their number and set of

functions. Furthermore, Valtolina and Fratus (2022)

investigated accessibility issues of local government

websites in Italy proposing an evaluation strategy via

two validators (AChecker and VaMolà) and

analysing 7,713 homepages against WCAG 2.0 and

Italian Law recommendations. Results confirmed a

low degree of web accessibility of municipal websites

However, most studies exploited only automated

analysis tools (validators). A recent systematic

literature review on automated tool utilization in web

accessibility research from 2002 to 2021 showed an

increasing trend in its use year by year (Macakoglu

and Peker, 2022). Considering the limits of automatic

evaluation of web accessibility (due to their inability

to check all aspects of accessibility automatically),

our study investigates the current degree of

accessibility of Italian PA services by the point of

view of users with disabilities, who can highlight

actual problems via an online survey further deepened

through semi-structured interviews.

3 THE STUDY

3.1 Methodology

This study investigates the accessibility of digital

services provided by PAs for people with disabilities

in Italy. At this aim, the study exploits two tools for

gathering participants’ feedback: an online survey

and semi-structured interviews.

The online survey was administered via Google

Forms, which supporting screen readers and keyboard

shortcuts offers a good degree of accessibility for

screen reading users (Leporini et al., 2018). The

questionnaire focuses on interaction aspects:

authentication, effectiveness in successfully

completing the service, ease to use of the interface,

overall user satisfaction.

To recruit potential end-users, an invitation letter

was defined, describing the study aim and the

process. The main Italian organizations of people

with disabilities (visual, motor, hearing, intellectual)

were contacted first by phone, and next via email

asking them to distribute the invitation letter (with the

questionnaire link) to their associates. This assures

the authors not having any direct contact with

participants.

The purpose of the semi-structured interviews was

to better understand the accessibility problems

encountered in interacting with government websites.

The interviews are designed to reveal the participants'

personal narratives, leaving them free to express their

opinions, considerations, and suggestions, that could

not emerge in closed questions.

Participants were recruited via e-mail or phone by

the associations for people with disabilities. Then, the

contacts of users available for further information

were provided by the associations to the authors.

Interviews were conducted via a video conferencing

tool (Google Meet) or by phone (preferred by visual

impaired users) by two of the authors: one conducted

the interview and the other one annotated answers,

comments and observations. After each interview,

both authors immediately checked and integrated the

annotated answers.

3.2 Questionnaire

The questionnaire, in Italian language, is composed

of eighteen questions organized in 3 sections. The

first section is composed of six questions to

characterize the sample, their use of the Internet and

their experience with most-used e-Government

services. Section 2 includes eight questions to

investigate the participants’ usability experience

when interacting with e-Government services: access,

effectiveness, ease to use, user satisfaction, support,

errors and customization. Last Section 3 proposes

four questions concerning authentication and citizen

rights.

The questionnaire’s content and language were

assessed by two accessibility experts and checked by

a totally blind person who verified its accessibility via

screen reader. The list of questions is presented in the

Accessibility of e-Government Websites in Italy: The User Experience of People with Disabilities

Appendix. Last, the heading of the questionnaire

describes:

• Who performed the study

• The research purpose: to understand the level

of accessibility and any issues encountered by

users with disabilities when interacting with

Public Administration websites

• The target users: persons with legal age (≥18)

and at least one type of disability

• Information about data treatment (no personal

data collected, no transfer to third parties).

3.3 Participants

A total of 76 participants filled out the questionnaire:

50 males and 26 females. Participants are aged

between 18 and more than 70 years: 2 (2,6%)

participants in the 18-29 range, 8 (10,5%) in the 30-

39 range, 20 (26,3%) in the 40-49 range, 30 (39,5%)

aged 50-59, 8 (10,5%) aged 60-69, and 8 (10,5%)

participants were 70 or older.

All participants indicated their gender: 2/3 are

males (50 out of 76) and 1/3 are female (25 out of 76).

Regarding the type of disability, most of

participants, are visually impaired (65 out of 76),

followed by people with motor disability (8 out of 76)

and three people having a mild cognitive impairment.

Last, two hearing impaired people who communicate

through the Italian sign language (signists) and one

with a behavioral disability complete the sample. It

was allowed to provide more than one answer in order

to correctly identify also multiple disabilities.

The sample composition suffers from an

imbalance in the representation of disability types,

with 81,25% of respondents being individuals with

visual disabilities. However, it should be taken into

consideration that several studies have shown that

users with visual impairment, and more specifically

blind users, encounter greater difficulties than users

with other disabilities when interacting with websites.

Visually-impaired users searching the Web content

took an average of 2,5 times longer than sighted users

(Craven and Brophy, 2003). An efficiency gap was

also confirmed by Ivory et al. (2004): when blind

people executed a search task, they took twice as long

as sighted users to explore search results and three

times as long to explore the corresponding web pages.

Petrie et al. in 2004 performed an accessibility test of

one hundred websites with users with visual, motor

and perceptual disabilities. Results showed a mean

task success rate of 76% which fall to 53%

considering only the totally blind (the lowest score of

all categories). Besides, regarding user satisfaction,

blind users confirmed the lowest score of all the user

categories (Petrie et al., 2004). Overall, blind users

interacting on the Web lose, on average, 30.4% of

time due to frustrating situations, such as page layout

causing confusing screen reader feedback, conflict

between screen reader and application, poorly

designed/unlabeled forms, no alternative text for

pictures, misleading links, inaccessible PDFs, and

screen reader crashes (Lazar et al., 2007). For these

reasons, we believe this study’s sample can offer

interesting insights on the accessibility of PA

services, including a high percentage of visually

impaired users, who experience the greatest

difficulties in web navigation.

The distribution of participants by their

occupation/job depicts that more than half of

participants (~54%) are employees of a PA (Italian

law requires PAs to recruit a percentage of people

with disabilities). Twenty-three participants (30,2%)

are retired, two participants are students (2,6%), five

participants (6,6%) are practitioners and five

unemployed people complete the sample.

Concerning the frequency of Internet use, most

(68 out of 76) participants use the Internet daily, three

weekly and two sometimes (Figure 1). Three

participants declared they use the Internet rarely. Two

of them are 70+ aged and one 50+.

In most cases, the Internet is an opportunity for

people with disabilities: accessible PA online services

can be performed autonomously, without the need for

an accompanying person, greatly empowering

personal autonomy. Autonomy is one of the most

important conditions for people with disability and

therefore it is very important to progressively increase

the portfolio of available services: investigating the

main reasons why users who could access such

services avoid using them could provide interesting

information helping to understand if there are barriers

or problems that can be solved.

Figure 1: Sample characterization: Participants’ Internet

use (Q5).

The users interact with several services of both

local and central public administrations (Figure 2).

Most frequently accessed services are job or pension,

Health, Local Administrations and taxation.

Daily

Weekly

Sometimes

Rarely

Internet Usage

WEBIST 2023 - 19th International Conference on Web Information Systems and Technologies

3.4 Results

Results are organized by analysing the answers to

sections 2 and 3 of the questionnaire.

Figure 2: Frequently accessed PA services (Q6).

3.4.1 Accessibility and Usability

Section 2 contains eight questions related to

participants’ usability experience when interacting

with e-Government services. Answers to question

Q7: “Have you had problems accessing the PA's

online services?” offer indications on the access

issues. It is remarkable that 89% of the sample

reported access problems (‘Sometimes’ for 46 out of

76 users, and ‘Often’ for 22). Only 11% of the users

(8 out of 76) never had an access problem (Fig. 3).

Figure 3: Frequency of problems reported/encountered by

participants when accessing e-Government services (Q7).

Regarding the effectiveness of interaction with

PA services (Q8), more than half participants

reported they were able to complete the requested

services always (6 users, 8%) or often (37 users,

49,3%), while 27 users (36%) only sometimes and 3

never, as shown in Figure 4.

Considering how frequently the interaction with

e-Government services was easy (Q9), most

participant were positive: 4 users (5,3%) found it

always simple and 37 users (48,7%) found it simple

in most cases, but it is relevant also the percentage of

participants that found it difficult in most cases

(32,9%, 25 users) or very difficult in all cases (13,2%,

10 users), as shown in Figure 5.

Figure 4: Effectiveness of PA service interaction (Q8).

Figure 5: Ease of the Interaction with PA services (Q9).

User satisfaction reported a consistent number of

participants (the majority) with a neutral position (27

users, 35,5%), as shown in Figure 6. Positive rating is

30,3% (satisfied, 23 users) and 7,9% (very satisfied,

6 users), while negative rating is 15,8% unsatisfied

(12 users) and 10,5 very unsatisfied (8 users).

Figure 6: User satisfaction about the Interaction with PA

service (Q10).

Three additional questions concern three usability

aspects: presence and clarity of documentation and

support, information, errors and recovery, and

possibility of customization. For evaluating the PA

website documentation and support (documentation

and manuals, pop-up messages, etc.) a 5-item Likert

scale (1 = Totally dissatisfied, to 5 = Truly satisfied)

has been administered.

0 102030405060

Others

Health

Municipality

Job/Pensions

Tax

Justice

Official Documents

Finance/Economy

School & Education

Frequently Accessed Sites

Yes, often

Sometimes

No, never

Access Problems

0 10203040

Yes, always

Often

Sometimes

Never

I've never tried

Effectiveness of PA service interaction

0 10203040

Always simple

Simple (in most cases)

Difficult

Very difficult

Ease of the interaction

0 102030

Very unsatisfied

Unsatisfied

Neutral

Satisfied

Very Satisfied

User satisfaction

Accessibility of e-Government Websites in Italy: The User Experience of People with Disabilities

Most users, 31 out 76, (i.e. 40,8%) chose the

neutral option (neither satisfied nor dissatisfied),

while the percentage of satisfied (22,4% satisfied and

7,9 very satisfied) is only slightly higher than the one

of dissatisfied (19,7% somewhat dissatisfied and

9,2% totally dissatisfied), as shown in Figure 7.

Figure 7: User satisfaction about documentation and help

(Q11).

Another question was about errors: if you made

mistakes, were you able to fix them easily (and were

the error messages clear and helpful)? Most of users

(40%) were able to correct the errors, but with

difficulties, 12% were unable to correct them, while

16% corrected the errors with ease (Figure 8).

Next question was: “If you made an error, were

you able to manage it and move forward?” and it aims

to understand if the management of errors is provided

through clear messages and interaction mechanisms.

Most users (40%, 30 users) reported that in case of

errors they could hardly correct them, reaching half

of the sample if we also consider the ones who said

they could not correct errors (12%, 9 users). The

neutral option (I don’t know, 17,3%, 13 users), “I

made mistakes, but I could easily correct them” (16%,

12 users) and “I don’t made mistakes” (14,7%, 11

users) complete the answers.

Figure 8: Errors visibility and easy of recovery (Q12).

Finally, concerning the ability to customize or

adapt the interface to the user's needs (Q13), one-fifth

of the sample (20,3%) was unable to adapt the

interface to their needs, 12,2 % were able in general,

while 31,1% only sometimes. One-fifth of the

participants (20,3%) did not try to modify the

interface (Figure 9).

Figure 9: Usability: interface customization (Q13).

About aspects to be improved urgently (Q14),

most of the users (45 out 76, i.e. 59,2%) suggested to

make the UIs easier to use and simplify the

interaction, while others suggested to add new online

services (25%) and shortening the service response

time (5,3%) as shown in Figure 10. Some skilled

users also suggested more specific improvements,

e.g., better structuring of the content, use of logical

templates to make it easier to find what you are

looking for, addition of instant chat for immediate

assistance, subtitling and possibly translation in the

sign language of videos.

Figure 10: Aspect to Improve. Q14.

3.4.2 Authentication & Citizen Rights

Section 3 of the questionnaire contains four questions

concerning authentication & citizen rights. The first

question (Q15) is about authentication via SPID

(Public Digital Identity System). Overall, 63

participants (82,9%) use SPID for accessing PA

services, 4 users (5,3%) have the authentication

credentials but do not use them, 6 users (7,9%) know

0 10203040

Totally dissatisfied

Dissatisfied

Neutral

Satisfied

Totally satisfied

Documentation and help

12%

40%

17%

16%

15%

I was not able to correct errors

I correct errors with difficulty

I don't know

I correct errors easily

I did not made errors

Errors

0 5 10 15 20 25

No, in general

Only in some sites

I don't know

Yes, generally

I have not tried

UI Customization

0 1020304050

Others

None, everything is fine

Response times

Increase number of services

Ease of use

Aspects to improve

WEBIST 2023 - 19th International Conference on Web Information Systems and Technologies

about the credentials but never requested them, and

only 3 users (3,9%) still do not know anything about

the SPID credentials.

Figure 11: Authentication: SPID credentials.

The other three questions regard citizen rights.

Almost all participants (72, 95%) know their right to

easily access all PA services (Q16).

Figure 12: Knowledge of citizen rights.

Q17 investigates the awareness of users on the

possibility of reporting accessibility problems to

AgID (Agency for digital Italy), the authority in

charge of monitoring and fuelling the digitalization of

the Italian PAs (Q17). Most of the participants

(48,7%) are aware of this possibility but did not report

any accessibility issues while 42,1% was not aware,

and only 9,2% sent some reports (Figure 13).

Figure 13: Reporting Accessibility feedback to AgID

(Q17).

Q18 was about the knowledge of the role of the

Public Advocate (Ombudsman) for the Digital Rights

and the possibility to provide feedback for

contributing to improve PA services. Most

participants (47 out of 76, 61,8%) are interested in

contributing, while 19 users (25%) do not consider

themselves able to do it, one has no time, and 9 are

not interested (11,8%).

Figure 14: Reporting Accessibility feedback to the Public

Advocate (Q18).

3.5 The Semi-Structured Interviews

The interviews started with four questions about

Difficulties, Effectiveness, Efficiency, Satisfaction of

use and Improvements:

1. What obstacles have you encountered in

accessing PA Web sites?

2. Were you able to do what you wanted, or did you

need help?

3. Was the interaction challenging, did it require

effort and/or a lot of time?

4. Were you satisfied or what would you like to

improve?

The sample consisted of 8 users: 3 visually

impaired (2 blind and one visually impaired that

navigates via screen reader), 2 motor impaired, and 3

old persons with mild cognitive and motor

impairment. Tab. 1 summarizes the key findings from

the interviews.

A common obstacle encountered in accessing PA

websites among all the users interviewed is the

difficulty in finding what they are looking for, due to

the lack of clarity in the contents and the complex

structure of the websites.

A totally blind user, who uses PA services daily,

reported two major problems: the presence of

advertisements on the sites that interfere with blind

users' listening to the content, and the difficulty in

using downloaded documents, as they are not

I don't know

what they

are

I know it, but

I don't own it

I own it, but I

don't use it

I own it and I

use it

SPID Credentials

Yes

95%

Citizen rights

No, I didn't know Yes, but I didn't

report it

Yes, and I made

some reports

Reporting Accessibility issues to Agid

0 1020304050

No, I'm not interested

No, I wouldn't be able

Yes, I would like to

participate

I don't have time to do it.

Feedback to improve PA services

Accessibility of e-Government Websites in Italy: The User Experience of People with Disabilities

Table 1: Key findings from the interviews.

Issues Affected tasks

Lack of clarity in the

contents

Exploring, searching

Complex structure of the

websites

Exploring, searching

Audio of the commercial

advertisements overlaps the

reading of the screen reader

Listening the vocal

synthesizer

Documents created as an

image (unreadable and

unstructured)

Reading a document

Request to change the

password frequently

Access to website areas

with authentication

Tight and insufficient time

for strong authentication

Access to website areas

with authentication

readable by the screen reader being not in text format

(e.g., PDF documents containing scans of paper

documents).

Most notably, many of the interviewed users

reported difficulties in accessing PA sites due to the

SPID authentication or the need to periodically

change their passwords, as required by the current

national security guidelines. In some cases,

participants were even unable to complete the service.

Such users often require the assistance of family

members or friends to accomplish these tasks.

An experienced blind user declared that the SPID

authentication was accessible for him only through a

specific provider. Indeed, such provider grants the

user with 1 minute and 30 seconds to enter and

confirm (SPID uses a two-factor authentication) the

user credentials, as opposed to the 30 seconds granted

by other providers.

4 DISCUSSION

To understand the accessibility of e-Government

services as perceived by participants with disability,

the proposed survey investigates the following

aspects: a) Ability to access to the PA service; b)

Effectiveness (to successfully complete the service);

c) Ease of use and user interaction; d) Adaptability

and help. The survey results have been also

corroborated by semi-structured interviews, as

presented in Section 3.5.

There are several previous studies investigating

accessibility of PA services in Italy, mostly exploiting

automatic tools for assessing the degree of

accessibility of PA web sites. It is interesting to

compare the results of the current survey with an

analogous one administered by the authors of this

paper in late 2017 on a sample of 68 participants with

disability (about 61% blind users, 20% motor

disabilities, 9% auditory, less than 5% cognitive and

behavioral) described in (Buzzi, 2019). Indeed, the

two studies have a similar number of participants (68

vs. 76) and both recruited mainly blind and visually

impaired (62% in the 2019 study and 85,5% in this

study). To make the results comparable, this study

only extends the number of questions of the original

questionnaire to collect more data (adding Q11, Q12,

and Q13) but keeps the original ones unchanged.

The previous study contains data collected in

2017, so in the following, we write “2017” to indicate

the results of (Buzzi et al, 2019) and “2023” for the

results of the current study.

Figure 15: Frequency of problems encountered when

accessing online PA services: 2017 vs. 2023.

Figure 16: Effectiveness of online PA Services: 2017 vs.

2023.

A main aspect concerns the participants’

experience in accessing PA websites. Most users

(89,4%) still are experiencing issues: 60,5% (vs

67,2% of 2017) of participants had problems

sometimes and 28,9 % (vs 19,7% of 2017) often, as

shown in Figure 15. The situation seems to have

gotten worse: the number of people who never

experienced problems has decreased by 2,6% (10,5 in

19,7%

67,2%

13,1%

28,9%

60,5%

10,5%

Yes, often Sometimes No, never

Access Problems

2017 2023

10,6%

51,5%

30,3%

3,0%

4,50%

8,0%

49,3%

36,0%

2,70%

Yes, always Often Sometimes Never I never tried

Effectiveness

2017 2023

WEBIST 2023 - 19th International Conference on Web Information Systems and Technologies

2023 vs 13,1 in 2017) and the total number of people

experiencing some issues has increased (89,4% of the

sample in 2023 vs 86,9% in 2017). This could be

caused by the new requirement of strong

authentication for accessing the PA services,

introduced by the European eIDAS Regulation on

electronic Identification, Authentication and trusts

services

. Indeed, in Italy, access to PA services

exploits digital identification through an Identity

Service Provider (SPID) or via an Electronic Identity

Card (CIE). As discovered in the interviews, strict

temporal limits can obstacle the access by blind users

interacting with such authentication mechanisms via

a screen reader, as discussed in the following.

Also regarding effectiveness, i.e. the ability to

complete a service, the situation seems slightly worse

than in 2017: only 8% of the participants (in 2023),

compared to 10,6% (in 2017), has always achieved

their goal, while the majority, i.e., 49,3% (in 2023),

compared 51,5% (in 2017), achieved it only often and

36% (in 2023) vs 30,3% (in 2017) only sometimes

(Figure 16). The total number of participants

experiencing effectiveness issues increased to 89,3%

in 2023 vs 84,8% in 2017. However, we have to

consider the narrow samples do not allow a

generalization of such results.

Figure 17: Ease of interaction in accessing online PA

Services: 2017 vs. 2023.

A light worsening was highlighted also regarding

the ease of use in the interaction with online PA

websites (Figure 17). Participants who evaluated the

interaction as simple decreased from 54,9% in 2017

(9,7% always simple + 45,2% simple) to 54,2% in

2023 (5,5% always simple + 48.7% simple).

Furthermore, 46,1% of 2023 participants found the

interaction quite difficult (32,9% difficult + 13,2%

very difficult) compared to 45,2% in 2017 (35,5%

difficult + 9,7% very difficult) of 2017.

https://digital-strategy.ec.europa.eu/en/policies/eidas-

regulation)

To measure the user satisfaction, we used a 5-item

Likert scale (1 = Very unsatisfied to 5 = Very

satisfied). Figure 18 shows that most participants

were not completely satisfied, 30,3% (vs 26,5% in

2017) were satisfied and only 7,9% (vs 6,3% in 2017)

very satisfied. However, since the percentage of

unsatisfied increased to 26,3% in 2023 vs 22,9% in

2017 the overall improvement is very little. Very

often, inexperienced people having problems

accessing online services believe that it is their fault,

being unable to correctly interact with the interface,

while the cause could be a poor usability design.

Figure 18: User satisfaction in accessing PA website: 2017

vs. 2023.

Overall, the collected users’ feedback concerning

the accessibility of PA websites, although on a small

sample, seems to indicate a worsening, even if only

slightly, compared to those of 2017, suggesting that,

despite the regulation’s intent, the problems seem not

under resolution.

Concerning the Authentication & citizen rights

session, with respect to the result of the 2017

questionnaire, the situation improved: 82,9% of

participants use SPID for accessing PA services

respect to only 5,9% in 2017, 5,3% have the

authentication credentials but do not use them vs.

8,8% in 2017, 7,9% know about the credentials are

but never requested them (vs 47,1 in 2017), and only

3,9% still do not know anything about the SPID

credentials (vs 37,2 in 2017). Analogously, most of

participants (96% vs 89,7 % in 2017) know their right

to easily access all PA services.

Concerning reports on accessibility issues of PA

services to AGID, awareness increased a lot. Most of

the participants (48,7%) are aware of this possibility

but did not report any accessibility issues (vs 29,4%

in 2017) while 42,1% were not aware in comparison

to 61,8% in 2017. However, few users exploit this

9,7%

45,2%

35,5%

9,7%

5,5%

48,7%

32,9%

13,2%

Always simple Simple Difficult Very difficult

Interaction

2017 2023

4,7%

17,2%

45,3%

26,5%

6,3%

10,5%

15,8%

35,5%

30,3%

7,9%

Very

unsatisfied

Unsatisfied Neutral Satisfied Very

Satisfied

2017 2023

User Satisfaction

Accessibility of e-Government Websites in Italy: The User Experience of People with Disabilities

possibility: only 9,2% in 2023 and 8,8 % in 2017.

Last, most of the participants would contribute by

sending feedback to the Public Advocate: 61,8% vs

47,1% in 2017, while 25% do not consider

themselves able to do it, respect 29,4% in 2017.

5 CONCLUSION

This study investigates the experience of people with

disabilities when accessing PA websites in Italy.

Results show that, although user awareness is

significantly increased and more users with disability

are able to benefit from PA digital services, user

satisfaction is only very slightly increased over the

last 6 years, effectiveness has slightly decreased, and

interaction seems more difficult, e.g., due to the

increased security and authentication constraints (as

the multi-factor authentication mechanisms)

introduced in the last years.

Furthermore, while the increasing number of

services offered through the PA websites is generally

a positive aspect for all users, often this implies

complex user interfaces that, if not suitably

structured, are more difficult to understand and

navigate via screen readers. Clearly, both the current

authentication processes and the user interfaces need

further design simplification efforts to make users

with disabilities able to successfully access the PA

services, asking for urgent improvement of usability

such as ease of use of the user interfaces that should

be logically organized for easier navigation, offering

more help and a simplified interaction (also for

authentication purposes).

The path for a truly accessible PA is still in

progress and needs considerable endeavour. New

initiatives inside the National Recovery and

Resilience Plan, which is part of the Next Generation

EU program, can place crucial resources and

accelerate this process in Europe, but it is important

to involve users with disabilities in these efforts since

their feedback must be at the heart of a truly inclusive

design process.

As future work, we intend to expand our survey

in two directions: a) involving a more significant and

balanced sample of users, especially with regard to

the types of disability; b) investigating why some

people do not use the internet (or just very rarely) in

order to understand if it is due to specific barriers

(technological, cultural, accessibility, etc.), or

personal issues (privacy concerns, lack of interest,

etc.) or other causes such a psychological technology

rejection.

ACKNOWLEDGEMENTS

We thank all the study participants for their valuable

contribution.

REFERENCES

Agrawal, G., Kumar, D. & Singh, M. Assessing the usability,

accessibility, and mobile readiness of e-government

websites: a case study in India. Univ Access Inf Soc 21,

737–748, 2022. https://doi.org/10.1007/s10209-021-

00800-8

Akram, M., Ali, G.A., Sulaiman, A. et al. Accessibility

evaluation of Arabic University websites for compliance

with success criteria of WCAG 1.0 and WCAG 2.0. Univ

Access Inf Soc, 2022. https://doi.org/10.1007/s10209-

022-00921-8

Alam, T.; Aftab, M.; Abbas, Z.; Ugli, K.M.M.; Bokhari,

S.A.A. Impact of E-Government Initiatives to Combat

Corruption Mediating by Behavioral Intention: A

Quantitative Analysis from Emerging Economies.

Sustainability 2023, 15, 2694.

https://doi.org/10.3390/su15032694

Alajarmeh, N. Evaluating the accessibility of public health

websites: An exploratory cross-country study. Univ

Access Inf Soc 21, 771–789 (2022).

https://doi.org/10.1007/s10209-020-00788-7

Alcaraz-Quiles, F. J., Urquia-Grande, E., Muñoz-Colomina,

C. I., & Rautiainen, A. E-Government Implementation:

Transparency, Accessibility and Usability of

Government Websites. In International E-Government

Development, 291-306, Palgrave Macmillan, Cham,

2018.

Al-Sakran, H. O., Alsudairi, M. A.: Usability and

Accessibility Assessment of Saudi Arabia Mobile E-

Government Websites, IEEE Access, vol. 9, 2021, DOI:

10.1109/ACCESS.2021.3068917

Barricelli, B. R., Sciarelli, P., Valtolina, S., & Rizzi, A.: Web

accessibility legislation in Italy: a survey 10 years after

the Stanca Act. Universal Access in the Information

Society, 17, 211-222, 2018.

Buzzi, M. C., Buzzi, M., & Ragni, F.: Accessibility of Italian

E-Government services: The Perspective of Users with

Disabilities. In Electronic Governance and Open Society:

Challenges in Eurasia, EGOSE 2018, Communications

in Computer and Information Science, vol 947. Springer,

2019 https://doi.org/10.10 07/978-3-030-13283-5_21

Craven, J., and Brophy, P.: “Non-visual access to the digital

library: the use of digital library interfaces by blind and

visually impaired people.” Technical report, Manchester:

Centre for Research in Library and Information

Management (CERLIM), 2003,

http://www.cerlim.ac.uk/pubs/index.php

European Union (2021). European Commission. Union of

equality: Strategy for the rights of persons with

disabilities 2021-2030. https://ec.europa.eu/social/

main.jsp?catId=1484

WEBIST 2023 - 19th International Conference on Web Information Systems and Technologies

Ferri, D., & Favalli, S. Web accessibility for people with

disabilities in the European Union: Paving the road to

social inclusion. Societies, 8(2), 40, 2018.

Galvez, R. A., Youngblood, N. E.: e-Government in Rhode

Island: what effects do templates have on usability,

accessibility, and mobile readiness? UAIS, June 2016,

Volume 15, Issue 2, pp 281–296

Ivory, M. Y., Yu, S., and Gronemyer, K.: “Search result

exploration: a preliminary study of blind and sighted

users' decision making and performance”. Extended

abstracts of CHI 2004, pp. 453-1456

Jaeger, P. T. (2022). Disability and the Internet. In Disability

and the Internet. Lynne Rienner Publishers.

Kous, K., Kuhar, S., Pavlinek, M., Heričko, M., & Pušnik,

M. (2021). Web accessibility investigation of Slovenian

municipalities’ websites before and after the adoption of

European Standard EN 301 549. Universal Access in the

Information Society, 20, 595-615.

Lazar, J., Allen, A., Kleinman, J., Malarkey, K. (2007) What

Frustrates Screen Reader Users on the Web: A Study of

100 Blind Users, International Journal of Human–

Computer Interaction, 22:3, 247-269, DOI:

10.1080/10447310709336964

Leporini, B., & Buzzi, M. (2018, April). Home automation

for an independent living: investigating the needs of

visually impaired people. In Proceedings of the 15th

International Web for All Conference (pp. 1-9).

Lynn, T., Kennedy, J., Rosati, P., Fox, G., O’Gorman, C.,

Curran, D. and Hynes, K. Web Accessibility of Irish

Local Government Websites, ICDS 2023 (Seventeenth

International Conference on Digital Society), 2023

Macakoglu, S.S. and Peker, S.: "Web accessibility

performance analysis using web content accessibility

guidelines and automated tools: a systematic literature

review," 2022 International Congress on Human-

Computer Interaction, Optimization and Robotic

Applications (HORA), Ankara, Turkey, 2022, pp. 1-8,

doi: 10.1109/HORA55278.2022.9799981.

Paul, S. Accessibility analysis using WCAG 2.1: evidence

from Indian e-government websites. Univ Access Inf

Soc, 2022. https://doi.org/10.1007/s10209-021-00861-9

Petrie, H., Hamilton, F., and King, N.: “Tension, what

tension?: Website accessibility and visual design”, Proc.

2004 International Cross-disciplinary Workshop on Web

Accessibility (W4A), 2004, pp. 13 – 18.

Seljan, S., Miloloža, I., & Pejić Bach, M.: e-Government in

European countries: gender and ageing digital divide.

Interdisciplinary Management Research, 16, 1563-1584,

2020.

Silcock, R.: What is e-government. Parliamentary Aff. 54(1),

88–101 2001

Siqueira, M.S.S., Dias, F.S., Rigatto, S.H., Carvalho, M.C.N.,

Marques, T.A.M. and Freire, A.P.: Who watches the

watchers? Accessibility of the public prosecutor's office

websites in Brazil and implications for e-government

accessibility surveillance policies, Electronic

Government Journal, Vol. 19, No. 1, pp 72-94, 2022,

https://doi.org/10.1504/EG.2023.127576

Valtolina, S. and Fratus, D.: Local Government Websites

Accessibility: Evaluation and Finding from Italy. Digit.

Gov.: Res. Pract. 3, 3, Article 17 (July 2022), 16 pages,

2022. https://doi.org/10.1145/3528380

W3C. Introduction to Web Accessibility. 31 March 2022

https://www.w3.org/WAI/fundamentals/accessibility-

intro/#evaluate

W3C. Web Content Accessibility Guidelines (WCAG) 2.2.

25 January 2023 https://www.w3.org/TR/WCAG22/

Yesilada, Y., & Harper, S. (2019). Web Accessibility.

London: Springer

APPENDIX

Survey on the Accessibility of e-Government

Websites in Italy

The purpose of this research conducted by the

University of L’Aquila and CNR is to understand the

level of accessibility and issues encountered by users

with disabilities when interacting with Public

Administration websites. The questionnaire is

addressed to adults and it is completely anonymous.

Data collected will be processed in aggregate form for

only research purposes. They will not be transferred

to third parties. Consent to data processing: I confirm

that I have read and understood the above information

and I understand that:

a) my participation is voluntary, and I can withdraw

at any time without giving any reason

b) my data are anonymous and no identifying

information will be made available to anyone.

I accept, I give my consent and I participate

in the questionnaire

I do not accept (exit the questionnaire)

Section 1: Sample Characterization

Q1 Gender:

Male

Female

I don’t want to specify.

Q2 Age:

18-29

30-39

40-49

50-59

60-69

≥70

Q3 Occupation/Job:

Student

Employee

Professional

Unemployed

Retired

Q4 Disability – multiple choice

Visual

Motor

Hearing

Accessibility of e-Government Websites in Italy: The User Experience of People with Disabilities

Intellectual

Behavioural

Q5 Internet Usage - How often do you use the

internet?

Almost Never

Sometimes

Weekly

Daily

Q6 Which Online PA Services do you Access most

Frequently? - Multiple choice:

Health

Municipality

Job/Pension

Tax

Justice

Official Documents

Finance/Economy

School & Education

Section 2: Interaction with e-Government services

Q7 Access Problems – Have you ever had problems

accessing the PA's online services?

Yes, often

Yes, sometimes

No, never

Q8 Effectiveness of PA Service Interaction - Were

you able to get what you wanted and to complete the

requested services?

Yes, always

Often

Sometimes

Never

I've never tried

Q9 Ease of the Interaction

Always simple

Simple (in most cases)

Difficult (It was quite difficult in most cases)

Very difficult (No, it was difficult in all cases and

sometimes I was unable to complete the service)

Q10 User satisfaction

Very unsatisfied

Unsatisfied

Neutral

Satisfied

Very Satisfied

Q11 Documentation and Help - Rate documentation

and help (manuals, interface messages, contacts, etc.)

1 = Totally dissatisfied

2 = Dissatisfied

3 = Neutral

4 = Satisfied

5 = Totally satisfied

Q12 Errors – Were you able to manage it and

move forward if you made mistakes? (clear

messages and interaction mechanisms)

I made mistakes and could not correct them

I made mistakes and could hardly correct them

I don't know

I made mistakes, but I could easily correct them

I did not make mistakes

Q13 User Interface Customization - Could the

interface be adapted to your needs? (e.g. increase font

size, change contrast, increase line spacing)

No, in general

Only in some sites

I don't know

Yes, generally

I have not tried

Q14 Aspects To Be Improved Urgently

None, everything is fine

Response times

Ease of use

Increase the number of online services to avoid face-

to-face services

Others

Section 3 -Authentication & Citizen Rights

Q15 Do you have SPID credentials?

I don't know what they are

I know what SPID is, but I don't own it

I own it, but I don't use it

I own it and I use it

Q16 Did you know that every citizen has the right

to easily access all PA services?

YES

Q17 Did you know that AgID (Agency for digital

Italy) is the authority in charge of receiving

reports on difficulties in accessing PA services?

NO, I didn't know

YES, but I didn't report it

YES, and I made some reports

Q18 Each Administration must identify a Digital

Public Advocate. Would you like to participate by

sending them your personal suggestions?

No, I'm not interested

No, I wouldn't be able

Yes, I would like to participate

I don't have time to do it.

WEBIST 2023 - 19th International Conference on Web Information Systems and Technologies