Evaluating Privacy Policy Summarization: An Experimental Study

among Japanese Users

Vanessa Bracamonte

, Seira Hidano

, Welderufael B. Tesfay

and Shinsaku Kiyomoto

KDDI Research, Inc., Saitama, Japan

Goethe University Frankfurt, Frankfurt, Germany

Keywords: Privacy Policy, User Perception, Privacy, Risk.

Abstract: Summarization and visualization applications can help users understand the content of privacy policies.

However, research has focused on English language privacy policies and has not considered users who are

not native English speakers nor the potential situation of encountering a privacy policy in a foreign

language. In this paper, we contribute to the research on privacy policy summarization by conducting an

experimental survey on Japanese users to assess their interest on using such an application, and the

influence of this application on their perception. We conducted an experimental survey among Japanese

participants, and evaluated their perception on different privacy policy languages (Japanese or English) and

risk levels, using PrivacyGuide. We found that PrivacyGuide can increase interest in the contents of the

addition, we found that respondents who indicated interest in using the application mentioned a wide variety

of scenarios for its use, while respondents who answered negatively or were hesitant mentioned lack of trust

and uncertainty about PrivacyGuide’s reputation and accuracy. We discuss these results and offer

suggestions for improving adoption of privacy policy summarization tools like PrivacyGuide.

1 INTRODUCTION

Privacy policies are often complicated and difficult

to read (Proctor et al., 2008; Sunyaev et al., 2015).

In an effort to address this situation, regulation like

the GDPR (EU, 2016) indicates that information

about privacy policies should be written in clear

language, but even so, the cost of reading every

Cranor, 2008).

Presenting privacy-related information in formats

that are easier to understand, such as shorter privacy

policies (Gluck et al., 2016) or graphical information

regarding privacy risks (Gideon et al., 2006) can

help users make better decisions regarding privacy.

In order to provide this more understandable

information, applications that can automatically

analyze existing privacy policies and present the

summarized results them have been introduced

(Harkous et al., 2018; Tesfay et al., 2018; Wilson et

al., 2016).

A limitation of these studies is that they focus on

English speaking users and English privacy policies.

However, users can access websites all over the

world. In Japan, for example, the top sites accessed

as ranked by Alexa

include international websites

—such as those of well-known companies based in

English speaking countries. These websites provide

local language versions of their privacy policies, but

these are often direct translations of the English

version, and therefore have the same shortcomings.

And if they do not provide a translated privacy

policy, it is possible that foreign users cannot read

them at all.

Applications that summarize privacy policies and

offer a structured presentation of results may be

useful to address these two scenarios, since a

standardized format and pre-defined categorizations

are suited for translation.

In this paper, we use PrivacyGuide (Tesfay et al.,

2018) to evaluate the effects of privacy policy

summarization in a non-English language context.

We consider the question of whether PrivacyGuide

will increase interest in the privacy policy and

whether it will effectively communicate risk to

users. In addition, we consider the question of

https://www.alexa.com/topsites/countries/JP (accessed 8.2.18)

370

Bracamonte, V., Hidano, S., Tesfay, W. and Kiyomoto, S.

Evaluating Privacy Policy Summarization: An Experimental Study among Japanese Users.

DOI: 10.5220/0007378403700377

In Proceedings of the 5th International Conference on Information Systems Security and Privacy (ICISSP 2019), pages 370-377

ISBN: 978-989-758-359-9

whether Japanese users would be interested in using

such an application and under which circumstances.

We addressed these questions by conducting an

experimental survey among Japanese users,

considering two scenarios: Japanese and English

language privacy policies. We found that

PrivacyGuide results increased interest in the

content of a privacy policy in both languages.

PrivacyGuide also communicated risk to users for

the English language privacy policy. In addition,

Japanese users mentioned a variety of possible uses

for PrivacyGuide, but also identified barriers to its

use, such as uncertainty about its trustworthiness,

reputation and accuracy of the results. These results

contribute evidence of the potential and challenges

for privacy policy summarization for non-English

speaking users. To the best of our knowledge, this is

the first user perception study on privacy policy

analysis tools such as PrivacyGuide. We discuss our

findings and possible ways to improve adoption of

2 BACKGROUND

2.1 User Perception of Privacy Policy

Summarization

Previous research has found that presenting

information in more understandable formats can

help users better understand the privacy practices of

websites. Icons indicating the privacy risk level

(Gideon et al., 2006) and detailed information of

what data is at risk (Harbach et al., 2014) can help

users make more privacy conscious decisions. In

particular for privacy policies, standardized

information in a format similar to a nutrition label

can more effective in helping users obtain accurate

information than a text privacy policy (Kelley et al.,

2010) and a shorter text can provide enough

information for users to understand risks in privacy

policies, compared to longer documents (Gluck et

al., 2016).

Although these studies demonstrated possible

ways to design privacy policies to effectively

communicate risk to users and make them aware of

privacy practices, the current situation is that privacy

policies are often complex and lengthy text-only

documents. However, there have been research

efforts to provide applications to automatically

analyze existing privacy policies and visually

present these results. These applications define a

categorization of the privacy policies’ contents,

although the basis for that categorization may be

different: based fair information practices and

policies (Zimmeck and Bellovin, 2014), on the

knowledge of privacy domain experts (Harkous et

al., 2018; Wilson et al., 2016) or based on regulation

such as the GDPR (Tesfay et al., 2018). However,

these studies have not included user validation

studies of the effectiveness of these applications for

communicating risk.

As far as we could determine, there are no

studies on privacy policies in foreign language in the

privacy literature. However, we consider that this is

a topic that merits research. Anecdotal evidence

indicates that Japanese users are concerned when

they see English privacy policies. This was the case

when the GDPR came into effect on May 2018 and

Japanese users received updates to privacy policies

in English, which they could not read.

In addition, currently the list of top smartphone

apps on Google Play and Apple’s App store for

Japan regularly includes apps from foreign

developers whose websites have only an English

language privacy policy. Although in general, very

few users check the privacy policies of the websites

they visit (Steinfeld, 2016), this is not an ideal

situation from the point of view of providing users

the information they need to make privacy related

decisions.

3 METHODOLOGY

3.1 PrivacyGuide

In this study, we use of PrivacyGuide (Tesfay et al.,

2018), a machine learning-based application for

automatically analyzing and summarizing privacy

policies written in English. The goal of

PrivacyGuide is to support users’ understanding of

the privacy policy and to elicit interest in the detail

of its contents, by providing risk-related information

about the privacy policy.

PrivacyGuide classifies the content of the

Tesfay et al., (2018) for details), which are based on

an analysis of criteria from the EU GDPR, and

determines a risk level for each privacy aspect. It

presents the result of this analysis with the use of

icons and colors: one icon for each privacy aspect in

green, yellow or red color corresponding to the risk

level identified. Privacy guide was considered

suitable for the purposes of this experiment due to its

structured result presentation format, which could be

straightforwardly translated to Japanese language.

In addition, although PrivacyGuide was

Evaluating Privacy Policy Summarization: An Experimental Study among Japanese Users

371

developed specifically for the analysis of English

language privacy policies following the GDPR, we

considered that this approach was compatible for

language privacy policies of international websites.

Currently, the content and structure of the English

privacy policies for these websites have been

developed with awareness of the GDPR. And as we

could confirm during the development of the

experiment’s privacy policies, the Japanese versions

of international websites’ privacy policy are in many

cases a direct translation of the English language

of PrivacyGuide was appropriate for the experiment.

3.2 Experiment Design

We designed an experimental survey to address the

following research questions:

 Will PrivacyGuide increase users’ interest in the

content of the privacy policy itself?

 Will the summarized privacy policy result

provided by PrivacyGuide correctly

communicate risk?

 Will users be interested in using PrivacyGuide

and if so, under which circumstances?

In addition, we considered the use case of English

language privacy policies for each of these

questions.

To address these questions, we conducted an

experimental survey where we asked participants to

answer an online questionnaire based on their

perception of a website page, which included a

defined four experimental conditions resulting from

the combination of privacy policy language (English

or Japanese) and risk level (low or high) factors.

The online questionnaire instructed participants

to imagine a situation where they had found a

website and were considering whether or not to

page and asked participants to view it as they would

in their normal internet use. The participants were

not primed to consider privacy in the instructions

and we did not ask them to read the privacy policy,

which would not have been possible for every

participant in the English language conditions.

After answering questions on their perception of

the website, the survey introduced PrivacyGuide as a

and instructed participants to take some time to

check the PrivacyGuide results. The participants

then answered the remaining questions.

The survey was conducted online, using a third-

party online survey company. The survey company

distributed the call for participation among their

subscribers. The sample is therefore a convenience

sample, although the recruitment process targeted a

pool with an equal distribution of gender and an age

distribution similar to that of the Japanese

population. Each participant was randomly assigned

to one of the four conditions, and viewed only one

version of the website page —therefore, only one

screen. All participants answered the same

questions.

3.3 Website Page

We developed a non-interactive website sign-in page

for the experiment. We used a fictional company to

control for reputation effects.

The page consisted of a simple online

registration form with first and last name, email

address and password input fields, a scrollable text

area with the experiment privacy policy, and a check

box to indicate agreement to the privacy policy.

Besides the privacy policy content, the design of the

website page was the same for all conditions.

We developed the page in Japanese and therefore

did not conduct a translation process, but we

reviewed the language and interface design

naturalness.

3.4 Privacy Policies

We constructed four privacy policies, corresponding

to the combination of languages and risk levels

determined for the experiment. The PrivacyGuide

result screen consists of eleven privacy aspects,

whose respective risk levels are indicated by icons in

three different colors: red, yellow and green,

depending on the content of the privacy policy.

However, we determined that it was not feasible to

test all combinations of privacy aspects and risk

levels, nor was it the goal of this experiment to

measure the effect of specific privacy aspects.

In order to facilitate the distinction, we

established two risk levels —low and high— for the

experiment. These levels were not intended to

represent an absolute scale, but rather to

approximate a risk level that users might realistically

encounter in normal circumstances.

We used the following procedure to construct the

low and high privacy policies for the experiment.

First, we obtained a list of the top 50 websites

accessed from Japan from the Alexa website (Alexa

ICISSP 2019 - 5th International Conference on Information Systems Security and Privacy

372

Internet, Inc., 2018) on August 8, 2018. From this

list, we selected those websites which provided an

English language privacy policy.

We identified ten privacy policies using these

criteria, which were then analyzed using

PrivacyGuide. The PrivacyGuide result for each

aspects and corresponding risk levels. We assigned a

value to each level (green=1; yellow=2; red=3) and

calculated a total risk value for each combination.

Higher values were considered to indicate a higher

risk privacy policy. We removed the privacy policies

with the highest and lowest risk values, and took the

next values as initial candidates for low and high

risk.

We then counted the frequency of each risk level

for each privacy aspect and identified a base pattern:

a group of privacy aspects with the same risk level

for most privacy policies in the list. To this base

pattern, and corresponding risk value, we added the

remaining privacy aspect levels at a risk level that

would help reach a value corresponding to a low and

a high-risk level privacy policy.

To create the text of the experiment privacy

policies we used fragments from privacy policies of

existing websites. We obtained these privacy

policies from websites which had been used to

develop PrivacyGuide and which had both a

Japanese and English language privacy policy, for

which the Japanese version was a translation of the

English version. A person fluent in Japanese and

English verified that the fragments included the

same content in both languages. We removed

references to the original website and put together

the fragments to create the English and Japanese low

and high-risk privacy policies for the experiment.

To minimize any possible influence due to

differences in privacy policy length, we verified that

the privacy policies created had a similar number of

lines, and word count (for the English version) or

character count (for the Japanese version).

3.5 PrivacyGuide Result

In order to construct the PrivacyGuide result screen

for the experiment, we first translated the

PrivacyGuide interface and privacy aspects to

Japanese, as follows. A person fluent in English and

Japanese translated the text elements of the

interface, such as page title, instructions and button

labels. A native Japanese speaker fluent in English,

with expert knowledge of Japanese and European

privacy regulation, translated the privacy aspects

name and description. A second native Japanese

speaker reviewed the translated PrivacyGuide

interface elements and privacy aspects, focusing on

understandability and naturalness. We addressed the

translation issues identified in this stage.

We then constructed two versions of the

PrivacyGuide result screen, corresponding to the low

and high risk levels described in the previous

subsection. During this development, we simplified

the presentation of PrivacyGuide results as a

consequence of feedback received from the Japanese

reviewer. PrivacyGuide shows the privacy aspect

description and a fragment of the analyzed privacy

policy that corresponds to the risk assessment when

the user hovers over an icon. For the experiment, the

non-interactive version of the result screen that we

constructed included icons as well as the privacy

aspect name and description of the identified risk

level for the privacy aspect. We did not include the

original privacy policy fragment since we did not

ask participants to read the privacy policy and

because it was not possible to ask them to compare

the accuracy of those results to the English language

3.6 Questionnaire

We addressed perception of the website through

three constructs: behavioral intention, risk

perception and privacy concern using items adapted

from previous research on user perception of

websites (Kim et al., 2008). We used a six-point

Likert scale for the items’ score; the scales ranged

from Completely Disagree to Completely Agree,

with the exception of risk items, which ranged from

Very Safe to Very Risky. In addition, we included

an open-ended question asking participants if they

were interested in using PrivacyGuide to analyze a

website’s privacy policy, and if so, which one.

We also included Likert-style questions about the

participants’ normal privacy policy-related behavior,

whether they would use websites in English and

whether they would read privacy policies in English.

We included these two last questions as proxy for

measuring self-perceived English ability related to

these tasks, in order to avoid self-effacing responses.

We also measured the number of times a

participant viewed the experiment website and

PrivacyGuide result, the time they spent viewing

those screens and the total time spend on the survey.

We initially developed the survey questionnaire

in English and then translated it to Japanese with the

following procedure. First, a native Japanese speaker

fluent in English forward translated the

questionnaire. Then, a second native Japanese

Evaluating Privacy Policy Summarization: An Experimental Study among Japanese Users

373

speaker —a person familiar with privacy research—

reviewed the translation with a focus on accuracy,

identifying and modifying any inaccuracies. A third

native Japanese speaker conducted an additional

review of the translation, which focused on

understandability and naturalness. The translated

questionnaire was then compared with the original

English one, by a native Japanese speaker and a

person fluent in English and Japanese.

At every stage, identified issues were discussed

and addressed by the translators and reviewers until

there was agreement about the questionnaire text.

3.7 Limitations

The methodology we used had the following

limitations. We used a convenience sample, obtained

from a limited pool of users that had subscribed to

participate in online surveys conducted by the third-

party survey company. This may have introduced

bias in the analysis; however, as the results in the

next section show, the sample age distribution

followed the Japanese population age distribution

and we obtained an equal number of male and

female respondents. In addition, the website page

developed for the experiment as well as the

PrivacyGuide result screen were non-interactive,

which limited the authenticity of the scenario

proposed to the participants. Finally, when we

assigned the privacy policies’ total risk values we

assumed that all privacy aspects had equal

importance, but users may have different priorities

and may consider a particular privacy aspect more

important than another.

4 ANALYSIS AND RESULTS

4.1 Data Validation

The responses were collected from August 30 until

August 31, 2018. We obtained a total of 1040

participant responses, with 260 participants in each

group, as predefined by the survey process.

We first analyzed these data to identify

suspicious response patterns. We used the criterion

of no variability of extreme responses —where the

answer to every question was either 1 or 6—to select

the initial candidates for elimination, and assessed

the total survey response time for these cases. All

cases identified by the no variability criterion

showed a low total survey response time. The total

sample after eliminating these cases was 984.

Construct validity was evaluated using

confirmatory factor analysis (CFA) using maximum

likelihood estimation with robust standard errors and

a Satorra-Bentler scaled test statistic to correct for

nonnormality (Curran et al., 1996; Rosseel, 2012).

Items loaded on their respective constructs with a

standardized loading higher than 0.7 and the model

showed good fit: RMSEA=0.06, CFI=0.97,

TLI=0.97, SRMR=0.03 (Kline, 2005). All constructs

showed good internal consistency, with a minimum

Cronbach's alpha value of 0.87. We then constructed

composite variables by summing the validated items

for each construct. We used non-parametric

statistical tests for the analysis due to non-normality

and we used the Benjamini–Hochberg procedure to

control for false positives (Benjamini and Hochberg,

1995) due to the multiple statistical tests,. The

adjusted p-values are reported for each test.

Regarding the sample characteristics, there was a

similar distribution of gender: 490 male (49.8%) and

494 female (50.2%) respondents. The minimum

respondent age was 18 and the maximum was 69.

We found no statistical differences in the

distribution of age and gender between experimental

condition groups. Half of respondents (52%)

indicated that they read the privacy policies of

websites at least occasionally. In addition, a majority

of respondents indicated they would not use English

websites or apps (81%), or read privacy policies in

English (88%).

4.2 Interest in the Privacy Policy

We first compared initial interest for the privacy

policies in the different language conditions. The

results of a Mann-Whitney U test showed that

interest in the contents of the Japanese privacy

policy was significantly higher (p=0.024) than

interest in the English one.

Next, we used separate Wilcoxon Signed-Rank

tests to evaluate interest in the privacy policy after

viewing PrivacyGuide, for both risks levels in each

language condition. Interest in the contents of the

Japanese privacy policy significantly increased for

both risk levels (low: p=0.0004; high: p=0.04). On

the other hand, interest in the English privacy policy

increased only after viewing the low risk results

(p=0.0002), but not for the high-risk result. The

results provide evidence that PrivacyGuide can

indeed promote interest in the privacy policy. In the

case of the high-risk English privacy policy, the lack

of interest may be due to respondents completely

dismissing the possibility of using the website itself

ICISSP 2019 - 5th International Conference on Information Systems Security and Privacy

374

and therefore considering that they no longer have to

worry about the contents of its privacy policy.

4.3 Website Perception

We evaluated website perception to address whether

PrivacyGuide communicated risk to respondents.

First, we compared the respondents’ initial

perception of the English and Japanese privacy

policy language versions of the website, using

separate Mann-Whitney U tests for the statistical

analysis. Respondents were less willing to register

on a website with an English privacy policy

(p=0.015) than a website with a Japanese privacy

policy, although we found no statistically significant

differences in (p=0.07) or privacy concern (p=0.34).

We also compared the time spent viewing the

experiment website, the PrivacyGuide result screen

and the total time taken to finish the survey, between

participants in different language conditions. Time

data was not normally distributed, so we used

separate one-tailed Mann-Whitney U tests for the

analysis. There were no significant differences for

any of the viewing times or for the total survey time

(p>0.05). Since the majority of participants indicated

low self-perceived English ability, this suggests that

participants’ lower behavioral intention was

influenced simply by seeing the English text.

We then evaluated the effect of PrivacyGuide on

perception of the website, using separate Wilcoxon

Signed-Rank tests for both risk levels in each

language condition. There was a significant increase

in behavioral intention (p=0.0004) and a significant

decrease in risk perception (p=0.004) for the low-

risk English privacy policy condition, but there were

no significant differences for any of the other

conditions.

Considering that behavioral intention towards the

websites with an English privacy policy was initially

lower, the results suggest that PrivacyGuide

effectively communicated risk level information to

respondents, whose perception of the website

improved for the low-risk condition but not for the

high-risk condition. This also might explain why

interest in the contents of the high-risk English

policy did not increase: users would not be

interested in the privacy policy of a website that they

are not considering using.

In the case of the Japanese privacy policy

conditions, the initial intention and risk perception

may not have greatly influenced by the privacy

policy itself, but rather by website unfamiliarity.

Therefore, although the respondents were more

interested about the contents of the Japanese privacy

policy, any additional information about the privacy

policy did not have significant influence on the

website itself. On the other hand, we did not find

any statistical difference in privacy concern for any

of the conditions.

4.4 Interest in Privacyguide

To address the question of interest in PrivacyGuide,

we coded the responses to the open-ended question.

A native Japanese speaker familiar with the goal and

structure of the survey coded the responses to the

open-ended question on participants’ interest in

trying PrivacyGuide according to whether they were

positive, negative or neutral (“I don’t know”). If the

answer did not correspond to either of these types, it

was coded as “other”. Table 1 shows the comments

by type (blank or “Other” responses are not

included).

Table 1: Interest in PrivacyGuide.

Positive Negative Don’t know

N % N % N %

EN-Low R 62 26% 123 51% 17 7%

EN-High R 46 19% 136 56% 14 6%

JP-Low R 67 27% 134 53% 8 3%

JP-High R 56 22% 138 55% 10 4%

The proportion of positive, negative and neutral

responses was similar for the same risk level of

different language groups. We validated this by

quantitatively testing for the difference in interest in

using PrivacyGuide between risk levels for each

language, using separate one-tailed Mann-Whitney

U tests. There was slightly higher interest in using

PrivacyGuide for a low risk English privacy policy,

but the difference was not statistically significant

(p>0.05). Similarly, the Japanese group showed no

statistical difference in interest between the risk

levels (p>0.05).

In addition, the results of cross tabulation

showed a significant relationship between higher

interest in PrivacyGuide (chi square=26.52, df=2,

p<0.001). Respondents who read privacy policies at

least some times are the ones more likely to be

interested in PrivacyGuide.

4.5 Use Cases and Barriers

In addition to the quantitative analysis, we also

qualitatively analyzed the content of responses.

Regarding which websites they would be interested

in analyzing, respondents in both language groups

mentioned a variety of use cases. They gave as an

Evaluating Privacy Policy Summarization: An Experimental Study among Japanese Users

375

example types of websites ranging from online

shopping and SNS websites to financial and

government websites.

In addition, they indicated interest in trying

PrivacyGuide on the privacy policies of websites

they frequently used —with mentions of Google,

Yahoo and Instagram, among other well-known

international websites—, but they also mentioned

wanting to use it on unfamiliar websites. In

particular, they mentioned wanting to use

PrivacyGuide when registering on a new website, if

they felt the website was asking for too much

personal information. Respondents mentioned

personal information in general, only specifying

address, phone number and credit card as examples.

Interestingly, respondents from the English language

groups mentioned an interest on trying PrivacyGuide

on Japanese websites; conversely, a respondent from

the Japanese language groups mentioned the

potential usefulness for analyzing foreign websites’

privacy policies.

With regards to respondents who indicated no

interest in using PrivacyGuide, for the most part they

did not specify a reason for their answer. We

consider that non-interactive PrivacyGuide result

screen could have limited further the interest shown

in the application. Next in frequency were

respondents who mentioned that they had concerns

regarding the trustworthiness and reputation

PrivacyGuide, and therefore would not use it.

Similarly, respondents who answered neutrally

mentioned that they would consider using

PrivacyGuide if it could be trusted, if it was

provided by a well-known company or “if

everybody used it”. Negative and neutral

respondents also mentioned that they did not know

the accuracy of PrivacyGuide, and therefore did not

know whether they could rely on its results. Other

respondents indicated that they did not need to use

PrivacyGuide because they would not use risky

websites in the first place.

5 DISCUSSION

The results indicate that privacy policy

summarization, as the one provided by

PrivacyGuide, has potential to be beneficial for

Japanese users, in particular for foreign language

privacy policies. PrivacyGuide effectively

communicated risk in case of the English language

website, although there were no changes for the

Japanese language privacy policy. In the case of

increase of interest in the privacy policy contents

and interest in using PrivacyGuide, results were

similar for the Japanese and English conditions.

In general, results are consistent with previous

research in other countries. In the case of privacy

concern, on which Privacy had no effect, the results

are similar to those found in (Gluck et al., 2016). We

consider the possibility that understanding of the

privacy practices of a website by itself cannot ease

users’ feelings of concern, in particular for an

unknown website. The results also indicate

challenges and areas for improvement. Addressing

lack of reputation is one way that could help

improve adoption among Japanese users. This could

be realized if well-known organizations or

companies provide or support these applications,

although what constitutes “well-known” has to

consider the local context.

An important challenge to consider is Japanese

users’ concern about the trustworthiness of

PrivacyGuide. We could not determine whether

there were specific aspects that would influence

trustworthiness perception, but we consider that

Japanese users’ comments on the accuracy of results

may be a factor. Research on automation and

machine learning-based applications suggests that

providing explanation of results can increase trust

(Lee and See, 2004), but there is need to consider

how much information to provide according to the

context and user expectations (Kizilcec, 2016). In

the case of the Japanese privacy policy, presenting

users the fragment of text used to decide on a

privacy aspect risk level would have provided some

additional information that would help them

evaluate accuracy. Future research is planned to

evaluate how to communicate trustworthiness and

accuracy of the application, in particular for users

who have to rely on it to understand foreign

language privacy policies.

6 CONCLUSIONS

We conducted an experimental survey among

Japanese users to evaluate of the effects of a privacy

policy summarization application, PrivacyGuide.

We considered two scenarios, native language

(Japanese) and foreign language (English) privacy

policies. The results showed that PrivacyGuide can

achieve its goal of increasing interest in the content

of privacy policy, for both languages. And in the

case of the English privacy policy, it can effectively

communicate risk and affect perception of a website.

In addition, we found that Japanese users would

ICISSP 2019 - 5th International Conference on Information Systems Security and Privacy

376

want analyze the privacy policies of different types

of website —familiar and unfamiliar, domestic and

foreign. On the other hand, users indicate that lack

of trustworthiness, reputation and explanation about

the results are barriers for use of the application. In

future research, we will address the barriers

identified by users, in particular regarding trust and

how to provide explanations for automated analysis.

ACKNOWLEDGEMENTS

We would like to thank Naonori Kato for his help

during the preparation of this study.

REFERENCES

Benjamini, Y., Hochberg, Y., 1995. Controlling the False

Discovery Rate: A Practical and Powerful Approach to

Multiple Testing. Journal of the Royal Statistical

Society. Series B (Methodological) 57, 289–300.

Curran, P.J., West, S.G., Finch, J.F., 1996. The

Robustness of Test Statistics to Nonnormality and

Specification Error in Confirmatory Factor Analysis.

Psychological methods 1, 16–29.

EU, 2016. REGULATION (EU) 2016/679 OF THE

EUROPEAN PARLIAMENT AND OF THE

COUNCIL on the protection of natural persons with

regard to the processing of personal data and on the

free movement of such data, and repealing Directive

95/46/EC (General Data Protection Regulation).

Gideon, J., Cranor, L., Egelman, S., Acquisti, A., 2006.

Power Strips, Prophylactics, and Privacy, Oh My!

Presented at the Symposium On Usable Privacy and

Security (SOUPS) 2006, ACM Press, p. 12.

Gluck, J., Schaub, F., Friedman, A., Habib, H., Sadeh, N.,

Cranor, L.F., Agarwal, Y., 2016. How Short Is Too

Short? Implications of Length and Framing on the

Effectiveness of Privacy Notices, in: Twelfth

Symposium on Usable Privacy and Security (SOUPS

2016). USENIX Association, Denver, CO, pp. 321–

340.

Harbach, M., Hettig, M., Weber, S., Smith, M., 2014.

Using Personal Examples to Improve Risk

Communication for Security & Privacy Decisions, in:

Proceedings of the SIGCHI Conference on Human

Factors in Computing Systems, CHI ’14. ACM, New

York, NY, USA, pp. 2647–2656.

Harkous, H., Fawaz, K., Lebret, R., Schaub, F., Shin, K.,

Aberer, K., 2018. Polisis: Automated Analysis and

Presentation of Privacy Policies Using Deep Learning

[WWW Document].

Kelley, P.G., Cesca, L., Bresee, J., Cranor, L.F., 2010.

Standardizing Privacy Notices: An Online Study of the

Nutrition Label Approach, in: Proceedings of the

SIGCHI Conference on Human Factors in Computing

Systems, CHI ’10. ACM, New York, NY, USA, pp.

1573–1582.

Kim, D.J., Ferrin, D.L., Rao, H.R., 2008. A Trust-Based

Consumer Decision-Making Model in Electronic

Commerce: The Role of Trust, Perceived Risk, and

Their Antecedents. Decision Support Systems 44, 544–

564.

Kizilcec, R.F., 2016. How Much Information?: Effects of

Transparency on Trust in an Algorithmic Interface, in:

Proceedings of the 2016 CHI Conference on Human

Factors in Computing Systems, CHI’16. pp. 2390–

2395.

Kline, R.B., 2005. Principles and practice of structural

equation modeling, 2nd ed, Principles and practice of

structural equation modeling, 2nd ed. Guilford Press,

New York, NY, US.

Lee, J.D., See, K.A., 2004. Trust in Automation:

Designing for Appropriate Reliance. Hum Factors 46,

50–80.

McDonald, A.M., Cranor, L.F., 2008. The Cost of

Reading Privacy Policies. ISJLP 4, 543.

Proctor, R.W., Ali, M.A., Vu, K.-P.L., 2008. Examining

Usability of Web Privacy Policies. International

Journal of Human–Computer Interaction 24

, 307–328.

Rosseel, Y., 2012. lavaan: An R Package for Structural

Equation Modeling. Journal of Statistical Software,

Articles 48, 1–36.

Steinfeld, N., 2016. “I agree to the terms and conditions”:

(How) do users read privacy policies online? An eye-

tracking experiment. Computers in Human Behavior

55, 992–1000.

Sunyaev, A., Dehling, T., Taylor, P.L., Mandl, K.D.,

2015. Availability and quality of mobile health app

privacy policies. J Am Med Inform Assoc 22, e28–e33.

Tesfay, W.B., Hofmann, P., Nakamura, T., Kiyomoto, S.,

Serna, J., 2018. PrivacyGuide: Towards an

Implementation of the EU GDPR on Internet Privacy

Policy Evaluation, in: Proceedings of the Fourth ACM

International Workshop on Security and Privacy

Analytics, IWSPA ’18. ACM, New York, NY, USA,

pp. 15–21.

Wilson, S., Schaub, F., Dara, A.A., Liu, F., Cherivirala, S.,

Giovanni Leon, P., Schaarup Andersen, M., Zimmeck,

S., Sathyendra, K.M., Russell, N.C., B. Norton, T.,

Hovy, E., Reidenberg, J., Sadeh, N., 2016. The

Creation and Analysis of a Website Privacy Policy

Corpus, in: Proceedings of the 54th Annual Meeting of

the Association for Computational Linguistics

(Volume 1: Long Papers). pp. 1330–1340.

Zimmeck, S., Bellovin, S.M., 2014. Privee: An

Architecture for Automatically Analyzing Web

Privacy Policies, in: Proceedings of the 23rd USENIX

Conference on Security Symposium, SEC’14. USENIX

Association, Berkeley, CA, USA, pp. 1–16.

Evaluating Privacy Policy Summarization: An Experimental Study among Japanese Users

377