Better Policies as a Result of Policy Evaluation

Irina Igorevna Nazarova and Elena Anatolyevna Barmina

Department of State and Municipal Administration, Vyatka State University, Russia

Keywords: Policy-making, policy evaluation, policy evaluation technics.

Abstract: This article will explore the role of the evaluation in the modern policy-making process and the ways in which

evaluation activities can contribute to the improvement of the public policies and programs. Therefore, the

theoretical part of the article I will, firstly, explore the role of evaluation on various stages of policy-making

process. Secondly, the advantages and disadvantages of using qualitative and quantitative approaches in

policy evaluation will be outlines as well as one of the possible solutions for overcoming the limitations of

both these approaches will be proposed. Thirdly, this article will concentrate on the explanation of the

necessity of ex-ante and ex-post policy evaluations. In the empirical part the article will concentrate of the

concrete methods of policy evaluation and their practical use in order to demonstrate their critical role in the

improvement of public policies and programs.

1 INTRODUCTION

Public policies and programs are aimed at solving

particular social problems and bringing benefits for

certain strata of population. However, the formulation

and implementation of a certain policy or program

does not guarantee that the intended outcome will be

achieved. Policy evaluation in its turn is conducted

for the purpose of making judgements over the

effectiveness of particular policy intervention and is,

therefore, not only a keystone for the improvement of

the public policies and programs but also an

important tool that enables policy-makers to

understand whether these policies and programs

generate intended effects and need to be implemented

or extended. Moreover, policy evaluation promotes

accountability in resource allocation across various

public programs and helps to understand whether the

measured changes in public well-being can be

attributed to certain policy intervention (Khandker et

al, 2010: 3). Thus, policy evaluation is inevitable part

of the policy-making process as it provides

systematic, objective and empirical examination of

the effects of the public policies and programs

(Parsons, 1995: 545).

Nevertheless, evaluation cannot directly affect the

decision-making as new policies and programs are

seldom based solely on the results of evaluation as a

number of other factors such as, for instance,

competing pressures of ideologies, institutional

constrains and interests tend to play significant role in

the design policy intervention. Moreover, evaluation

is not often directly utilised by governments due to

the lack of institutionalised procedures and channels

connecting evaluation results with the arenas in which

decisions are being made (Weiss, 1999: 477-479).

However, by finding out and analysing the outcomes

of government interventions evaluation provides the

directions for the improvement of policies and

programs. Furthermore, policy-makers tend to pay a

significant attention to evaluation not only with the

intention to create the wisest and best policies but also

searching for justification for their actions. In

addition, some types of policy evaluation include

cost-effectiveness and cost-benefit analysis that

enable the policy-makers to make decisions based on

the finical constraints and choose the programs

according to the level of expenditure (Weiss, 1999:

473-474). Thus, although policy evaluation does not

directly affect the decision-making, it is still very

important in the policy-making process providing

politicians with the knowledge important for the

creation of the beneficial programs and policies as

well as with the justification for their actions.

2 MATERIALS AND METHODS

Three major groups of approaches to the evaluation

analysis can be identified: techniques measuring the

relations of utility and benefits to costs, techniques

measuring performance and techniques using

Nazarova, I. and Barmina, E.

Better Policies as a Result of Policy Evaluation.

DOI: 10.5220/0010698600003169

In Proceedings of the International Scientiﬁc-Practical Conference "Ensuring the Stability and Security of Socio-Economic Systems: Overcoming the Threats of the Crisis Space" (SES 2021),

pages 275-280

ISBN: 978-989-758-546-3

275

experiments in evaluating programs and policies

(Parsons, 1995: 545). Each of these techniques plays

a significant role in different phases of the policy-

making cycle. To begin with, in the agenda setting

and problem definition phases of policy cycle policy

evaluation diagnoses social conditions with intention

to provide information over the necessity of new

program and, therefore, is used to compare, identify

and prioritise the needs inside as well as outside

program areas. Moreover, evaluation research is

involved in defining the targets such as geographic

areas, groups and individuals and physical units.

Good specification of targets appropriately

establishes boundaries allowing the intervention to

correctly address the target population (Rossi et all,

2004: 130-131). Moreover, techniques of decision

analysis play major role in policy design phase by

identifying and assessing various alternatives of

achieving program outcomes with the aim of

achieving cost-effective alternative (Parsons, 1995:

546). Thus, in the in the initial phases of the policy

cycle evaluation assists in adequate identification of

social problems and specification of target groups for

intervention that are essential for the effective

operation of a policy or a program. In addition, the

techniques aimed at measuring the relationship of

costs to utility allow policy-makers to choose the best

alternative with the restricted budget.

Formative evaluation occurs in the intermediate

stages of the policy cycle. Its major aim is to unsure

that a program or a policy is feasible before it is fully

implemented. This mode of evaluation typically

addresses such questions as whether the program is

reaching the targeted population, the amount of the

resources being spent in the conduct of the policy or

whether the services are delivered according to

program or policy design specification (Parsons,

1995: 547). Thus, formative evaluation plays an

important role in the policy implementation stage as

through monitoring the way in which the program is

managed and administrated, evaluators provide

policy-makers, managers and stakeholders with

feedback enabling them to control and correct the

process of delivery of particular policy or program

more effectively.

While formative evaluations aim to assist the

program developers in improving the program in the

early stages, summative evaluation is conducted after

the curriculum is finished and is designated to help

with decisions about whether the program should be

continued or extended to other locations (Weiss,

1998:31). Summative evaluation reviews such

categories as goals, methods, results and

implementation in order to draw conclusion whether

the operation of program or policy was consistent

with the initial goals. Summative evaluation is

initially a comparative mode of inquiry as in order to

measure the actual impact of the intervention analysts

usually compare the situation before and after the

policy or program implementation in terms of

relevant outcomes. Thus, for instance, the impact of

intervention on one group can be defined by

comparing the group subjected to intervention with a

control group (Parsons, 1995: 550). Summative

evaluation in addition to evaluating the achievement

of aims also takes into consideration unintended

consequences. Therefore, a policy or a program may

be considered to be justified even if it has not

achieved the effects that it was set to achieve but has

had other beneficial effects on society (Spicker, 2006:

167-168).

Thus, evaluation research undoubtedly plays and

important role in every phase of the policy cycle.

Therefore, the difference between various phases of

policy cycle provides one of the explanations of the

choice of techniques used in evaluation. Thus, for

instance, there is no point in commencing expensive

experimental approach the results of which may not

be available for considerable period of time, if it is

necessary to obtain a quick evaluation of the short-

term impact of the program or policy in its early

stages to modify it or justify its extension. At the same

time when evaluation is conducted with the intention

to provide a guidance over the expansion of initially

limited policy or program, it is necessary to choose an

appropriate technique allowing to ensure that the

expanded policy or program would bring expected

outcomes (Hogwood and Gunn, 1984:228).

Similarly, the purpose of the evaluation

determines the appropriateness of the choice between

qualitative and quantitative research designs.

Whereas qualitative evaluators tend to use

observational techniques relying on detailed

knowledge of the processes by which the program or

policy is shaped and how program or policy

stakeholders are affected by these processes,

quantitative approaches use the statistical methods of

analysis as the data collected by quantitative

evaluators can be transformed into numerical value.

The quantitative results are found to be more

authoritative as qualitative evaluation allows to reach

conclusions with known degree of confidence.

However, qualitative investigation can provide more

dynamic information and a richness of detail.

Therefore, when the central focus of inquiry is related

to the program process qualitative approach is

generally more preferable. Moreover, in case of

introducing a new complex and innovative policy or

SES 2021 - INTERNATIONAL SCIENTIFIC-PRACTICAL CONFERENCE "ENSURING THE STABILITY AND SECURITY OF

SOCIO - ECONOMIC SYSTEMS: OVERCOMING THE THREATS OF THE CRISIS SPACE"

276

program qualitative investigation can answer the

questions concerning possible problems that can arise

or how such new entry takes shape. However, when

addressing the outcomes of the policy intervention

quantitative methods are more preferable as they can

provide accurate data of a number of people benefited

from a particular policy or program (Weiss, 1998:85-

86). Thus, both qualitative and quantitative methods

have inherited strengths are drawbacks and the choice

between these methods should be determined by the

initial goal of evaluation.

Furthermore, the weaknesses of the policy

evaluation research design can be minimised when

qualitative and quantitative methods are mixed. The

insights of the policy or program evaluation can be

yielded by integration of qualitative and quantitative

approaches that neither method would produce on its

own. While quantitative approaches provide a tool kit

that is very useful in evaluating the impact of the

policy or program, it still has significant limitations

in social science that can be overcome by

incorporating qualitative methods (Rao and

Woolcock, 2003:165). The combination of

qualitative research with quantitative tools results in

greater coverage and generalisability making the

most of comparative advantages of both approaches.

Thus, for instance, Community Score Card that

combines both qualitative and quantitative data and

analysis is successfully used as an interactive

monitoring instrument increasing accountability of

service providers. While quantitative methods are

used in scoring the particular qualities of service

provision, qualitative approaches then utilises the

generated scores for defining and diagnosing the

problems and identifying solutions (Garbarino and

Holland, 2009:16).

Both ex-ante and ex-post policy evaluations play

a critical role in the improvement of policies and

programs. While ex-ante evaluation by involving

simulations seeks to measure the anticipated impact

of future policy intervention, ex-post evaluation

measures the actual impact of the policy or program

on targeted area or population (Khandker et al, 2010:

20-21). Therefore, ex-ante evaluation makes it

possible to design a policy or a program that would

maximise the benefits at a given cost. Moreover, it

may help to avoid some initially inefficient policies

and programs providing some evidence on the

impacts that should be expected after the program

implementation. In addition, ex-ante evaluation

provides the information on how the modification of

some parameters of the policy or program would

influence the general impacts. Thus, for instance, Job

Training Partnership Act in the US was replaced

largely because the experimental evaluation

demonstrated that in addition to be very costly the

program was not beneficial for a number of

stakeholders (Todd and Wolpin, 2008: 265). Ex-post

evaluation in its turn plays a major role not only in

assessment of achievement of intended objectives but

also by identifying weak and strong points of

intervention this mode of evaluation allows to

indicate the directions for improvement of future

policies programs. Thus, ex-ante and ex-post

evaluations allow the policy-makers to improve both

existing and future policies and programs.

There is a wide variety of methods used in modern

policy evaluation, however, given the initial

complexity of the subject this article will demonstrate

the practical use only of two of them. To begin with,

one of the methods frequently used in assessing social

programs is participatory evaluation. This evaluation

model is best suited to formative evaluation and,

therefore, most frequently occurs in the intermediate

phases of policy cycle. Participatory evaluation is

conducted in order to provide opportunity for policy

or program primary users to participate in the

research process (Cousins and Earl, 1992:400).

Evaluation is viewed by participatory model as a

collaborative effort involving individuals both

internal and external to organisation such as

professionally trained evaluation personnel, program

developers, practice-based decision makers and

program or policy stakeholders in co-learning

relationship and power sharing (Harris, 2010: 7).

Participation occurs in all stages of evaluation process

from identification of relevant questions to

dissemination of outcomes and preparation of the

plan for the program or policy improvement. Thus,

the focus of policy evaluation, its design and

outcomes are determined by participants within their

own cultural, political and socioeconomic

environments. Overall, participatory evaluation is

intended to understand the preferences, voices,

perspectives and decisions of program beneficiaries

and most affected stakeholders (Zukosi and

Luluquisen, 2002:2-3).

Consequently, such guiding principles of

participatory evaluation as participant focus and

ownership, negotiation, learning and flexibility can

be identified. These core principals explain far-

reaching benefits of this model of evaluation. To

begin with, by allowing local participants to identify

the most relevant questions participatory approach

ensures that the evaluation meets the needs of policy

or program developers and beneficiaries. Secondly,

participatory evaluation provides an opportunity for

policy or program stakeholders to reflect on its

Better Policies as a Result of Policy Evaluation

277

process and apply generated knowledge to make mid-

course improvements to program or policy

performance. Thus, this model of evaluation can be

characterised as action-oriented and reflective.

Thirdly, participatory approach empowers policy or

program beneficiaries to control the process of

evaluation giving them a sense of ownership over the

evaluation results. The recognition of expertise of

local talents brings pride and confidence among

participants and in the community. Moreover,

participatory approaches enables participant learning

by providing an opportunity to introduce and develop

evaluation skills that can result in better

understanding of their environment and may lead to

active involvement of local people in advocacy for

policy change. Finally, participatory evaluation

promotes organisational learning and growth through

creating knowledge base among local organisations

and people that can be further applied on other

projects and programs (Zukosi and Luluquisen,

2002:3).

However, the benefits of participatory model of

evolution are neither guaranteed nor automatic (Guijt,

2014: 18). The major challenges to participatory

approaches may be summarised as time and

commitment, resources and conflict. Thus,

participatory evaluation requires significant time and

commitment from a number of players as it involves

coordination and training of various stakeholders

with diverse backgrounds. Differences in

participants’ backgrounds can in their turn provoke

conflicts among as well as within groups. Therefore,

foreseeing possible ways of conflict resolution among

the participants involved is required in the initial

stages of the evaluation planning. In addition, the

involvement of a number of people is required by

evaluation process available resources and funds

should be allocated realistically (Zukosi and

Luluquisen, 2002:4). Nevertheless, an appropriate

use and design of participatory evaluation enables to

overcome the challenges of this evaluation model and

benefit from its strengths.

Thus, for instance, participatory impact

evaluation approach was utilised to track progress and

change as a result of the Northern Region Rural

Integrated Programme in Ghana. Village

Development Capacity Index evaluation framework

developed by trained evaluation team in partnership

with various program stakeholders including local

rural representatives was used to assess the impact of

local capacity building to improved sanitation

services and water supply. Village representatives

provided feedback on data collected through focus

group discussions, interviews with key informants

and households, and observations in the field. The

scores on each of these indicators were allocated to

the villages under investigation by the program major

stakeholders. Consensus building and dialogue

among different stakeholders were facilitated by the

scoring process. The scores were further used to track

the program impacts and changes over time in order

to evaluate the progress by monitoring team (Estrella

and Gaventa, 1998:8). Another example of practical

use of participatory evaluation approaches comes

from Latin America. Colombia's Association of

Indigenous Councils of Northern Cauca (ACIN)

consistently participates in monitoring and evaluation

of the regional development plan. ACIN is involved

in systematic comparison of the actual results with

intended outcomes and in assessment of links

between productivity and environmental and cultural

factors. These activities have helped in recognition of

communities strengths as well in the improvement of

their management capabilities. The links between a

number of communities provide the concerted

necessary in negations with provincial and national

governments as well as with private sector (Guijt and

Gaventa, 1998:4).

Realistic evaluation is another approach widely

used in assessing public policies and programs. This

evaluation model developed by Pawson and Tilley in

1997 is focused not only on the question whether the

policy intervention produced required outcomes but

also under which conditions and how these outcomes

were produced (Gill and Turbin, 1999:181). Realistic

evaluation seeks to explain why policy or program

works through understanding mechanisms actions.

Therefore, evaluators need to concentrate on how

causal mechanisms generating behavioural and social

problems are countered or removed by the

mechanisms introduced by particular policy or

program. Moreover, this model of evaluation seeks to

understand in what circumstances and for whom the

intervention provides intended outcomes using the

context-mechanism-outcome configurations enabling

evaluators to develop cumulative and transferable

lessons from research (Pawson and Tilley, 1997:215-

218).

3 CONCLUSIONS

There are both advantages and disadvantages of this

method deserving consideration. The major strength

of realistic evaluation is its attempt to link particular

concepts to mechanisms through generative

causation. Thus, it enables evaluator to deepen the

research that is not possible with the use of orthodox

SES 2021 - INTERNATIONAL SCIENTIFIC-PRACTICAL CONFERENCE "ENSURING THE STABILITY AND SECURITY OF

SOCIO - ECONOMIC SYSTEMS: OVERCOMING THE THREATS OF THE CRISIS SPACE"

278

evaluation methods (Crabbé and Leroy, 2012: 121).

Moreover, realistic evaluation provides a consistent

and coherent framework for evaluation in various

stages of policy cycle. In addition, it promotes

maximisation of learning across practice, policy and

organisational boundaries. Overall, realistic

evaluation ‘provides a principle steer from failed one-

size-fits-all ways of responding to problems’ (Pawson

and Tilley, 2004:22). However, this evaluation model

has some distinct limitations. To begin with, there is

no a general formula that can provide step-by-step

instructions for delivering findings. Therefore,

realistic evaluation requires sustained theoretical

understanding, abilities to design techniques and

research to analyse data (Pawson and Tilley,

2004:22). Finally, this evaluation model does not

allow generalisation of findings as the context is

regarded to be one of the most important explanatory

factors (Crabbé and Leroy, 2012: 121). Nevertheless,

realistic evaluation method can be argued to be

promising for future policy evaluations.

Pawson and Tilley (1997:78-82) demonstrate the

practical use of realistic evaluation for assessing the

installation of closed-circuit television (CCTV) as a

measure against the crime in car parks. The authors

providing lists of mechanisms and contexts assess not

only the final outcomes of using CCTV for crime

reduction, but also explain how and under which

conditions the introduction of this particular measure

promotes the decrease of crimes in the car parks.

Thus, for instance, one if the mechanisms outlined is

what the authors refer to ‘memory jogging’. This

mechanism emphasise the role of CCTV as an

indicator for reminding the drivers about the

vulnerability of their cars. Thereby, drivers may be

prompted to take greater care to lock their vehicles, to

remove items that can be easily stolen from view or

to purchase additional security devises.

Evaluation of Aboriginal Parental Engagement

Program (APEP) is another empirical example

demonstrating the beneficial use of realistic approach

to policy evaluation. APEP program was funded by

federal government Department of Education and

Employment Relations of Australia with the purpose

to enhance readiness of 0-5 year old Aboriginal

children for school by increasing the level of parental

engagement in education. Ex-post realistic approach

was conducted both to assess the program outcomes

and to specify the underlying mechanisms and

contexts determining the program impact. The results

of the evaluation demonstrate that multiple

mechanisms and contexts contributes to parental

engagement. Thus, for instance, the analysis of the

surveys enabled evaluators to draw the conclusions

on how the program outcomes varied according the

families’ initial circumstances. The results of this

evaluation can be used by policy-makers and program

developers to improve future policies taking into

consideration defined mechanisms and contexts

(Cargo and Warner, 2013).

REFERENCES

Cargo, M., Warner, L., 2013. Realist evaluation” in action:

a worked example of the Aboriginal Parental

Engagement Program. MELBOURNE:

AUSTRALIAN INSTITUTE OF FAMILY STUDIES.

https://www2.aifs.gov.au.

Carol, H., Weiss, 1998. Evaluation, Second Edition.

LONDON: PRENTICE HALL.

Cousins, J.B., Earl, L.M., 1992. The case for participatory

evaluation. In Educational evaluation and policy

analysis. 14(4). pp. 397-418.

Crabbé, A., Leroy, P., 2012. The handbook of

environmental policy evaluation. In Earthscan.

Estrella, M., Gaventa, J., 1998. Who counts reality? In

Participatory monitoring and evaluation: a literature

review. BRIGHTON: INSTITUTE OF

DEVELOPMENT STUDIES.70.

Garbarino, S., Holland, J., 2009. Quantitative and

qualitative methods in impact evaluation and measuring

results.

Gill, M., Turbin, V., 1999. Evaluating “realistic

evaluation”: evidence from a study of CCTV. In Crime

Prevention Studies. 10(1). pp. 179-199.

Guijt, I., Gaventa, J., 1998. Participatory Monitoring and

Evaluation: Learning from Change.

https://www.ids.ac.uk.

Guijt, I., 2014. Participatory Approaches. In

Methodological Briefs. IMPACT EVALUATION 5,

UNICEF OFFICE OF RESEARCH, FLORENCE.

http://devinfolive.info.

Harris, M. J., 2010. Evaluating public and community

health programs. JOHN WILEY & SONS.

Hogwood, B. W., Gunn, L., 1984. Policy Analysis for the

Real World. LONDON: OXFORD UNIVERSITY

PRESS.

Khandker, S.R., Koolwal, G.B., Samad, H.A., 2010.

Handbook on impact evaluation: quantitative methods

and practices. WASHINGTON, DC: THE WORLD

BANK.

Parsons, W., 1995. Public policy: An introduction to the

theory and practice of policy analysis.

Rao, V., Woolcock, M., 2003. Integrating qualitative and

quantitative approaches in program evaluation. In The

impact of economic policies on poverty and income

distribution: Evaluation techniques and tools. pp. 165-

190.

Pawson, R., Tilley, N., 1997. Realistic Evaluation.

LONDON: SAGE.

Pawson, R., Tilley, N., 2004. Realist Evaluation.

http://www.communitymatters.com.au.

Better Policies as a Result of Policy Evaluation

279

Rossi, P.H., Lipsey, M.W., Freeman, H.E., 2004.

Evaluation: A systematic approach. SAGE

PUBLICATIONS.

Spicker, P., 2006. Policy analysis for practice: Applying

social policy. In Public Administration. 85. pp. 561-63.

Todd, P.E., Wolpin, K.I., 2008. Ex ante evaluation of social

programs. In Annales d'Economie et de Statistique. pp.

263-291.

Weiss, C. H., 1999. The interface between evaluation and

public policy. In Evaluation. 5(4). pp. 468-486.

Zukoski, A., Lulaquisen, M., 2002. Participatory

Evaluation. What is it? Why do it? What are the

challenges? In Community-based public health: Policy

and practice. http://depts.washington.edu.

SES 2021 - INTERNATIONAL SCIENTIFIC-PRACTICAL CONFERENCE "ENSURING THE STABILITY AND SECURITY OF

SOCIO - ECONOMIC SYSTEMS: OVERCOMING THE THREATS OF THE CRISIS SPACE"

280