Growing Together

A Model of Critical Research Design

Wakhid Nashruddin

1, 2

Charles Darwin Unversity, Australia

IAIN Syekh Nurjati, Cirebon, Indonesia

wakhid.nashruddin

cdu.edu.au, wakhid.nashruddin

ekhnur

ati.ac.id

Keywords: Engagement, academics, research design.

Abstract: This paper illustrates a research design that models direct engagement of academics teaching in English

Language Teacher Education (ELTE) departments in implementing national policies, in this case, the

integrating of National Standards of Higher Education (NSHE) into undergraduate courses. The key

difference in the design between the present study and conventional research design is in that it did not begin

with a question about its form, quantitative and qualitative. Furthermore, in Education, research design is

largely limited to surveys and a testing of simple hypotheses. Unlike these traditional approaches, the present

study engaged ELTE academics directly in the process of collaborative reflection on the meaning of the new

higher education policies from the perspective of their own contexts. Key issue in designing the study was to

conceptualise a perspective that will help inform this process of engagement, academics’ reflections and

further learning. Complex analyses were conducted to reach this point. Again, this was a very different process

than simply listing “previous research”. The presentation will briefly illustrate the background thinking behind

the study design, the design and the impacts that the chosen design has helped to achieve.

1 INTRODUCTION

Critical research design might not be a popular design

in educational research. The term critical itself is

based on critical theory (Gelo, 2012, p.123) and aims

at raising question “what should be?” (Tracy, 2013,

p.48). The focus of this kind of research aims at

transformation or improvement of practice (Tracy,

2013, p.48).

The research design in this paper is a design used

in a PhD project entitled in “Developing

undergraduate research and inquiry in English

language teacher education programs in Indonesia:

A case study for empowerment and sustainability.”

The research and the data collections of the research

have been completed. In this research project, a quite

new issue that has not been well addressed in the

context of English Language Teacher Education

(ELTE) in Indonesia was raised. In this study, the

participants were asked to challenge their beliefs in

integrating IQF into their teaching practice.

2 ONTOLOGICAL AND

EPISTEMOLOGICAL

FRAMING

The present study draws on the principles of a

dialogic model of inquiry explicated by Lian (2006,

2012) in relation to research. Lian herself draws on

Calhoun’s model (1995) theorised for the purpose of

critical social sciences and Latour (2003, 2004, 2010)

in the philosophy of science. As illustrated by Lian

(2012, p. 2), from the perspective of ontology, a

dialogic inquiry resists the temptation to “seek [...] the

umpire’s chair” (Calhoun, 1995, p. 11) and the idea

that researchers “move simply from false

propositions to true ones” (Calhoun, 1995, p. 7),

“claim[ing] – like Sherlock Holmes – to be working

with “nothing but the facts” (Calhoun, 1995, p. 5).

Instead, it is conscious of its own historicity and

draws a distinction between the interpretations which

guide researchers’ questions and those of his or her

informants (who have different stakes than the

researcher). The inquiry serves as a tool for the

researcher to identify the possibilities as well as the

limits of his or her perspectives. A dialogic inquiry

therefore neither represents voices nor gives voice; it

256

Nashruddin, W.

Growing Together - A Model of Critical Research Design.

In Proceedings of the Annual Conference on Social Sciences and Humanities (ANCOSH 2018) - Revitalization of Local Wisdom in Global and Competitive Era, pages 256-261

ISBN: 978-989-758-343-8

explores the relevance of its own voice: “You and

your informants have different concerns—when they

intersect it’s a miracle, and miracles, in case you don’t

know, are rare” (Latour, 2003, p. 71).

In terms of epistemology, this self-reflective

(critical) process originates in the perception of

conflict (Lian, 2012, p. 2) which dialogic researchers

purposefully seek out in order to “connect widely

different phenomena” and build increasingly

informed perspectives on the issues of interest

(Latour, 2004). This process results in the

construction of the points “from which positions, or

possibilities, become more perceptible” (Hobson, as

cited in Lian, 2012, pp. 2-3). Lian (2012, p. 3)

explains, this process is oriented toward “discovering

our limits rather than affirming our possibilities”

(Calhoun, 1995, p. 13), i.e. an objective which gives

the inquiry its critical process.

The dialogic approach provides this study with a

conceptual framework specifying its purpose and the

methodology. In accordance with the dialogic model,

in terms of purpose (i.e. what is possible), the study

will engage academics teaching in English language

teacher education programs in Indonesia in a

capacity-building context. Academic staff teaching in

these programs will be involved in constructing,

implementing and evaluating conceptual frameworks

and strategies for supporting the culture and practices

of research pedagogy in their undergraduate

programs. In terms of methodology (self-reflection),

the study will engage the participants in collaborative

and learning opportunities, with the participants

looking for ways to question what they know and

how, without requiring from anyone to adopt a

specific worldview or theory. The study will evaluate

the impact of this learning process in relation to (a)

feedback provided by the participants, and (b) the

concepts that underpin current thinking and research

in the area of research pedagogy in English language

teacher education programs.

3 STUDY DESIGN AND

INSTRUMENTS FOR DATA

COLLECTION

The study aims to engage academics teaching in

English teacher education programs across Indonesia.

In order to make the study manageable but also

reflective of the diversity of higher education

intuitions in Indonesia, the study identifies a sample

of institutions from the public and private sectors,

with both secular and religious missions (three private

universities and one Islamic university took part in

the study). The institutions that were invited to take

part in the study are already part of the network of the

researcher’s home university; others will be contacted

by the researcher.

In different terms, this critical research may be

called an action research project. While most

traditional action research uses researchers’

solution(s) for identified problems in those research,

or a group of people (Tomal, 2010, p. 129) that

discussed the problem to offer solutions, this current

study facilitates the research participants in the study

took a large control over their action plans, with as

minimum interference as possible to make sure the

actions are developed by the participants themselves.

Such condition is expected to make the participants

more creative and innovative in grappling with the

issue of integration of research and inquiry in their

context where they use their own networking, IT, and

supports they already have.

Questionnaires, workshop, and interview are used

to collect the data. Questionnaires are conducted

before, during, and after the implementation of the

integration of research and inquiry in English

Language Teacher Education programs.

3.1 Questionnaire 1: Initial

Questionnaire

The questions in the questionnaires address

international developments and, specifically, issues

raised in regard to research culture in undergraduate

programs in the Boyer Report (Boyer Commission,

1998), USA; Bradley Report (Bradley, Noonan,

Nugent, and Scales, 2008), Australia; and in the

various documents informing the development of

universities in the ASEAN member countries (e.g.

ASEAN Secretariat, 2009). The preliminary literature

review conducted thus far to examine these

developments in the USA and Australia showed a

number of areas where progress is slow, with studies

like Barrie et al. (2014) reporting that supporting

research and inquiry in undergraduate programs is not

without its problems. Thus, questions reflecting on

the relationship between the graduate outcomes show

to be important and will be explored.

To frame the questionnaire, “three levels of the

architecture of a job” (HBS, 2010) were used. Firstly,

exploration on “what is the job to be done” was used

to capture the impact of the implementation of IQF

(Indonesian Government, 2013) and Decree 44

(MRTHE, 2015). Secondly, this identification of the

jobs to be done provided reflections on “what are the

experiences” in relation to what academic staff doing

Growing Together - A Model of Critical Research Design

257

research in IQF and Decree 44 implementation.

Finally, the reflections was expected to lead to

thinking of “what and how we must integrate” IQF

and graduate competencies in the curriculum

supporting undergraduate research. The topics of the

questionnaire are based on the Boyer report’s

recommendation (The Boyer Commission, 1998).

Research question 1: What challenges does the

integration of graduate skills into degree curricula

present to academics teaching in English language

teacher education programs in Indonesian research

universities?

An anonymous questionnaire is developed and

sent to departments teaching English language

teacher education programs in universities from the

public and private sectors, with both secular and

religious missions across Indonesia. The objective of

the questionnaire is to (a) gather evidence about the

broader context that impacts on the design and

workings of the English language teacher education

programs in Indonesia, including the knowledge of

the international context and its impact on national

(and university) policies and strategic planning; and

(b) to make space for academics to critically approach

current developments, practices and beliefs. The data

from initial questionnaire informs Stage 2 of the

study, reflection workshops.

The following list is initial questionnaire

questions:

1) Please identify your academic level!

• Level E (Guru Besar/Gol.IV.E)

• Level D (Guru Besar/Gol.IV.D)

• Level C (Lektor Kepala)

• Level B (Lektor)

• Level A (Asisten Ahli)

2) Briefly describe yourself in terms of teaching,

research interests and any information you wish

to share? What research have you produced

relating to your teaching?

3) In as many words as you wish, please describe

your experiences with Graduate Competencies!

In your response, you may wish to consider the

following guiding points:

• Briefly describe your views of the

Graduate Competencies.

• Are you using Graduate Competencies in

your teaching?

• In your opinion, what impact do Graduate

Competencies have on the planning of the

English degree program and the units that

you teach?

• Are your students aware of Graduate

Competencies? If so, how do you

perceive the Graduate Competencies

impact on their learning?

• In your research, do you make references

to Graduate Competencies? Briefly

describe how.

• What problems do you find with Graduate

Competencies? Explain your views.

• Have you been supported by your

university or Department in working with

Graduate Competencies? How have these

supports been made available?

3.2 A Three-Day Workshop

Research question 2: In what ways can a theoretical

approach to graduate skills contribute to the building

of a 21

century model of English teacher education

programs relevant to the Indonesian context?

In order to generate opportunities for critical

reflection, the researcher organised a three-day

workshop. The workshop invited volunteering

participants representing the universities participating

in this study and working as lecturers in English

language teacher education programs. The workshop

was conducted from 20 to 24 December 2016 with

seventeen participants attended the workshop.

The general goal of the workshop was to invite

the participants to (a) approach graduate outcomes

theoretically, rather than heuristically as this has been

the case in Indonesia (Susilo, 2015) and in Australia

(Lian, 2012); (b) identify what academic staff views

as “best practice” models for integrating the IQF; (c)

inquire about the support that lecturers design for

their students; and (d) identify how the concern with

research and inquiry skills support impacts on their

research and teaching.

3.3 Questionnaire 2: Workshop

Evaluation Questionnaire

To evaluate the workshop program, participants were

asked to reflect on the program they have joined. This

reflection functions to see whether or not the learning

has taken place. In evaluating a workshop, the main

concern is the relevance of the workshop (curriculum,

materials) with the jobs to be done by the participants

Jolles (2005, p. 261, p. 270).

The questions in the workshop evaluation

questionnaire were developed by looking at the

relevant aspects of the participants learning

experience, including their opinions about the

workshop materials and their free comments about

the workshop. The questions for the questionnaire

are:

ANCOSH 2018 - Annual Conference on Social Sciences and Humanities

258

• Was the workshop a worthwhile experience?

• In what ways do you think the workshop

relevant or irrelevant to you?

• What would you like to suggest the workshop

could have done better?

• Please comment on the activities and materials

in the first day of the workshop about

"questionnaire findings: mapping your voice"

• Please comment on the activities and materials

in the second day of the workshop about

"literature review"

• Please comment on the activities and materials

in the third day of the workshop about "planning

strategies"

• Any other comments?

3.4 Questionnaire 3: Challenges of the

Integration of Research and

Inquiry in English Language

Education Teacher Programs in

Indonesia

During the workshop, the participants were asked to

design their strategic planning of the integration of R

and I to be applied after they return to their respective

universities. It was unfortunate that they could not

make it because they still had no clear direction on

what units they were going to teach in the next

coming semester. However, they did this strategic

planning design after the workshop. The strategic

planning in this study is the participants’ unit

syllabus.

A questionnaire was used to reframe their

thinking and refresh their ideas about the integration

of R and I. The questions were developed based on

the important elements of syllabuses. The questions

raised in the questionnaire are:

• How do you identify your unit outcomes (has to

be 6 outcomes)?

• How do you integrate unit outcomes into the

unit description?

• How do you link the structure and the key ideas

of the syllabus with the unit outcomes?

• What is the general logic that informs your

choice of unit materials in your unit?

• How do you integrate research and inquiry

(higher order skills in the IQF and graduate

competencies) into your teaching plan?

• How do you develop assessment tasks?

• How is your assessment addressing the key

outcomes?

• Other challenges (Please identify if any).

3.5 Questionnaire 4: Evaluation on the

Integration of Research and

Inquiry in English Language

Education Teacher Programs in

Indonesia

To review what the participants have done so far, in

the end of the project, by the end of July 2017, a

questionnaire on evaluation on the integration of

research and inquiry in English language education

teacher programs in Indonesia were sent to the

participants.

Research question 3: What are the optimal

conditions required to support research and inquiry in

undergraduate English language teacher education

degree structures?

The participants of the study were requested to

provide feedback on their experiences in

implementing new teaching support, which was

agreed upon during the workshop, in their teaching

programs. To obtain the data, a questionnaire was

sent out with questions which will (a) include the

criteria identified during the workshop as appropriate

for evaluating the impact of this support; and (b)

identify questions for future research; and (c) request

feedback for evaluating the relevance of these new

findings in the light of the findings of the initial

questionnaire. The aim of this stage is to generate

reflection upon the relationship between what seems

feasible and the status quo reflected in the findings of

the initial questionnaire.

Research question 4: In what ways did the

“community-building” approach of the study assist

(or prevent) the academic staff in identifying and

implementing best practice models for working with

graduate skills to support research and inquiry in

undergraduate programs?

Together with the questionnaire relating to

Question 3, the researcher also explored the

participants’ feelings about the relevance of their

participation in the study as individuals and as

members of their university community. The

questions will inquire about the research capacity (as

per graduate capabilities identified in the IQF and

Decree 44) that the study generated in the research

participants and their colleagues.

1) Please, identify your academic level!

• Level E (Guru Besar/Gol.IV.E)

• Level D (Guru Besar/Gol.IV.D)

• Level C (Lektor Kepala)

• Level B (Lektor)

• Level A (Asisten Ahli)

Growing Together - A Model of Critical Research Design

259

2) How do you identify your unit outcomes (has to

be 6 outcomes)?

3) How do you integrate unit outcomes into the

unit description?

4) How do you link the structure and the key ideas

of the syllabus with the unit outcomes?

5) What is the general logic that informs your

choice of unit materials in your unit? Identify

the number and specify if they all were from

your university.

6) How do you integrate research and inquiry

(higher order skills in the IQF and graduate

competencies) into your teaching plan?

7) How do you develop assessment tasks?

8) How is your assessment addressing the key

outcomes?

9) Other challenges (Please identify if any).

3.6 Interview

The interview questions were developed from the

result of discussion. The interview questions come

from the workshop participants. The purpose of this

is to provide them space for thinking and allow them

to develop as a part of their learning and

collaboration. During the workshop, the participants

were asked to suggest what aspects should be

included in the evaluation of the study. After they

provide some aspects, these aspects were discussed

with three active participants of the study.

The participants were asked to brainstorm and

lists the issues that need addressing in the further data

collection. After that, the researcher help them to

make the questions, and the questions were revised by

them. From the discussion, the questions emerged are

as follow:

1) How do the staff respond to the integration

of research and inquiry in ELTE programs?

2) How do the students respond to the

integration of research and inquiry in ELTE

programs?

3) How did you work with the syllabus? Do

you see the relevance of syllabi and the

application of the syllabi plans?

4) Do you see the importance of synchronizing

learning materials across units in the

curriculum? Please explain.

5) Would you describe challenges in the

integration of research and inquiry in ELTE

programs’ units?

6) Would you describe the inquiry processes

happened in the field (classes)?

7) What kinds of support or resources do you

need to support the integration of research

and inquiry in ELTE programs’ units?

8) How do you design the assessment? How do

you relate it with the course description?

9) Is there anything else you would like to say

regarding this project?

4 DATA ANALYSIS

In this step, the findings are not displayed as they are

based on questions given, rather, the findings are

displayed by categorisation according to two

frameworks: a) questionnaires were analysed using

HBS’ framework in “integrating around the job to be

done” (2010) and b) interviews were analysed using

Lian and Pertiwi’s framework about “theorizing for

innovation in English Language Teacher Education”

(2017). The two frameworks are considered to be the

most relevant following the nature of data collection

purposes.

5 MAKING CONCLUSION

The conclusions were drawn from the understanding

of all data interpretation in relation to framework

developed which are based on IQF and graduate

competencies. Question 5 of this study was addressed

in this section.

Research question 5: What strategies can be

applied to change the culture and practices of research

pedagogy in undergraduate programs and to

contribute to the building of a 21

century model of

English teacher education programs relevant to the

Indonesian context?

In order to respond to this question, the study

summarises its findings by evaluating their

significance in relation to the vision for research

universities described in Indonesian and ASEAN

policy documents.

6 CONCLUSIONS

This paper has illustrated a research procedure where

the participants are the center of the problem solving

process, giving them opportunity to grapple with

important issues by considering their own contexts.

While most research in education tend to position

researchers as experts (identifying problems, doing

literature review, and judge what kinds of treatments

ANCOSH 2018 - Annual Conference on Social Sciences and Humanities

260

will work), this current research design allow

research participants to learn, reflect their learnings in

relation to their own experience and learn from the

whole process of research and they become

knowledgeable in coping with issues in front of

themselves. After all, they may have their own

research agenda as a follow up after their reflection

after taking part in handling issues in this current

research.

By involving the participants in the research, the

researcher give them opportunities to learn, think

critically, develop their capacity, and build their

networks. As such, using critical research design,

rather than putting participants and people involved

in the study as “objects” of research, they have a

chance to be “subject” of the research, where they

have their right to take actions and have an equal

chance to grow together.

REFERENCES

ASEAN Secretariat. 2009. ASEAN socio-cultural

community blueprint, ASEAN Secretariat. Jakarta.

Retrieved May 3, 2016 from:

http://www.asean.org/?static_post=asean-socio-

cultural-community-blueprint-2025.

Barrie, S., Hughes, C., Crisp, G., 2014. Assessing and

assuring Australian graduate learning outcomes:

principles and practices within and across disciplines.

Final Report OLT Australia.

Boyer Commission., 1998. Reinventing undergraduate

education: A blueprint for America’s research

universities, The Boyer Commission on Educating

Undergraduates in the Research University. Stony

Brook, US.

Bradley, D., Noonan, P., Nugent, H., Scales, B., 2008.

Review of Australian higher education: Final report,

Commonwealth of Australia. Canberra.

Calhoun, C., 1995. Critical social theory, Blackwell.

Oxford.

Gelo, O.C.G., 2012. On research methods and their

philosophical assumptions: Raising the consciousness

of researchers, again. Psychotherapie and

Sozialwissenschaft (2), (109-128).

HBS (Harvard Business School), 2010. Integrating around

the job to be done. Retrieved May 2, 2016 from:

http://www.public.navy.mil/fltfor/nwdc/CRIC%20Arti

cles/Integrating_Around_the_Job_to_Be_Done[1].pdf

Indonesian Government., 2013. Indonesian Qualifications

Framework. Jakarta

Jolles, R.L., 2005. How to run seminars and workshops:

Presentation skills for consultants, trainers, and

teachers, John Wiley and Sons, Inc. Hoboken, New

Jersey, 3

edition.

Latour, B., 2003. A Dialog on Actor Network Theory with

a (Somewhat) Socratic Professor. In C. Avgerou, C.

Ciborra, F. F. Land (Eds.), The social study of

information and communication study (pp. 62-76),

Oxford University Press. Oxford.

Latour, B., 2004. How to talk about the body: The

normative dimension of science studies, Body and

Society, 10(2-2), 205-229.

Latour, B., 2010. An Attempt at a “Compositionist

Manifesto”, New Literary History, 41, 471-490.

Lian, A.B., 2006. A dialogic model of inquiry in second

language teaching: Toward the concept of a critical

approach to pedagogic research. (PhD), The

University of Queensland. Queensland.

Lian, A.B., 2012. A dialogic framework for embedding

graduate attributes in discipline-based degree curricula,

Rangsit Journal of Arts and Sciences, 2(1), 1-14.

Lian, A., Pertiwi, W.H.S., 2017. Theorising for Innovation:

Implications for English Language Teacher Education,

GEMA Online® Journal of Language Studies, 17(3), 1-

17. http://doi.org/10.17576/gema-2017-1703-01.

MRTHE (The Ministry of Research, Technology, and

Higher Education), 2015. The Ministry of Research,

Technology, and Higher Education Decree No.44 Year

2015 on National Higher Education Standards.

Susilo, M-P., 2015. Curriculum of EFL teacher education

and Indonesian Qualification Framework: A blip of the

future direction, Dinamika Ilmu, 15(1), 11-24.

Tomal, D.R., 2010. Action Research for Educators,

Plymouth, UK. Rowman and Littlefield Education, 2

edition.

Tracy, S.J., 2013. Qualitative research methods: Collecting

evidence, crafting analysis, communicating impact,

Wiley-Blackwell. West Sussex.

Growing Together - A Model of Critical Research Design

261