Applying Knowledge Codification in a Post-mortem Process

A Practical Experience

Erivan Souza da Silva Filho, Davi Viana and Tayana Conte

USES Research Group, Instituto de Computação, Universidade Federal do Amazonas, Manaus, Brazil

Keywords: Knowledge Management, Post-mortem Analysis, KJ Method, Ishikawa's Diagram, Coding Knowledge.

Abstract: In information systems, acquiring experiences in projects can result in new knowledge to people or the

organization. Knowledge Management analyzes such experiences as a significant resource to the

organization. Through the Post-mortem analysis, people can remember experiences and situations that they

had during a software development project. In order to support such analysis, the PABC-pattern structure

proposes codifying knowledge, assisting practitioners in registering key elements to facilitate the

understanding of that experience. This paper proposes a process of Post-mortem Analysis based on the KJ

method. We have integrated the PABC-Pattern approach as a final product in order to record the

experiences and gathered information.

1 INTRODUCTION

In software projects, the team members acquire new

knowledge and experiences that can benefit future

projects and their professional skills (Birk et al.,

2002). Many businesses are human and knowledge

intensive, such as software organization. Knowledge

intensive organizations have been noticing that a

large amount of problems is attributed to un-

captured and un-shared product and process

knowledge (Lindvall et al., 2003).

Knowledge Management is the process of

creating, validating, representing, distributing and

applying knowledge (Bhatt, 2001). Knowledge

Management also refers to identifying and

increasing the collective knowledge of an

organization to help it keeping a competitive

advantage (Alavi and Leidner, 2001).

Another common factor to Knowledge

Management is learning with the past successes and

failures. The goal of this learning is to improve the

future of the software development (Dingsøyr,

2005). Identifying these experiences in projects and

codifying them can help to prevent mistakes in

future projects. By avoiding such mistakes, we can

reduce re-work while repeating well-succeeded

processes in order to increase productivity and the

probability of achieving success (Rus and Lindvall,

2002).

Retrospective analysis (known as Post-mortem)

is one approach to support remembering knowledge

from projects. According to Scott and Stalhane

(2003), during a Post-mortem analysis the

participants of an ongoing or finished project are

reunited and they are asked to identify: (a) which

aspects of the project went well and must be

repeated, and (b) which aspects of the project went

bad and must be avoided.

Once the Post-mortem captures the knowledge, it

is necessary to codify it. There are techniques

regarding capturing knowledge (Bjørnson and

Dingsøyr, 2008). A solution proposed to capture and

code knowledge is employing the PABC-Pattern

(Rabelo et al., 2014). PABC-Pattern is an approach

to codify the lessons learned in a structured form.

The elements from such form aim to facilitate the

knowledge storage in a software organization.

Adopting a Post-mortem approach combining

with a coding method can lead to improvements in a

project. This paper describes an experience of using

retrospective Post-mortem analysis at the end of

each of the iterations of a project, and applying the

PABC-Pattern coding structure. By describing such

Post-mortem analysis process, we intend to provide

practitioners with an example they can replicate in

other software projects.

The rest of this paper is organized as follows.

Section 2 shows the background of this work. In

Section 3, we present the definition of the Post-

mortem process. Next, Section 4 describes the

153

Souza da Silva Filho E., Viana D. and Conte T..

Applying Knowledge Codiﬁcation in a Post-mortem Process - A Practical Experience.

DOI: 10.5220/0005376301530165

In Proceedings of the 17th International Conference on Enterprise Information Systems (ICEIS-2015), pages 153-165

ISBN: 978-989-758-097-0

 2015 SCITEPRESS (Science and Technology Publications, Lda.)

application process in a case study while Section 5

discusses the case study results. Finally, Section 6

shows our conclusions and future work.

2 BACKGROUND

Individual knowledge is necessary for the

development of the organizational knowledge

(Bhatt, 2001). Knowledge within an organization is

a collection of knowledge, experiences and

information which people or groups use to carry out

their tasks (Vasconcelos et al., 2005). In this section,

we present the concepts that we applied as

theoretical basis for this work.

2.1 Knowledge Management

The human resources is the main asset of many

companies where knowledge has to be preserved and

passed from the individual to the organizational

level, allowing continuous learning and

improvements (Lindvall et al., 2003). Companies

generally understand the term “knowledge” as

codified information with a high proportion of

aggregated human value including perception,

interpretation, context, experience and wisdom

(Davenport and Völpel, 2001).

Davenport and Prusak (1998) placed the

knowledge in three distinct items: data, information

and knowledge. Data is a group of distinct facts and

goals related to events. Information aims at changing

the way in which the receiver sees something,

exercising some impact on his/her judgment and

behavior. Knowledge is the fluid mixture of

condensed experience, values, contextual

information and experienced insight which gives us

a structure to the evaluation and incorporation of

new experiences and information (Davenport and

Prusak, 1998).

Nonaka and Takeuchi (1995) characterized

knowledge into two types: explicit and tacit. Explicit

or codified knowledge can be articulated in formal

or textual language. Tacit knowledge is the personal

knowledge, incorporated to the individual

experience and which involves intangible factors

(e.g. personal beliefs, perspectives and value

systems).

Knowledge Management is a method that

simplifies the process of sharing, distributing,

creating and comprehending a company's knowledge

(Bjørnson and Dingsøyr, 2008). Its goal is to solve

problems regarding the identification, localization

and usage of the knowledge (Rus and Lindvall,

2002). Furthermore, Knowledge Management is

concerned with aspects regarding how to collect

and/or make explicit the experiences of the projects

to be used by others (Dingsøyr et al., 2001).

In the case of finished projects, we can

remember the experiences that the participants had

during the project. Therefore, such knowledge can

be reutilized. One approach to guarantee the

collection of such experiences is through the Post-

mortem Analysis.

2.2 Post-mortem Analysis

Post-mortem analysis is the most common name

given to retrospective analysis (Myllyaho et al.,

2004). Post-mortem is the activity of gathering

knowledge which can be organized for a project that

is in its final stage or finished (Dingsøyr, 2005). The

goal of a Post-mortem must be to learn not to

evaluate. In this sense, an evaluation can make

people restrain their experiences because they might

think that sharing such experiences could be

embarrassing (Desouza et al., 2005).

There are many ways to apply a Post-mortem

Analysis. Dingsøyr (2005) shows two different

proposals of how to conduct a Post-mortem in small

or medium companies:

1. Post project review process by Neal Whitten: (1)

Declare intent – send a message to all

participants indicating that a Project

retrospective meeting will be made; (2) Select

participants – the key project participants must

be selected; (3) Prepare for workshop – the

participants must prepare a presentation

answering a series of question such as: “What

was the productivity level achieved by your

task?”; (4) Conduct workshop – the workshop

must take from one to two days, the participants

present their answers and a list of possible

improvements regarding the project; (5) Present

results – the results of the post-mortem are

presented to the project leaders and then, to all

project members; (6) Adopt recommendations

– A post-mortem report is created, containing all

the information of the workshop and

recommendations to the project leaders.

2. Retrospective Meeting by Collison and Parcell:

(1) Call to the meeting – right after the Project

ends, the participants are summoned; (2) Invite

the right people – the key people are selected by

priority; (3) Appoint a facilitator – a person

who is not close to the project is chosen to

conduct the meeting; (4) Revisit the objectives

and deliverables of the project – find the

ICEIS2015-17thInternationalConferenceonEnterpriseInformationSystems

154

original success criteria and ask if they were

achieved; (5) Revisit the project plan or

process – this step must be useful in the

construction of a flow where the activities,

deliveries and decisive points will be showed; (6)

Ask “What went well” – an answer list must be

generated and complemented with a “why?”; (7)

Find out why these aspects went well, and

express the learning as advice for the future –

identify the success factors and

recommendations; (8) Ask “What could have

gone better?” – it must start with the project

leader and then continue with the rest of the

people in the meeting; (9) Find out what the

difficulties were – identify problems and

barriers that must be avoided; (10) Ensure that

the participants leave the meeting with their

feelings acknowledged – ask each person

regarding his/her impressions on the meeting;

(11) “What next” – if someone starts a new

project, a session is recommended showing the

retrospectives; (12) Recording the meeting – a

structure must contain all the main aspects,

artifacts and documents that characterize that

project.

Birk et al., (2002) defined the Post-mortem analysis

in three phases:

1. Preparation: during the preparation phase, a

whole project retrospective will be made to

provide a better understanding. All project

documents like reports and project plan will be

reviewed. Also a goal for the Post-mortem will

be defined, for example, “Identify the bigger

achievements and improvements on the project”.

2. Data Collection: all the relevant experiences of

the Project are collected. The project participants

and other stakeholders will make a group

session. Some techniques can be applied in that

phase such as: (a) Semi-structured Interviews –

the facilitator will make a list of questions to the

participants; (b) Facilitated group discussion –

the facilitator will conduct the discussion while

(s)he documents the main results on a

whiteboard; (c) KJ Method – the participants

will write 4 positive and negative experiences

regarding the project in notes. This method will

be furthered detailed in the next section.

3. Analysis: The facilitator conducts a feedback

session with the participants aiming to identify if

(s)he understood what they said and if (s)he

discussed all the relevant facts. When the data is

sufficient, an Ishikawa’s diagram is created to

find the causes to the positive and negative

experiences. The Ishikawa’s diagram will be

further detailed in Section 2.4.

The Post-mortem reports may vary in size, variety,

scope and deepness (Desouza et al., 2005).

According to Birk et al., (2002), the facilitator writes

the Post-mortem analysis results in an experience

report which must contain:

• A description of the project, including the

developed product, development methods, time

and necessary effort.

• The project’s main problems, with diagrams

representing the problems (e.g. an Ishikawa’s

diagram).

• The project’s success factors, also with diagrams

representing them.

Transcribing what was said in the meetings helps to

contextualize the information for future readings

(Dingsøyr, 2005). The person responsible for writing

the Post-mortem report can be someone of the

group, or the own group reunited, or even an

external facilitator, depending on project’s nature.

(Desouza et al., 2005).

2.3 KJ’s Method

KJ’s Method, also known as the Affinity Diagram, is

a discussion tool created by Jiro Kawakita (Kim and

Kim, 2014). The KJ’s Method usually employs

individual descriptions on notes (e.g. stick notes) or

cards. Then, with the help of the other participants,

the notes are grouped by similar points (Widjaja and

Sawamura, 2014).

The KJ Method’s application includes four steps

(Scupin, 1997): (1) Card creation, (2) Card

grouping, (3) Card unification by similar patterns,

and (4) Written or oral explanation. In the literature,

the way in which these four steps are applied may

vary according to the process by each author, as

shown in the following approaches:

Kokogawa et al., (2012) describe the standard

process to perform KJ’s Method as follows: (1)

Prepare the cards with ideas referring to a specific

theme; (2) Form a group of cards with similar ideas;

(3) Identify each group with a name; (4) Position

each group into a diagram, showing the relationship

between them; (5) Describe sentences to express

what the diagram means.

Kim and Kim (2014) describe the steps in a

similar way to Kokogawa et al., (2012), adding some

new steps: (1) A topic is selected for discussion; (2)

Participants generate ideas through brainstorming;

(3) A card is created for each of the participant’s

ideas; (4) After all ideas have been discussed, the

cards that are similar are grouped (in case an idea

ApplyingKnowledgeCodificationinaPost-mortemProcess-APracticalExperience

155

does not fit any of the previously created groups, it

enters a new group separately); (5) Each group of

cards is given a representative name; (6) The groups

are classified as “main”, “superior”, “medium” and

“minors”, and the similar groups are combined; (7)

The classified cards are then reorganized in a board

or a big sheet of paper.

The advantage of the KJ’s Method is that it

allows the discovery of problems even in chaotic

situations. Such method can help generating new

ideas and identifying the essence of the problems

correctly, breaking any stereotypes (Kim and Kim,

2014).

2.4 Ishikawa’s Diagram

Ishikawa’s Diagram, also known as the fishbone

diagram (due to its shape), is a diagram that aims at

showing the causes and effects of a problem

(Stålhane et al., 2003). In the diagram, a line is

drawn indicating a problem to be discussed. Then,

other diagonal lines are drawn pointing to the main

line (Dingsøyr et al., 2001). The attached lines (in

fishbone form) represent the causes that leaded to

the problem or experience (Birk et al., 2002).

2.5 PABC-Pattern

The literature presents different approaches such as

formal routines, causal map, concept map, models

with case-based reasoning and other proprietary

tools (Rabelo et al., 2014). Additionally, we decided

to apply PABC-Pattern because this technique is not

associated with a tool or specific software process.

Thus, it is possible to integrate it with other

approaches or processes that work with knowledge,

such as a Post-Mortem Analysis.

PABC-Pattern is an approach to codify the

lessons learned in a structured way. Its elements aim

to facilitate knowledge storage in a software

organization (Rabelo et al., 2014). The knowledge

codification process can enable registering the

lessons learned for further consultation in order to

achieve personal learning or to apply it in new

projects. The elements that compose the structure of

the PABC-Pattern are:

• Title: the lesson’s name and a brief description;

• Problem: a detailed description of a problem or

a question that the lesson learned tries to solve;

• Cause of the Problem: details the cause of the

problem, this description must contain what

caused the problem;

• Consequence(s) of the Problem: description of

the consequences of the problem. In other words,

what happened after the problem occurred;

• Action: provides details of the solution to the

problem, (i.e. highlights of an activity that was

applied to solve the problem);

• Benefit: description containing the effects

(positive and/or negative) caused by the action;

• Keyword: it describes the keywords that identify

the lesson;

• Relation with Other Lessons: lists the

identification code of other lessons that are

related to the described lesson;

• Context: characterizes the environment in which

the action was executed. The context can be

described in terms of:

- Project Type: Selection of the Project type

(development project, maintenance project,

or both);

- Project Size: Selection of the project’s size

(small project, medium project, or big);

- Project Phase: Indication of the Project

phase (requirements elicitation, analysis,

coding, test, implantation, management

activities, supporting activities, others);

• Role of the Lesson’s Creator: description of the

role of the person who created the lesson;

• Related Domain: description of the domain

where the lesson can be applied;

• Other Relevant Information: description of

other information that the lesson’s creator judges

necessary.

We chose this approach based on the positive

empirical results collected regarding the PABC-

Pattern (Rabelo et al., 2014). According to the

subjects of the empirical studies, the reasons for

these positive results of the PABC-Pattern are:

• A more detailed description of the knowledge

scenario, becoming more enlightening;

• The possibility to acquire more information

applying less effort;

• The ability to describe the problem and the

solution in the same record.

The experience collected in a post-mortem needs to

be storage in order to help other projects in the

future. Therefore, a knowledge codification

technique can be necessary.

3 THE POST-MORTEM PROCESS

The Post-mortem is an important activity in a

project’s retrospective. Executing a process means

ICEIS2015-17thInternationalConferenceonEnterpriseInformationSystems

156

following the steps to achieve a goal, in this case,

remember and register experiences. The process of

Post-mortem applied in this work is based on the

works by Birk et al., (2002) and Dingsøyr (2005).

The process will be composed of 3 stages:

(1) Data Collection – where data from the

participants will be collected;

(2) Data Analysis – once the information is

collected, it will be analyzed, interpreted and

codified;

(3) Results – creation of an experience report, final

corrections and feedback from the members of

the project.

3.1 Data Collection

As suggested by Dingsøyr (2005), a facilitator is

placed to conduct the four activities of the Post-

mortem data collection. According to Birk et al.,

(2002), (s)he can be someone from outside the

project whom the participants trust. (S)he will be

responsible to let the session flow and intervene

whenever a problem arises.

The facilitator’s role is to guide the project’s

participants so that the session flows clearly. The

facilitator will be answering any doubts that occur

and organizing people in the environment to

promote the participants’ interaction in the best way

possible. Also, (s)he may interfere if necessary (i.e.

when (s)he finds problems that could interrupt the

meeting).

Figure 1 shows the first activity of the data

collection stage, “1 – Preparation of the

Participants”. This activity starts the Post-mortem

process. In this activity, we explain to the

participants, through a presentation, how the Post-

mortem process will be made, why it is important

and what they should do.

Figure 1: Data Collection Process.

The “2 – KJ Session” activity uses the KJ

Method, one of the techniques that we can apply to

make the information gathering in a Post-mortem

process (Birk et al., 2002). It is complemented in

parallel with the “3 – Recording” and “4 –

Annotations” activities. The KJ session will be

detailed in Subsection 3.3.1.

The “3 - Recording” activity registers all KJ

session. The recording depends on the environment,

as it is preferable that the environment is calm. Each

participant must be requested to keep a voice level

that can be recorded by the recording tool.

The “4 - Annotations” activity aims at

highlighting ideas, conclusions or facts that can be

relevant to be further recorded in the Data Analysis

stage. This activity is optional; the facilitator can

choose to keep notes (or not) of anything during the

KJ session or only focus in other activities.

3.1.1 KJ Session

The session using the KJ method is based on the

steps presented in Kim and Kim (2014). Figure 2

shows the steps of the KJ method that are adopted in

our work:

Figure 2: KJ Session Steps.

Step I – “What is the question to be

answered?” - The participants are guided to answer

a question. In our work, the question was “What

were the positive and negative experiences learned

in this project?”. Thus, the participants were guided

to answer this question. By guiding the participants,

we intend to gather specific data to solve the

problem.

Step II – “The group is organized” - In projects

with many participants it may not be able to have all

participants in a session. Therefore, the participants

must be chosen in a way that diverse points of view

of the project are collected. In addition, time must be

planned accordingly to the organization’s context. In

our work, all participants were invited and this

guaranteed that all points of view of the project were

visited. Also, we planned the session for one hour.

Step III – “The experiences are collected in

notes” – In this step, we distributed a predefined

number of notes to all participants so they could

write their experiences. As suggested by Birk et al.

(2002) we initially provided four notes to be filled.

Then, we informed the participants that they had 20

ApplyingKnowledgeCodificationinaPost-mortemProcess-APracticalExperience

157

minutes to fill the notes in order to answer the

question from step I. However, as some participants

requested more notes, we allowed them to feel free

in writing further notes answering the question.

Step IV – “The participants discuss the notes”

One by one, the participants should choose a note

and discuss it with the team. The team starts a debate

of ideas regarding such note. The process is repeated

until all the notes or the estimated time for the

meeting ends.

Step V – “The notes are grouped by similar

experiences”, the facilitator identifies the described

experiences on the notes that might be similar and

should ask the group if they agree with this

interpretation. In case of disagreement, the notes are

grouped in a different way. This step must be made

until every note is discussed.

Step VI – “We put the notes on a board” -

Discussed notes are pasted on a board so all the

participants can see them. Notes that describe

similar experiences notes are put close to each other.

The participants are free to disagree on any decision

at any moment.

Step VII – “The notes are grouped under a

theme” - Here we will decide if the experience

happened in a common context and after, they are

grouped under a common theme.

Step VIII – “The notes groups are named” The

groups are named with words or texts that identify

and generalize such group of experiences.

Step XI – “Prioritizing the notes groups” At

the end, the participants must prioritize the

experiences. Making the participants visualize the

negative experiences guides them to tackle the main

problems of the project. Positive experiences that

must be replicated are exhibited aiming to show the

evolution and the success of the team.

3.2 Data Analysis

Data analysis is the process of gathering all the

acquired information and codifying it to a model,

form or document. In our work, we chose the

PABC-Pattern approach. We employ the term

“codifier” to refer to the person who is going to

analyze the recorded data, notes, annotations and to

write the codified experience. Figure 3 presents the

data analysis process.

The “4 – Mapping notes, recordings and

notifications” activity results in a transcription of all

notes gathered in the KJ’s session. Each note, the

transcription of what the participant wrote, and the

starting time of the recording in which it was

discussed must be stored. After the codifier analyses

the information and then, executes activity 7, (s)he

must map the identification of the codified document

with the note.

Figure 3: Data Analysis Process.

The “5 – Listening to the recordings” activity

will be the process of listening to the KJ’s session

recordings. It will serve also as a complement to the

first codifications in order to generate new ones.

During the discussions, the participants might report

some new tacit experiences that were not written in

the notes.

The “6 – Analyzing the annotations” activity

will bring support to the codifications in a sense of

remembering a related fact that the facilitator

pointed out during the session or some relevant

information that was not in the notes from the

participants. This activity is optional, because the

facilitator might choose not to write annotations

during the Post-mortem session.

The “7 – Codifying Knowledge applying

PABC-Pattern” activity is in constant execution

during the stage of data analysis. The codifier, based

on activities 4, 5 and 6, has the role of describing the

experiences in the PABC-Pattern in an impartial way

and relating them to all the points discussed in the

post-mortem session.

The optional “8 – Generating an Ishikawa’s

diagram” activity is an option for the codifier when

(s)he does not understand an experience. This

misunderstanding might be caused due to difficulties

in understanding or when the participants have many

different points of view regarding an experience.

The diagram will map all the opinions, reports or

solutions, in order to find what problem is being

approached. The codifier will have at his disposal a

general view of the causes for the experiences and

then, complete the codification.

3.3 Results

After finishing all the codifications to the PABC-

Pattern, a report with the Post-mortem results will be

ICEIS2015-17thInternationalConferenceonEnterpriseInformationSystems

158

generated (see Figure 4). We chose a traditional way

to generate the Post-mortem report, based on the

advantages and disadvantages presented by Desouza

et al., (2005). The main advantages in a traditional

report are that it is highly structured, easy to

comprehend and low codification cost. On the other

hand, conclusions are less easy to remember.

Figure 4: Detailed process of results and experiences.

The “9 – Generating report” activity resulted in

the Post-mortem report of the project. In our work,

to compose the report’s structure, we have chosen

the following topics from the literature (based on

Dingsøyr et al., 2001; Birk et al., 2002; and

Dingsøyr, 2005): (1) a general description of the

project, (2) the time of the project execution, (3) the

developed product, .(4) the main problems, (5) the

improvement opportunities., (6) the key artifacts, (7)

the people involved, and, (8) a list of positive and

negative points, in our case, the codified

experiences.

The “10 – Participant’s feedback” activity is a

last review made by members of the project. They

can add or change something that has not been

discussed in the Post-mortem meeting. This activity

is optional, because it is necessary to allocate extra

time with all the project participants. However, the

results will pass through a detailed review. After the

end of the results stage, the final report is delivered

to the organization.

4 CASE STUDY: APPLICATION

OF THE POST-MORTEM

PROCESS IN A SMALL-SIZED

PROJECT

The case used in this research is regarding a

software development project on a system focusing

in supporting the daily care of the elderly. Two

distributed teams were involved in the system

development. One team was located in the north of

Brazil and it was responsible for the user interface

design. The other team was located in the south of

the same country and it was responsible for the

software implementation (coding). The information

exchange between both teams was done through the

project managers and leaders. Thus, meetings were

made possible by online conferences and documents,

information or questions exchange were made by e-

mail. Both teams responded to the representatives of

a transnational company.

The Post-mortem was applied on the team

responsible for modeling the user interface. The

team was composed of six members and the project

duration was six months. Each month, the team

delivered the results from a set of activities defined

by the project schedule. The term “Sprint” was

applied by the project team to denote the activities to

be performed in a month, including its results.

When a Sprint ended, a Post-mortem session was

scheduled and attended by all project participants.

4.1 Planning the Post-mortem Session

The first post-mortem session was held in a meeting

room with a white board on the wall. The

participants received a presentation from the

facilitator and tools like the whiteboard, a notepad

and a recorder were used. The time limit to the first

post-mortem session was free, because the process

hasn’t been defined yet.

Based on the average time of the other teams

meetings, it had been stipulated a time length of 60

minutes on the second and third Post-mortem

session where 20 minutes were reserved to the

filling of the notes and 40 minutes to the debate and

discussion of the notes. It also has the assistance of a

projector to the presentation which served as a visual

guide visual to the participants.

4.2 Post-mortem Session in Sprint 1

The first Post-mortem session was made with

empirical purposes; the process had not been defined

yet. The participants gathered in a meeting room

where the whole session was recorded, the session

took around 26 minutes to be made. In the

experiences capture process, some aspects related to

applying a Post-mortem were employed, for

example, the intervention of a facilitator and the

recordings where:

1. A list of pre-defined topics was written on a

whiteboard. Those topics had a group of

activities made in the work as basis;

2. For each topic, each participant was asked to

answer what the experiences (s)he had regarding

such topic;

3. The participant explained to everyone his/her

experience, and everyone was free to

complement or debate the presented ideas;

ApplyingKnowledgeCodificationinaPost-mortemProcess-APracticalExperience

159

4. The facilitator took annotations in a notepad

about the experiences that the participants

described.

There were difficulties in executing the Post-mortem

session. The participants needed a method that

assisted them in remembering their experiences.

Allowing the participants to talk freely was not

resulting in a dynamic enough session to capture the

experiences. After the session it was evident that a

more structured Post-mortem session needed to be

adopted.

4.3 Post-mortem Session in Sprint 2

In the second Post-mortem session that had been

made for Sprint 2, we used the first version of the

Post-mortem process (Birk et al., 2002) described in

this work. The use of the KJ’s method to guide the

session was chosen for being the most disseminated

method in the results from our literature review

regarding this work. The session took around 1 hour.

The steps of the process were described to the

participants in a presentation and all participants

were asked to stand up and explain his experience.

The gathered experiences after the session

demonstrated that the use of notes between the

participants stimulated them to discuss more than in

the previous session. On the other hand, during the

data analysis stage, codifying the experiences to the

PABC-Pattern became problematic since the data in

the notes was not enough to fill some fields of the

PABC-Pattern form. For example: when a

participant described a problem, (s)he (and the rest

of the team) did not discuss the action(s) that could

solve that problem.

Another problem with the second session was the

wrong use of the whiteboard. The notes were

exhibited on a table and notes for the generated

groups were not created, as seen in Figure 5. That

led to the non-execution of the “Prioritize groups”

step. The missed step generated an absence of

information in the report where improvements

opportunities in future sprints and projects should be

described.

Figure 5: Experience notes disposed on a table in the

Sprint 2.

4.4 Post-mortem Section in Sprints 3

and 4

The third Post-mortem session was employed for

Sprints 3 and 4 simultaneously. This decision was

made as there was not time to make a session

between Sprint 3 and 4 due to deadlines and the

need for developing the deliveries of the project. In

this session, one of the team members left the team,

which caused a difference in terms of data

collection. That means that the maximum number

possible generated notes was reduced by four (since

the team member that left was not contributing for

the generation of ideas) and there would be one less

point of view during the debate and discussions. The

whiteboard was improved with the use of a

cardboard and tape. Also, notes with different colors

were used to differentiate the groups names, as

showed in Figure 6.

Figure 6: Whiteboard for Sprint 3 and Sprint 4.

During the filling of the notes, a participant

finished filling the four notes and asked if (s)he

could fill one more, besides the already four filled

notes. The facilitator allowed it and gave him/her

another blank note to be filled. In this case, the

participants were still employing the 20 minutes

given for filling the notes. The facilitator must know

when (s)he should or should not allow the filling of

more notes by considering requirements such as

available time for the meeting, the dynamism

between the participants and others.

The facilitator now had a group of pre-

estabilished questions to ask to the participants.

(S)he could use these questions while a participant

was reporting his/her experience if (s)he noticed a

lack of information that could make it difficult to

condify the knowledge using the PABC-Pattern. The

questions were:

1. What were the causes of this experience?

2. What were the consequences of this experience?

3. What would you do to solve it?

4. What should your team do to solve it?

ICEIS2015-17thInternationalConferenceonEnterpriseInformationSystems

160

5. What were the benefits of doing it?

The groups names were created by the facilitator.

(S)he suggested a name and the team decided if that

name was proper. After the end of the Post-mortem

session, a questionaire was given to the participants

aiming at gathering the impressions and opinions

about the post-mortem sessions completion until that

moment. Table 1 shows the questionnaire that was

used.

Table 1: Feedback Questionnaire of the Post-mortem.

Id Question

Regarding the usage of the notes to remember

experiences, do you believe that notes helped you to

remember your experiences? Please comment.

Still regarding the notes, what is your opinion about

the limit of 4 notes per participant? Do you think this

limit helps the Post-Mortem session to be more

dynamic or this limits the number of experiences that

you would like to write?

About the use of brainstorming for each note. Does

explaining the note to the group help to pass the

experience correctly? If you disagree or agree,

please, explain why.

The discussion about a note helps you to complete

your understanding about an experience? With the

discussion, do you feel that you would remember

new facts that would help you to complete that

experience?

Using a Whiteboard with attached notes is important

to you? And does prioritizing experiences help you

understand the main problems of the project?

In your opinion, which aspects were positive and

negative during the Post-mortem session applying

the KJ method?

Would you change any step or activity of the Post-

mortem session applying the KJ Method? Any

suggestions for improvement?

4.5 Analysis from the Session’s Data

The first analysis from the Post-mortem session of

Sprint 1 had as data the collected annotations made

in paper by the facilitator and the recordings. We

tried to verify and identify positive and negative

experiences only on the data collection’s stage. In

the end, a document was created using the PABC-

Pattern structure.

The lesson learned through the data analysis was

in identifying if the experience was positive or

negative in the recordings. The description of the

experience became very dependent of the

interpretation of whom analyzed the data on the

recordings and because of that, it was transferred to

the data collection stage.

Another lesson learned from the data collection

was the use of the Ishikawa’s Diagram when

mapping the experiences with many points of view

from the participants. Since by using the Ishikawa’s

Diagram it was possible to understand different

points of view, we attached it to the Post-mortem

process.

On the second data analysis from the Post-

mortem session in Sprint 2, the analysis process was

changed. We included a step to transcribe the

recordings and codifying what the participants had

said. A second version of the PABC-Pattern

document was generated considering that there were

mistakes on how the structure was crafted. For

example, the document did not correctly highlight

the context structures and the identification field was

omitted.

The lesson learned on the data analysis stage was

related to the transcription step. The transcription

was hard and time demanding with little benefits.

Analyzing and codifying the recordings has high

cost – low effectiveness activity. Also, only listening

to the recordings was enough to retrieve the data.

Therefore, the Post-mortem process was altered

again by excluding this activity.

Figure 7: Sample of transcription notes from Session 3.

The third data analysis from the Post-mortem

session in Sprints 3 and 4 was described in this work

(see Section 4.4). In this third analysis, the notes

were transcribed to a slideshow tool where the notes

were mapped. As shown in Figure 7, the group title

stays above the notes. Each note is identified with a

positive/negative id defined by one of the

participants of the post-mortem session. Next to the

identification, their respective codifications in

PABC-Pattern were made. After that, we added what

was written on the participant’s note followed by the

beginning time of the recording. Notes with similar

ApplyingKnowledgeCodificationinaPost-mortemProcess-APracticalExperience

161

descriptions of experiences were piled one over

another and have the same identification on the

PABC-Pattern.

The lesson learned through the data analysis was

that with this mapping it was possible to identify a

note with its respective codification and it became

more intuitive.

5 RESULTS

In this section, we show the results obtained in the

Post-mortem process defined in this work. We also

present the results regarding the opinion of the

participants.

5.1 Results of the Data Collection:

Post-mortem Sessions

In this subsection, we will show the results obtained

from the questions in Table 1. Table 2 shows a

summary of our key findings. We present the details

of such findings below. For confidentiality reasons,

people’s names and organizations will be omitted on

the data presentation of this research.

Table 2: Findings of the Post-mortem session.

Key Findings

Using notes helps remembering experiences.

The limit of notes did not make it difficult for the participants

to describe their experiences

Describing a note helps to transfer the experience to the group

Discussions help to complement the experience

The prioritization of notes is an important step

Regarding the use of the white board, the participants' opinion

were divergent.

Mentioning other people’s mistakes can be uncomfortable to

the participants

The discussion can be an obstacle to shy people

A weekly e-mail is necessary to support remembering of

experiences.

Did the notes help you remember the

experiences? (Q1): Two participants agreed that the

use of notes help remembering experiences, but

there were improvements that could be applied:

“Yes, because they made me think. But if the

same notes had small sentences guiding me to

remember experiences it could be better”.

Participant 3

Does the notes limit help the session be more

dynamic? (Q2): All participants described that the

limit of notes did not trouble make it difficult to

describe the experiences. As described in Section

3, the participants were allowed to describe more

experiences in case there still was time for this task:

One of the participants (participant 2) who used this

exception claimed that:

“In fact, there was not such limitation, because I

was told that I could fill more or less notes”.

Does the explanation of the note help to correctly

transfer an experience? (Q3): All participants agreed

that describing a note helps to transfer the

experience to the group. Only writing could not be

enough for a participant to describe the situation nor

for the group to understanding it. By describing the

note, the participants can complement with

something that they did not write on the note:

“I think it helps because people explain what

they wrote (because it might not be so clear). It

helps to remember other experiences.”

Participant 1

Does the discussion complement the understanding

about the experience? (Q4): Again, all participants

agreed that the discussion helped them

complement the experience. Discussing a note

made the other participants complement it with their

point of view:

“The advantage of discussing a note is that the

member (participant) has got the team’s general

view about the referred experience and I believe

that this makes him/her remember about related

experiences and think about them”. Participant 4

Regarding the usefulness of the whiteboard and the

prioritization of the notes (Q5): two participants

agreed with using the whiteboard while another one

thought it could be inconvenient. Furthermore, all of

the participants were unanimous about the opinion

that the prioritization of the notes is an important

step, because prioritizing makes the team have a

general view of what the main problems are to be

solved in the next stages of the project:

“Yes, I believe that prioritizing [notes] helps a

lot, because then I define what we have to pay

more attention to during the project’s

execution”. Participant 3

Regarding positive and negative aspects of the

applied post-mortem (Q6) complementing with

some alterations of the KJ session (Q7) we observed

that the discussion can be an obstacle to shy

people. These people have experiences that they

want to express to the group, but talking to people is

an obstacle that can make them feel uncomfortable

when remembering the experiences:

ICEIS2015-17thInternationalConferenceonEnterpriseInformationSystems

162

“I can never remember the four things. I don’t

like going in front of everybody to speak.

However, I do think that the discussion helps

other people”. Participant 1

Another negative issue refers to mentioning other

people’s errors, because it can be uncomfortable

to the participants. Additionally, exposing that a

participant had a negative result can create

omissions:

“Sometimes having to expose someone is

annoying and uncomfortable”. Participant 3

Finally, a suggestion is a weekly e-mail about the

experiences that the participant would have liked to

share, but could not remember from the Post-

mortem experiences:

“I think that a weekly e-mail could be necessary

(…) that can motivate the participants to

describe their experiences”. Participant 4

5.2 Results of the Data Analysis

Through the first data analysis of the Post-mortem

session for Sprint 1 it was possible to obtain 10

experiences that were codified through the PABC-

Pattern approach. In the first analysis, there were

data recordings and annotations, but only listening to

the recordings without any visual guide made the

analysis costly. It was very difficult to understand

where it finished and where the experience report

from another participant started.

The usage of Ishikawa’s Diagram aimed at

identifying what was the problem of an experience

where many participants commented and each one

explained from a different point of view. It allowed

us to obtain a general view of all opinions and

conclusions about the problem. After understanding

the problem, it was possibile to codify to the PABC-

Pattern, easily. Figure 8 presents an example of the

usage of the Ishikawa’s Diagram.

Figure 8: Ishikawa’s Diagram for the Sprint 1

Post-mortem.

The second data analysis from the Post-mortem

session for Sprint 2 resulted in a total of 15 codified

experiences. With the use of notes and recordings, it

was easy to know what the experience was,

including when each participant described his/her

note. Also, it became easier to locate the experience

on the recordings to know about what experience

(s)he referred to.

Two Ishikawa’s Diagrams were needed to

understand two experiences. Even without the notes

aid, there were situations in which the notes were

incorrect to reflect a situation. Also, the point of

view changed through the discussion between the

participants. Aditionally, by mapping the diagram

with all the participants’ opinions, it was possible to

discover and conclude what was the problem that

they were refering to.

The third data analysis from the Post-mortem

session regarding Sprints 3 and 4 resulted in a total

of 12 codified experiences. In the Ishikawa’s

Diagram that was created for this analysis, we did

not have difficulties in understanding the described

experiences.

After mapping the notes in a slideshow

presentation, the process of the analyses from the

data became easier and faster. It was possible to

know which codification on the PABC-Pattern

belonged to each note and when was the time in

which the note was explained in the recordings,

which means that reviewing the experiences was

made easier.

6 CONCLUSIONS AND FUTURE

WORK

A process for a Post-mortem analysis helps us to

more easily carry out the knowledge collection so

we can codify it later. The first conclusion is that

mentioning wrong experiences with other

participants is still an obstacle to be avoided during

the Post-mortem sessions. The fear of causing

personal problems makes them retain those sorts of

experiences in the post-mortem sessions. Even if the

facilitator reinforces that the goal of the post-mortem

session is not to judge the participant’s acts but to

think about the project’s errors, this situation is not

reversed.

Regarding shy participants, we observed that

they do not like to talk to large crowds, even

knowing about the benefits of the debate. This type

of participants needs to have a better treatment by

the facilitator, because they need to be constantly

stimulated to expose their experiences so that they

feel comfortable to speak.

ApplyingKnowledgeCodificationinaPost-mortemProcess-APracticalExperience

163

During the step of creating the names for the

groups of experiences on the whiteboard, the

facilitator wrote a name and asked if the team agreed

with that title. We conclude that the participants

must propose the name of the theme of the groups

on the whiteboard. After that, the participants attach

their experiences in the group that they think is more

relevant.

Finally, an issue that was raised is how to know

that using an Ishikawa’s Diagram before codifying

through the PABC-Pattern is necessary in a data

analysis process. The advantage of using an

Ishikawa’s Diagram was that understanding

problems with multiple points of view became

easier. If we needed to codify the lessons learned

directly into the PABC-Pattern structure, the

understanding process would have taken longer and

mistakes would have been introduced.

As future work, the process proposed in this

work will be evolved to make it more adequate to

capturing experiences. Some of the identified

improvement opportunities are:

• To stimulate shy participants so they expose their

experiences. It is necessary to find ways to help

these participants in remembering and

externalizing their experiences to the group.

Identifying isolated experiences is not an

alternative solution since describing an

experience to the group helps everyone

understand the experience (Section 4).

• Remembering the main points of the project

before the description of the experiences. It may

help the participants write better notes and help

the facilitator to better understand the project’s

context.

• To carry out the results stage of the Post-mortem

process and finish the experiences report of the

project. After finishing, carry out an evaluation

to know if the chosen report structure to this

work is efficient.

ACKNOWLEDGEMENTS

We would like to thank Jacilane Rabelo for her

support on PABC-Pattern and the collaborators who

participated in the case study. Also, we would like to

thank the support granted by CAPES process AEX

10932/14-3 and by FAPEAM through processes:

062.00146/2012; 062.00600/2014; 062.00578/2014;

01135/2011; Edital 009/2012 - RHTI Doutorado.

REFERENCES

Alavi, M., Leidner, D. E., 2001. Review: Knowledge

management and knowledge management systems:

Conceptual foundations and research issues. In MIS

Quarterly Vol. 25 No. 1, pp. 107-136.

Bhatt, G. D., 2001. Knowledge management in

organizations: examining the interaction between

technologies, techniques, and people. In Journal of

knowledge management, v. 5, n. 1, p. 68-75.

Birk, A., Dingsøyr, T., Stålhane, T., 2002. Postmortem:

Never leave a project without it. In IEEE software, v.

19, n. 3, p. 43-45.

Bjørnson, F. O., Dingsøyr, T., 2008. Knowledge

management in software engineering: A systematic

review of studied concepts, findings and research

methods used. In Information and Software

Technology, 50(11), 1055-1068.

Davenport, T. H., Prusak, L., 1998. The Book. Working

Knowledge: How Organizations Manage What They

Know, Harvard Business School Press, Boston, USA.

Davenport, T.H., Völpel, S.C., 2001. The rise of

knowledge towards attention management. In Journal

of Knowledge Management, Vol. 5 Iss 3 pp. 212 - 222.

Desouza, K., Dingsoyr, T., Awazu, Y., 2005. Experiences

with conducting project postmortems: Reports vs.

Stories and practitioner perspective. In: HICSS'05.

IEEE. Pp. 1-10

Dingsøyr, T., Moe, N. B., Nytrø, Ø., 2001. Augmenting

experience reports with lightweight postmortem

reviews. In Product Focused Software Process

Improvement. Springer Berlin Heidelberg. p. 167-181.

Dingsøyr, T., 2005. Postmortem reviews: purpose and

approaches in software engineering. In Information

and Software Technology, v. 47, n. 5, p. 293-303.

Kim, S. H., Kim, W. J., 2014. Development of Operational

Quality Measurement Attributes of Application

Software Using KJ Method. In Int. J. of Software

Eng.& Its Applications, v. 8, n. 3. pp. 171-188.

Kokogawa, T., Maeda, Y., Ajiki, T., Itou, J., Munemori,

J., 2012. The effect to quality of creativity with

sampling partial data from a large number of idea

cards. In Proc of the ACM 2012 CSCW. pp. 147-150.

Lindvall ,M., Rus ,I., Sinha, S.S., 2003. Software systems

support for knowledge management. In Journal of

Knowledge Management, Vol. 7 Iss: 5, pp.137 - 150.

Myllyaho, M., Salo, O., Kääriäinen, J., Hyysalo, J.,

Koskela, J., 2004. A review of small and large post-

mortem analysis methods. In Proceedings of the

ICSSEA, Paris.pp.1-8

Nonaka, I., Takeuchi, H., 1995. The Book. The

Knowledge-Creating Company: how Japanese

companies create the dynamics of innovation”. New

York: Oxford University Press, 1995, 284p.

Rabelo, J., Viana, D., Santos, G., Conte, T., 2014. Using

Pattern-PABC to Coding Knowledge: An

Experimental Study. In XIII SBQS, p. 13-27. (in

Portuguese)

Rus, I., Lindvall, M. 2002. Knowledge management in

software engineering. In IEEE Software, v. 19, n. 3,

ICEIS2015-17thInternationalConferenceonEnterpriseInformationSystems

164

pp. 26-38.

Scott, L., Stålhane, T., 2003. Experience Repositories and

the Postmortem. In Wissensmanagement. p. 79-82.

Scupin, R., 1997. The KJ Method: A Technique for

Analyzing Data Derived from Japanese ethnology. In

Human Organization, vol. 56, pp. 233-237.

Stålhane, T., Dingsøyr, T., Hanssen, G. K., Moe, N. B.,

2003. Post mortem–an assessment of two approaches.

In Empirical Methods and Studies in Software

Engineering (pp. 129-141). Springer.

Vasconcelos, J.B., Seixas, P.C., Lemos, P.G., Kimble, C.,

2005. Knowledge Management in Non-Governmental

Organisations: A Partnership for the Future. In Proc.

of the 7th International Conference, Enterprise

Information Systems (ICEIS), Miami, USA.

Widjaja, W., Sawamura, M., 2014. Bring your own

device: ubiquitous approach to digital affinity diagram

collaboration. In Proc. of the 2014 ACM International

Joint Conference on Pervasive and Ubiquitous

Computing: Adjunct Publication. pp. 287-290.

ApplyingKnowledgeCodificationinaPost-mortemProcess-APracticalExperience

165