Applying Knowledge Codification in a Post-mortem Process
A Practical Experience
Erivan Souza da Silva Filho, Davi Viana and Tayana Conte
USES Research Group, Instituto de Computação, Universidade Federal do Amazonas, Manaus, Brazil
Keywords: Knowledge Management, Post-mortem Analysis, KJ Method, Ishikawa's Diagram, Coding Knowledge.
Abstract: In information systems, acquiring experiences in projects can result in new knowledge to people or the
organization. Knowledge Management analyzes such experiences as a significant resource to the
organization. Through the Post-mortem analysis, people can remember experiences and situations that they
had during a software development project. In order to support such analysis, the PABC-pattern structure
proposes codifying knowledge, assisting practitioners in registering key elements to facilitate the
understanding of that experience. This paper proposes a process of Post-mortem Analysis based on the KJ
method. We have integrated the PABC-Pattern approach as a final product in order to record the
experiences and gathered information.
In software projects, the team members acquire new
knowledge and experiences that can benefit future
projects and their professional skills (Birk et al.,
2002). Many businesses are human and knowledge
intensive, such as software organization. Knowledge
intensive organizations have been noticing that a
large amount of problems is attributed to un-
captured and un-shared product and process
knowledge (Lindvall et al., 2003).
Knowledge Management is the process of
creating, validating, representing, distributing and
applying knowledge (Bhatt, 2001). Knowledge
Management also refers to identifying and
increasing the collective knowledge of an
organization to help it keeping a competitive
advantage (Alavi and Leidner, 2001).
Another common factor to Knowledge
Management is learning with the past successes and
failures. The goal of this learning is to improve the
future of the software development (Dingsøyr,
2005). Identifying these experiences in projects and
codifying them can help to prevent mistakes in
future projects. By avoiding such mistakes, we can
reduce re-work while repeating well-succeeded
processes in order to increase productivity and the
probability of achieving success (Rus and Lindvall,
Retrospective analysis (known as Post-mortem)
is one approach to support remembering knowledge
from projects. According to Scott and Stalhane
(2003), during a Post-mortem analysis the
participants of an ongoing or finished project are
reunited and they are asked to identify: (a) which
aspects of the project went well and must be
repeated, and (b) which aspects of the project went
bad and must be avoided.
Once the Post-mortem captures the knowledge, it
is necessary to codify it. There are techniques
regarding capturing knowledge (Bjørnson and
Dingsøyr, 2008). A solution proposed to capture and
code knowledge is employing the PABC-Pattern
(Rabelo et al., 2014). PABC-Pattern is an approach
to codify the lessons learned in a structured form.
The elements from such form aim to facilitate the
knowledge storage in a software organization.
Adopting a Post-mortem approach combining
with a coding method can lead to improvements in a
project. This paper describes an experience of using
retrospective Post-mortem analysis at the end of
each of the iterations of a project, and applying the
PABC-Pattern coding structure. By describing such
Post-mortem analysis process, we intend to provide
practitioners with an example they can replicate in
other software projects.
The rest of this paper is organized as follows.
Section 2 shows the background of this work. In
Section 3, we present the definition of the Post-
mortem process. Next, Section 4 describes the
Souza da Silva Filho E., Viana D. and Conte T..
Applying Knowledge Codification in a Post-mortem Process - A Practical Experience.
DOI: 10.5220/0005376301530165
In Proceedings of the 17th International Conference on Enterprise Information Systems (ICEIS-2015), pages 153-165
ISBN: 978-989-758-097-0
2015 SCITEPRESS (Science and Technology Publications, Lda.)
application process in a case study while Section 5
discusses the case study results. Finally, Section 6
shows our conclusions and future work.
Individual knowledge is necessary for the
development of the organizational knowledge
(Bhatt, 2001). Knowledge within an organization is
a collection of knowledge, experiences and
information which people or groups use to carry out
their tasks (Vasconcelos et al., 2005). In this section,
we present the concepts that we applied as
theoretical basis for this work.
2.1 Knowledge Management
The human resources is the main asset of many
companies where knowledge has to be preserved and
passed from the individual to the organizational
level, allowing continuous learning and
improvements (Lindvall et al., 2003). Companies
generally understand the term “knowledge” as
codified information with a high proportion of
aggregated human value including perception,
interpretation, context, experience and wisdom
(Davenport and Völpel, 2001).
Davenport and Prusak (1998) placed the
knowledge in three distinct items: data, information
and knowledge. Data is a group of distinct facts and
goals related to events. Information aims at changing
the way in which the receiver sees something,
exercising some impact on his/her judgment and
behavior. Knowledge is the fluid mixture of
condensed experience, values, contextual
information and experienced insight which gives us
a structure to the evaluation and incorporation of
new experiences and information (Davenport and
Prusak, 1998).
Nonaka and Takeuchi (1995) characterized
knowledge into two types: explicit and tacit. Explicit
or codified knowledge can be articulated in formal
or textual language. Tacit knowledge is the personal
knowledge, incorporated to the individual
experience and which involves intangible factors
(e.g. personal beliefs, perspectives and value
Knowledge Management is a method that
simplifies the process of sharing, distributing,
creating and comprehending a company's knowledge
(Bjørnson and Dingsøyr, 2008). Its goal is to solve
problems regarding the identification, localization
and usage of the knowledge (Rus and Lindvall,
2002). Furthermore, Knowledge Management is
concerned with aspects regarding how to collect
and/or make explicit the experiences of the projects
to be used by others (Dingsøyr et al., 2001).
In the case of finished projects, we can
remember the experiences that the participants had
during the project. Therefore, such knowledge can
be reutilized. One approach to guarantee the
collection of such experiences is through the Post-
mortem Analysis.
2.2 Post-mortem Analysis
Post-mortem analysis is the most common name
given to retrospective analysis (Myllyaho et al.,
2004). Post-mortem is the activity of gathering
knowledge which can be organized for a project that
is in its final stage or finished (Dingsøyr, 2005). The
goal of a Post-mortem must be to learn not to
evaluate. In this sense, an evaluation can make
people restrain their experiences because they might
think that sharing such experiences could be
embarrassing (Desouza et al., 2005).
There are many ways to apply a Post-mortem
Analysis. Dingsøyr (2005) shows two different
proposals of how to conduct a Post-mortem in small
or medium companies:
1. Post project review process by Neal Whitten: (1)
Declare intent – send a message to all
participants indicating that a Project
retrospective meeting will be made; (2) Select
participants – the key project participants must
be selected; (3) Prepare for workshop – the
participants must prepare a presentation
answering a series of question such as: “What
was the productivity level achieved by your
task?”; (4) Conduct workshop – the workshop
must take from one to two days, the participants
present their answers and a list of possible
improvements regarding the project; (5) Present
results – the results of the post-mortem are
presented to the project leaders and then, to all
project members; (6) Adopt recommendations
– A post-mortem report is created, containing all
the information of the workshop and
recommendations to the project leaders.
2. Retrospective Meeting by Collison and Parcell:
(1) Call to the meeting – right after the Project
ends, the participants are summoned; (2) Invite
the right people – the key people are selected by
priority; (3) Appoint a facilitator – a person
who is not close to the project is chosen to
conduct the meeting; (4) Revisit the objectives
and deliverables of the project – find the
original success criteria and ask if they were
achieved; (5) Revisit the project plan or
process – this step must be useful in the
construction of a flow where the activities,
deliveries and decisive points will be showed; (6)
Ask “What went well” – an answer list must be
generated and complemented with a “why?”; (7)
Find out why these aspects went well, and
express the learning as advice for the future
identify the success factors and
recommendations; (8) Ask “What could have
gone better?” – it must start with the project
leader and then continue with the rest of the
people in the meeting; (9) Find out what the
difficulties were – identify problems and
barriers that must be avoided; (10) Ensure that
the participants leave the meeting with their
feelings acknowledged – ask each person
regarding his/her impressions on the meeting;
(11) “What next” – if someone starts a new
project, a session is recommended showing the
retrospectives; (12) Recording the meeting – a
structure must contain all the main aspects,
artifacts and documents that characterize that
Birk et al., (2002) defined the Post-mortem analysis
in three phases:
1. Preparation: during the preparation phase, a
whole project retrospective will be made to
provide a better understanding. All project
documents like reports and project plan will be
reviewed. Also a goal for the Post-mortem will
be defined, for example, “Identify the bigger
achievements and improvements on the project”.
2. Data Collection: all the relevant experiences of
the Project are collected. The project participants
and other stakeholders will make a group
session. Some techniques can be applied in that
phase such as: (a) Semi-structured Interviews
the facilitator will make a list of questions to the
participants; (b) Facilitated group discussion
the facilitator will conduct the discussion while
(s)he documents the main results on a
whiteboard; (c) KJ Method – the participants
will write 4 positive and negative experiences
regarding the project in notes. This method will
be furthered detailed in the next section.
3. Analysis: The facilitator conducts a feedback
session with the participants aiming to identify if
(s)he understood what they said and if (s)he
discussed all the relevant facts. When the data is
sufficient, an Ishikawa’s diagram is created to
find the causes to the positive and negative
experiences. The Ishikawa’s diagram will be
further detailed in Section 2.4.
The Post-mortem reports may vary in size, variety,
scope and deepness (Desouza et al., 2005).
According to Birk et al., (2002), the facilitator writes
the Post-mortem analysis results in an experience
report which must contain:
A description of the project, including the
developed product, development methods, time
and necessary effort.
The project’s main problems, with diagrams
representing the problems (e.g. an Ishikawa’s
The project’s success factors, also with diagrams
representing them.
Transcribing what was said in the meetings helps to
contextualize the information for future readings
(Dingsøyr, 2005). The person responsible for writing
the Post-mortem report can be someone of the
group, or the own group reunited, or even an
external facilitator, depending on project’s nature.
(Desouza et al., 2005).
2.3 KJ’s Method
KJ’s Method, also known as the Affinity Diagram, is
a discussion tool created by Jiro Kawakita (Kim and
Kim, 2014). The KJ’s Method usually employs
individual descriptions on notes (e.g. stick notes) or
cards. Then, with the help of the other participants,
the notes are grouped by similar points (Widjaja and
Sawamura, 2014).
The KJ Method’s application includes four steps
(Scupin, 1997): (1) Card creation, (2) Card
grouping, (3) Card unification by similar patterns,
and (4) Written or oral explanation. In the literature,
the way in which these four steps are applied may
vary according to the process by each author, as
shown in the following approaches:
Kokogawa et al., (2012) describe the standard
process to perform KJ’s Method as follows: (1)
Prepare the cards with ideas referring to a specific
theme; (2) Form a group of cards with similar ideas;
(3) Identify each group with a name; (4) Position
each group into a diagram, showing the relationship
between them; (5) Describe sentences to express
what the diagram means.
Kim and Kim (2014) describe the steps in a
similar way to Kokogawa et al., (2012), adding some
new steps: (1) A topic is selected for discussion; (2)
Participants generate ideas through brainstorming;
(3) A card is created for each of the participant’s
ideas; (4) After all ideas have been discussed, the
cards that are similar are grouped (in case an idea
does not fit any of the previously created groups, it
enters a new group separately); (5) Each group of
cards is given a representative name; (6) The groups
are classified as “main”, “superior”, “medium” and
“minors”, and the similar groups are combined; (7)
The classified cards are then reorganized in a board
or a big sheet of paper.
The advantage of the KJ’s Method is that it
allows the discovery of problems even in chaotic
situations. Such method can help generating new
ideas and identifying the essence of the problems
correctly, breaking any stereotypes (Kim and Kim,
2.4 Ishikawa’s Diagram
Ishikawa’s Diagram, also known as the fishbone
diagram (due to its shape), is a diagram that aims at
showing the causes and effects of a problem
(Stålhane et al., 2003). In the diagram, a line is
drawn indicating a problem to be discussed. Then,
other diagonal lines are drawn pointing to the main
line (Dingsøyr et al., 2001). The attached lines (in
fishbone form) represent the causes that leaded to
the problem or experience (Birk et al., 2002).
2.5 PABC-Pattern
The literature presents different approaches such as
formal routines, causal map, concept map, models
with case-based reasoning and other proprietary
tools (Rabelo et al., 2014). Additionally, we decided
to apply PABC-Pattern because this technique is not
associated with a tool or specific software process.
Thus, it is possible to integrate it with other
approaches or processes that work with knowledge,
such as a Post-Mortem Analysis.
PABC-Pattern is an approach to codify the
lessons learned in a structured way. Its elements aim
to facilitate knowledge storage in a software
organization (Rabelo et al., 2014). The knowledge
codification process can enable registering the
lessons learned for further consultation in order to
achieve personal learning or to apply it in new
projects. The elements that compose the structure of
the PABC-Pattern are:
Title: the lesson’s name and a brief description;
Problem: a detailed description of a problem or
a question that the lesson learned tries to solve;
Cause of the Problem: details the cause of the
problem, this description must contain what
caused the problem;
Consequence(s) of the Problem: description of
the consequences of the problem. In other words,
what happened after the problem occurred;
Action: provides details of the solution to the
problem, (i.e. highlights of an activity that was
applied to solve the problem);
Benefit: description containing the effects
(positive and/or negative) caused by the action;
Keyword: it describes the keywords that identify
the lesson;
Relation with Other Lessons: lists the
identification code of other lessons that are
related to the described lesson;
Context: characterizes the environment in which
the action was executed. The context can be
described in terms of:
- Project Type: Selection of the Project type
(development project, maintenance project,
or both);
- Project Size: Selection of the project’s size
(small project, medium project, or big);
- Project Phase: Indication of the Project
phase (requirements elicitation, analysis,
coding, test, implantation, management
activities, supporting activities, others);
Role of the Lesson’s Creator: description of the
role of the person who created the lesson;
Related Domain: description of the domain
where the lesson can be applied;
Other Relevant Information: description of
other information that the lesson’s creator judges
We chose this approach based on the positive
empirical results collected regarding the PABC-
Pattern (Rabelo et al., 2014). According to the
subjects of the empirical studies, the reasons for
these positive results of the PABC-Pattern are:
A more detailed description of the knowledge
scenario, becoming more enlightening;
The possibility to acquire more information
applying less effort;
The ability to describe the problem and the
solution in the same record.
The experience collected in a post-mortem needs to
be storage in order to help other projects in the
future. Therefore, a knowledge codification
technique can be necessary.
The Post-mortem is an important activity in a
project’s retrospective. Executing a process means
following the steps to achieve a goal, in this case,
remember and register experiences. The process of
Post-mortem applied in this work is based on the
works by Birk et al., (2002) and Dingsøyr (2005).
The process will be composed of 3 stages:
(1) Data Collection – where data from the
participants will be collected;
(2) Data Analysis – once the information is
collected, it will be analyzed, interpreted and
(3) Results – creation of an experience report, final
corrections and feedback from the members of
the project.
3.1 Data Collection
As suggested by Dingsøyr (2005), a facilitator is
placed to conduct the four activities of the Post-
mortem data collection. According to Birk et al.,
(2002), (s)he can be someone from outside the
project whom the participants trust. (S)he will be
responsible to let the session flow and intervene
whenever a problem arises.
The facilitator’s role is to guide the project’s
participants so that the session flows clearly. The
facilitator will be answering any doubts that occur
and organizing people in the environment to
promote the participants’ interaction in the best way
possible. Also, (s)he may interfere if necessary (i.e.
when (s)he finds problems that could interrupt the
Figure 1 shows the first activity of the data
collection stage, “1 – Preparation of the
Participants”. This activity starts the Post-mortem
process. In this activity, we explain to the
participants, through a presentation, how the Post-
mortem process will be made, why it is important
and what they should do.
Figure 1: Data Collection Process.
The “2 – KJ Session” activity uses the KJ
Method, one of the techniques that we can apply to
make the information gathering in a Post-mortem
process (Birk et al., 2002). It is complemented in
parallel with the “3 – Recording” and “4 –
Annotations” activities. The KJ session will be
detailed in Subsection 3.3.1.
The “3 - Recording” activity registers all KJ
session. The recording depends on the environment,
as it is preferable that the environment is calm. Each
participant must be requested to keep a voice level
that can be recorded by the recording tool.
The “4 - Annotations” activity aims at
highlighting ideas, conclusions or facts that can be
relevant to be further recorded in the Data Analysis
stage. This activity is optional; the facilitator can
choose to keep notes (or not) of anything during the
KJ session or only focus in other activities.
3.1.1 KJ Session
The session using the KJ method is based on the
steps presented in Kim and Kim (2014). Figure 2
shows the steps of the KJ method that are adopted in
our work:
Figure 2: KJ Session Steps.
Step I – “What is the question to be
answered?” - The participants are guided to answer
a question. In our work, the question was “What
were the positive and negative experiences learned
in this project?”. Thus, the participants were guided
to answer this question. By guiding the participants,
we intend to gather specific data to solve the
Step II – “The group is organized” - In projects
with many participants it may not be able to have all
participants in a session. Therefore, the participants
must be chosen in a way that diverse points of view
of the project are collected. In addition, time must be
planned accordingly to the organization’s context. In
our work, all participants were invited and this
guaranteed that all points of view of the project were
visited. Also, we planned the session for one hour.
Step III – “The experiences are collected in
notes” – In this step, we distributed a predefined
number of notes to all participants so they could
write their experiences. As suggested by Birk et al.
(2002) we initially provided four notes to be filled.
Then, we informed the participants that they had 20
minutes to fill the notes in order to answer the
question from step I. However, as some participants
requested more notes, we allowed them to feel free
in writing further notes answering the question.
Step IV – “The participants discuss the notes
One by one, the participants should choose a note
and discuss it with the team. The team starts a debate
of ideas regarding such note. The process is repeated
until all the notes or the estimated time for the
meeting ends.
Step V – “The notes are grouped by similar
experiences”, the facilitator identifies the described
experiences on the notes that might be similar and
should ask the group if they agree with this
interpretation. In case of disagreement, the notes are
grouped in a different way. This step must be made
until every note is discussed.
Step VI – “We put the notes on a board” -
Discussed notes are pasted on a board so all the
participants can see them. Notes that describe
similar experiences notes are put close to each other.
The participants are free to disagree on any decision
at any moment.
Step VII – “The notes are grouped under a
theme” - Here we will decide if the experience
happened in a common context and after, they are
grouped under a common theme.
Step VIII – “The notes groups are named” The
groups are named with words or texts that identify
and generalize such group of experiences.
Step XI – “Prioritizing the notes groups” At
the end, the participants must prioritize the
experiences. Making the participants visualize the
negative experiences guides them to tackle the main
problems of the project. Positive experiences that
must be replicated are exhibited aiming to show the
evolution and the success of the team.
3.2 Data Analysis
Data analysis is the process of gathering all the
acquired information and codifying it to a model,
form or document. In our work, we chose the
PABC-Pattern approach. We employ the term
“codifier” to refer to the person who is going to
analyze the recorded data, notes, annotations and to
write the codified experience. Figure 3 presents the
data analysis process.
The “4 – Mapping notes, recordings and
notifications” activity results in a transcription of all
notes gathered in the KJ’s session. Each note, the
transcription of what the participant wrote, and the
starting time of the recording in which it was
discussed must be stored. After the codifier analyses
the information and then, executes activity 7, (s)he
must map the identification of the codified document
with the note.
Figure 3: Data Analysis Process.
The “5 – Listening to the recordings” activity
will be the process of listening to the KJ’s session
recordings. It will serve also as a complement to the
first codifications in order to generate new ones.
During the discussions, the participants might report
some new tacit experiences that were not written in
the notes.
The “6 – Analyzing the annotations” activity
will bring support to the codifications in a sense of
remembering a related fact that the facilitator
pointed out during the session or some relevant
information that was not in the notes from the
participants. This activity is optional, because the
facilitator might choose not to write annotations
during the Post-mortem session.
The “7 – Codifying Knowledge applying
PABC-Pattern” activity is in constant execution
during the stage of data analysis. The codifier, based
on activities 4, 5 and 6, has the role of describing the
experiences in the PABC-Pattern in an impartial way
and relating them to all the points discussed in the
post-mortem session.
The optional “8 – Generating an Ishikawa’s
diagram” activity is an option for the codifier when
(s)he does not understand an experience. This
misunderstanding might be caused due to difficulties
in understanding or when the participants have many
different points of view regarding an experience.
The diagram will map all the opinions, reports or
solutions, in order to find what problem is being
approached. The codifier will have at his disposal a
general view of the causes for the experiences and
then, complete the codification.
3.3 Results
After finishing all the codifications to the PABC-
Pattern, a report with the Post-mortem results will be
generated (see Figure 4). We chose a traditional way
to generate the Post-mortem report, based on the
advantages and disadvantages presented by Desouza
et al., (2005). The main advantages in a traditional
report are that it is highly structured, easy to
comprehend and low codification cost. On the other
hand, conclusions are less easy to remember.
Figure 4: Detailed process of results and experiences.
The “9 – Generating report” activity resulted in
the Post-mortem report of the project. In our work,
to compose the report’s structure, we have chosen
the following topics from the literature (based on
Dingsøyr et al., 2001; Birk et al., 2002; and
Dingsøyr, 2005): (1) a general description of the
project, (2) the time of the project execution, (3) the
developed product, .(4) the main problems, (5) the
improvement opportunities., (6) the key artifacts, (7)
the people involved, and, (8) a list of positive and
negative points, in our case, the codified
The “10 – Participant’s feedback” activity is a
last review made by members of the project. They
can add or change something that has not been
discussed in the Post-mortem meeting. This activity
is optional, because it is necessary to allocate extra
time with all the project participants. However, the
results will pass through a detailed review. After the
end of the results stage, the final report is delivered
to the organization.
The case used in this research is regarding a
software development project on a system focusing
in supporting the daily care of the elderly. Two
distributed teams were involved in the system
development. One team was located in the north of
Brazil and it was responsible for the user interface
design. The other team was located in the south of
the same country and it was responsible for the
software implementation (coding). The information
exchange between both teams was done through the
project managers and leaders. Thus, meetings were
made possible by online conferences and documents,
information or questions exchange were made by e-
mail. Both teams responded to the representatives of
a transnational company.
The Post-mortem was applied on the team
responsible for modeling the user interface. The
team was composed of six members and the project
duration was six months. Each month, the team
delivered the results from a set of activities defined
by the project schedule. The term “Sprint” was
applied by the project team to denote the activities to
be performed in a month, including its results.
When a Sprint ended, a Post-mortem session was
scheduled and attended by all project participants.
4.1 Planning the Post-mortem Session
The first post-mortem session was held in a meeting
room with a white board on the wall. The
participants received a presentation from the
facilitator and tools like the whiteboard, a notepad
and a recorder were used. The time limit to the first
post-mortem session was free, because the process
hasn’t been defined yet.
Based on the average time of the other teams
meetings, it had been stipulated a time length of 60
minutes on the second and third Post-mortem
session where 20 minutes were reserved to the
filling of the notes and 40 minutes to the debate and
discussion of the notes. It also has the assistance of a
projector to the presentation which served as a visual
guide visual to the participants.
4.2 Post-mortem Session in Sprint 1
The first Post-mortem session was made with
empirical purposes; the process had not been defined
yet. The participants gathered in a meeting room
where the whole session was recorded, the session
took around 26 minutes to be made. In the
experiences capture process, some aspects related to
applying a Post-mortem were employed, for
example, the intervention of a facilitator and the
recordings where:
1. A list of pre-defined topics was written on a
whiteboard. Those topics had a group of
activities made in the work as basis;
2. For each topic, each participant was asked to
answer what the experiences (s)he had regarding
such topic;
3. The participant explained to everyone his/her
experience, and everyone was free to
complement or debate the presented ideas;
4. The facilitator took annotations in a notepad
about the experiences that the participants
There were difficulties in executing the Post-mortem
session. The participants needed a method that
assisted them in remembering their experiences.
Allowing the participants to talk freely was not
resulting in a dynamic enough session to capture the
experiences. After the session it was evident that a
more structured Post-mortem session needed to be
4.3 Post-mortem Session in Sprint 2
In the second Post-mortem session that had been
made for Sprint 2, we used the first version of the
Post-mortem process (Birk et al., 2002) described in
this work. The use of the KJ’s method to guide the
session was chosen for being the most disseminated
method in the results from our literature review
regarding this work. The session took around 1 hour.
The steps of the process were described to the
participants in a presentation and all participants
were asked to stand up and explain his experience.
The gathered experiences after the session
demonstrated that the use of notes between the
participants stimulated them to discuss more than in
the previous session. On the other hand, during the
data analysis stage, codifying the experiences to the
PABC-Pattern became problematic since the data in
the notes was not enough to fill some fields of the
PABC-Pattern form. For example: when a
participant described a problem, (s)he (and the rest
of the team) did not discuss the action(s) that could
solve that problem.
Another problem with the second session was the
wrong use of the whiteboard. The notes were
exhibited on a table and notes for the generated
groups were not created, as seen in Figure 5. That
led to the non-execution of the “Prioritize groups”
step. The missed step generated an absence of
information in the report where improvements
opportunities in future sprints and projects should be
Figure 5: Experience notes disposed on a table in the
Sprint 2.
4.4 Post-mortem Section in Sprints 3
and 4
The third Post-mortem session was employed for
Sprints 3 and 4 simultaneously. This decision was
made as there was not time to make a session
between Sprint 3 and 4 due to deadlines and the
need for developing the deliveries of the project. In
this session, one of the team members left the team,
which caused a difference in terms of data
collection. That means that the maximum number
possible generated notes was reduced by four (since
the team member that left was not contributing for
the generation of ideas) and there would be one less
point of view during the debate and discussions. The
whiteboard was improved with the use of a
cardboard and tape. Also, notes with different colors
were used to differentiate the groups names, as
showed in Figure 6.
Figure 6: Whiteboard for Sprint 3 and Sprint 4.
During the filling of the notes, a participant
finished filling the four notes and asked if (s)he
could fill one more, besides the already four filled
notes. The facilitator allowed it and gave him/her
another blank note to be filled. In this case, the
participants were still employing the 20 minutes
given for filling the notes. The facilitator must know
when (s)he should or should not allow the filling of
more notes by considering requirements such as
available time for the meeting, the dynamism
between the participants and others.
The facilitator now had a group of pre-
estabilished questions to ask to the participants.
(S)he could use these questions while a participant
was reporting his/her experience if (s)he noticed a
lack of information that could make it difficult to
condify the knowledge using the PABC-Pattern. The
questions were:
1. What were the causes of this experience?
2. What were the consequences of this experience?
3. What would you do to solve it?
4. What should your team do to solve it?
5. What were the benefits of doing it?
The groups names were created by the facilitator.
(S)he suggested a name and the team decided if that
name was proper. After the end of the Post-mortem
session, a questionaire was given to the participants
aiming at gathering the impressions and opinions
about the post-mortem sessions completion until that
moment. Table 1 shows the questionnaire that was
Table 1: Feedback Questionnaire of the Post-mortem.
Id Question
Regarding the usage of the notes to remember
experiences, do you believe that notes helped you to
remember your experiences? Please comment.
Still regarding the notes, what is your opinion about
the limit of 4 notes per participant? Do you think this
limit helps the Post-Mortem session to be more
dynamic or this limits the number of experiences that
you would like to write?
About the use of brainstorming for each note. Does
explaining the note to the group help to pass the
experience correctly? If you disagree or agree,
please, explain why.
The discussion about a note helps you to complete
your understanding about an experience? With the
discussion, do you feel that you would remember
new facts that would help you to complete that
Using a Whiteboard with attached notes is important
to you? And does prioritizing experiences help you
understand the main problems of the project?
In your opinion, which aspects were positive and
negative during the Post-mortem session applying
the KJ method?
Would you change any step or activity of the Post-
mortem session applying the KJ Method? Any
suggestions for improvement?
4.5 Analysis from the Session’s Data
The first analysis from the Post-mortem session of
Sprint 1 had as data the collected annotations made
in paper by the facilitator and the recordings. We
tried to verify and identify positive and negative
experiences only on the data collection’s stage. In
the end, a document was created using the PABC-
Pattern structure.
The lesson learned through the data analysis was
in identifying if the experience was positive or
negative in the recordings. The description of the
experience became very dependent of the
interpretation of whom analyzed the data on the
recordings and because of that, it was transferred to
the data collection stage.
Another lesson learned from the data collection
was the use of the Ishikawa’s Diagram when
mapping the experiences with many points of view
from the participants. Since by using the Ishikawa’s
Diagram it was possible to understand different
points of view, we attached it to the Post-mortem
On the second data analysis from the Post-
mortem session in Sprint 2, the analysis process was
changed. We included a step to transcribe the
recordings and codifying what the participants had
said. A second version of the PABC-Pattern
document was generated considering that there were
mistakes on how the structure was crafted. For
example, the document did not correctly highlight
the context structures and the identification field was
The lesson learned on the data analysis stage was
related to the transcription step. The transcription
was hard and time demanding with little benefits.
Analyzing and codifying the recordings has high
cost – low effectiveness activity. Also, only listening
to the recordings was enough to retrieve the data.
Therefore, the Post-mortem process was altered
again by excluding this activity.
Figure 7: Sample of transcription notes from Session 3.
The third data analysis from the Post-mortem
session in Sprints 3 and 4 was described in this work
(see Section 4.4). In this third analysis, the notes
were transcribed to a slideshow tool where the notes
were mapped. As shown in Figure 7, the group title
stays above the notes. Each note is identified with a
positive/negative id defined by one of the
participants of the post-mortem session. Next to the
identification, their respective codifications in
PABC-Pattern were made. After that, we added what
was written on the participant’s note followed by the
beginning time of the recording. Notes with similar
descriptions of experiences were piled one over
another and have the same identification on the
The lesson learned through the data analysis was
that with this mapping it was possible to identify a
note with its respective codification and it became
more intuitive.
In this section, we show the results obtained in the
Post-mortem process defined in this work. We also
present the results regarding the opinion of the
5.1 Results of the Data Collection:
Post-mortem Sessions
In this subsection, we will show the results obtained
from the questions in Table 1. Table 2 shows a
summary of our key findings. We present the details
of such findings below. For confidentiality reasons,
people’s names and organizations will be omitted on
the data presentation of this research.
Table 2: Findings of the Post-mortem session.
Key Findings
Using notes helps remembering experiences.
The limit of notes did not make it difficult for the participants
to describe their experiences
Describing a note helps to transfer the experience to the group
Discussions help to complement the experience
The prioritization of notes is an important step
Regarding the use of the white board, the participants' opinion
were divergent.
Mentioning other people’s mistakes can be uncomfortable to
the participants
The discussion can be an obstacle to shy people
A weekly e-mail is necessary to support remembering of
Did the notes help you remember the
experiences? (Q1): Two participants agreed that the
use of notes help remembering experiences, but
there were improvements that could be applied:
Yes, because they made me think. But if the
same notes had small sentences guiding me to
remember experiences it could be better”.
Participant 3
Does the notes limit help the session be more
dynamic? (Q2): All participants described that the
limit of notes did not trouble make it difficult to
describe the experiences. As described in Section
3, the participants were allowed to describe more
experiences in case there still was time for this task:
One of the participants (participant 2) who used this
exception claimed that:
In fact, there was not such limitation, because I
was told that I could fill more or less notes”.
Does the explanation of the note help to correctly
transfer an experience? (Q3): All participants agreed
that describing a note helps to transfer the
experience to the group. Only writing could not be
enough for a participant to describe the situation nor
for the group to understanding it. By describing the
note, the participants can complement with
something that they did not write on the note:
I think it helps because people explain what
they wrote (because it might not be so clear). It
helps to remember other experiences.
Participant 1
Does the discussion complement the understanding
about the experience? (Q4): Again, all participants
agreed that the discussion helped them
complement the experience. Discussing a note
made the other participants complement it with their
point of view:
The advantage of discussing a note is that the
member (participant) has got the team’s general
view about the referred experience and I believe
that this makes him/her remember about related
experiences and think about them”. Participant 4
Regarding the usefulness of the whiteboard and the
prioritization of the notes (Q5): two participants
agreed with using the whiteboard while another one
thought it could be inconvenient. Furthermore, all of
the participants were unanimous about the opinion
that the prioritization of the notes is an important
step, because prioritizing makes the team have a
general view of what the main problems are to be
solved in the next stages of the project:
Yes, I believe that prioritizing [notes] helps a
lot, because then I define what we have to pay
more attention to during the project’s
execution”. Participant 3
Regarding positive and negative aspects of the
applied post-mortem (Q6) complementing with
some alterations of the KJ session (Q7) we observed
that the discussion can be an obstacle to shy
people. These people have experiences that they
want to express to the group, but talking to people is
an obstacle that can make them feel uncomfortable
when remembering the experiences:
I can never remember the four things. I don’t
like going in front of everybody to speak.
However, I do think that the discussion helps
other people”. Participant 1
Another negative issue refers to mentioning other
people’s errors, because it can be uncomfortable
to the participants. Additionally, exposing that a
participant had a negative result can create
Sometimes having to expose someone is
annoying and uncomfortable”. Participant 3
Finally, a suggestion is a weekly e-mail about the
experiences that the participant would have liked to
share, but could not remember from the Post-
mortem experiences:
I think that a weekly e-mail could be necessary
(…) that can motivate the participants to
describe their experiences”. Participant 4
5.2 Results of the Data Analysis
Through the first data analysis of the Post-mortem
session for Sprint 1 it was possible to obtain 10
experiences that were codified through the PABC-
Pattern approach. In the first analysis, there were
data recordings and annotations, but only listening to
the recordings without any visual guide made the
analysis costly. It was very difficult to understand
where it finished and where the experience report
from another participant started.
The usage of Ishikawa’s Diagram aimed at
identifying what was the problem of an experience
where many participants commented and each one
explained from a different point of view. It allowed
us to obtain a general view of all opinions and
conclusions about the problem. After understanding
the problem, it was possibile to codify to the PABC-
Pattern, easily. Figure 8 presents an example of the
usage of the Ishikawa’s Diagram.
Figure 8: Ishikawa’s Diagram for the Sprint 1
The second data analysis from the Post-mortem
session for Sprint 2 resulted in a total of 15 codified
experiences. With the use of notes and recordings, it
was easy to know what the experience was,
including when each participant described his/her
note. Also, it became easier to locate the experience
on the recordings to know about what experience
(s)he referred to.
Two Ishikawa’s Diagrams were needed to
understand two experiences. Even without the notes
aid, there were situations in which the notes were
incorrect to reflect a situation. Also, the point of
view changed through the discussion between the
participants. Aditionally, by mapping the diagram
with all the participants’ opinions, it was possible to
discover and conclude what was the problem that
they were refering to.
The third data analysis from the Post-mortem
session regarding Sprints 3 and 4 resulted in a total
of 12 codified experiences. In the Ishikawa’s
Diagram that was created for this analysis, we did
not have difficulties in understanding the described
After mapping the notes in a slideshow
presentation, the process of the analyses from the
data became easier and faster. It was possible to
know which codification on the PABC-Pattern
belonged to each note and when was the time in
which the note was explained in the recordings,
which means that reviewing the experiences was
made easier.
A process for a Post-mortem analysis helps us to
more easily carry out the knowledge collection so
we can codify it later. The first conclusion is that
mentioning wrong experiences with other
participants is still an obstacle to be avoided during
the Post-mortem sessions. The fear of causing
personal problems makes them retain those sorts of
experiences in the post-mortem sessions. Even if the
facilitator reinforces that the goal of the post-mortem
session is not to judge the participant’s acts but to
think about the project’s errors, this situation is not
Regarding shy participants, we observed that
they do not like to talk to large crowds, even
knowing about the benefits of the debate. This type
of participants needs to have a better treatment by
the facilitator, because they need to be constantly
stimulated to expose their experiences so that they
feel comfortable to speak.
During the step of creating the names for the
groups of experiences on the whiteboard, the
facilitator wrote a name and asked if the team agreed
with that title. We conclude that the participants
must propose the name of the theme of the groups
on the whiteboard. After that, the participants attach
their experiences in the group that they think is more
Finally, an issue that was raised is how to know
that using an Ishikawa’s Diagram before codifying
through the PABC-Pattern is necessary in a data
analysis process. The advantage of using an
Ishikawa’s Diagram was that understanding
problems with multiple points of view became
easier. If we needed to codify the lessons learned
directly into the PABC-Pattern structure, the
understanding process would have taken longer and
mistakes would have been introduced.
As future work, the process proposed in this
work will be evolved to make it more adequate to
capturing experiences. Some of the identified
improvement opportunities are:
To stimulate shy participants so they expose their
experiences. It is necessary to find ways to help
these participants in remembering and
externalizing their experiences to the group.
Identifying isolated experiences is not an
alternative solution since describing an
experience to the group helps everyone
understand the experience (Section 4).
Remembering the main points of the project
before the description of the experiences. It may
help the participants write better notes and help
the facilitator to better understand the project’s
To carry out the results stage of the Post-mortem
process and finish the experiences report of the
project. After finishing, carry out an evaluation
to know if the chosen report structure to this
work is efficient.
We would like to thank Jacilane Rabelo for her
support on PABC-Pattern and the collaborators who
participated in the case study. Also, we would like to
thank the support granted by CAPES process AEX
10932/14-3 and by FAPEAM through processes:
062.00146/2012; 062.00600/2014; 062.00578/2014;
01135/2011; Edital 009/2012 - RHTI Doutorado.
Alavi, M., Leidner, D. E., 2001. Review: Knowledge
management and knowledge management systems:
Conceptual foundations and research issues. In MIS
Quarterly Vol. 25 No. 1, pp. 107-136.
Bhatt, G. D., 2001. Knowledge management in
organizations: examining the interaction between
technologies, techniques, and people. In Journal of
knowledge management, v. 5, n. 1, p. 68-75.
Birk, A., Dingsøyr, T., Stålhane, T., 2002. Postmortem:
Never leave a project without it. In IEEE software, v.
19, n. 3, p. 43-45.
Bjørnson, F. O., Dingsøyr, T., 2008. Knowledge
management in software engineering: A systematic
review of studied concepts, findings and research
methods used. In Information and Software
Technology, 50(11), 1055-1068.
Davenport, T. H., Prusak, L., 1998. The Book. Working
Knowledge: How Organizations Manage What They
Know, Harvard Business School Press, Boston, USA.
Davenport, T.H., Völpel, S.C., 2001. The rise of
knowledge towards attention management. In Journal
of Knowledge Management, Vol. 5 Iss 3 pp. 212 - 222.
Desouza, K., Dingsoyr, T., Awazu, Y., 2005. Experiences
with conducting project postmortems: Reports vs.
Stories and practitioner perspective. In: HICSS'05.
IEEE. Pp. 1-10
Dingsøyr, T., Moe, N. B., Nytrø, Ø., 2001. Augmenting
experience reports with lightweight postmortem
reviews. In Product Focused Software Process
Improvement. Springer Berlin Heidelberg. p. 167-181.
Dingsøyr, T., 2005. Postmortem reviews: purpose and
approaches in software engineering. In Information
and Software Technology, v. 47, n. 5, p. 293-303.
Kim, S. H., Kim, W. J., 2014. Development of Operational
Quality Measurement Attributes of Application
Software Using KJ Method. In Int. J. of Software
Eng.& Its Applications, v. 8, n. 3. pp. 171-188.
Kokogawa, T., Maeda, Y., Ajiki, T., Itou, J., Munemori,
J., 2012. The effect to quality of creativity with
sampling partial data from a large number of idea
cards. In Proc of the ACM 2012 CSCW. pp. 147-150.
Lindvall ,M., Rus ,I., Sinha, S.S., 2003. Software systems
support for knowledge management. In Journal of
Knowledge Management, Vol. 7 Iss: 5, pp.137 - 150.
Myllyaho, M., Salo, O., Kääriäinen, J., Hyysalo, J.,
Koskela, J., 2004. A review of small and large post-
mortem analysis methods. In Proceedings of the
ICSSEA, Paris.pp.1-8
Nonaka, I., Takeuchi, H., 1995. The Book. The
Knowledge-Creating Company: how Japanese
companies create the dynamics of innovation”. New
York: Oxford University Press, 1995, 284p.
Rabelo, J., Viana, D., Santos, G., Conte, T., 2014. Using
Pattern-PABC to Coding Knowledge: An
Experimental Study. In XIII SBQS, p. 13-27. (in
Rus, I., Lindvall, M. 2002. Knowledge management in
software engineering. In IEEE Software, v. 19, n. 3,
pp. 26-38.
Scott, L., Stålhane, T., 2003. Experience Repositories and
the Postmortem. In Wissensmanagement. p. 79-82.
Scupin, R., 1997. The KJ Method: A Technique for
Analyzing Data Derived from Japanese ethnology. In
Human Organization, vol. 56, pp. 233-237.
Stålhane, T., Dingsøyr, T., Hanssen, G. K., Moe, N. B.,
2003. Post mortem–an assessment of two approaches.
In Empirical Methods and Studies in Software
Engineering (pp. 129-141). Springer.
Vasconcelos, J.B., Seixas, P.C., Lemos, P.G., Kimble, C.,
2005. Knowledge Management in Non-Governmental
Organisations: A Partnership for the Future. In Proc.
of the 7th International Conference, Enterprise
Information Systems (ICEIS), Miami, USA.
Widjaja, W., Sawamura, M., 2014. Bring your own
device: ubiquitous approach to digital affinity diagram
collaboration. In Proc. of the 2014 ACM International
Joint Conference on Pervasive and Ubiquitous
Computing: Adjunct Publication. pp. 287-290.