![](bg6.png)
line of code may execute an action with different pa-
rameters depending on the value of some variable.
The second execution of that line would then increase
the action coverage, but not the instruction or branch
coverage.
If all parameters of the executable actions are dis-
crete and have sufficiently few values, each possible
action parametrization can correspond to a coverage
point. The coverage calculation and implementation
are straight-forward in this case.
Example 4. Consider again the simple environment
from Example 2 with three tables and two object
types. Also consider a two-handed robot acting in this
environment. The robot may pick an object from any
of the tables or place an object on a table. The action
abstracts from the exact position on the table. It is pa-
rameterized by its type (pick or place), the table, the
object type and the arm that is used. This allows for a
total of 2· 3· 2·2 = 24 distinct actions to be performed,
resulting in 24 coverage points.
However, in many cases there will be continuous
parameters or ones with a lot of possible values. In
these cases a straight-forward approach will still work
to some extend, but due to the extremely high or even
infinite amount of possible actions, the overall cov-
erage will be either very close to zero or undefined.
To avoid this problem, we suggest to form buckets
of similar actions and create one coverage point per
bucket.
A bucket is a set of actions that are sufficiently
similar in their parameters. The space of all pos-
sible actions should be divided into a finite set of
buckets such that each action belongs to exactly one
bucket. After an action is executed, the respective
bucket is marked as executed. In our implementation
of coverage-guided fuzzing, each bucket would have
its own counter that is incremented whenever an ac-
tion from that bucket is executed.
The choice of buckets is highly domain-specific
and may depend on the plan and environment under
observation. This obviously makes it harder to com-
pare the quality of different plans acting in different
environments. Still, the comparability of different test
sets for the same plan is preserved and the metric is
well suited to guide a fuzzer.
Example 5. Consider again the environment and ac-
tions from the previous example. Now, assume an ad-
ditional navigation action that will navigate the robot
to a continuous coordinate within the room. This re-
sults in an infinite number of distinct actions. To re-
duce the number of coverage points to a manageable
amount, the navigation action is divided into 4 buck-
ets depending on its target position. There is one
buckets for each table and its surrounding area and
one bucket for all positions not adjacent to a table.
This increases the total number of coverage points to
28.
Action coverage can be used in combination with
coverage-guided fuzzing as presented in the previous
section, but also independently. Like other coverage
metrics it may be used to judge the quality of hand-
written or (semi-)automatically generated test cases.
We believe that action coverage measures the di-
versity of plan executions more closely than structural
coverage metrics, since the focus is on the actual be-
havior of the robot in its environment, and not just on
the control flow of the underlying program.
6 EXPERIMENTAL EVALUATION
This section describes our experimental evaluation.
We evaluate both our approach to coverage-guided
fuzzing for plan-based robotics in general and the
combination with action coverage in particular. In
Section 6.1 we present the plan and environment that
was used for the evaluation. Afterwards, we discuss
our results in Section 6.2.
6.1 Robotic Plan and Environment
We evaluate our approach on a CPL plan that is set
in a warehouse-inspired environment. The static part
consist of a table and a shelf with three boards in a
rectangular room. The dynamic part contains a vari-
able number of objects with three types (milk, cereal
and bowl). Initially, the objects may be on any of the
shelf boards or on the table. The plan is supposed
to sort the objects onto the shelf boards. Each object
type has a corresponding board on the shelf. It does
so by first moving all objects to the table, clearing the
shelf in the process, and then moving them to their re-
spective shelf boards. To save trips between the shelf
and table, the robot will always transport two objects
at once if possible. Due to the width of the shelf, the
robot is not able to reach all positions on it from the
same point. A series of case distinctions is responsi-
ble for picking the right position for the robot to pick
or place both of its objects.
In total, the plan involves 1785 bytecode instruc-
tions, 52 branching instructions and 6 different action
types. These are the move-torso, park-arms, detect-
objects, navigate, pickup and place action.
For the action coverage, we decided on a total of
87 buckets. One bucket belongs to each of the move-
torso, park-arms and detect-objects actions. The nav-
igate action has 6 buckets, which are distinguished
by their target position. The pickup action also has
Coverage-Guided Fuzzing for Plan-Based Robotics
111