Definition and Classification of Primitives for the Robotic Unfolding of a
Piece of Clothing
Dimitra Triantafyllou and Nikos A. Aspragathos
Department of Mechanical Engineering and Aeronautics, University of Patras, Rio, Greece
Clothes, Unfolding, Primitives.
In this paper a dictionary of primitives is proposed that define critical features for the representation of a cloth’s
configuration so as to unfold it using a general manipulator. The primitives are defined, detected and classified
according to geometrical criteria. Experimental results demonstrate the effectiveness of the detection and
classification approach. Finally, applications are presented where the dictionary of the proposed primitives
can be used, such as the definition of different types of folds of an unfolded fabric that can facilitate the
unfolding task. Conclusion and suggestions for future work completes this paper.
Recent years robotic scientists tend to give augment-
ing attention to the field of service robots. Robots
that engage house works could relief people from a
burden in their everyday life. A challenging area of
house works for robotic scientists is the manipulation
of non-rigid objects, such as clothes. Modelling and
predicting the behaviour of fabrics presents huge dif-
ficulties due to the large deformations and the very
low bending resistance of the fabrics.
A challenging task for the robotic handling of gar-
ments is their unfolding from a random configuration
to a fully unfolded flat state. Since there are infi-
nite possible configurations in which a garment can be
found and there is no stable posture for grasping, as
in rigid objects, researchers propose ways to diminish
the garments’ possible configurations. The transition
of the garments into a group of known configurations
facilitates the recognition of their state, hence their
In order to achieve the reduction of the configura-
tions, Osawa et al. (Osawa et al., 2007) proposed a
technique, where the robot grasps a piece of clothing
from a random point, picks it up, detects its lowest
hanging corner, grasps it and then detects and grasps
its new lowest hanging corner. In this way, garments
result half folded in known configurations, facilitat-
ing their classification to a type of clothing and their
unfolding. Cusumano-Towner et al. (Cusumano-
Towner et al., 2011) suggested re-grasping the cloth’s
lowest hanging point to reduce the possible configu-
rations. The goal was to recognize the specific article,
by matching its outline with existing templates, and
plan the sequence of manipulations, which bring the
article to the desired unfolded state.
On the other hand, Kaneko et al. (Kaneko and
Kakikura, 2001) used the relationships between the
shadows that appear on clothes and the convex shape
at the outline of clothing to detect points of the hem-
lines. Furthermore, they proposed a classification
process, which matches the half folded clothes that
occur after grasping the garment from two hemline
points to predefined templates that suggest the type
of clothing and its configuration. Maintin-Shepard et
al. (Maitin-Shepard et al., 2010) implemented a vi-
sion algorithm that takes into account the sharpness
of curvature of the cloth in order to find two neigh-
bour corners and manage the unfolding of towels.
Besides the unfolding task, some researchers de-
tect characteristic features met on clothes, such as
wrinkles or loose folds (Han and Zhu, 2005),(Ya-
mazaki and Inaba, 2009), (Yamazaki et al., 2011)
. Han et al (Han and Zhu, 2005) suggested a two
level generative model for cloth representation us-
ing shape from shading technique. An interesting as-
pect in their work is that they propose a dictionary of
shading and fold primitives in order to detect loose
folds on drapery and clothes. In previous work (Tri-
antafyllou and Aspragathos, 2011) we tried to define
and classify critical features on a piece of fabric. In
this paper, a better classification with fewer categories
is introduced, simplifying the problem, and an effi-
cient implementation methodology is presented. Ad-
Triantafyllou D. and A. Aspragathos N..
Definition and Classification of Primitives for the Robotic Unfolding of a Piece of Clothing.
DOI: 10.5220/0005155704170422
In Proceedings of the International Conference on Knowledge Engineering and Ontology Development (KEOD-2014), pages 417-422
ISBN: 978-989-758-049-9
2014 SCITEPRESS (Science and Technology Publications, Lda.)
ditionally, possible applications of the methodology
are proposed.
Particularly, in this paper, a dictionary of primi-
tives for critical features that define the configuration
of a piece of clothing is introduced. The ”words” of
the dictionary include types of junctions, such as ”L”
and ”arrow” junctions, which constitute components
that are used in bibliography (Shapiro and Stockman,
2001) for perceiving 3D objects in 2D images. In-
spired by this approach, we use the same primitives to
define the configuration of a piece of clothing. Nev-
ertheless, in the presented case the primitives do not
represent the joins of the facets of a 3D object but in-
dicate the existence of folds, overlappings, edges and
corners on a garment. Furthermore, machine vision
techniques are used to detect and classify these crit-
ical features into categories. The dictionary can be
used for any configuration under the restriction that
it occurs from holding the garment in the air from its
outline and then laying it onto a table. The main dif-
ference from previous works that aimed at unfolding
clothes is that we try to analyse and comprehend the
configuration of the garment no matter its shape. We
do not use predefined templates, so even pieces of fab-
ric with unusual shapes can be handled.
The outline of this paper is as follows. Section 2
presents the dictionary of primitives in detail, while
in section 3 possible applications are described. In
section 4, the detection and classification of the prim-
itives is implemented. Finally, section 5 includes ex-
perimental results and section 6 some conclusions and
ideas for future work.
The infinite degrees of freedom of clothes result in
infinite possible configurations. The introduced prim-
itives concern only configurations, where gravity is
used to convert crumbled clothes (Fig 1.b) into a more
straightened, crisp state. In particular, the accepted
configurations can occur after lifting the garment in
the air from a point at its outline when it is fully un-
folded (Fig 1.a), so that all major wrinkles are re-
moved, and then lay it on a working table, as in Fig
1.c. In this paper, configurations on a table are con-
sidered, although the differences when holding the
garment in the air are small. An easy way to de-
tect an outline point is to apply the lowest hanging
point technique as suggested in (Osawa et al., 2007),
(Cusumano-Towner et al., 2011). So, when a garment
grasped from a random point is hanging, its lowest
point is an outline point.
Figure 1: Configurations of the cloth: (a) the outline of the
cloth when it is fully unfolded, (b) a piece of clothing in a
crumbled configuration, (c) the piece of clothing after it is
held in the air from its outline and laid on a working table.
Figure 2: Critical features for the unfolding task and their
characteristic points of intersection: (a) a fold, (b) an over-
The primitives of the proposed dictionary are geo-
metric features that describe folds, overlappings, parts
of edges and corners on the surface of the lying piece
of clothing. We define as a fold the result of bending
a cloth over upon itself (Fig 2.a). On the other hand,
an overlapping occurs when a part of a cloth covers
partly another one, if the two participating parts are
not located next to each other in the garment’s un-
folded configuration (Fig 2.b). To indicate the exis-
tence of a fold or overlapping, the detection of the
junction, where the participating layers intersect is
sufficient (Fig 2). Furthermore, the term ”corners”
refers only to the corners that are located at the out-
line ( Fig 1.a)of the unfolded cloth.
The geometrical features, that constitute the in-
troduced primitives, are types of junctions which are
classified according to the number of edges and the
size of the angles they form into ”I”, ”L”, ”T” and
”arrow” junctions (Table 1). So, a fold is detected
by an ”arrow junction”, an overlapping is met when a
”T” junction occurs, a part of an edge is an ”I” junc-
Table 1: The primitive features for the unfolding task.
Features fold overlapping corner edge
Junction’s type ”arrow” junction ”T” junction ”L” junction ”I” junction
Junction’s type
Junction’s schema
tion, while the corners of the garment are indicated
by ”L” junctions. Furthermore, an ”L” junction could
also indicate a ”hidden fold”, which is actually a fold
whose folded part is hidden behind the visible part
of the garment. One way to distinguish a hidden fold
from a cloth’s corner is to lift the ”L” junction with the
robot and check if an ”arrow” junction was hidden at
the side of the cloth that was not visible. In addition,
the edges of a ”T” junction dictate which part of the
cloth is overlapped and which one overlaps the other
part. So, if we consider the ”T” junction as an in-
tersection of one edge to the middle of another edge,
then the part of the cloth that includes the first edge
is overlapped by the part of the cloth with the second
The relationships described in the previous para-
graph cover a big part of the junctions that can be
found on a lying piece of clothing. Other types of
junctions could rarely occur in the configurations un-
der consideration thanks to gravity and for now are
not taken into account. In addition, we cannot ex-
clude the possibility that the types of junctions used as
primitives could occur from other circumstances than
the mentioned ones. Nonetheless, the reasonable lo-
cation on the cloth and the appropriate combination of
primitives can largely diminish false correspondences
between the primitives and their meaning. In section
3, apart from examples that prove that these mistakes
can be avoided, different ways that the primitives can
be appropriately connected and form layers are de-
The introduced features can provide information for
the extraction of the garment’s layers in order to facil-
itate the unfolding task. The goal of this paragraph is
not to classify the layers based on the proposed primi-
tives, although it will be presented in future work. The
aim is to demonstrate the utility of the proposed dic-
tionary by showing how its primitives, detected in ap-
propriate combinations, can describe different types
of layers. So, in the same way that words put in an
appropriate order form sentences with different mean-
ings, the primitives of the proposed dictionary, de-
tected in an appropriate order, define layers that occur
from different configurations.
Figure 3: Primitives and their role to extracting edges for
the unfolding task: (a) extraction of a layer that can be un-
folded, (b) detection of an overlapped layer and a layer that
can be unfolded, (c) the effect of a ”loose fold”, (d) extrac-
tion of a layer when the cloth is in a half folded configura-
In Fig 3.a layer L can be defined by a sequence
of the proposed primitives. Starting from one of the
arrow junctions (e.g. a1), which implies the existence
of a fold, and following its internal edge by detecting
”I” junction primitives, we result in the ”L” junction
c1, meaning a corner. Continuing in the same way,
arrow junction a2 is met and finally we result again to
arrow junction a1. This sequence of primitives sug-
gests the existence of a layer that can be unfolded,
since it does not include any ”T” junction that implies
overlapping. Similarly, in Fig 3.b, starting from arrow
junction a3 and following its internal edge we reach
a ”T” junction. The edge that led to the ”T” junction
implies that the layer that we started detecting (L2) is
overlapped by another layer (L1), as a result it can-
not be unfolded. On the other hand, the extraction of
the layer L1 can be conducted in a similar way as in
Fig 3.a, since, in this case, the ”T” junction suggests
that layer L1 overlaps another layer. So, depending
on the orientation of a ”T” junction different conclu-
sions about the unfoldability of a layer can occur. In
addition, in Fig3.c, the ”T” junction created by the
”loose fold” cannot affect the correct extraction of an
unfoldable layer since its position does not suggest
that is overlapped.
In addition, the primitives’ dictionary could also
be used for unfolding in situations that the garment
is brought in a half folded configuration by using
the ”lowest hanging point” technique, as suggested
in (Osawa et al., 2007), (Cusumano-Towner et al.,
2011). The detection of the layer to unfold in addi-
tion to the detection of possible overlaps can be con-
ducted in a similar way as described in the previous
paragraph (Fig 3.d).
Finally, a scenario that could be investigated is the
detection of folds indicated by ”arrow” junctions, in
order to find points at the outline of the garment un-
der unfolding so as to recognize the type of clothing
and unfold it as in (Kaneko and Kakikura, 2001). In
(Kaneko and Kakikura, 2001) the detection of such
points is made using the shadows created on the gar-
ment in combination with the convex shape created at
the outline of the hanging cloth near the fold. Since
folds do not always create sufficient shadow for de-
tection we believe that the detection of ”arrow” junc-
tions can provide more candidates for outline points,
but this is left as future work.
In this section a method for the implementation of the
primitives’ detection and classification is presented.
The experiments are made on monochromatic clothes
using RGB images; nevertheless we have experimen-
tal evidence that the same procedure could be suc-
cessful on multicolour garments with a stereo camera.
The pictures are taken from a 20 cm distance from the
working table where the clothes are lying and no spe-
cific lighting conditions are used.
The first step is to extract edges from the grayscale
picture depicting the primitive under detection. Bilat-
eral filtering (Tomasi and Manduchi, 1998) , which
enhances the edges, is applied (the Gaussian window
is 20 pixels, the spatial domain standard deviation is
2, while intensity domain standard deviation is 0.6).
Figure 4: Examples of primitive features and their edge de-
tection: (a) a part of a towel’s edge, (b) the edge detection
of the towel’s edge, (c) a fold on a fleece piece of clothing,
(d) the edge detection of the fleece’s fold.
Figure 5: The detected edges from Ransac algorithm and
the clusters of their intersection points.
Then an automated procedure is used for a Canny de-
tector threshold that diminishes the noise, while ex-
tracting the edges. Particularly, the threshold is re-
duced until the ratio of the pixels that belong to the
edges and the noise to the rest pixels of the image be-
comes smaller than 0.025. In this way, while the ra-
tio is reduced, the noise pixels that are decreased are
more than the decreased edge pixels, since they are
enhanced by the bilateral filter. This procedure pro-
vides satisfactory edge detection for different kinds of
clothes regardless if their surface is smooth or rough
(e.g. towel). For example in Fig 4 primitives from
a towel and a fleece piece of clothing are extracted
without particular noise.
To proceed, different edges are detected using
Ransac algorithm (Fig 5). Up to ve edges are de-
tected in order to reduce computational cost and avoid
the detection of small edges created from noise. Sev-
eral of the detected edges might be double, meaning
Figure 6: Examples of the detection and classification of primitives. In the first row the grayscale images of the primitives
are presented while in the second row the corresponding detected features are depicted. For each image different edges are
depicted with different colors while the cluster that represents the final point of intersection is marked with a blue circle. The
third row contains the results of the classification.
that an edge and its shadow can be detected as two
different edges, that Ransac split an edge in two parts,
that the cloth has a thick hemline creating two edges
etc. Such examples can be seen in Fig 6. Further-
more, the detected edges that participate in a junction
usually do not intersect to only one point but within a
small area (cluster 1 in Fig 5). To detect this area the
points of the edges’ intersections are calculated and
clustered according to their distance from each other
using the single-link hierarchical clustering method
(the maximum relative distance of the cluster’s ele-
ments is 25 pixels). The cluster including the highest
number of elements is chosen as the junction of inter-
est (Fig 5).
Then, the number of the edges participating to
the junction is calculated. Double edges are unified,
based on their location and inclination, since they are
almost parallel and close to each other (eg in Fig 5
edge 3 and 4). Once the junction of interest and the
number of intersecting edges is found, the angles cre-
ated by the edges of the junction are measured to de-
termine the type of junction. So, according to its ge-
ometry, the junction is classified to one of the primi-
tives defined in section 2.
The detection and classification algorithm was tested
in 40 different samples using 4 different monochro-
matic fabrics. In 80% of the cases, the junctions were
detected and classified into the correct category of
primitives. Examples depicting the correct detection
and the classification of the primitives can be seen in
Fig 6. On the other hand, in 7, 5% the arrow junc-
tions were mistaken as other type of junction because
of the big curvature of the bended layer of fabric near
the junction and the shadow created (Fig. 7.a). With
straightening movements made by the robot, this cur-
vature could be diminished and better results would
be obtained. Furthermore, in 7.5% the noise created
problems extracting wrong edges and in 5% the junc-
tion was near the frame of the picture, so one of the
participating edges was not big enough to be detected
(Fig. 7.b). The additional evaluation of a second pic-
ture, slightly dislocated from the first one, is consid-
ered as a measure that could enhance the results and
will be investigated in the future.
Figure 7: Examples where the method failed to categorize
correctly the primitives: (a) the folded part creates shade be-
cause of the big curvature near the point the fold is formed,
(b) one of the edges participating to the arrow junction is
not big enough to be detected.
In this paper, a dictionary of primitives, that define
critical features on a piece of clothing for its unfold-
ing, is proposed. The dictionary can be used for
pieces of clothing and fabrics of various shapes, since
we do not use restricting predefined templates. The
primitives are detected and classified based on geo-
metrical criteria.
The combination of primitives can represent dif-
ferent configurations of a garment laid properly on
a table. Layers that can be unfolded, folds, over-
lappings and corners of the garment can be detected
and used for planning manipulations for the unfolding
task. The detection and classification of the primitives
is implemented using machine vision techniques. Re-
sults show the effectiveness of the proposed approach.
As future work, the implementation of layers’ ex-
traction and manipulation in order to unfold a piece
of clothing is planned. Additionally, a scenario to be
investigated is the detection of folds, in order to find
points at the outline of the garment under unfolding
so as to recognize the type of clothing and unfold it.
Cusumano-Towner, M., Singh, A., Miller, S., Brien, J. O.,
and Abbeel, P. (2011). Bringing clothing into desired
configurations with limited perception. In IEEE Inter-
national Conference on Robotics and Automation.
Han, F. and Zhu, S. (2005). Cloth representation by shape
from shading with shading primitives. In Conference
in Computer Vision and Pattern Recognition.
Kaneko, M. and Kakikura, M. (2001). Planning strategy for
putting away laundry - isolating and unfolding task. In
IEEE International Symposium on Assembly and Task
Maitin-Shepard, J., Cusumano-Tower, M., Lei, J., and
Abbeel, P. (2010). Cloth grasp point detection based
on multiple-view geometric cues with application to
robotic towel folding. In IEEE International Confer-
ence on Robotics and Automation.
Osawa, F., Seki, H., and Kamiya, Y. (2007). Unfolding of
massive laundry and classification types by dual ma-
nipulator. In Journal of Advanced Computational In-
telligence and Intelligent Informatics.
Shapiro, L. and Stockman, G. (2001). Computer Vision.
Prentice Hall, 1st edition.
Tomasi, C. and Manduchi, R. (1998). Bilateral filtering for
gray and colour images. In IEEE International Con-
ference on Computer Vision.
Triantafyllou, D. and Aspragathos, N. A. (2011). A vision
system for the unfolding of highly non-rigid objects
on a table by a manipulator. In International Confer-
ence on Intelligent Robotics and Applications.
Yamazaki, K. and Inaba, M. (2009). A cloth detection
method on image wrinkle feature for daily assistive
robots. In Conference on Machine Vision Applica-
Yamazaki, K., Nagahama, K., and Inaba, M. (2011). Daily
clothes observation from visible surfaces based on
wrinkle and cloth-overlap detection. In Conference
on Machine Vision Applications.