The line segment is defined as the edge with a pre-
cise length. Using the two vanishing points, the ho-
mography matrix which defines the image projection
is estimated and the image is rectified. Next in the
rectified image, the unwanted regions such as the sky,
the road and the neighbourhood area are recognized
by using the line segment spatial distribution and the
luminance features. As a result, the region of inter-
est where the building facade locates is extracted for
further analysis tasks.
The assumption and the requirement for this
method is that the facade images need to be taken in
a cloudy morning, or evening, of later Spring or early
Autumn, when the building facades are relatively il-
luminated homogeneously and less occluded by veg-
etation. In addition, the image needs to be taken with
large angle objective so the facade will be contained
in one single image. These restrictive conditions en-
sure that the facade image will preserve maximum de-
tails and different parts of the facade can be relatively
easy to differentiate in terms of geometry, color and
texture.
1.1 Related Work
The perspective rectification and image segmentation
for natural and urban scene interpretation have been
studied intensively in computer vision.
The general procedure for the interpretation in-
volves three steps, feature extraction, feature process-
ing based on mathematical modeling of the problem
and the final inference. The selection of features are
crucial and problem-specific. The widely used fea-
tures include shape, color and textures. Those visual
appearances are extracted straightforwardly and com-
pared directly using various error metrics. However,
in complicated urban scenes, where human made ob-
jects could be highly textured and occluded, those
visual appearance features are not suitable for direct
comparison. For instance, a blazing window glass can
reflect any visual content from the sky or neighbour-
hood which is interfering and not related to the true
characteristics so that two identical windows may ap-
pear different visually.
The edges, which are the discontinuities over the
borders of objects on images, can provide strong clues
for object shape and location information. This is es-
pecially apparent in man-made objects where many
rectangular surfaces exist such that horizontal and
vertical edges are often seen. For complex shape de-
tection, one can take advantage of processing edges
elaborately. Similarly, proper clustering of the edges
can be utilized to analyze the overall typology of the
man made objects such as architectures where repe-
tition and symmetry of primitive geometries are the
typical composition characteristics. In addition, the
distribution of edges could reveal other useful infor-
mation such as the orientation of the scene.
Traditionally, the detection of line segment, is
based on edge detection and the Hough trans-
form (Matas et al., 1998). And the line segment hy-
pothesis is usually validated by hard decisions on the
gap and length thresholds. As a result, this detec-
tion often produces false line segments, which can be
harmful to further edge feature processing. Rafael et
al (Grompone von Gioi et al., 2010) proposed a fast
line segment detection algorithm which provides al-
most no false line and the computation time is linear.
This algorithm takes gradient as input and searches
for line region by considering less false detection. The
output from this algorithm is close to human percep-
tion such that in case of a noisy like texture, there will
be no line segment detected. With this line segment
detector, the line segment is robust and less biased for
computer vision problems.
In processing architectural photos, the successful
perspective rectification depends on accurate estima-
tion of vanishing points, which are intersecting points
from line segments on images. In highly rectangular
textures (Liebowitz and Zisserman, 1998), lines can
be easily extracted and rectification is completed by
combining affine rectification with vanishing points
and metric rectification with other priors like known
angles between lines. However, generally, such prior
information can not be obtained automatically and
needs human interaction. In general case (Kalantaria
et al., 2008), the image is rectified by using two van-
ishing points and solving the homography matrix di-
rectly. The certainty of less interference from false
line detection is provided by using RANSAC algo-
rithm. However, the rectification is less efficient due
to abundant false line segments.
The segmentation of outdoor scene could also
benefit from line segment detection. Derek Hoeim
(Hoiem et al., 2005) proposed a decision tree based
machine learning algorithm for outdoor scene seg-
mentation. In his method, the direction of edges
is considered as key feature and was shown to pro-
vide useful information in differentiating objects. De-
spite the method’s general application, the amount of
manual labeling work can be terrifying. Similarly in
the unsupervised approaches, the interpretation of the
sky can rely on edge analysis (Laungrungthip et al.,
2008). In this case, the sky can be thought of as
clean regions against the ground, different to previ-
ous assumption that the sky is more blue than the
ground (Laungrungthip, 2008) (Schmitt and Priese,
2009) (Zafarifar and de With, 2006).
AUTOMATIC FACADE IMAGE RECTIFICATION AND EXTRACTION USING LINE SEGMENT FEATURES
105