CONFIDENCE-BASED DENOISING RELYING ON A
TRANSFORMATION-INVARIANT, ROBUST PATCH SIMILARITY
Exploring Ways to Improve Patch Synchronous Summation
Cesario V. Angelino, Eric Debreuve and Michel Barlaud
Laboratoire I3S, Universit
´
e de Nice-Sophia Antipolis/CNRS, 2000 route des lucioles, 06903 Sophia Antipolis, France
Keywords:
Denoising, Image patch, Patch denoising confidence, Robust patch similarity.
Abstract:
Patch-based denoising techniques have proved to be very efficient. Indeed, they account for the correlations
that exist among the patches of natural images, even when degraded by noise. In this context, we propose
a denoising method which tries to minimize over-smoothing of textured areas (an effect observed with NL-
means), to avoid staircase effects in monotonically varying areas (an effect observed with BM3D), and to limit
spurious patterns in areas with virtually no variations. The first step of the proposed method is to perform
patch denoising by averaging similar patches of the noisy image (the equivalent in the space of patches to
synchronous summation for temporal signals). From there, our contribution is twofold. (a) We proposed to
combine the resulting overlapping denoised patches accounting for an assessed patch denoising confidence.
(b) Since a crucial aspect is the definition of a similarity between two patches, we defined a patch similarity
invariant to some transformations and robust to noise thanks to a polynomial patch approximation, instead of
a classical weighted L
2
-similarity. The experimental results show an arguably better visual quality of images
denoised using the proposed method compared to NL-means and BM3D. In terms of PSNR, the results are
significantly above NL-means and comparable to BM3D.
1 INTRODUCTION
Some studies on the distribution of patches form-
ing natural images (Carlsson et al., 2008; Huang
and Mumford, 1999; Lee et al., 2003) have moti-
vated recent patch-based processing methods, for ex-
ample, for image and video denoising (Buades et al.,
2005; Awate and Whitaker, 2006; Dabov et al., 2007;
Boulanger et al., 2007) or inpainting (Criminisi et al.,
2004). Indeed, these studies showed that there exist
correlations among patches composing images. As
a consequence, the probability is high that patches
similar to a given image patch be encountered in the
image itself. It is reasonable to think that these cor-
relations remain in the presence of (some reasonable
amount of) noise or after similar image degradations,
offering the opportunity to reduce noise or to recover
missing information in a patch. As a matter of fact,
the nonlocal means algorithm (NL-means) (Buades
et al., 2005) and BM3D (3D transform-domain col-
laborative filtering) (Dabov et al., 2007) proved to
be successful in image denoising. Given an image
patch in a noisy image, the main step of NL-means
is to perform patch denoising by averaging some
patches of the noisy image similar to the given one.
This can be viewed as the equivalent in the space of
patches to synchronous summation for temporal sig-
nals. Then, the denoised patch is projected onto the
pixel space by simply extracting the central denoised
pixel. Some works have established a variational jus-
tification relying on an information-theoretic energy
to such an approach (Awate and Whitaker, 2006; An-
gelino et al., 2008). Nevertheless, some developments
rely on other projections which make better use of
the denoised patches (Dabov et al., 2007; Salmon and
Strozecki, 2010). In this context, our contribution is
twofold. (a) The definition of a similarity between
two patches is crucial for the denoising efficiency. In-
stead of a classical weighted L
2
-norm (Buades et al.,
2005) (possibly computed after a dimension reduction
such as a PCA (Principal Component Analysis)) (Tas-
dizen, 2008) or an SVD (Singular Value Decomposi-
tion) (Orchard et al., 2008)), we defined a patch simi-
larity invariant to some transformations (rotation and
symmetries) and robust to noise thanks to a polyno-
mial approximation of the patches. Note that the idea
of transforming the patches before similarity compu-
tation has been proposed for a reduced set of trans-
65
V. Angelino C., Debreuve E. and Barlaud M..
CONFIDENCE-BASED DENOISING RELYING ON A TRANSFORMATION-INVARIANT, ROBUST PATCH SIMILARITY - Exploring Ways to Improve
Patch Synchronous Summation.
DOI: 10.5220/0003374400650071
In Proceedings of the International Conference on Imaging Theory and Applications and International Conference on Information Visualization Theory
and Applications (IMAGAPP-2011), pages 65-71
ISBN: 978-989-8425-46-1
Copyright
c
 2011 SCITEPRESS (Science and Technology Publications, Lda.)

formations (Sun et al., 2009). (b) Moreover, we pro-
pose to project, or otherwise stated, combine the over-
lapping denoised patches accounting for an assessed
patch denoising confidence. The notion of denoising
confidence for patch combination has also been pro-
posed at the patch level (Dabov et al., 2007) and at the
pixel level (Salmon and Strozecki, 2010). The confi-
dence proposed here is patchwise and it is adapted to
the chosen patch denoising procedure. These contri-
butions aim at minimizing over-smoothing of textured
areas (an effect observed with NL-means), avoid-
ing staircase effects in monotonically varying areas
(an effect observed with BM3D), and limiting spuri-
ous patterns in areas with virtually no variations (a
phenomenon observed in (Brox and Cremers, 2007)
when iterating NL-means, as proposed in (Awate and
Whitaker, 2006)).
The remainder of the paper is organized as fol-
lows: Section 2 reminds some results about a varia-
tional interpretation of NL-means based on an energy
defined as a conditional entropy; Section 3 describes
the proposed confidence-based denoising procedure;
Section 4 describes the proposed robust patch similar-
ity; Section 5 presents some experimental results for
a synthetic noise on standard benchmark images and
a digital camera noise on a professional benchmark
photography; and Section 6 provides a summary and
discusses some points of the proposed method.
2 DENOISING USING PATCH
SIMILARITIES
The inverse problem of image restoration can be for-
mulated as a minimization problem. As mentioned in
Section 1, natural images exhibit correlation among
the patches which compose them. This correlation
should be accounted for in deriving a restoration pro-
cedure.
Let us consider the conditional entropy h of
patches, i.e. the uncertainty on the color of a pixel
when its neighborhood is known. Let us model an
image as a random field X . Let T be the set of pix-
els of the image and let C
t
be a neighborhood of pixel
t ∈ T . The random vector Y (t) = {X(u)}
u∈C
t
con-
tains the set of intensities or colors of the neighbors
C
t
of pixel t. The random vector Z(t) = (X(t),Y (t))
denotes the corresponding patch, i.e. the pixel color
combined with those of its neighborhood. The recov-
ered image ideally satisfies
X(t)
∗
= argmin
X
h(X|
˜
Y =
˜
Y (t)) (1)
for all t ∈ T , where
˜
Y is the observed noisy neighbor-
hood.
Classically, we propose to use a gradient descent
procedure to solve (1). As a consequence, we need to
determine the derivative of the conditional entropy of
the color of a pixel knowing its neighborhood.
Entropy, whether conditional or not, can be ap-
proximated by the Ahmad-Lin estimator (Ahmad and
Lin, 1976)
h(X|
˜
Y = y
i
) = −
1
|T |
∑
t
j
∈T
log p(x
j
|y
i
) (2)
where the color x
j
is encountered at pixel t
j
, y
i
is the
set of (noisy) colors in the neighborhood of t
j
, and
p(s|y
i
) =
1
|T
y
i
|
∑
t
m
∈T
y
i
K(s − x
m
) (3)
is the kernel-based estimate of the underlying condi-
tional probability density function (PDF), with T
y
i
the
set of pixels which have the same neighborhood y
i
, K
a symmetric kernel, and x
m
the color encountered at
pixel t
m
. The derivative of (2) is
∂h(X|
˜
Y = y
i
)
∂x
i
= −
1
|T |
∇p(z
i
)
p(z
i
)
·
∂z
i
∂x
i
(4)
where p(Z) is the joint PDF of the high dimensional
random vector Z. The solution to (1) can be computed
by gradient descent





X
(0)
=X
0
x
(n+1)
i
=x
(n)
i
+ β
∇p

z
(n)
i

p

z
(n)
i

·
∂z
(n)
i
∂x
i
. (5)
3 CONFIDENCE-AWARE
DENOISING
3.1 Patch Synchronous Summation
Unlike denoising methods such as (Awate and
Whitaker, 2006; Buades et al., 2005) which actually
implement pixel-based iterative procedures relying on
patches, as done by (5), we propose to convert (5)
into a patch-based iterative procedure. This means
that in the update equation of (5) we discard the pro-
jection term ∂z
(n)
i
/∂x
i
, which projects the patch space
(Z) gradient vector into the pixel space (X) derivative.
Thus, we get





X
(0)
=X
0
z
(n+1)
i
=z
(n)
i
+ β
∇p

z
(n)
i

p

z
(n)
i

. (6)
In practice, we have noticed that performing only
one iteration is sufficient. The normalized derivative
IMAGAPP 2011 - International Conference on Imaging Theory and Applications
66

in (6) can be approximated by a mean shift (Fukunaga
and Hostetler, 1975; Comaniciu and Meer, 2002) on
the high dimensional joint PDF of Z. In the k-th near-
est neighbor (kNN) framework, it can be expressed
as (Angelino et al., 2008)
∇p(z
i
)
p(z
i
)
=
d + 2
ρ
2
k
(z
i
)
1
k
∑
z
j
∈kNN(z
i
)
(z
j
− z
i
) (7)
where d is the dimension of Z, kNN(z) denotes the
set of the k closest patches of z and ρ
k
(z
i
) is the kNN
patch distance.
By setting β = ρ
2
k
(z
i
)/(d + 2) and using (7), it is
clear that one iteration of the procedure (6) is equiva-
lent to
z
∗
i
=
1
∑
z
j
∈kNN(z
i
)
w
j
∑
z
j
∈kNN(z
i
)
w
j
z
j
(8)
where the weights w
j
account for the fact that, among
the patches of kNN(z
i
), patches farther away might
not correspond as well to noisy versions of z
i
as do
closer patches (they may also be slightly structurally
different).
Similarly to NL-means (Buades et al., 2005),
w
j
= exp(−|z
i
− z
j
|
2
/σ
2
w
), (9)
σ
w
being a parameter.
3.2 Patch Combination
Denoised patches obtained in (8) overlap each other.
In consequence, there is some redundancy in the de-
noising process. Indeed, for a given pixel x
i
, we ob-
tain several estimators, one for each patch to which
x
i
belongs (if the patch size is N
p
× N
p
, we have N
2
p
estimators.).
Clearly, among all the patches containing x
i
, some
will lead to an accurate denoising at x
i
, some might
not while leading to an accurate denoising in other
pixels.
Since the noise is assumed to be uncorrelated, the
synchronous summation (8) reduces the amount of
noise (as characterized by its variance) by a factor of
1/c
i
, where
c
i
=

∑
z
j
∈kNN(z
i
)
w
j

2
∑
z
j
∈kNN(z
i
)
w
2
j
. (10)
We call c
i
the patch denoising confidence.
In summary, for each patch z
i
of the noisy im-
age, we are able to compute with the confidence c
i
(see (10)) a denoised patch z
∗
i
(see (8)). These patches
are then combined (or aggregated) according to their
confidence term. Among a plethora of methods for
combining estimators, we used, as in (Dabov et al.,
2007), a linear combination of denoised patches.
The denoised patches z
∗
i
are then aggregated as
follows: starting from an aggregation image of zeros
and a confidence map of zeros, a denoised patch z
∗
i
is
added, after weighting by c
i
, to the aggregation image
at its original location, and a constant patch equal to c
i
is added to the confidence map at the same location.
After dealing with all the patches, the denoised im-
age is defined as the pointwise division between the
aggregation image and the confidence map.
For a given pixel i, the aggregation writes
x
i
=
1
∑
N
2
p
q=1
c
i,q
N
2
p
∑
q=1
c
i,q
x
∗
i,q
, (11)
where c
i,q
is the confidence of patch q among the N
2
p
patches which contain pixel i.
4 ROBUST PATCH SIMILARITY
The key idea of patch-based denoising algorithm is
image self similarity. On an image, one can find sev-
eral small areas or patches that are similar to each
other. Suppose that we have several noisy realizations
of the same patch. Assuming an additive zero mean
noise, we can denoise such a patch by taking the av-
erage of the noisy patches. The (dis)similarity func-
tion between two patches plays an important role in
choosing the best candidates (in our case the k nearest
neighbors) for the denoising task. The most popular
choice is the L
2
-distance between the patches. How-
ever, when the noise is high, such a distance becomes
unreliable. Moreover, two patches can be similar only
up to some transformation such as rotation. There-
fore, we propose to define a patch similarity invariant
to some patch transformations and applied to polyno-
mial approximations of the patches. This represents
an improvement over (Angelino et al., 2010).
4.1 Robustness to Noise
Before trying to look for similar patches, the patches
are modified in order to ensure some level of robust-
ness with respect to noise. Given a noisy patch con-
sidered as a surface, a polynomial surface approxi-
mation is computed with independent degrees in both
directions. Each degree can range from 1 to 10. The
best approximation has been defined as the one pro-
ducing a zero-mean noise between the noisy patch
and its approximation. The approximations of all
the patches are then used, possibly after geometrical
CONFIDENCE-BASED DENOISING RELYING ON A TRANSFORMATION-INVARIANT, ROBUST PATCH
SIMILARITY - Exploring Ways to Improve Patch Synchronous Summation
67

transformations (see Section 4.2), to look for self sim-
ilarities.
4.2 Transformation Invariance
Standard patch-based denoising algorithms compute
the L
2
-distance blockwise. This means that the simi-
lar patch search is performed allowing only for patch
translation. However, more similar patches can often
be found if allowing for rotation or symmetry of the
patches (Sun et al., 2009). Building on this idea, we
propose an extended set of patch transformations:
• symmetries with respect to the horizontal and ver-
tical axes and the diagonals;
• rotations by π/2, π, and 3π/2;
• rotations between −α and α with a step of dα.
Note that when comparing two patches, the rotations
π/2, π, and 3π/2 are often not the transformation
leading to the best matching (contrary to the rotations
between −α and α with α around π/6). However,
they are computed cheaply and exactly (no interpola-
tion required).
In summary, let S be the set of patches in the im-
age and T the set of transformations (symmetries and
rotations), then the set in which we look for similar
patches is the augmented set S ∪ T (S).
5 EXPERIMENTAL RESULTS
5.1 Effect of the Transformations:
A Toy Example
A white Gaussian noise was added to a synthetic im-
age of a disk. The PSNR of the noisy image was 14.9.
This image was denoised without and with transfor-
mation invariant patch similarity (see Section 4.2).
The positive effect of using patch transformations can
be seen on Figure 1. Objectively, the denoised im-
age obtained with transformations has a PSNR of 29.1
while the image obtained without transformations has
a PSNR of 27.7.
5.2 Synthetic Noise
We tested our Patch Denoising Confidence with Ro-
bust Similarity (PDC-RS) algorithm on several im-
ages and compared to BM3D and NL means algo-
rithms. Each image was corrupted with four levels of
AWGN noise. We set BM3D in normal profile with
standard parameters provided by authors. We chose
for both PDC-RS and NL means a patch radius of 7
Figure 1: Effect of the transformations: A toy example.
In lexicographic order: Original, Noisy, Denoised with no
transformations, Denoised with transformations.
Figure 2: PSNR plot for the image Elaine.
and a search radius of 15. Furthermore, as suggested
from authors, we set h = σ, h being the filtering pa-
rameter of NL means and σ the noise standard devia-
tion. Figure 2 shows the Peak Signal to Nose Ratio
(PSNR) as function of the noise standard deviation for
the three compared algorithms. PDC-RS outperforms
NL-means and is close in terms of PSNR to both our
previous method PCkNN (Angelino et al., 2010) and
BM3D. However, as shown in Figure 3, the image
obtained with PDC-RS looks more natural than the
ones obtained with the other algorithms. In partic-
ular, the residual noise is quite natural and does not
exhibit spurious patterns, thus leading to a denoised
image with a very natural appearance. NL-means is
clearly oversmoothed and BM3D presents many flat-
tened regions in smoothly varying areas (see Fig. 4),
IMAGAPP 2011 - International Conference on Imaging Theory and Applications
68

Figure 3: Visual comparison on the image Elaine for
σ = 25. In lexicographic order: Noisy, Original, BM3D,
PCkNN, NL-means, and PDC-RS.
giving a somewhat unnatural, cartoon effect to the de-
noised image. This is the major drawback of this al-
gorithm for high noise levels that might be due to the
thresholding in the wavelet domain. To illustrate the
cartoon effect, Figure 4 shows a close-up on the im-
age Elaine of Figure 3 and the corresponding isolevel
lines. The orientation and density of these lines pro-
vide an indication on the direction and the norm of
the gray level gradient. PDC-RS preserved very well
the original isolevel line configuration while BM3D
created a “patchwork” of flattened regions.
Figure 5 clearly shows how the denoised images
well preserve textures and edges, avoiding any car-
toon effect for all the noise levels.
5.3 Digital Camera Noise
Digital camera noise is very well approximated with
an independent Gaussian stochastic process. How-
ever, contrary to the classical additive white Gaus-
sian noise model with constant variance (used in Sec-
tion 5.2 for comparison purposes), the variance of the
digital camera noise can be modeled as an affine func-
Figure 4: A close-up on the image Elaine of Figure 3. From
top to bottom: Original, BM3D, and PDC-RS. Left: image
alone; right: isolevel lines superimposed on the image.
tion of the signal intensity x: σ
2
(x) = γx + δ. More-
over, most digital cameras acquire images using a
single image sensor overlaid with a color filter array
(CFA) which produces images with a single red, blue,
or green component per pixel. The process of com-
puting the missing 2 color components at each pixel is
called demosaicing. Unfortunately, demosaicing in-
troduces correlation among neighborhood pixels and
hence correlates the noise. The result is a “structured
noise” which is not Gaussian and not independent
anymore. Removing this noise is a harder task since
algorithms usually rely on a hypothesis of indepen-
dence. Thus, we applied it to each color channel of the
raw image, i.e., before demosaicing. Furthermore, we
adapted our denoising algorithm to the varying vari-
ance model by making σ
2
w
in (9) equal to γ ¯x + δ, ¯x
being the average patch intensity. After denoising the
CONFIDENCE-BASED DENOISING RELYING ON A TRANSFORMATION-INVARIANT, ROBUST PATCH
SIMILARITY - Exploring Ways to Improve Patch Synchronous Summation
69

Figure 5: Denoising of the image Elaine using the proposed method, PDC-RS. The image was corrupted by an additive white
Gaussian noise with successive standard deviations σ of 10, 15, 20, and 25. In lexicographic order: Noisy (σ = 25), Original,
denoised image for σ = 10, 15, 20, and 25.
Figure 6: Professional benchmark image (Courtesy of DxO Labs). RAW image (left) and denoised image (right).
image has been demosaiced. Figure 6 shows a result
on a 1024x1024-crop out of a 4416x3312-DxO Labs
benchmark image taken with a Canon G10 at ISO
1600. Again, the denoised image has a very natural
appearance without flattened regions.
6 DISCUSSION
This paper presented PDC-RS a fully patch-based de-
noising algorithm. Denoised patches are provided
with a confidence term which measures the quality of
denoising. The denoised patches are then combined
together based on their respective confidence terms in
IMAGAPP 2011 - International Conference on Imaging Theory and Applications
70

order to reconstruct the denoised image. Moreover,
we introduced a new patch similarity measure invari-
ant to transformation and robust to noise. Experimen-
tal results show the good quality of PDC-RS denoised
images w.r.t. state of the art denoising techniques es-
pecially for high level of noise. In particular, denoised
images have a very natural appearance. Image details
are well preserved and there is no cartoon effect even
in high levels of noise. PDC-RS has also been tested
on a professional DxO Labs benchmark image giv-
ing very good and promising result. As mentioned
in Section 5.3, digital cameras image denoising is a
challenging task since noise variance is function of
the signal.
Concerning the future works, let us mention the
measure used to search for similar patches in the im-
age. Currently, this search is done in the L
2
sense.
Certainly, a better notion of visual similarity such as
the SSIM (Wang et al., 2004) could be used instead.
ACKNOWLEDGEMENTS
The authors would like to thank DxO Labs for provid-
ing the raw test image of Fig. 6 and the demosaicing
software.
REFERENCES
Ahmad, I. A. and Lin, P. (1976). A nonparametric esti-
mation of the entropy for absolutely continuous distri-
butions. IEEE Transactions On Information Theory,
pages 372–375.
Angelino, C. V., Debreuve, E., and Barlaud, M. (2008). Im-
age restoration using a knn-variant of the mean-shift.
In IEEE ICIP, San Diego, CA, USA.
Angelino, C. V., Debreuve, E., and Barlaud, M. (2010).
Patch confidence k-nearest neighbors denoising. In
IEEE ICIP, Hong Kong.
Awate, S. P. and Whitaker, R. T. (2006). Unsupervised,
information-theoretic, adaptive image filtering for im-
age restoration. IEEE Trans. Pattern Anal. Mach. In-
tell., 28(3):364–376.
Boulanger, J., Kervrann, C., and Bouthemy, P. (2007).
Space-time adaptation for patch-based image se-
quence restoration. IEEE Trans. On Pattern Analysis
And Machine Intell., 29:1096–1102.
Brox, T. and Cremers, D. (2007). Iterated nonlocal means
for texture restoration. In Scale Space and Variational
Methods in Computer Vision, First International Con-
ference, SSVM 2007, Ischia, Italy, May 30 - June 2,
2007, Proceedings, pages 13–24.
Buades, A., Coll, B., and Morel, J. (2005). A non-local
algorithm for image denoising. In IEEE CVPR, pages
60–65, Washington, DC, USA.
Carlsson, G., Ishkhanov, T., de Silva, V., and Zomorodian,
A. (2008). On the local behavior of spaces of natural
images. Int. J. Comput. Vision, 76:1–12.
Comaniciu, D. and Meer, P. (2002). Mean shift: A robust
approach toward feature space analysis. IEEE Trans.
Pattern Anal. Mach. Intell., 24:603–619.
Criminisi, A., P
´
erez, P., and Toyama, K. (2004). Region
filling and object removal by exemplar-based image
inpainting. IEEE Trans. on Image Process., 13:1200–
1212.
Dabov, K., Foi, A., K., V., and Egiazarian, K. (2007). Image
denoising by sparse 3d transform-domain collabora-
tive filtering. IEEE Trans. Image Process., 16:2007.
Fukunaga, K. and Hostetler, L. D. (1975). The estimation
of the gradient of a density function, with applications
in pattern recognition. IEEE Trans. On Inf. Theory,
21:32–40.
Huang, J. and Mumford, D. (1999). Statistics of natural
images and models. In IEEE CVPR, pages 541–547.
Lee, A. B., Pedersen, K. S., and Mumford, D. (2003). The
nonlinear statistics of high-contrast patches in natural
images. Int. J. Comput. Vision, 54(1-3):83–103.
Orchard, J., Ebrahimi, M., and Wong, A. (2008). Efficient
nonlocal-means denoising using the svd. In IEEE In-
ternational Conference on Image Processing, pages
1732–1735, San Diego (CA), USA.
Salmon, J. and Strozecki, Y. (2010). From patches to pixels
in non-local methods: Weighted-average reprojection.
In IEEE International Conference on Image Process-
ing, Hong Kong, China.
Sun, W., Peng, Y., and Hwang, W. (2009). Modified similar-
ity metric for non-local means algorithm. Electronics
Letters, 45(25):1307–1309.
Tasdizen, T. (2008). Principal components for non-local
means image denoising. In IEEE International Con-
ference on Image Processing, pages 1728–1731, San
Diego (CA), USA. Electr. & Comput. Eng. Dept.,
Univ. of Utah, UT.
Wang, Z., Bovik, A. C., Sheikh, H. R., and Simoncelli, E. P.
(2004). Image quality assessment: From error visi-
bility to structural similarity. IEEE Trans. Image Pro-
cess., 13:600–612.
CONFIDENCE-BASED DENOISING RELYING ON A TRANSFORMATION-INVARIANT, ROBUST PATCH
SIMILARITY - Exploring Ways to Improve Patch Synchronous Summation
71