Signal Activity Estimation with Built-in Noise Management in Raw
Digital Images
Angelo Bosco
1
, Davide Giacalone
1
, Arcangelo Bruna
1
, Sebastiano Battiato
2
and Rosetta Rizzo
2
1
STMicroelectronics, AST-Computer Vision Group, Catania, Italy
2
University of Catania, Dept. of Mathematics and Computer Science, Catania, Italy
Keywords: Signal Activity, Bayer Pattern, CFA, Raw, Noise.
Abstract: Discriminating smooth image regions from areas in which significant signal activity occurs is a widely
studied subject and is important in low level image processing as well as computer vision applications. In
this paper we present a novel method for estimating signal activity in an image directly in the CFA (Color
Filter Array) Bayer raw domain. The solution is robust against noise in that it utilizes low level noise
characterization of the image sensor to automatically compensate for high noise levels that contaminate the
image signal.
1 INTRODUCTION
Digital images are usually acquired by means of
image sensors covered by a CFA (Color Filter
Array) which enables sensitivity to only one color
component per pixel, either Red, Green, or Blue;
demosaicing is eventually required to obtain a color
image. Because of the subsampling in the CFA
pattern, thin edges or texture may occupy just a few
pixels in the subsampled lattice, making edges hard
to detect (
Chen, 2006). Discrimination between areas
with signal activity from homogeneous areas can be
difficult especially when the signal to noise ratio is
low; noise may overpower the image signal or it
may have a spatial structure that is similar to texture;
this makes it difficult to discern useful signal from
noise.
In this paper we propose a method that works
directly in the raw CFA domain and exploits the
image sensor noise characterization in order to
robustly compensate for signal degradation caused
by noise. This technique enables early detection of
signal activity in the imaging pipeline, allowing
subsequent algorithms (e.g. demosaicing, noise
filtering) to optimally adapt to the image content.
2 NOISE MODEL
Signal amplification at image sensor level is a blind
process that amplifies both image signal and noise
by means of an analog gain usually expressed in
terms of the ISO setting. The acquired image is
contaminated by various sources of noise that are
usually modeled as zero mean additive white
Gaussian noise; a Poissonian noise component is
also present (
Foi 2007, 2008; Bosco 2010). In general,
the standard deviation of the underlying Gaussian
noise distribution is assumed as a measure of the
noise level. The signal dependent noise model can
be expressed as (1):
,
∙
(1)
where ∊
0,…,2
1
is the recorded signal
intensity; is the image bitdepth and ,∈
.
The coefficients and depend on the sensor gain
(i.e. ISO). As the ISO increases, the and
coefficients generate noise curves with increasing
values.
The a and b coefficients can be determined in an
offline sensor characterization phase repeated by
varying the amplification gain.
3 PROPOSED METHOD
The proposed solution, rather than partitioning
image pixels into flat and non-flat classes, estimates
a measure of flatness. A block diagram of the
proposed solution is illustrated in Figure 1.
118
Bosco A., Giacalone D., Bruna A., Battiato S. and Rizzo R..
Signal Activity Estimation with Built-in Noise Management in Raw Digital Images.
DOI: 10.5220/0004280301180121
In Proceedings of the International Conference on Computer Vision Theory and Applications (VISAPP-2013), pages 118-121
ISBN: 978-989-8565-47-1
Copyright
c
2013 SCITEPRESS (Science and Technology Publications, Lda.)
The processing kernel is split into 4 layers
,…,
;
for each layer, a flatness degree measure
is
estimated, for i=1,..,4. Finally, the Signal Activity
Estimator block computes the Signal Activity
Degree by combining the layers flatness degrees.
Figure 1: A kernel moves across the CFA data. Four
layers are extracted; for each layer, a flatness Degree
(
) is estimated. The Signal Activity Estimator Block
combines the four estimations according to equation 10.
Measuring flatness can be seen as the dual problem
of estimating signal activity; by normalizing the
flatness degree
between 0 (no flatness) and 1
(max flatness), the layer signal activity measure
can be simply expressed as (2):
1
(2)
In the rest of the paper we will calculate the flatness
degree and obtain the corresponding signal activity
by simply applying equation (2).
3.1 Layer Interpolator
The input to the proposed estimator consists of a
CFA raw Bayer image that is pixel-wise processed
by a moving kernel; a flatness or, dually, signal
activity degree is hence assigned to each pixel. The
processing kernel is shown in Figure 2.
Figure 2: Processing Kernel contains pixels from all CFA
channels.
According to Figure 2 we define a kernel
(
), such that 20, 9. From this
kernel, four subkernels are generated; each
subkernel has size , 21≤ and is centered
in
,
. The first subkernel,
, is
obtained by simply subsampling
as indicated
in Figure 2 and Eq. 3:
,
|
,
,…,
2,2
,
,…,
(3)
The elements
,0,…,8) of
with = 3
are shown in Figure 2. The other three subkernels
are generated by interpolating the complementary
CFA channels in the same spatial positions of
. The three interpolated subkernels are all
centered in
≡
,
and, taken together,
they constitute 9 virtual pixels for which all the CFA
information is available. Basically, a trivial
demosaicing is performed in the same spatial
locations of
; for example the complementary
CFA information in
is interpolated by averaging
the nearest pixels of the same color to be
interpolated:
,
,
,
,
,
,
,
for the green-blue, blue, and red CFA channels
respectively. The procedure is similar for the other
non-central pixels. Though other more sophisticated
interpolation choices are possible, simple averaging
provides enough precision for our purposes. The
four subkernels share the same spatial locations,
hence they can be considered as superimposed
“layers”. The usage of the interpolated layers is
necessary because tiny lines may occupy just a few
pixels in the Bayer grid, hence the use of adjacent
pixels to
does not provide satisfactory results.
3.2 Layer Flatness Degree Estimator
The Layer Flatness Degree Estimator block is
illustrated in Figure 3.
Figure 3: The Layer Flatness Degree Estimator Block
processes a layer and produces the flatness degree
estimation for the input layer according to equation 8.
SignalActivityEstimationwithBuilt-inNoiseManagementinRawDigitalImages
119
As the kernel moves pixelwise across the raw image,
four layers
1,…4
are obtained at each
new position of the kernel; the standard deviation of
each layer is computed, i.e. for
∊
,0,…,
1,1,…,4:
(4)
The minimum value of each layer
is computed:
min
,0,…,1
(5)
Then, the noise level associated to the layer
minimum pixel value is retrieved taking into account
the acquisition gain :
,
,
∙
(6)
The values of equation (6) can be stored in a LUT
instead of computing them. The reference noise
level
is the one associated to a flat area
whose value is equal to the minimum pixel value in
the layer, with the given ISO gain ; the minimum
value of the layer is chosen to avoid flatness
overestimation.
The value
is computed as:
,
,
(7)
The value
is ISO dependent by means of the
factor, hence
is implicitly compensated as the
noise in the image increases or decreases. The basic
idea underlying the equation (7) is that if the area is
homogeneous, then the ratio between the standard
deviation of the pixels in the layer (
) and the
minimum noise standard deviation (
) of the
layer, should be close to 1. Ideally, when
exceeds 1, the central pixel of the layer should be
classified as belonging to a non-flat area because the
standard deviation of the pixels in the layer
overpowers the standard deviation of the noise in a
flat area whose value corresponds to the minimum
value in the layer; however, as detailed in paragraph
4, this is an ideal condition because some biasing
effects need to be considered.
In particular, a threshold
must be
defined for each layer to compensate the biasing
factors:
1
∆
1
0
(8)
The term is used to avoid a strong binary
classification; its value can be chosen to define a
linearly fading zone that extends beyond
.
3.3 Signal Activity Estimator
The final flatness degree
associated to the
central pixel
,
of the kernel, is obtained
by summation of the flatness degrees
of all
layers:
∙
(9)
Where
is a weighting coefficient, currently set to
“1” for i=1,2,3,4.
Finally, we obtain the signal activity degree
associated to the central pixel of the kernel
,
as:
4
∙
(10)
4 COMPENSATING THE
BIASING EFFECTS
As described in (Kim, 2005), two areas can have the
same standard deviation but different pixel
arrangements, such that one area contains some
detectable pattern whereas the other area does not;
we mitigate this problem by using the four data
layers. Additionally, the assumption that
is
close to 1 for flat areas is only ideal;
must be
compared with
1, compensating the two
other important biasing factors: the small kernel size
and the layers interpolation process. After
compensating for these biases, the thresholds are
mathematically defined and they do not need to be
changed when noise levels change. In case a
different image sensor is used, only the sensor noise
profile LUT generated during the sensor
characterization process needs to be updated.
5 EXPERIMENTAL RESULTS
Experiments have been performed using a noise
profiled image sensor for mobile devices.
VISAPP2013-InternationalConferenceonComputerVisionTheoryandApplications
120
Figure 4: Scaled output examples. Top: original CFA ISO
100 (left) and ISO 1600 (right). Center: estimated signal
activity maps. Bottom: Two magnified crops of the signal
activity maps.
Figure 4 shows the results of the signal activity
estimator for two CFA images taken at ISO 100 and
ISO 1600. Images are scaled to fit them in the
figure.
Areas with signal activity are shown using
increasing values of grey, whereas flat classified
pixels are shown using low values of grey. It can be
observed how the method works very well in the
detection of the signal activity related to the texture
of the floor tiles, without misclassifying it as noise
(Figure 4 (bottom left)).
6 CONCLUSIONS
A method for estimating the signal activity in the
CFA raw domain has been presented. The solution
works prior any pre-processing algorithm. The
image sensor noise profiling is embedded in the
estimation process; hence the solution is robust in
the presence of noise and does not require tuning
thresholds at different noise levels. Experimental
results show the high robustness of the estimator
also in presence of high levels of noise. Future work
includes further refinements of the solution and
incorporating the method in noise filters and
demosaicing algorithms.
REFERENCES
Chia-Hsiung Chen, Sao-Jie Chen, Pei-Yung Hsiao, 2006,
“Edge Detection on the Bayer Pattern”, Circuits and
Systems. APCCAS 2006. IEEE Asia Pacific
Conference on.
A. Foi, S. Alenius, V. Katkovnik, K.Egiazarian, “Noise
Measurement for Raw-Data of Digital Imaging
Sensors by Automatic Segmentation of Nonuniform
Targets”, IEEE Sensors Journal, Volume 7, Issue: 10,
Oct. 2007.
A. Foi, M. Trimeche, V. Katkovnik , K. Egiazarian, 2008,
“Practical Poissonian-Gaussian noise modeling and
fitting for single-image raw-data”, Image Processing,
IEEE Transactions on, Volume: 17 Issue:10, Oct.
2008
A. Bosco, R. A. Bruna, D. Giacalone, S. Battiato, R.
Rizzo, 2010, “Signal-dependent raw image denoising
using sensor noise characterization via multiple
acquisitions”, Proc. SPIE 7537, 753705
Yeong-Hwa Kim, Jaeheon Lee, “Image feature and noise
detection based on statistical hypothesis tests and their
applications in noise reduction”. Consumer
Electronics, IEEE Transactions on, Vol. 51 Issue: 4,
Nov. 2005.
SignalActivityEstimationwithBuilt-inNoiseManagementinRawDigitalImages
121
