COMPUTER VISION BASED SORTING OF ATLANTIC
SALMON
(SALMO SALAR) ACCORDING TO SIZE AND SHAPE
Ekrem Misimi
2,1
John R. Mathiassen
1
, Ulf Erikson
1
1. SINTEF Fisheries and Aquaculture, N-7465 Trondheim, Norway
Amund Skavhaug
2. Department of Engineering Cybernetics, Norwegian University of Science and Technology, N-7491 Trondheim, Norway
Keywords: Computer vision, feature extraction, fish grading, processing line, Atlantic salmon.
Abstract: Intensive use of manual labour is necessary in the majority of operations in today’s fish processing plants,
incurring high labour costs, and human mistakes in processing, evaluation and assessment. Automatization
of processing line operations is therefore a necessity for faster, low-cost processing. In this paper, we
present a computer vision system for sorting Atlantic salmon according to size and shape. Sorting is done
into two grading classes of salmon: “Production Grade” and “Superior/Ordinary Grade”. Images of salmon
were segmented into binary images, and then feature extraction was performed on the geometrical
parameters to ensure separability between the two grading classes. The classification algorithm was a
threshold type classifier. We show that our computer vision system can be used to evaluate and sort salmon
by shape and deformities in a fast and non-destructive manner. Today, the low-cost of implementing
advanced computer vision solutions makes this a real possibility for replacing manual labour in fish
processing plants.
1 INTRODUCTION
During the last few decades, the number of whitefish
processing plants in Norway has diminished
considerably for several reasons. In aquaculture,
although the production volume of salmonids has
increased tremendously over the same period of
time, most of the fish are currently exported as raw
material, i.e. gutted fresh or frozen. In both sectors,
particularly due to the high labour costs, fish
processing is often unprofitable. For instance, for
slaughtering of farmed salmonids, the needed
manpower is typically 25-40 persons per shift to
process 40-100 tons of bled, gutted fish packed in
ice. Therefore, greater automatization of various unit
operations, preferably at low investment costs,
represents a common strategy within the fish
processing industry today. A fish processing line
consists of several separate unit operations.
Arnarson et al. (1988) reviewed and outlined a
number of possibilities for implementing computer
vision for automation and improving product quality
in the fish processing sector. However, several unit
operations in a fish processing line still rely on, at
least in part, repetitive manual labour. Manual
processing and grading has several drawbacks. It is
influenced by human factors such as mistakes,
occasional omissions in processing and fatigue.
These factors may result in imperfections that
decrease product quality and thereby reduce profit
(Pau and Olafsson, 1991). Therefore, there is a need
for automation of basic processing operations to
obtain faster processing and a more objective and
consistent quality determination (Strachan and
Murray, 1991; Gunnlaugsson, 1997; Brosnan and
Sun, 2002). Here computer vision can contribute to
further improving the quality of fish products. With
the latest developments in camera technology and
the continuous increases in CPU speed, computer
vision technology has become increasingly more
relevant.
265
Misimi E., R. Mathiassen J., Erikson U. and Skavhaug A. (2006).
COMPUTER VISION BASED SORTING OF ATLANTIC SALMON (SALMO SALAR) ACCORDING TO SIZE AND SHAPE.
In Proceedings of the First International Conference on Computer Vision Theory and Applications, pages 265-270
DOI: 10.5220/0001370002650270
Copyright
c
 SciTePress

Today computer vision solutions are easy to
implement with high flexibility and low cost. Until
recently, the cost of high-resolution, high-speed
cameras has been comparatively high. These factors
imply that computer vision can be used effectively
for online processing of fish (Arnason et al., 1988).
The non-destructive nature and the sheer speed at
which the quality of fish can be evaluated and sorted
are other important factors that encourage the use of
computer vision based solutions.
Computer vision has proven successful for online
process control and inspection of food and
agricultural products with applications ranging from
simple automatic visual inspection to more complex
vision control (Gunasekaran, 2001). Strachan and
Murray (1991) describe how they developed a
machine, based on image analysis, for
discriminating mature herring by sex using infrared
light.
Computer vision algorithms for automated
processing of channel catfish (Ictalurus punctatus)
have been developed to detect fish orientation,
identify the head, tail and fins, and to determine
cutting lines for deheading, detailing, and defining
(Jia et al., 1996). Moreover, automated separation
has been developed for several marine fish species
(Wagner et al., 1987; Strachan and Murray, 1991;
Strachan, 1993) and for freshwater species such as
carp (Cyprinus carpio), St. Peter’s fish
(Oreochromis sp.) and grey mullet (Mugil cephalus)
(Zion et al., 1999). Walkott (1996) gives examples
on how shape region features can be used for object
recognition.
When farmed salmonids are slaughtered, the fish
size distribution approximately follows a Gaussian
distribution curve. From a processing point of view,
a uniform fish size is much favored. This has to do
with production planning including issues such as
the correct adjustment of gutting machines, possible
further processing to a certain uniform product (e.g.
fillet) and delivery of chilled gutted fish of a given
weight class to a specific costumer. Another factor is
that a certain fraction of the fish carries different
kinds of blemishes that originate from the farming
period. Sexually mature fish, fish with different
body deformities (‘short tails’ and ‘humpbacks’)
Figure 2: Superior class salmon.
Figure 3: Production class salmon.
(fig. 3) and skin defects (excessive loss of scales,
wounds, etc) all occur. Accordingly, our goals were
to develop computer vision based methods able to
(fig. 1):
(i) reject sexually mature fish and sort/grade
fish with deformities in shape. Such a
sensor system should be placed prior to fish
processing since such fish are not worth
processing.
(ii) grade fish according to these shape
parameters.
Today, salmonids in Norway are graded
according to external quality as follows: ‘Superior’
(no blemishes), ‘Ordinary’ (minor degree of
blemishes), (fig. 2), ‘Production’ (part of the fish
may be used for human consumption) (fig. 3) and
‘Rejected’ (not for human consumption, see (i)).
Humpback
Short tail
Streamlined shape
Long tail
Figure 1: Stages of classifier design.
VISAPP 2006 - IMAGE ANALYSIS
266

2 MATERIALS AND METHODS
2.1 Fish and fish sampling
Atlantic salmon (Salmo salar) from one fish
processing plant were used. Group I: Nine
‘Production Grade’ (weight: 3.58 ± 0.23 kg; length:
50 ± 2 cm; were selected from the slaughter line on
12 Oct 2003. The fish were bled and gutted at the
plant.
Figure 4: Shape parameters for feature extraction.
Group II: Fourteen “Superior/Ordinary Grade” fish
were collected from the same commercial
processing line on 12 Oct 2003. Thus, the fish
(‘Superior Grade’) had been bled and gutted at the
plant. The mean fish weight and length were 4.60 ±
0.4 kg and 59 ± 3 cm, respectively.
2.2 Image Acquisition
The images, intended for feature extraction, were
captured using an image acquisition system for a
digital colour camera (Nikon Coolpix 5000, Japan)
at a resolution of 1600 x 1200 pixels and acquired in
the JPEG format. These were still images. However,
ccommercial industrial full frame digital cameras
with comparable resolution are available at near
real-time speeds (HVDUO-5M, HanVision Co,
Korea). The use of line-scan colour cameras is most
likely preferable in an industrial setting, due to their
high-speed and the fact that fish in most cases are
transported on conveyor belts. The white balance of
the camera was set using the camera’s automatic
white balance. The fish were illuminated using only
one illumination setup. This setup used two parallel
halogen lamps under a white glass board to provide
the necessary illumination, with colour temperature
2900 K. The lamps were placed 30 cm below the
fillet. The images were acquired with a camera
mounted in a stand on a 90˚ angle, 100 cm above the
fillet. Images were processed with Adobe Photoshop
prior to processing with Matlab Development
Environment 7.01 (Mathworks, Natick,
Massachusetts, USA). Images were filtered, scaled
and rotated appropriately in the Matlab Image
Processing toolbox. Images originally had random
orientation, with a different angle to the horizontal
axis. Some images were in the flipped orientation.
By using and writing Matlab functions, all these
images were oriented in the same direction prior to
the feature extraction procedure in Matlab.
2.3 Feature Extraction
The features are derived from the geometry of the
salmon’s shape. Standards that the fish processing
industry uses for classification of “Production
Grade” and “Superior/Ordinary Grade” are also
based on the geometrical parameters of salmon
shape. An inspector at the processing line usually
looks after parameters such as ‘humpback’, ‘short
tail’ and ‘sexual maturity’ when he wants to detect
and grade a “Production Grade” salmon.
“Superior/Ordinary Grade” salmon has a
‘streamlined’ shape and with a ‘long tail’ and
reduced ‘roundness’ compared to “Production
Grade” salmon.
Based on the industrial standards and the
geometrical parameters defining the shape of salmon
(fig. 4), four features were chosen for extraction,
which would allow us to classify the salmon. The
first parameter was the ratio (
lw
R ):
max
W
L
R
t
lw
= (1)
where
t
L is the total length of fish from its nose to
the end of the tail, and
max
W is the maximum width
of fish.
The second parameter was the area ratio (
r
A ):
2
1
A
A
A
r
= (2)
where
1
A
is the area on the front half of the fish and
2
A is the area on the back.
The third parameter was the ratio (
r
W ):
min
max
W
W
r
W =
(3)
COMPUTER VISION BASED SORTING OF ATLANTIC SALMON (SALMO SALAR) ACCORDING TO SIZE AND
SHAPE
267

where
max
W is the maximum width of fish and
min
W is the minimum length of the fish.
The final parameter was the ratio (
tl
R ):
t
l
tl
L
T
R = (4)
where
l
T is the tail length, and
t
L is the total length
of the fish.
In this way we used a total of four features
i
x ,
4,3,2,1=i :
lw
Rx =
1
(5)
r
Ax =
2
(6)
r
Wx =
3
(7)
tl
Rx =
4
(8)
creating the (1x4) feature vector:
[]
4321
,,, xxxxx = (9)
Figure 5: Segmented binary image.
The geometrical parameters in figure 4, which
are used in the feature’s definition, were derived in
Matlab from the segmented binary image of the
salmon (fig. 5). The size of the image was defined
with the pair
() cr, , where
r
is the total number of
rows, and
c
is the total number of columns. The
images were cropped and scaled in Matlab in such a
manner that the first column is the start point of the
nose of the fish, and the last column corresponds to
the end of the tail. Consequently the total length
t
L ,
which is the length from the nose to the end of the
tail, was defined as equal to the total number of
columns in the image:
cL
t
= (10)
The width W of fish is the width of the fish at
any point. In Matlab it was calculated as the number
of pixels equal to one (=1) in the row direction at the
given column position. The maximum width of fish,
and the appropriate column position, where the
maximum width occurred, was defined as:
[
][]
WJW
j
maxarg,
max
=
(11)
The maximum width occurred at the column
position located between the dorsal fin (fig. 4) and
the belly. The minimum width of the fish was
defined in the same fashion, where we ensured that
the searching was done on the back side of the fish,
from column
J to the end of the tail. The minimum
width of a fish, together with the position where it
occurred, was:
[
][]
WKW
j
minarg,
max
=
(12)
In the
r
Ax
=
2
feature,
1
A in figure 4 was
defined as the area of the front half of the fish, from
the head of the fish until the midpoint
J at the dorsal
fin, where the maximum width occurred.
2
A
, on the
other hand, is the area portion of the back half of the
fish from the midpoint position
J from the dorsal
fin to the end of the tail. The reason why the area
ratio was recorded as a feature was that the ratio
aspect analysis indicated that the “Production
Grade” fish was rounder than the
“Superior/Ordinary Grade”. The mean area ratio for
“Production Grade” fish was 1.3 ± 0.183, while for
“Superior/Ordinary” fish the mean area ratio was 0.9
± 0.15.
Tail length
l
T (fig. 4), was defined as the
difference:
esl
TTT
−
=
(13)
where,
s
T was the position calculated as the
beginning of the tail, seen from the tail side of the
fish, and which was calculated as the difference
between the total length of the fish
t
L
and the value
which was 10% of the
t
L .
10
t
ts
L
LT −=
(14)
The point position
e
T was designated as the end
of the tail and was located at the ventral fin.
Calculating this involved using more parameters.
The ventral fin of salmon served as the boundary for
the tail length. After localizing the point
K
, where
the minimum width occurred, the middle
position
m
R was found, which was the row point at
half of the
min
W
. By scanning the binary image from
the midpoint
J to the point
K
we found the point
e
T
where the width of the fish was 50% bigger than
2
min
W
W
half
=
:
VISAPP 2006 - IMAGE ANALYSIS
268

⎥
⎦
⎤
⎢
⎣
⎡
≤≤≥= KjJWWWT
half
j
e
,
2
3
;arg (15)
2.4 Training of the Classifier
A dataset consisting of 23 labeled binary images of
salmon was used to train the classifier. Nine images
of “Production Grade” label salmon and fourteen
“Superior/Ordinary Grade” label salmon were used
for this purpose. Prior to training we had to decide
what type of classifier was most suitable for this
case. By analyzing the adopted criteria for feature
extractions one by one, we determined how good
these criteria were if used as a single classification
criterion.
Using only a single criterion for classification
was ineffective. We could not reliably separate the
“Production Grade” from the “Superior Grade”
salmon. By combining two or more criteria, the
separability between classes was more reliable. By
applying aspect ratio
lw
R in combination with the
area ratio
r
A , the separability of classes improved
(fig. 6). Similar results were obtained with the other
combinations of features.
The decision boundary in figure 6 implied that
a linear classifier might perform the classification
quite well. Therefore, we applied Linear
Discriminant Analysis – LDA to train the classifier
and took into consideration all four features. The
function written in Matlab was based on the Fisher’s
linear discriminant (Theodoridis and Koutroumbas,
2003):
2
2
2
1
2
21
)(
σσ
µµ
+
−
=FDF
(16)
Testing of the classifier’s performance was
done with the Leave One Out (LOO) method
(Theodoridis and Koutroumbas, 2003). Training of
the algorithm was done with N-1 samples and the
test was carried out using the excluded sample. If
1
X and
2
X were the respective data for classes 1-
“Production Grade” and 2-“Superior Grade”, then
the training was done using
[]
)(
1
jXX − and
[]
)(
2
jXX − samples respectively and the test was
carried out with the excluded sample
)( jX .
⎥
⎥
⎥
⎥
⎥
⎥
⎥
⎥
⎦
⎤
⎢
⎢
⎢
⎢
⎢
⎢
⎢
⎢
⎣
⎡
N
j
X
X
X
X
,1
,1
2,1
1,1
......
......
Train with (N-1)
⎥
⎥
⎥
⎥
⎥
⎥
⎥
⎥
⎦
⎤
⎢
⎢
⎢
⎢
⎢
⎢
⎢
⎢
⎣
⎡
+
N
j
X
X
X
X
,1
1,1
2,1
1,1
......
........
Test
Figure 6: Features of aspect ratio
lw
R and area ratio
r
A .
The dark line could serve as a decision boundary for our
classifier. Classification error was lower than when we
used only one feature.
Figure 7: Linear discriminant analysis for training the
algorithm with all four features used.
3 RESULTS AND DISCUSSION
Twenty three salmon of “Production Grade” and
“Superior/Ordinary Grade” were sorted according to
four features. The classification error was three, two
from “Production Grade” and one from
“Superior/Ordinary Grade”. In percent this classifier
has an 87% (20 out of 23) sorting reliability as
estimated using the Leave One Out method.
One of the two “Production Grade” salmons
which are not correctly classified lies further to the
right (fig. 7). From the data log we have from the
day we picked the fish at the processing plant, the
existing ‘outlier’ has neither ‘humpback’ nor ‘short
tail’. It was classified as “Production Grade” salmon
[
]
j
X
,1
COMPUTER VISION BASED SORTING OF ATLANTIC SALMON (SALMO SALAR) ACCORDING TO SIZE AND
SHAPE
269

from the production chief because it had a ‘black
head’. The work presented, carried out in laboratory
conditions, with this classification reliability has to
be repeated with a bigger dataset and repeated in the
working conditions in the fish processing plant. The
work shows a feasibility of sorting one type of fish
into different grading classes based on the standards
specified by the fish processing industry. There are
several problems on which one must focus attention
when doing image acquisition of salmon and
labelling them into grading classes:
1. The illumination/backlighting system has
to be carefully set in order to provide easy
thresholding and segmentation of fish
images.
2. Labelling of salmon, for the training
phase, into grading classes has to be
carried out by experts; otherwise one
might end up with fish having, for
instance, a wrong class label without
satisfying any of the parameters defining
that class.
4 CONCLUSION
A computer vision system and algorithm for sorting
Atlantic salmon into two grading classes is
described. This classification algorithm works with
an estimated sorting reliability of 87%. An improved
version of this system can potentially be used to
substitute manual inspectors in the fish processing
line. Further work is required in acquiring a bigger
dataset and expert help on the correct, unmistakable
labelling of grading classes, before building a
prototype.
ACKNOWLEDGEMENTS
The project was funded by the Research Council of
Norway (NFR project No. 145634/140 –‘Efficient
and economic sustainable fish processing industry’).
REFERENCES
Arnarson, H., Bengoetxea, K. and Pau, L.F. 1988. Vision
applications in the fishing and fish product industries.
International Journal of Pattern Recognition and
Artificial Intelligence 2, 657-671.
Brosnan, T., Sun, W. D., 2002. "Inspection and grading of
agricultural and food products by computer vision
systems--a review." Computers and Electronics in
Agriculture 36(2-3), 193-213.
Gunasekaran, S., 2001. In: Nondestructive Food
Evaluation Techniques to Analyze Properties and
Quality.Marcel Dekker, New York.
Gunnlaugsson, G.A. 1997. Vision technology: intelligent
fish processing systems. In: Seafood from producer to
consumer, integrated approach to quality (Eds. J.B.
Luten, T. Børresen and J. Oehlenschläger), Elsevier
Science B.V., 351-359.
HVDUO-5M, HanVision CO, Korea, Retrieved from
http://www.alt-vision.com/documentation/8208-0004-11-
00%20HVDUO-5M%20PD.pdf
Jia, P., Evans, M.D. and Ghate, S.R. 1996. Catfish feature
identification via computer vision. Transactions of the
ASAE 39,
1223-1931.
Theodoridis, S., Koutroumbas, K., 2003. Pattern
Recognition,Academic Press, San Diego, 2
nd
edition..
Pau, L. F., Olafsson, R., 1991. Fish Quality Control by
Computer Vision. New York, Marcel Dekker, 23-38
.
Strachan, N.J.C. and Murray, C.K. 1991. Image analysis in
the fish and food industries. In: Fish quality control by
computer vision (Eds. L.F. Pau and R. Olafsson), M.
Dekker, New York, 209-223.
Strachan, N.J.C. 1993. Recognition of fish species by
colour and shape. Image Vision and Computing 11, 2-
10.
Wagner, H., Schmidt, U. and Rudek, J.H. 1987. Zur
Artenunterscheidung von Seefischen.
Lebensmittelindustrie 34, 20-23.
Walkott, A. P., 1996. Object recognition using colour,
shape and affine invariant ratios. In BMVC96,
Edinburgh, Scotland, 273-282.
Zion, B. Shklyar, A. and Karplus, I. 1999 Sorting fish by
computer vision. Computers and Electronics in
Agriculture 23, 175-187.
VISAPP 2006 - IMAGE ANALYSIS
270