A Low-Cost Process for Plant Motion Magniﬁcation

for Smart Indoor Farming

Danilo Pena

, Parinaz Dehaghani

, Oussama Hadj Abdelkader

, Hadjer Bouzebiba

and A. Pedro Aguiar

Faculty of Engineering, University of Porto, 4200-465 Porto, Portugal

Keywords:

Phase-based Motion Magniﬁcation, Plant Sensing, Non-Invasive Sensing, Plant Monitoring, Small Motions,

Leaf Movements, Eulerian Magniﬁcation.

Abstract:

Smart indoor farming promises to improve the capacity to feed people in urban centers in future production.

Non-invasive sensing and monitoring technologies play a crucial role in enabling such controlled environ-

ments. In this paper, we propose a new architecture to magnify subtle movements of plants in videos, high-

lighting non-perceptible motions that can be used for analyzing and obtaining characteristic traits of plants.

We investigate the limitations of the technique with synthetic and real data and evaluate different plant sam-

ples. Experimental results present leaf movements from short videos that could not be noticed before the

magniﬁcation.

1 INTRODUCTION

Vertical farming and modern greenhouses have

worked to make their production more efﬁcient,

adaptable, and less harmful to the environment. They

cultivate plants using optimized control and improved

monitoring techniques, pursuing the ideal growth

conditions to exploit the resources efﬁciently and in-

crease production yield. The plant observations are

fundamental to provide data for precise control and

management systems for these environments. The

RGB cameras are increasingly contributing to moni-

toring key metrics and traits in a non-invasive manner

in this ﬁeld. Plant movements are visible responses

in the environment to optimize their survival, growth,

and reproduction. Hence, they may reveal helpful in-

formation from the stimulus or genetic factors such

as moving toward the light, growth in search of wa-

ter and nutrients, or sensing changes in the environ-

ment (Bhatla and A. Lal, 2018). Although the plants

can move their position and behavior by sensing ex-

ternal environment changes, these movements are of-

ten prolonged and seldom detectable (Bhatla and A.

https://orcid.org/0000-0003-2767-5764

https://orcid.org/0000-0002-9932-0690

https://orcid.org/0000-0002-6722-4960

https://orcid.org/0000-0002-5915-9441

https://orcid.org/0000-0001-7105-0505

Lal, 2018; Rasti et al., 2021).

Studies have presented tools to assist plant growth,

such as leaf area monitoring (Ngo et al., 2022), water

and nutrient content estimation (Li et al., 2020), and

health assessment (Chouhan et al., 2020). Moreover,

efforts have been shown to facilitate plant evaluations

over time using image processing techniques by ecol-

ogists, farmers, and researchers, such as leaf segmen-

tation (Ghazal et al., 2020), texture and shape selec-

tion (Shah et al., 2017; Siricharoen et al., 2016), and

leaf tracking (Gelard et al., 2018). However, these

plant assessments and measurements are limited to

their time evolution, which usually takes many days to

present relevant changes in common species (Forterre

et al., 2016; Skotheim and Mahadevan, 2005).

One way to evaluate the plant traits in a faster

manner is by amplifying small changes in image ac-

quisitions (Wu et al., 2012). Eulerian motion magniﬁ-

cation is a method to amplify small motions in a set of

frames and is helpful in various applications (Wadhwa

et al., 2013), including but not limited to health care

and monitoring, biology, structural analysis, and me-

chanical engineering. When ampliﬁed, the tiny mo-

tions may reveal information that seems invisible or

imperceptible in normal videos, presenting meaning-

ful temporal variations. This technique requires no

special equipment, and it can be used by regular cam-

eras at usual frame rates. The method was applied

to tiny leaf color changes to detect the photosynthesis

378

Pena, D., Dehaghani, P., Abdelkader, O., Bouzebiba, H. and Aguiar, A.

A Low-Cost Process for Plant Motion Magniﬁcation for Smart Indoor Farming.

DOI: 10.5220/0011717100003417

In Proceedings of the 18th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications (VISIGRAPP 2023) - Volume 4: VISAPP, pages

378-384

ISBN: 978-989-758-634-7; ISSN: 2184-4321

 2023 by SCITEPRESS – Science and Technology Publications, Lda. Under CC license (CC BY-NC-ND 4.0)

process (Taj-Eddin et al., 2017), aiming to monitor

plants.

Motivated by the above, we propose a robust mo-

tion ampliﬁcation method to reveal leaf movements

that are not perceptible by the naked eye. The video

magniﬁcation approach assumes that the object of in-

terest has small motions and background regions re-

main still (Le Ngo and Phan, 2020; Elgharib et al.,

2015). Thus, video stabilization is a crucial step in

this magniﬁcation process to reach accurate results.

We apply video stabilization as a preprocess to re-

move the undesirable large handshake based on a ro-

bust feature trajectories algorithm (Ken-Yi Lee et al.,

2009). Moreover, we utilize leaf extraction and mask

to segment the region of interest (ROI) and apply

magniﬁcation to only this region, achieving higher

motion magniﬁcation quality. We added a plant seg-

mentation algorithm (Riehle et al., 2020) to obtain an

appropriate segmentation for our approach.

The system proposed by this paper can visualize,

with a smartphone camera, without any additional in-

strument, the small leaf movements of the plant in

a non-destructive manner. Such movements may be

associated with the growth rate of the plant, help-

ing researchers and farmers to investigate and design

control systems based on faster metric acquisitions.

We also explore the magniﬁcation bounds and robust-

ness of our approach to provide a measurement tool

for controlled plant environments, where the different

aforementioned plant assessment systems can beneﬁt

from our method.

In summary, the main contributions of this paper

include the proposed architecture system with special

target to magnify small motions of plants using a low-

cost setup, and its robustness and performance eval-

uation. We show that the proposed architecture at-

tenuates considerable distortions related to the back-

ground noise or shaking artifacts, and increases the

magniﬁcation quality when compared with a standard

Eulerian phase-based technique (Wu et al., 2012).

The dataset and its acquisition parameters are pro-

vided in a public repository (Pena et al., 2022).

The paper is organized as follows. In Section 2 the

traditional phase-based motion magniﬁcation is dis-

cussed. Section 3 describes the proposed system and

its architecture, presenting all steps to visualize the

ampliﬁed plant motions. Experimental results are pre-

sented in Section 4, with bound evaluation, and com-

parisons between real and artiﬁcial plants. Finally, in

Section 5, we present our ﬁnal remarks.

2 EULERIAN PHASE-BASED

METHOD

The Eulerian phase-based method (Wadhwa et al.,

2013) is a technique to magnify the small changes in

short videos. The algorithm ampliﬁes the phase dif-

ference value computed between the current and ﬁrst

frames. Initially, the video is decomposed in spatial

frequencies by a complex-valued steerable pyramid.

This pyramid allows us to measure and modify local

motions since its ﬁlters have an impulsive response

with ﬁnite spatial support. Then, the local phase rep-

resented by the spatial scale and orientation of the

steerable pyramid is isolated to a speciﬁc temporal

frequency by using a temporal bandpass ﬁlter. The

frame I

(x,t) of the complex steerable pyramid, ﬁl-

tered for frequency ω, has a local displacement δ(t)

modeled as an intensity of complex sinusoid, given

(x,t) = A

jω(x+δ(t))

. (1)

The temporal ﬁlter removes the DC component

and allows the ampliﬁcation of the displacement δ(t)

by a factor α for each sub-band, producing αδ(t) pix-

els of shift. The sub-band, with the increase in phase

is given by

(x,t) = A

jω(x+(1+α)δ(t))

. (2)

3 PROPOSED METHOD

The proposed method aims to amplify plant move-

ments in a shorter duration than many days of time-

lapse acquisitions. We claim that in short video ac-

quisition, we can highlight small plant movements

for analysis. The architecture is separated into three

steps, described in Figure 1, as: 1) The frames are

stabilized using point feature matching. 2) We mag-

nify the video motion using a modiﬁed phase-based

Eulerian technique. 3) We use a mask based on the

leaf segmentation technique, applying the magniﬁca-

tion only to the ROI during the magniﬁcation.

3.1 Stabilization Process

The stabilization process is essential to avoid artifacts

or ampliﬁcation of the background shaking in the mo-

tion magniﬁcation stage. First, interest points are

identiﬁed in the frame by corner detection using the

features-from-accelerated-segment test (FAST) algo-

rithm (Rosten and Drummond, 2006). Then, we esti-

mate a transformation that corrects the distortion be-

A Low-Cost Process for Plant Motion Magniﬁcation for Smart Indoor Farming

379

Figure 1: Overview of the proposed system.

tween the frames for these points. This afﬁne trans-

form provides the correspondence points of a frame

in the next frame (Ken-Yi Lee et al., 2009). We

compute the matching using the Hamming distance

in a Fast Retina Keypoint (FREAK) descriptor cen-

tered around each point. Finally, we estimate the ge-

ometric transformation between two images using the

M-estimator SAmple Consensus (MSAC) algorithm.

The afﬁne transform that produces the closest inliers

between two sets of points is given by



cos(θ

) −s

sin(θ

) ∆

x,t

sin(θ

) s

cos(θ

) ∆

x,t



, (3)

where ∆

x,t

represents translation motion, s

scale fac-

tor, and θ

indicates the rotation angle. Thus the

cumulative transformation chain relative to the ﬁrst

frame is the product of all preceding inter-frame trans-

forms. The compensated current frame

I(t) is ob-

tained by

∏

t−1

n=0

containing the parameters of the

afﬁne transform of Equation 3.

3.2 Eulerian Phase-Based Motion

Magniﬁcation

We use the Eulerian phase-based technique (Wad-

hwa et al., 2013) to magnify the small changes in

the plant. Due to the high signal-to-noise ratio of

pixel intensities reachable by controlled scenarios and

the magnitude noise efﬁciency of the phase-based

method (Wadhwa et al., 2013), we pay more atten-

tion to any undesirable motion that is not related to

the ROI or even error in video stabilization that can

cause large artifacts in the magniﬁcation. The current

approaches fail to exclude irrelevant motions occur-

ring at frequencies similar to the target motion fre-

quency (Le Ngo and Phan, 2020). Thus, we model

the magniﬁed phase including motions from different

sources, η(t), related to shaking, distortions, and un-

desired movements by representing phase values in

the frames that are not from the ROI motions, given

(x,t) = A

jω(x+(1+α)(δ(t)+η(t)))

. (4)

The distortion η(t) is magniﬁed with the target

motion, producing the magniﬁed image with α(δ(t)+

η(t)) displacement. Hence, the stabilization process

should ensure the removal of irrelevant motions from

the frames as much as possible. In Section 4.3, we

evaluate the inﬂuence of this noise background mo-

tion in the motion magniﬁcation of a synthetic video,

simulating the translation motion not corrected by the

stabilization process.

3.3 Plant Segmentation and Mask

The magniﬁcation is performed only in the ROI of

the frame in order to obtain higher ampliﬁcation fac-

tors and avoid artifacts in the video. We use a mask

based on plant semantic segmentation (Riehle et al.,

2020) to achieve the foreground segmentation from

the frame. Such technique segments the ROI into

two steps: pre-segmentation based on index, and seg-

mentation using color space with a threshold. First,

the method computes the excess green minus excess

red index, an index-based segmentation method com-

monly used in plant applications based on the differ-

ence of another two indexes: the excess green index

(ExG) and the excess red vegetative index (ExR). The

ExG is calculated from an RGB frame by the follow-

ing equation:

ExG = 2G −R − B, (5)

where R, G, and B are normalized RGB values. The

ExR, an index that accentuates the redness of a frame,

is computed by:

ExR = 1.4R −B. (6)

The combination of both indexes, ExG minus

ExR, produces the ExGR, an approach developed to

segment plants from background in images:

ExGR = ExG − ExR. (7)

The mentioned indexes are used as detectors with

different thresholds to produce the mask. Addition-

ally, one can obtain segmentation with different sen-

sitivity and speciﬁcity by using two different color

VISAPP 2023 - 18th International Conference on Computer Vision Theory and Applications

380

spaces (Riehle et al., 2020): HSV or CieLab. Depend-

ing on the dataset and the requirements, the combina-

tion of threshold and color space is chosen to reach

the most representative frames. In our studies, we ﬁnd

the ExGR with a constant zero threshold and CieLab

as most adequate for our acquisitions. However, for

young leaves, we use only the ExGR index with a con-

stant zero threshold, since they were not segmented in

the HSV and CieLab color space histograms.

4 EXPERIMENTAL RESULTS

Indoor experiments were carried out using a regular

camera of a smartphone connected to a ring light tri-

pod, as presented in Figure 2. The ring light has 10

inches in diameter and provides a led power of 10

W with 6500 K of temperature. After acquisition

and video processing using our proposed approach,

we obtained a total of 20 short videos at 30 frames

per second each, as described in Table 1, originally

recorded at 12 seconds per frame, from 3 different

plant samples: Peperomia, Fern, and artiﬁcial. The

dataset and its acquisition parameters are provided in

a public repository (Pena et al., 2022).

Total acquisition:

30 min

Short video:

5 secs

Take

pictures

every 12

seconds

Figure 2: Acquisition setup.

The camera used was from a Samsung smart-

phone with 12 megapixels, 26 mm of focal length,

1.5 of aperture, and optical stabilization. The device

recorded in ﬂight mode while the software acquired

pictures at a uniform and constant sampling frequency

with an automatic and ﬁxed focus on the object.

Unless mentioned otherwise, the evaluations were

performed using videos that present plants with 30

min of recording. We computed the magniﬁcation us-

ing a difference of Butterworths ﬁlter with a quarter-

octave bandwidth pyramid and eight orientations. The

processing was performed in YIQ color space in the

three channels independently, using 150 frames of

640x480. The frequency of the temporal bandpass

ﬁlter was deﬁned as the lowest component possible

since we are interested in slow movements in the video.

Table 1: Parameters used in the acquisition and the magni-

ﬁcation method.

Parameter Values

Frame rate 30 frames per second

Time duration 30 min

Time-lapse 12 seconds per frame

Resolution 640 x 480

Magniﬁcation factor 100

Filter frequencies 0.01 - 0.2 Hz

4.1 Comparisons with Eulerian Method

The small motions from the plants in the frames of-

ten produce distortions in the traditional motion mag-

niﬁcation method. Such slow motions occur at low-

frequency components in time, where also occur mo-

tions related to the background noise or shaking ar-

tifacts. Therefore, we present a comparison between

the Eulerian phase-based magniﬁcation and the pro-

posed architecture with the difference between the

ﬁrst and last frames in the magniﬁed video, illustrated

in Figure 3.

(a) (b)

Figure 3: Comparisons between traditional magniﬁcation

and the proposed method, where: a) it is the ﬁrst frame

of the original video; b) difference of frames for Eulerian

phase-based magniﬁcation; c) difference of frames in the

proposed method without masking; d) difference of frames

in the proposed method with mask enabled.

Figure 3 presents the Eulerian technique magnify-

ing motions that are not related to the plant. The shak-

ing motion can be perceived in the horizontal line in

the background, where even a rigid tripod could not

ensure stability for the magniﬁcation. However, the

proposed architecture attenuates such distortions due

to the stabilization process. Moreover, the masking

increases the magniﬁcation quality by enabling the

A Low-Cost Process for Plant Motion Magniﬁcation for Smart Indoor Farming

381

system to magnify only the ROI.

4.2 Noise Evaluation

A simulated study of the effect of image noise was

performed to assess the robustness of the proposed

technique to different noise conditions, as presented

in Figure 4. Since the magniﬁcation is phase-based,

the noise is not magniﬁed but phase-shifted instead.

Unlike a linear magniﬁcation method, which ampli-

ﬁes the noise linearly, the proposed technique, based

on the phase method, presents a low error increase, as

expected (Wadhwa et al., 2013).

0.01 0.02 0.03 0.04 0.05 0.06 0.07 0.08 0.09 0.10

mean

100

105

110

115

120

MSE

Figure 4: Error evaluation as a function of noise.

4.3 Stabilization Assessment

We evaluate the stabilization process and magniﬁca-

tion of the proposed method using a synthetic video

containing a green circle representing the ROI, and

triangles representing the background. The ROI has

its motion modeled by δ(t) + η(t) shift, where δ(t)

is an oscillating motion, while the background has

only translational η(t) displacement, representing the

distortion. The ROI and background share the same

noise shift value integrated at time t. Figure 5 shows

the ground truth, a shaking motion, and a stabilized

version being magniﬁed.

Figure 6 presents the mean square error (MSE)

image over noise (σ) in η(t). The evaluation shows

that the stabilization produces lower MSE for high

values of phase noise as expected.

4.4 Ampliﬁcation Analysis

For non-distorted video, the quarter-octave bandwidth

steerable pyramid provides approximately 4 periods

of a sinusoid under the Gaussian envelope of the Gar-

bor ﬁlter (Wadhwa et al., 2013). Using the Gaussian

window, with width σ, to determine the bound, we ob-

tain 4σ ≈

on the frequency ω

, and the maximum

displacement is given by

αδ(t) < λ, (8)

Figure 5: Synthetic frames: a) Ground truth; b) Shaking ef-

fect; c) Shaking motion after stabilization; d) Ground truth

magniﬁed; e) Shaking video magniﬁed; f) Stabilized video

magniﬁed.

0.5 1 1.5 2 2.5 3 3.5 4 4.5 5

MSE

Shaking

Stabilized

Figure 6: MSE between ground truth magniﬁed as a refer-

ence, and noise and stabilized videos magniﬁed.

where λ is the spatial wavelength. In an acquisition

with the ROI at 20 cm of distance and a calibration

factor of approximately 0.5 mm/pixel, we can de-

termine the fastest plant movement at 10 seconds or

slower for one pixel of displacement, based on the

physical limitation of the plant speed (Forterre et al.,

2016).

The results of our method indicate fast movements

from young leaves after magniﬁcation that were not

visible in original acquisitions. We use frames col-

lected with a non-uniform background, facilitating for

the stabilization process to track any translation in the

frames. Figure 7 presents the movements from the

growth process of the leaves by computing the differ-

ence between the ﬁrst and last frames. The original

video seems to be a static scene with no movements,

as demonstrated in the supplementary material (Pena

et al., 2022).

4.5 Comparisons with Artiﬁcial Plant

We tested the proposed system using videos contain-

ing simultaneously real and artiﬁcial plants. Fig-

ure 8 presents leaf growth motions from the Peper-

omia, while the artiﬁcial plant stays static. In the

traditional Eulerian method, both plants present large

VISAPP 2023 - 18th International Conference on Computer Vision Theory and Applications

382

Figure 7: Motion highlighted in difference of frames.

motions magniﬁed due to the shaking, while the pro-

posed technique highlights the magniﬁcation on the

young leaf.

Figure 8: Young leaf growth magniﬁed.

5 CONCLUSION

This paper proposes a non-destructive and low-cost

technique to highlight the small movements of plants

in short videos. The technique uses mainly the Eu-

lerian magniﬁcation method to amplify phase differ-

ences with the stabilization process and automatic

segmentation of the ROI. The method was tested in

different samples and presented fast and magniﬁed

movements in short videos. The proposed method can

be used for plant assessment systems in order to ex-

tract hidden metrics.

Future studies may investigate the performance of

the technique over raw data, avoiding post-processing

from the camera’s Image Signal Processor (ISP).

Moreover, the technique can be evaluated in growing

and plant disease monitoring, obtaining in advance

relevant features in such applications.

ACKNOWLEDGEMENTS

The authors acknowledge the partial support of

ARISE Associated Laboratory LA/P/0112/2020, and

the R&D Unit SYSTEC - Base - UIDB/00147/2020

and Programmatic - UIDP/00147/2020 funds, and

also the support of projects SNAP - NORTE-

01-0145-FEDER-000085, RELIABLE - PTDC/EEI-

AUT/3522/2020.

REFERENCES

Bhatla, S. C. and A. Lal, M. (2018). Plant Physiology, De-

velopment and Metabolism. Springer Singapore, Sin-

gapore, 1 edition.

Chouhan, S. S., Singh, U. P., and Jain, S. (2020). Applica-

tions of Computer Vision in Plant Pathology: A Sur-

vey. Archives of Computational Methods in Engineer-

ing, 27(2):611–632.

Elgharib, M. A., Hefeeda, M., Durand, F., and Freeman,

W. T. (2015). Video Magniﬁcation in Presence of

Large Motions. In Proceedings of the IEEE Con-

ference on Computer Vision and Pattern Recognition

(CVPR).

Forterre, Y., Marmottant, P., Quilliet, C., and Noblin, X.

(2016). Physics of rapid movements in plants. Euro-

physics News, 47(1):27–30.

Gelard, W., Herbulot, A., Devy, M., and Casadebaig, P.

(2018). 3D Leaf Tracking for Plant Growth Monitor-

ing. In 2018 25th IEEE International Conference on

Image Processing (ICIP), pages 3663–3667. IEEE.

Ghazal, M., Mahmoud, A., Shalaby, A., Shaker, S., Khe-

liﬁ, A., and El-Baz, A. (2020). Precise Statistical Ap-

proach for Leaf Segmentation. In 2020 IEEE Interna-

tional Conference on Image Processing (ICIP), pages

2985–2989. IEEE.

Ken-Yi Lee, Yung-Yu Chuang, Bing-Yu Chen, and Ming

Ouhyoung (2009). Video stabilization using robust

feature trajectories. In 2009 IEEE 12th International

Conference on Computer Vision, pages 1397–1404.

IEEE.

Le Ngo, A. C. and Phan, R. C. (2020). Seeing the Invisible.

ACM Computing Surveys, 52(6):1–20.

Li, D., Li, C., Yao, Y., Li, M., and Liu, L. (2020). Mod-

ern imaging techniques in plant nutrition analysis: A

review. Computers and Electronics in Agriculture,

174:105459.

Ngo, U. Q., Ngo, D. T., Nguyen, H. T., and Bui, T. D.

(2022). Digital image processing methods for esti-

mating leaf area of cucumber plants. Indonesian Jour-

nal of Electrical Engineering and Computer Science,

25(1):317.

Pena, D., Lobo Pereira, F., and Binandeh, P. (2022).

Plant motion magniﬁcation for indoor farming.

https://doi.org/10.6084/m9.ﬁgshare.19237245.v1.

Rasti, S., Bleakley, C. J., Holden, N., Whetton, R., Lang-

ton, D., and O’Hare, G. (2021). A survey of high res-

olution image processing techniques for cereal crop

A Low-Cost Process for Plant Motion Magniﬁcation for Smart Indoor Farming

383

growth monitoring. Information Processing in Agri-

culture.

Riehle, D., Reiser, D., and Griepentrog, H. W. (2020). Ro-

bust index-based semantic plant/background segmen-

tation for RGB- images. Computers and Electronics

in Agriculture, 169:105201.

Rosten, E. and Drummond, T. (2006). Machine learning

for high-speed corner detection. In Leonardis, A.,

Bischof, H., and Pinz, A., editors, Computer Vision

– ECCV 2006, pages 430–443, Berlin, Heidelberg.

Springer Berlin Heidelberg.

Shah, M. P., Singha, S., and Awate, S. P. (2017). Leaf

classiﬁcation using marginalized shape context and

shape+texture dual-path deep convolutional neural

network. In 2017 IEEE International Conference on

Image Processing (ICIP), pages 860–864. IEEE.

Siricharoen, P., Scotney, B., Morrow, P., and Parr, G.

(2016). Texture and shape attribute selection for plant

disease monitoring in a mobile cloud-based environ-

ment. In 2016 IEEE International Conference on

Image Processing (ICIP), volume 2016-Augus, pages

489–493. IEEE.

Skotheim, J. M. and Mahadevan, L. (2005). Physical limits

and design principles for plant and fungal movements.

Science, 308(5726):1308.

Taj-Eddin, I. A. T. F., Aﬁﬁ, M., Korashy, M., Ahmed, A. H.,

Cheng, N. Y., Hernandez, E., and Abdel-Latif, S. M.

(2017). Can we see photosynthesis? Magnifying the

tiny color changes of plant green leaves using Eulerian

video magniﬁcation. Journal of Electronic Imaging,

26(06):1.

Wadhwa, N., Rubinstein, M., Durand, F., and Freeman,

W. T. (2013). Phase-based video motion processing.

ACM Transactions on Graphics, 32(4):1–10.

Wu, H. Y., Rubinstein, M., Shih, E., Guttag, J., Durand, F.,

and Freeman, W. (2012). Eulerian video magniﬁca-

tion for revealing subtle changes in the world. ACM

Transactions on Graphics, 31(4).

VISAPP 2023 - 18th International Conference on Computer Vision Theory and Applications

384