Automatic Soft Gripper Implementation on Foods Using Machine

Vision

Suharyadi Pancono, Wahyu Adhie Candra and Andi Muhammad Aqil

Department of Automation and Mechatronics Engineering, Bandung Polytechnic of Manufacturing, Bandung, Indonesia

Keywords: Deformable Food Handling, Soft Gripper, Pick and Place, Object Identification.

Abstract: The rigid frame of conventional robots makes the food industry still need a lot of human involvement. Soft

gripper is created as one of the solutions for food handling automation system. Various studies have developed

a variety of soft grippers in food pick-and-place process. The gripper with 2 finger configurations showing

better performance compared to the gripper with single finger configuration because it can handle more

diverse object, but the configuration of the fingers is still selected and changed manually. This study proposed

a method to automatically selecting and changing the finger configuration. This study developed 4-finger soft

gripper with a conversion mechanism for 2 finger configurations. The finger configuration is selected based

on the object’s length-to-width ratio that calculated using machine vision. Canny edge detection method is

used to detect edge contours from the object’s image. The results obtained in this study show that the finger

configuration conversion mechanism on the soft gripper made has good repeatability performance, the length-

to-width ratio measurement using machine vision has accuracy up to 97,81% compared to manual

measurement, and food pick-and-place experiment with 100% success rate.

1 INTRODUCTION

Traditional rigid end-effectors are used widely on

automatic production lines, which can perform lots of

repetitive lifting operations. However, it is difficult

for them to handle small, fragile, and deformable

objects due to its bulky and rigid body (Zhong et al.,

2019). Therefore, handling deformable objects such

as foods still need a lot of human involvement. To

reduce labor cost, food handling automation systems

are highly demanded by food industry (Zhongkui

Wang et al., 2016).

Soft grippers replaced rigid body on conventional

robots with a structure made of elastic materials that

deform continuously in response to the interaction

with the objects. Soft grippers are able to

continuously vary their shape without requiring

complex multi-joint mechanisms, have the potential

to provide greater adaptability while presenting lower

costs, simpler structures, and simpler control

algorithms than hard end-effectors. Soft grippers can

be categorized into three groups based on grasping

principle: gripping by actuation, gripping by

controlled stiffness, and gripping by controlled

adhesion. The most suitable group for fruit handling

are the first group because of large lifting capabilities,

good response time, and have the highest techonology

readiness level compared to the other groups (Navas

et al., 2021; Shintake et al., 2018).

Various studies have developed a variety of soft

grippers. The results show that 3 and 4-finger soft

grippers with static finger configuration only capable

to lift up spherical objects (Minh Dang et al., 2021;

Zhangkui Wang et al., 2018; Zhongkui Wang et al.,

2016; Zhongkui Wang, Zhu, et al., 2017). To handle

more diverse object, several studies developed 4-

finger soft gripper with 2 finger configurations

(Zhongkui Wang et al., 2020; Zhongkui Wang,

Torigoe, et al., 2017). Compared to grippers with

static finger configuration, soft gripper with 2 finger

configurations successfully handle spherical and

elongated objects, but the finger configuration on the

mentioned study is still selected and changed

manually.

This study proposed a method to automatically

selecting and changing the finger configuration. This

study developed a 4-finger soft gripper with

conversion mechanism for 2 finger configurations.

The object’s shape tendency is affected by its

elongation factor. Elongation factor (EF) is defined as

the ratio of the longest segment within an object to the

mean length of the perpendicular segment (Patel et

Pancono, S., Candra, W. and Aqil, A.

Automatic Soft Gripper Implementation on Foods Using Machine Vision.

DOI: 10.5220/0011973700003575

In Proceedings of the 5th International Conference on Applied Science and Technology on Engineering Science (iCAST-ES 2022), pages 971-975

ISBN: 978-989-758-619-4; ISSN: 2975-8246

971

al., 2021). The finger configuration is selected based

on the object’s EF that calculated using machine

vision. Before calculating the EF, the edge contours

from the object’s image is traced using Canny edge

detection method.

2 SYSTEM OVERVIEW

Image from the object is captured using webcam. The

webcam send the data to the laptop for digital image

processing to estimate the object’s EF and selecting

the suitable finger configuration. Then the laptop

send the command to the controller based on the

selected finger configuration via serial

communication. Next, the controller sends PWM

signal to the servos and trigger signal to arm robot’s

I/O port. Servo’s shaft rotation will physically

determine finger configuration and actuate the fingers

on the soft gripper. Soft gripper attached to arm robot

and will perform pick-and-place sequence together.

Figure 1: System overview.

3 IMPLEMENTATION

3.1 Soft Gripper with Conversion

Mechanism

Soft gripper in this study is created using 3D Print

method. The body parts are created using PLA

filament with 25% infill and 1,2mm wall thickness.

The finger parts are created using TPU filament with

10% infill and 0,8mm wall thickness. The mechanical

construction of the soft gripper is refer to reference

(Mishra et al., 2017) with modifications.

Figure 2: Soft grippers CAD Model

Rotary base is rotatable to provide 2 finger

configurations as illustrated on figure 3.

(a) (b)

Figure 3: (a) Perpendicular and (b) Parallel finger

configurations on the gripper.

Perpendicular configuration used for spherical

objects while parallel configuration used for

elongated objects. Commonly used finger model for

“gripping by actuation” soft grippers are pneumatics

(Minh Dang et al., 2021; Zhangkui Wang et al., 2018;

Zhongkui Wang et al., 2016; Zhongkui Wang, Zhu, et

al., 2017; Zhong et al., 2019) and tendon-driven

(Gafer et al., 2020; Hussain et al., 2021; Mishra et al.,

2017). Tendon-driven actuation model is used on the

gripper in this study because it needs less component

and easier to produce than the pneumatics model, also

the pneumatics model is susceptible for leakage in the

air chamber (N. Tan et al., 2018; Zaidi et al., 2021).

Two servo motors set on the gripper. The first one

is TowerPro MG90S located on the lower base and

used to rotate the rotary base. Another servo is

TowerPro MG995 located on the upper base and used

to pull the wire for finger actuation.

3.2 Elongation Factor Estimation

Machine vision is used to estimate the elongation

factor (EF) of the object. Spherical object’s EF is

approximately 1 (one). Before the EF calculation, the

edges contour of the object is traced using Canny

Edge Detection method. Canny is used because canny

is the best edge detection method compared to

iCAST-ES 2022 - International Conference on Applied Science and Technology on Engineering Science

972

Prewitt, Roberts, Sobel, and Laplacian of Gaussian

(LOG) edge detection method (Ansari et al., 2017;

Kumar Shah et al., 2020) and reference (Devi et al.,

2017; Luo et al., 2021; Phate et al., 2019; S. H. Tan

et al., 2021) used canny edge detection to detect the

edge contours of the food which is the object in this

study. The program is created on pycharm software

using OpenCV and NumPy library. Below are the

steps of EF estimation in this study.

Figure 4: Steps of EF estimation using machine vision.

The length-to-width ratio of the bounding

rectangle is the EF estimation of the object.

4 RESULTS

4.1 Conversion Mechanism

Repeatability Test

The conversion mechanism of the gripper is tested

with repeatability test. The test is executed by

changing the finger configuration 20 times. The

distance between fingers is measured before and after

the test. The expected result from this test is the

distance difference between fingers before and after

the test is as small as possible that show the

conversion mechanism has good repeatability

performance. Each finger is given number as

illustrated on figure 5.

Figure 5: Gripper's finger numbering.

The distance between finger 1-4 and 2-3 is

measured before and after the test. The results

provided on table 1.

Table 1: Distance between fingers.

Condition

Finger 1–4

Distance

Finger 2-3

Distance

Before

Parallel 21 m

21 m

Perpendi

cula

49 mm 49 mm

After

Parallel 21 m

21 m

Perpendi

cula

49 mm 49 mm

The results show no difference before and after

the test.

4.2 Machine Vision Accuracy Test

The test is excecuted by comparing the object’s EF

estimation by machine vision and manual

measurement. The results provided on table 2.

Table 2: EF estimation using manual measurement and

machine vision comparation.

Test

Elongation Factor

Error

(%)

Manual

Machine

Vision

1 1,04 1,04 0,00

2 1,01 1,01 0,00

3 1,03 1,01 1,94

4 1,04 1,05 0,96

Automatic Soft Gripper Implementation on Foods Using Machine Vision

973

able 2: EF estimation using manual measurement and

achine vision comparation. (cont.)

5 1,01 1,01 0,00

6 1,34 1,3 2,99

7 1,00 1,03 3,00

8 1,82 1,86 2,20

9 1,39 1,32 5,04

10 1,44 1,43 0,69

11 1,00 1,02 2,00

12 1,49 1,47 1,34

13 1,00 1,02 2,00

14 3,49 3,62 3,72

15 1,33 1,3 2,26

16 1,31 1,25 4,58

17 1,01 1,05 3,96

18 1,06 1,04 1,89

19 1,32 1,25 5,30

20 1,28 1,28 0,00

Average Error Rate 2,19

Based on the results, the machine vision has

97,81% accuracy compared to manual measurement.

4.3 Pick-and-Place Test

This test is executed by pick-and-place experiment

using bananas, oranges, round buns, and hot dog buns

5 times for each objects. The gripper attached to

UR5e arm robot.

Figure 6: Work Area.

If the object’s EF estimation is below 1.2,

perpendicular finger configuration will be used. If the

object’s EF is ≥ 1.2, parallel finger configuration will

be used. The representative result for each objects

provided on table 3.

Table 3: Pick-and-place results.

Documentation Status Documentation Status

Success

The results show 100% success rate at pick-and-

place test. Experiments using round buns showing

some damaged part at the top of the buns. Based on

observation during experiments, it is caused by the

size of the buns is bigger than the opening size of the

gripper’s finger.

5 CONCLUSION

The results obtained in this study show that the

conversion mechanism of the finger configurations

on the soft gripper made has good repeatability

performance, the elongation factor estimation using

machine vision has accuracy up to 97,81% compared

to manual measurement, and food pick-and-place

experiment with 100% success rate. For upcoming

research, the opening size of the fingers suggested to

be bigger than the size of the objects to prevent

damage on the upper parts of the object.

ACKNOWLEDGEMENTS

This research is conducted in Automation and

Mechatronics Department’s sensor laboratory at

Bandung Polytechnic of Manufacturing. Thank you

for being allowed to use the laboratory.

iCAST-ES 2022 - International Conference on Applied Science and Technology on Engineering Science

974

REFERENCES

Ansari, Mohd. A., Kurchaniya, D., & Dixit, M. (2017). A

Comprehensive Analysis of Image Edge Detection

Techniques. International Journal of Multimedia and

Ubiquitous Engineering, 12(11), 1–12. doi:

10.14257/ijmue.2017.12.11.01

Devi, T. G., Neelamegam, DR. P., & Sudha, S. (2017).

Image Processing System for Automatic Segmentation

and Yield Prediction of Fruits using OpenCV.

International Conference on Current Trends in

Computer, Electrical, Electronics and Communication.

Gafer, A., Heymans, D., Prattichizzo, D., & Salvietti, G.

(2020). The Quad-Spatula Gripper: A Novel Soft-Rigid

Gripper for Food Handling. 2020 3rd IEEE

International Conference on Soft Robotics (RoboSoft).

doi: 10.1109/RoboSoft48309.2020.9115968

Hussain, I., Malvezzi, M., Gan, D., Iqbal, Z., Seneviratne,

L., Prattichizzo, D., & Renda, F. (2021). Compliant

gripper design, prototyping, and modeling using screw

theory formulation. International Journal of Robotics

Research, 40(1), 55–71. doi: 10.1177/0278364

920947818

Kumar Shah, B., Kedia, V., Raut, R., Ansari, S., & Shroff,

A. (2020). Evaluation and Comparative Study of Edge

Detection Techniques Related papers Evaluation and

Comparative Study of Edge Detection Techniques.

IOSR Journal of Computer Engineering (IOSR-JCE),

22(5), 6–15. doi: 10.9790/0661-2205030615

Luo, L., Liu, W., Lu, Q., Wang, J., Wen, W., Yan, D., &

Tang, Y. (2021). Grape berry detection and size

measurement based on edge image processing and

geometric morphology. Machines, 9(10). doi:

10.3390/machines9100233

Minh Dang, H., Thanh Vo, C., Tran Trong, N., Duc

Nguyen, V., Binh Phung, V., Minh City, C., & Chi

Minh City, H. (2021). Design and development of the

soft robotic gripper used for the food packaging system.

In Journal of Mechanical Engineering Research and

Developments (Vol. 44, Issue 3).

Mishra, A. K., del Dottore, E., Sadeghi, A., Mondini, A., &

Mazzolai, B. (2017). SIMBA: Tendon-driven modular

continuum arm with soft reconfigurable gripper.

Frontiers Robotics AI, 4(FEB). doi: 10.3389/

frobt.2017.00004

Navas, E., Fernández, R., Sepúlveda, D., Armada, M., &

Gonzalez-De-santos, P. (2021). Soft grippers for

automatic crop harvesting: A review. In Sensors (Vol.

21, Issue 8). MDPI AG. doi: 10.3390/s21082689

Patel, K. K., Kar, A., & Khan, M. A. (2021). Rapid

Assessment of Some Physical Parameters of Mangoes

Using Monochrome Computer Vision. Agricultural

Research, 10(3), 468–482. doi: 10.1007/s40003-020-

00517-9

Phate, V. R., Malmathanraj, R., & Palanisamy, P. (2019).

Classification and weighing of sweet lime (Citrus

limetta) for packaging using computer vision system.

Journal of Food Measurement and Characterization,

13(2), 1451–1468. doi: 10.1007/s11694-019-00061-3

Shintake, J., Cacucciolo, V., Floreano, D., & Shea, H.

(2018). Soft Robotic Grippers. In Advanced Materials

(Vol. 30, Issue 29). Wiley-VCH Verlag. doi:

10.1002/adma.201707035

Tan, N., Gu, X., & Ren, H. (2018). Design, characterization

and applications of a novel soft actuator driven by

flexible shafts. Mechanism and Machine Theory, 122,

197–218. doi: 10.1016/j.mechmachtheory.2017.12.021

Tan, S. H., Lam, C. K., Kamarudin, K., Ismail, A. H.,

Rahim, N. A., Azmi, M. S. M., Yahya, W. M. N. W.,

Sneah, G. K., Seng, M. L., Hai, T. P., & Lye, O. T.

(2021). Vision-Based Edge Detection System for Fruit

Recognition. Journal of Physics: Conference Series,

2107(1). doi: 10.1088/1742-6596/2107/1/012066

Wang, Zhangkui, & Hirai, S. (2018). Geometry

and Material Optimization of A Soft Pneumatic

Gripperfor Handling Deformable Object.

International Conference on Robotics and Biomimetics.

Wang, Zhongkui, Chathuranga, D. S., & Hirai, S. (2016).

3D Printed Soft Gripper for Automatic Lunch Box

Packing. International Conference on Robotics and

Biomimetics, 503–508. doi: 10.1109/ROBIO.

2016.7866372

Wang, Zhongkui, Or, K., & Hirai, S. (2020). A dual-mode

soft gripper for food packaging. Robotics and

Autonomous Systems, 125. doi: 10.1016/j.

robot.2020.103427

Wang, Zhongkui, Torigoe, Y., & Hirai, S. (2017). A

Prestressed Soft Gripper: Design, Modeling,

Fabrication, and Tests for Food Handling. IEEE

Robotics and Automation Letters, 2(4), 1909–1916.

doi: 10.1109/LRA.2017.2714141

Wang, Zhongkui, Zhu, M., Kawamura, S., & Hirai, S.

(2017). Comparison of different soft grippers for lunch

box packaging. Robotics and Biomimetics, 4(1). doi:

10.1186/s40638-017-0067-1

Zaidi, S., Maselli, M., Laschi, C., & Cianchetti, M. (2021).

Actuation Technologies for Soft Robot Grippers and

Manipulators: A Review. Current Robotics Reports,

2(3), 355–369. doi: 10.1007/s43154-021-00054-5

Zhong, G., Hou, Y., & Dou, W. (2019). A soft pneumatic

dexterous gripper with convertible grasping modes.

International Journal of Mechanical Sciences, 153–154,

445–456. doi: 10.1016/j.ijmecsci.2019.02.028.

Automatic Soft Gripper Implementation on Foods Using Machine Vision

975