Fusion algorithm in multi-target apple recognition and localization in natural scene

doi:10.14088/j.cnki.issn0439-8114.2022.07.027

Abstract

Abstract: The accurate recognition and three-dimensional positioning of apple picking targets in natural scenes are the key technologies of apple intelligent picking equipment. Integrating YOLOv3 algorithm and binocular vision technology, the samples in a variety of natural scenes were trained through YOLOv3 algorithm, the recognition model was constructed, apple imageswere obtained by binocular vision, the two-dimensional coordinates of the target apple in the image were obtained by using the YOLOv3 model, and then the depth coordinate information was obtained by using the binocular vision parallax principle to realize the three-dimensional spatial positioning of the target apple. The algorithm was applied to apple recognition and location in different natural scenes, and the recognition effect and location accuracy were evaluated. The results showed that the minimum relative error was 0.193% and the maximum relative error was 3.670% under the condition of uneven illumination, shadow on the fruit and mutual occlusion; In the case of insufficient illumination and mutual occlusion at night, the minimum relative error was 0.831% and the maximum relative error was 4.417%; The minimum relative error was 0.176% and the maximum relative error was 4.205% when dew forms reflection on the surface of apple and the fruits block each other; When the light was weak, the shadow was small and there was occlusion, the minimum relative error was 0.168%, and the maximum relative error was 3.776%. The algorithm used in the study only needed an appropriate number of samples to meet the recognition and positioning training in different scenes, and mAP （mean Average Precision） in different scenes was 96.60%. The algorithm has strong stability, can identify the target apples in different application scenarios such as overlap, night and light refraction, and can better meet the accuracy requirements of intelligent robot in recognition and positioning.

Key words: apple intelligent picking, binocular vision, YOLOv3, depth information, algorithm

CLC Number:

S225

CAO Chun-qing, ZHANG Wu-ping, LI Fu-zhong, HAN Ji-wan, YU Ting-xi, LIU Shuai. Fusion algorithm in multi-target apple recognition and localization in natural scene[J]. HUBEI AGRICULTURAL SCIENCES, 2022, 61(7): 145-151.

References

[1] 初广丽,张伟,王延杰,等.基于机器视觉的水果采摘机器人目标识别方法[J].中国农机化学报,2018,39(2):83-88.
[2] 王鑫,赵莹,杨简.基于视觉技术的苹果分拣系统设计[J].中国农机化学报,2014,35(5):169-172.
[3] 车金庆,王帆,吕继东,等.重叠苹果果实的分离识别方法[J].江苏农业学报,2019,35(2):469-475.
[4] 仲崇迪. 苹果采摘机械臂避障系统设计及研究[D].河北秦皇岛:燕山大学,2020.
[5] JI W,ZHAO D,CHENG F,et al.Automatic recognition vision system guided for apple harvesting robot[J]. Computers & electrical engineering, 2012, 38(5):1186-1195.
[6] 夏守城. 基于双目视觉的目标识别与定位及机械臂的抓取研究[D].广州:华南理工大学,2020.
[7] 赵利平,吴德刚.融合GA的三点定位夜间苹果目标的识别算法研究[J].中国农机化学报,2020,41(5):134-138.
[8] JIANG G Q, ZHAO C J. Apple recognition based on machine vision[J/OL].https://ieeexplore.ieee.org/document/6359517,2012-12-26.
[9] GONGAL A, SILWAL A, AMATYA S, et al.Apple crop-load estimation with over-the-row machine vision system[J]. Computers and electronics in agriculture, 2016,120:26-35.
[10] 卢军,桑农.变化光照下树上柑橘目标检测与遮挡轮廓恢复技术[J].农业机械学报,2014,45(4):76-81,60.
[11] 司永胜,乔军,刘刚,等.苹果采摘机器人果实识别与定位方法[J].农业机械学报,2010,41(9):148-153.
[12] 宋怀波,张卫园,张欣欣,等.基于模糊集理论的苹果表面阴影去除方法[J].农业工程学报,2014,30(3):135-141.
[13] 徐越,李盈慧,宋怀波,等.基于Snake模型与角点检测的双果重叠苹果目标分割方法[J].农业工程学报,2015,31(1):196-203.
[14] 吕继东,赵德安,姬伟.苹果采摘机器人目标果实快速跟踪识别方法[J].农业机械学报,2014,45(1):65-72.
[15] 赵德安,吴任迪,刘晓洋,等.基于YOLO深度卷积神经网络的复杂背景下机器人采摘苹果定位[J].农业工程学报,2019, 35(3):164-173.
[16] 汪宋,费树岷. SSD(Single Shot MultiBox Detector)目标检测算法的研究与改进[J].工业控制计算机,2019,32(4):103-105.
[17] LIU W, ANGUELOV D, ERHAN D, et al. SSD: Single shot multiBox Detector[J/OL]. https://link.springer.com/chapter/10.1007/978-3-319-46448-0_2, 2016.
[18] REDMON J,DIVVALA S,GIRSHICK R,et al. You only look once: Unified, real-time object detection[J/OL]. https://ieeexplore.ieee.org/document/7780460, 2016-12-12.
[19] LIU Y F, LU B H, PENG J Y, et al.Research on the use of YOLOv5 object detection algorithm in mask wearing recognition[J]. World scientific research journal,2020,6(11):276-284.
[20] 李振雨,王好臣.基于视觉识别定位的苹果采摘系统研究[J].图学学报,2018,39(3):493-500.
[21] 刘兆祥, 刘刚, 乔军. 苹果采摘机器人三维视觉传感器设计[J]. 农业机械学报, 2010(2):171-175.
[22] 高智伟,谭晓东,刘客.基于双目视觉的物体识别定位与抓取[J].科学技术与工程,2020,20(20):8285-8291.
[23] LIN T Y, DOLLAR P, GIRSHICK R, et al. Feature pyramid networks for object detection[J/OL].https://ieeexplore.ieee.org/document/8099589, 2017-07-26.
[24] 武星,齐泽宇,王龙军,等.基于轻量化YOLOv3卷积神经网络的苹果检测方法[J].农业机械学报,2020,51(8):17-25.
[25] 姚晓莉. 基于双目视觉的人形机器人物体定位与抓取[J].兰州工业学院学报,2020,27(6):67-71.
[26] 夏守城. 基于双目视觉的目标识别与定位及机械臂的抓取研究[D].广州:华南理工大学,2020.
[27] 王纪程,郁汉琪,房家军.基于双目立体视觉的工件识别与定位研究[J].机械制造与自动化,2020,49(6):162-165.