基于不同拍摄角度多幅图像的果园视觉导航

doi:10.12398/j.issn.2096-7217.2024.04.004

智能化农业装备学报（中英文） ›› 2024, Vol. 5 ›› Issue (4): 51-65.DOI: 10.12398/j.issn.2096-7217.2024.04.004

基于不同拍摄角度多幅图像的果园视觉导航

<a href="http://znhnyzbxb.niam.com.cn/CN/article/advancedSearchResult.do?searchSQL=(((马锃宏[Author]) AND 1[Journal]) AND year[Order])" target="_blank">马锃宏</a>1,2,3(<a href="mailto:mzh2018@zstu.edu.cn"><img src="/images/email.png" border="0" /></a>), <a href="http://znhnyzbxb.niam.com.cn/CN/article/advancedSearchResult.do?searchSQL=(((岳佳文[Author]) AND 1[Journal]) AND year[Order])" target="_blank">岳佳文</a>1,2,3, <a href="http://znhnyzbxb.niam.com.cn/CN/article/advancedSearchResult.do?searchSQL=(((殷程[Author]) AND 1[Journal]) AND year[Order])" target="_blank">殷程</a>1,2,3, <a href="http://znhnyzbxb.niam.com.cn/CN/article/advancedSearchResult.do?searchSQL=(((赵润茂[Author]) AND 1[Journal]) AND year[Order])" target="_blank">赵润茂</a>1,2,3, CHANDA Mulongoti1,2,3,   <a href="http://znhnyzbxb.niam.com.cn/CN/article/advancedSearchResult.do?searchSQL=(((杜小强[Author]) AND 1[Journal]) AND year[Order])" target="_blank">杜小强</a>1,2,3(<a href="mailto:xqiangdu@zstu.edu.cn"><img src="/images/email.png" border="0" /></a>)

^1.浙江理工大学机械工程学院，浙江杭州，310018
^2.浙江省种植装备技术重点实验室，浙江杭州，310018
^3.农业农村部东南丘陵山区农业装备重点实验室（部省共建），浙江杭州，310018

收稿日期:2023-10-25 修回日期:2023-12-29 出版日期:2024-11-15 发布日期:2024-11-15
通讯作者: DU Xiaoqiang， PhD， Professor， research interests： machine vision and agricultural robot， agricultural machinery equipment design. E-mail： xqiangdu@zstu.edu.cn

Visual navigation in orchard based on multiple images at different shooting angles

MA Zenghong¹^,²^,³(), YUE Jiawen¹^,²^,³, YIN Cheng¹^,²^,³, ZHAO Runmao¹^,²^,³, CHANDA Mulongoti¹^,²^,³, DU Xiaoqiang¹^,²^,³()

^1.School of Mechanical Engineering，Zhejiang Sci-Tech University，Hangzhou 310018，China
^2.Key Laboratory of Transplanting Equipment and Technology of Zhejiang Province，Hangzhou 310018，China
^3.Key Laboratory of Agricultural Equipment for Hilly and Mountainous Areas in Southeastern China（Co-construction by Ministry and Province），Ministry of Agriculture and Rural Affairs，Hangzhou 310018，China

Received:2023-10-25 Revised:2023-12-29 Online:2024-11-15 Published:2024-11-15
Contact: DU Xiaoqiang
About author:MA Zenghong， PhD， Associate Professor， research interests： agricultural machinery navigation and unmanned driving. E-mail： mzh2018@zstu.edu.cn
Supported by:
National Key Research and Development Program of China(2022YFD2202103);National Natural Science Foundation of China(31971798);Zhejiang Provincial Key Research & Development Plan(2023C02049);SNJF Science and Technology Collaborative Program of Zhejiang Province(2022SNJF017);Hangzhou Agricultural and Social Development Research Project(202203A03)

摘要/Abstract

摘要： 果园通常地形崎岖，树冠茂密，杂草丛生。由于信号遮挡、多径效应和射频干扰，GNSS难以在果园中进行自主导航。为了实现果园的自主导航，本文提出了一种基于不同拍摄角度下多幅图像的视觉导航方法。一种动态图像捕捉设备是为安装相机而设计的，可以在不同角度拍摄多个图像。首先，将获得的果园图像分为天空和土壤检测阶段。每个图像被转换到HSV空间，并通过中值滤波和形态学处理初步分割为天空、树冠和土壤区域。其次，利用最大连通区域算法提取天空和土壤区域，并利用Canny算子对区域边缘进行检测和滤波。第三，通过对区域坐标点的拟合，提取当前帧数中的导航线。然后使用动态加权滤波算法分别提取土壤和天空检测阶段的导航线，并将天空检测阶段导航线镜像到土壤区域。最后，利用卡尔曼滤波算法对最终导航路径进行融合提取。对200幅图像的测试结果表明，视觉导航路径拟合的准确率为95.5%，单帧图像处理耗时60 ms，满足导航的实时性和鲁棒性要求。在油茶园进行的视觉导航实验表明，当行驶速度为0.6 m/s时，无杂草和杂草环境下视觉导航的最大跟踪偏移量分别为0.14 m和0.24 m，均方根误差分别为30 mm和55 mm。

关键词: 果园, 视觉导航, 多拍摄角度, 区域分割, 卡尔曼滤波器

Abstract: The orchards usually have rough terrain， dense tree canopy and weeds. It is hard to use GNSS for autonomous navigation in orchard due to signal occlusion， multipath effect， and radio frequency interference. To achieve autonomous navigation in orchard， a visual navigation method based on multiple images at different shooting angles is proposed in this paper. A dynamic image capturing device is designed for camera installation and multiple images can be shot at different angles. Firstly， the obtained orchard images are classified into sky and soil detection stage. Each image is transformed to HSV space and initially segmented into sky， canopy and soil regions by median filtering and morphological processing. Secondly， the sky and soil regions are extracted by the maximum connected region algorithm， and the region edges are detected and filtered by the Canny operator. Thirdly， the navigation line in the current frame is extracted by fitting the region coordinate points. Then the dynamic weighted filtering algorithm is used to extract the navigation line for the soil and sky detection stage， respectively， and the navigation line for the sky detection stage is mirrored to the soil region. Finally， the Kalman filter algorithm is used to fuse and extract the final navigation path. The test results on 200 images show that the accuracy of visual navigation path fitting is 95.5%， and single frame image processing costs 60 ms， which meets the real-time and robustness requirements of navigation. The visual navigation experiments in Camellia oleifera orchard show that when the driving speed is 0.6 m/s， the maximum tracking offset of visual navigation in weed-free and weedy environments is 0.14 m and 0.24 m， respectively， and the RMSE is 30 mm and 55 mm， respectively.

Key words: orchard, visual navigation, multiple shooting angles, region segmentation, Kalman filter

中图分类号:

马锃宏, 岳佳文, 殷程, 赵润茂, CHANDA Mulongoti, 杜小强. 基于不同拍摄角度多幅图像的果园视觉导航[J]. 智能化农业装备学报（中英文）, 2024, 5(4): 51-65.

MA Zenghong, YUE Jiawen, YIN Cheng, ZHAO Runmao, CHANDA Mulongoti, DU Xiaoqiang. Visual navigation in orchard based on multiple images at different shooting angles[J]. Journal of Intelligent Agricultural Mechanization, 2024, 5(4): 51-65.

图/表 25

Figure 1 Algorithm flow chart of the orchard navigation path extraction

Figure 2 Dynamic image capturing device1. Monocular industrial camera　2. Camera bracket　3. Rod 4. Shaft　5. Bearing seat　6. Support frame　7. Electric motor 8. Electric motor bracket　9. Coupling　10. Bearing seat 11. Crank　12. Link　13. Angle sensor

Table 1 Technical parameters of MV-CA013-20GC industrial camera

Parameter	Numerical or description
Sensor type	CMOS, global shutter
Pixel size/（ $µ m × µ m$ ）	$4.8 × 4.8$
Target surface size	$1 / 3.6''$
Resolution	1 280×1 024
Maximum frame rate/fps	116
Data interface	GigE
Power/VDC	12
Dimensions/（mm×mm×mm）	29×29×42

Table 1 Technical parameters of MV-CA013-20GC industrial camera

Parameter	Numerical or description
Sensor type	CMOS, global shutter
Pixel size/（ $µ m × µ m$ ）	$4.8 × 4.8$
Target surface size	$1 / 3.6''$
Resolution	1 280×1 024
Maximum frame rate/fps	116
Data interface	GigE
Power/VDC	12
Dimensions/（mm×mm×mm）	29×29×42

Figure 3 Image acquisition at different pitch angles of the camera

Figure 4 Original image

Figure 5 Preliminary scene extraction at sky detection stage

Figure 6 Preliminary image extraction at soil scene detection stage

Figure 7 Image scene segmentation

Figure 8 Region boundary fitting of the images with pitch angles of +60° and -60°

Figure 9 Discretely select the central reference points at sky and soil detection stages

Figure 10 Navigation line extraction results in one cycle

Figure 11 Mirror projection of the navigation line at sky detection stage

Figure 12 Crawler-type fruit collector1. Dynamic image capturing device　2. Industrial camera　3. Upper computer controller　4. STM32 lower computer　5. Steering solenoid valves

Figure 13 Overall framework of the navigation hardware system

Figure 14 Interface of the orchard navigation control software

Figure 15 Comparison results of posteriori estimation slope and original slope parameters

Figure 16 Comparison results of posteriori estimation intercept and original intercept parameters

Figure 17 Navigation path extraction after KF fusion

Figure 18 Navigation line fitting in different weather

Figure 19 Reliability test of the navigation path extraction algorithm Table 2　Results of orchard navigation path extraction

Figure 20 Navigation test scenario

Figure 21 Actual navigation path in the orchard

Figure 22 Inter-row navigation data graph

Figure 23 Navigation accuracy box plot

Figure 24 Navigation accuracy box plot in weedy environment

参考文献 26

1	VERBIEST R， RUYSEN K， VANWALLEGHEM T， et al. Automation and robotics in the cultivation of pome fruit： Where do we stand today？［J］. Journal of Field Robotics， 2020.
2	MARINOUDI V， CLAUS G. SØRENSEN， PEARSON S，et al. Robotics and labour in agriculture. A context consideration ［J］. Biosystems Engineering， 2019， 184： 111-121.
3	FUMIOMI T， WEI Y， CHANGYING L， et al. Applying new technologies to transform blueberry harvesting ［J］. Agronomy， 2017， 7（2）： 33.
4	NAVAS E， ROEMI FERNÁNDEZ， DELIA SEPÚLVEDA， et al. Soft grippers for automatic crop harvesting： A review ［J］. Sensors， 2021， 21（8）： 2689.
5	YANG Q， DU X， WANG Z， et al. A review of core agricultural robot technologies for crop productions ［J］. Computers and Electronics in Agriculture， 2023， 206： 107701.
6	LECHNER W， BAUMANN S. Global navigation satellite systems ［J］. Computers and Electronics in Agriculture， 2000， 25： 67-85.
7	GAN-MOR S， CLARK R L， UPCHURCH B L. Implement lateral position accuracy under RTK-GPS tractor guidance ［J］. Computers and Electronics in Agriculture， 2007， 59： 31-38.
8	LIAO J， WANG Y， YIN J， et al. An integrated navigation method based on an adaptive federal Kalman filter for a rice transplanter ［J］. Transactions of the ASABE， 2021， 64（2）： 389-399.
9	HELLSTRÖM T. Autonomous navigation for forest machines ［J］. Pre-Study Report， Department of Computing Science， Umea University， Sweden， 2002.
10	LI M， IMOU K， WAKABAYASHI K， et al. Review of research on agricultural vehicle autonomous guidance ［J］. International Journal of Agricultural and Biological Engineering， 2009， 2（3）： 1-16.
11	BENGOCHEA-GUEVARA J M， CONESA-MUÑOZ J， ANDÚJAR D， et al. Merge fuzzy visual servoing and GPS-based planning to obtain a proper navigation behavior for a small crop-inspection robot ［J］. Sensors， 2016， 16（3）： 276.
12	OPIYO S， OKINDA C， ZHOU J， et al. Medial axis-based machine-vision system for orchard robot navigation ［J］. Computers and Electronics in Agriculture， 2021， 185： 106153.
13	SHARIFI M， CHEN X Q. A novel vision based row guidance approach for navigation of agricultural mobile robots in orchards ［C］// 2015 6th International Conference on Automation， Robotics and Applications （ICARA）. IEEE， 2015： 251-255.
14	RADCLIFFE J， COX J， BULANON D M. Machine vision for orchard navigation ［J］. Computers in Industry， 2018， 98： 165-171.
15	ZHOU M， XIA J， YANG F， et al. Design and experiment of visual navigated UGV for orchard based on Hough matrix and RANSAC ［J］. International Journal of Agricultural and Biological Engineering， 2021， 14（6）： 176-184.
16	MALAVAZI F B P， GUYONNEAU R， FASQUEL J B， et al. LiDAR-only based navigation algorithm for an autonomous agricultural robot ［J］. Computers and Electronics in Agriculture， 2018， 154： 71-79.
17	BLOK P M， SUH H K， VAN BOHEEMEN K， et al. Autonomous in-row navigation of an orchard robot with a 2D LIDAR scanner and particle filter with a laser-beam model ［J］. Journal of Institute of Control， Robotics and Systems， 2018， 24（8）： 726-735.
18	HAN Z H， LI J， YUAN Y W， et al. Path recognition of orchard visual navigation based on U-Net ［J］. Transactions of the Chinese Society of Agricultural Machinery， 2021， 52（1）： 30-39.
19	BI S， WANG Y H. Inter-line pose estimation and fruit tree location method for orchard robot ［J］. Transactions of the Chinese Society of Agricultural Machinery， 2021， 52（8）： 1-26， 39.
20	YANG Y， ZHANG Y L， MIAO W， et al. Accurate identification and location of corn rhizome based on faster R-CNN ［J］. Transactions of the Chinese Society of Agricultural Machinery， 2018， 49（10）： 46-53.
21	BLOK P M， VAN BOHEEMEN K， VAN EVERT F K， et al. Robot navigation in orchards with localization based on Particle filter and Kalman filter ［J］. Computers and Electronics in Agriculture， 2019， 157： 261-269.
22	BERGERMAN M， MAETA S M， ZHANG J， et al. Robot farmers： Autonomous orchard vehicles help tree fruit production ［J］. IEEE Robotics & Automation Magazine， 2015， 22（1）： 54-63.
23	SHALAL N， LOW T， MCCARTHY C， et al. Orchard mapping and mobile robot localisation using on-board camera and laser scanner data fusion-Part B： Mapping and localisation ［J］. Computers and Electronics in Agriculture， 2015， 119： 267-278.
24	MA Z， YIN C， DU X， et al. Rice row tracking control of crawler tractor based on the satellite and visual integrated navigation ［J］. Computers and Electronics in Agriculture， 2022， 197： 106935.
25	HAN Leng， HE Xiongkui， WANG Changling， et al. Key technologies and equipment for smart orchard construction and prospects ［J］. Smart Agriculture， 2022， 4（3）： 1-11.
26	LIU Limin， HE Xiongkui， LIU Weihong， et al. Autonomous navigation and automatic target spraying robot for orchards ［J］. Smart Agriculture， 2022， 4（3）： 63-74.

[1]	李建平, 李绍波, 杨欣, 张阔, 谢金燕, 刘树腾, 姜尊豪, 张詝, 王朋. 苹果园生产管理智能机械化技术与装备研究进展与展望[J]. 智能化农业装备学报（中英文）, 2024, 5(4): 66-83.
[2]	刘羊, 刘宇洋, 陈江春, 李景彬, 林小卫, 蔺广兆. 标准化果园多功能作业平台关键技术研究现状与趋势[J]. 智能化农业装备学报（中英文）, 2024, 5(1): 31-39.
[3]	金永奎, 薛新宇, 孙竹. 果园水肥一体管控系统设计与应用^*[J]. 智能化农业装备学报（中英文）, 2020, 1(2): 19-25.

基于不同拍摄角度多幅图像的果园视觉导航

Visual navigation in orchard based on multiple images at different shooting angles

RichHTML

PDF

可视化

摘要/Abstract

引用本文

使用本文

图/表 25

参考文献 26

相关文章 3

编辑推荐

Metrics

本文评价