外文翻譯-果樹采摘機(jī)器人及控制系統(tǒng)研制【文獻(xiàn)翻譯中英文】,文獻(xiàn)翻譯中英文,外文,翻譯,果樹,采摘,機(jī)器人,控制系統(tǒng),研制,文獻(xiàn),中英文 【中文3880字】 果樹采摘機(jī)器人及控制系統(tǒng)研制 趙德安，呂繼東，紀(jì)偉，張英，陳瑜 江蘇省電氣與信息工程學(xué)院江蘇省鎮(zhèn)江市學(xué)富路301號212013 摘要 機(jī)器設(shè)備組成的操縱者,效應(yīng)器和基于圖像的視覺伺服控制系統(tǒng)是為了采摘蘋果而開發(fā)的。5自由度的機(jī)械手PRRRP結(jié)構(gòu)幾何優(yōu)化提供準(zhǔn)線性行為和簡化控制策略。匙形效應(yīng)器與氣動驅(qū)動爪為了滿足采摘蘋果的要求。蘋果采摘機(jī)器人自主完成收集任務(wù)使用一個應(yīng)用模塊。通過使用支持向量機(jī)與徑向基函數(shù),果實識別算法開發(fā)了檢測并自動查找蘋果在樹上?？刂葡到y(tǒng),包括工業(yè)電腦和交流伺服驅(qū)動程序,進(jìn)行了機(jī)械手和效應(yīng)器融合、將蘋果摘下來的。原型機(jī)器人裝置的有效性經(jīng)實驗室測試和現(xiàn)場實驗的領(lǐng)域。機(jī)器人采摘蘋果的成功率是77%,平均采摘時間大約是15秒/蘋果。 1 序言 在中國,農(nóng)村經(jīng)濟(jì)的快速發(fā)展和水果種植領(lǐng)域結(jié)構(gòu)的不斷調(diào)整,如蘋果、柑橘和梨自1993年以來達(dá)到8 - 9百萬公頃,占四分之一世界上的水果種植總面積。然而,水果收集任務(wù),以50%-70%/小時,仍然依靠體力勞動。所以收獲將自動化,因為農(nóng)業(yè)人口在中國逐漸減少。此外,由于果樹高,采摘工作已經(jīng)將使用梯子,這使得手工收割十分危險和低效。因此,未來需要對蘋果采摘進(jìn)入機(jī)械化和自動化。機(jī)械采摘實驗在一些地區(qū)已經(jīng)進(jìn)行了假設(shè)模擬收獲,但開發(fā)這一戰(zhàn)略還沒有廣泛。有選擇性的采摘,這是必要的,需要復(fù)雜的機(jī)器人技術(shù)?？傊?有必要設(shè)計一個與人類感知能力相似的智能機(jī)器人。以下這個為實例,這臺機(jī)器需要檢測水果、計算水果的位置,然后選擇不破壞果皮或果樹，從而進(jìn)行采摘。 研究水果收獲機(jī)器人發(fā)生在1980年代。河村建夫、Namikawa Fujiura,Ura所言(1984)首次開發(fā)了一個果園機(jī)器人。后來, Rabatel,Pellenc、Journeau Aldon(1987)開發(fā)了一個機(jī)器人。從那時起,他們關(guān)于這方面的知識進(jìn)行了開創(chuàng)性的研究。此外,一些相關(guān)的研究農(nóng)業(yè)機(jī)器人在溫室進(jìn)行了。例如,番茄收獲,黃瓜收獲,櫻桃收獲,草莓收獲。然而,大多數(shù)水果收獲機(jī)器人的文獻(xiàn)目前沒有進(jìn)入生產(chǎn)或銷售。相反,他們?nèi)匀惶幱谘芯堪l(fā)展階段。為此,支持進(jìn)一步的研究和開發(fā)是很重要的,以提高性能和減少這些機(jī)器人的初始安裝費用。 基于上述概念,本研究打算開發(fā)和評估競爭低價設(shè)備自動收獲,即,一個蘋果采摘機(jī)器人。首先,一個完整的機(jī)器人包括組件描述的操縱者,效應(yīng)器和基于圖像的視覺伺服控制的系統(tǒng)描述。其次,機(jī)械手的幾何優(yōu)化以獲得準(zhǔn)線性行為并簡化描述的控制策略。第三,氣動驅(qū)動的結(jié)構(gòu)爪設(shè)計來滿足采摘蘋果的要求。基于這個設(shè)計,采摘機(jī)器人的自動執(zhí)行部分使用一個應(yīng)用模塊來檢測和采集任務(wù)找到蘋果在樹上,控制系統(tǒng)進(jìn)行機(jī)械手和效應(yīng)器方法來摘蘋果。研究采摘機(jī)器人,來有效的驗證實驗室測試和現(xiàn)場實驗測試的領(lǐng)域執(zhí)行。實驗結(jié)果是本文的重要貢獻(xiàn)。 本文組織如下:例如在第二節(jié)中,機(jī)械手,效應(yīng)器和基于圖像的視覺伺服控制系統(tǒng)。在第三節(jié)中討論實驗結(jié)果顯示的可行性機(jī)器人系統(tǒng)的提出。最后,在第四節(jié)中總結(jié)對機(jī)器人未來研究的建議。 2 材料和方法 2．1 蘋果采摘機(jī)器人的機(jī)械結(jié)構(gòu) 蘋果采摘機(jī)器人的原型主要考慮的是模型設(shè)計效率和成本效益。它主要由自主車,5自由度機(jī)械手,效應(yīng)器,傳感器、視覺系統(tǒng)和控制系統(tǒng)組成。自主研發(fā)水果采摘的機(jī)械結(jié)構(gòu)如圖1所示。 1.移動車輛 2.水果籃子 3.活動帶 4.末端效應(yīng)器 5.采集單元 6. 電動絞盤棒 7.運動小手臂 8.大臂 9.運動大手臂 10.活動的腰部 11.腰部 12. 升降平臺 13.中心控制系統(tǒng) 圖1 蘋果采摘機(jī)器人的原理圖 2.1.1 自主移動車輛 車輛的移動方式為履帶式移動。氣動泵供應(yīng)電力，電子硬件數(shù)據(jù)采集和控制,機(jī)械手的效應(yīng)器切水果。全球定位系統(tǒng)(GPS)技術(shù)用于自治導(dǎo)航的移動車輛,其典型的速度1.5m/s。 2.1.2 機(jī)械手 與其他結(jié)構(gòu)相比 ,聯(lián)合結(jié)構(gòu)是有效的三維位置和方向空間。采摘機(jī)器人是一個隨機(jī)的操作空間分布,可能存在很多障礙機(jī)器人。聯(lián)合操縱與自由的多具有任意曲線擬合功能。因此很容易躲開障礙,操作時相應(yīng)的關(guān)節(jié)效應(yīng)器到達(dá)指定的位置。因此,采摘機(jī)器人與五自由度機(jī)械手 (PRRRP)結(jié)構(gòu)安裝在自主移動車輛設(shè)計。第一個自由度是用于提出整個機(jī)械手。Xo、Xc、Yc為攝像機(jī)坐標(biāo)系的軸。L1,L2,L3為機(jī)器人的腰,大臂和手臂。q1、q2為第三關(guān)節(jié)的腰和手臂。水平垂直。U0為平面坐標(biāo)軸,中心坐標(biāo)xg,yg，投影中心用來協(xié)調(diào)目標(biāo),按照目標(biāo)蘋果的圖像特性的差異來進(jìn)行辨別，要采摘的目標(biāo)就會出現(xiàn)在顯示器上。主要的直臂k1,k2武器的控制參數(shù)Dd，根據(jù)運動度的像素單元就會調(diào)整一個合適的角度。交流模擬信號A / D、數(shù)字CCD電荷裝置，D /數(shù)字模擬直流電流。要求自由度GPS全球定位系統(tǒng)目標(biāo)的的顏色,強(qiáng)度、飽和IBVS基于圖像的視覺伺服，PBVS定位視覺伺服PRRRP棱鏡的轉(zhuǎn)動 ，從而進(jìn)行判斷。中間三個自由度的旋轉(zhuǎn),其中,第二個驅(qū)動臂設(shè)計旋轉(zhuǎn)腰部,第三和第四的轉(zhuǎn)動軸移動終端上下操作符。這個自由度允許效應(yīng)器朝任意方向移動。最后,手臂的靈活用于伸長, 機(jī)器人控制命令會根據(jù)實測值的誤差達(dá)到目標(biāo)位置,從而實現(xiàn)蘋果的采摘。上面的討論表明,五自由度機(jī)械手的設(shè)計應(yīng)足以執(zhí)行蘋果采摘的操作。機(jī)械手的機(jī)械結(jié)構(gòu)如圖2所示。機(jī)械手的升降是通過升降平臺來進(jìn)行的,能夠應(yīng)付特殊情況下蘋果的采摘。旋轉(zhuǎn)接頭和靈活的關(guān)節(jié)是由伺服電機(jī)驅(qū)動的。機(jī)器人機(jī)械手的運動參數(shù)和機(jī)械結(jié)構(gòu)如表一所示。 圖2 機(jī)械手的照片 2.1.3 實測值的誤差 效應(yīng)器機(jī)制是由生物學(xué)特性來操作目標(biāo)對象的。蘋果采摘機(jī)器人的操作對象主要是像蘋果一樣的球形。匙形效應(yīng)器(圖3所示)根據(jù)球面生物學(xué)特性而設(shè)計的水果,通過切斷莖來實現(xiàn)的。 效應(yīng)器包含以下部分:鉗子用于掌握水果，電動切割裝置將蘋果從枝椏上切斷。效應(yīng)器的打開和關(guān)閉是由一些適合快速行動的氣動設(shè)備,快速響應(yīng)特征效應(yīng)器的開關(guān)控制。轉(zhuǎn)移模式實現(xiàn)能量傳遞的動力是使用壓縮氣體壓力傳輸。蘋果根莖是安裝在夾具機(jī)制的電動刀切斷的。蘋果被抓住時,電源通過直流電使得爪刀旋轉(zhuǎn),切斷效應(yīng)器感應(yīng)到的目標(biāo)。 2.2 傳感器 非結(jié)構(gòu)性和不確定的操作特性，個體差異和隨機(jī)的操作對象的不同,使得蘋果采摘機(jī)器人應(yīng)該足夠應(yīng)對復(fù)雜的環(huán)境。在水果夾緊過程中,水果的生物學(xué)特性,包括果皮的薄和脆弱需要效應(yīng)器的高度掌控。它要求傳感器來控制高精度的把握力。此外,手臂的旋轉(zhuǎn),目標(biāo)的位置和準(zhǔn)確捕獲還需要經(jīng)過傳感器來檢測和定位。此外,為了避免損壞設(shè)備,導(dǎo)致受傷,未能確定水果、避免手臂的碰撞也需要傳感器有效的感知操作環(huán)境。 2.2.1 效應(yīng)器上的傳感器 傳感器的上的效應(yīng)器,包括攝像頭傳感器、位置傳感器、碰撞傳感器和壓力傳感器。如圖4所示。視覺傳感器,它使用高分辨率的電荷耦合裝置(CCD)攝像機(jī)，采集系統(tǒng)通過串行總線(USB)接口的視頻窗口捕獲技術(shù)形成圖像, 在完成圖像采集、水果搜索和認(rèn)可扮演一個重要角色。獲得廣泛應(yīng)用。視覺傳感器的位置是在一個眼手并用的模式。在圖4中可以看出的兩對紅外雙光電位置傳感器光電電池。此外, 傳感器的開關(guān)位置通常是用來限制安裝在手臂上的電切刀。手臂開始減速時,效應(yīng)器通過視覺傳感器上傳來的采摘對象的圖像。手臂停了下來,鉗子夾水果當(dāng)蓄電池的兩雙都是模糊的。在這一點上,壓力和碰撞傳感器采用力敏感電阻。當(dāng)壓力爪感到一定的壓力傳感器,電動切割器旋轉(zhuǎn)和切斷花梗。開關(guān)位置傳感器便會操作電切刀停止工作。碰撞傳感器用于采摘過程中躲避障礙物。模擬信號來自力敏感電阻和紅外線光電管，在工業(yè)計算機(jī)與數(shù)據(jù)采集模塊通常是不相容的。因此,他們之間需要調(diào)制傳輸之前數(shù)據(jù)采集模擬信號。圖5顯示了傳感器信號調(diào)制電路。 圖3 末端效應(yīng)器的圖片 圖4 末端傳感器的布局 2.2.2 傳感器為避免操作的碰撞 控制旋轉(zhuǎn)關(guān)節(jié)的角度和位置用來控制關(guān)節(jié)的靈活度?；魻杺鞲衅靼惭b在腰的轉(zhuǎn)動關(guān)節(jié),主要負(fù)責(zé)小手臂關(guān)節(jié)的兩端。在工作環(huán)境中,小手臂的運動空間大。其中位置傳感器，壓力傳感器，圖像傳感器，碰撞傳感器圖，效應(yīng)器的傳感器。圖5 為信號調(diào)制電路。 圖5 傳感器信號的調(diào)制電路 2.3 視覺系統(tǒng) 蘋果采摘機(jī)器人視覺系統(tǒng)的關(guān)鍵成分是公認(rèn)的蘋果圖像處理方法。它影響機(jī)器人的可靠性和直接決定的能力,快速、準(zhǔn)確進(jìn)行水果識別。然而,在早期的研究中,存在一些尚未解決的問題，低準(zhǔn)確率和時間消耗等在一定程度上限制了蘋果采摘機(jī)器人在自然環(huán)境中的實時和多任務(wù)處理能力。 為了克服這些缺點,運用識別視覺系統(tǒng)組成的彩色CCD相機(jī)自動獲取原始的蘋果圖像和一個工業(yè)計算機(jī)處理圖像識別定位水果。因為富士蘋果在中國是最受歡迎的,我們研究集中這種多樣性。識別和定位過程如下。 首先,由于自然環(huán)境的形象采集設(shè)備使用,原始的未經(jīng)加工的蘋果圖像不可避免的包含噪聲,影響其質(zhì)量。一個向量中值濾波器應(yīng)用于圖像增強(qiáng)預(yù)處理。它不僅可以有效地去除噪聲突顯出蘋果果實的前景,也能保持良好的圖像邊緣。 其次,大多數(shù)的蘋果采摘在自然圖像條件下通常包括樹枝和樹葉讓問題變得更加復(fù)雜。只通過傳統(tǒng)的形象分割算法,很難達(dá)到預(yù)期的效果?；陬伾狈綀D色調(diào)的統(tǒng)計,強(qiáng)度飽和模型,圖像分割算法用來開發(fā)雙閾值和區(qū)域增長，從復(fù)雜的背景識別蘋果果實。色度組件是適用于提取輕色調(diào)的蘋果,這避免了不同的照明水平對圖像的影響。這種算法簡單,所需的處理時間簡短。 蘋果提取通過不同的特征來確定空間位置,并提供手臂相應(yīng)的運動參數(shù)。對于顏色特征提取,色度組件色調(diào)和飽和度,通常與顏色特征提取識別不同。然而,在我們的研究中,蘋果果實、樹枝和葉有特定形狀,及其不同的形狀是巨大的。因此,蘋果的形狀特性對于對象識別是非常重要。形狀特征的選擇規(guī)則基于旋轉(zhuǎn)不變性和規(guī)模。考慮到蘋果果實圖像的特點,圓形的方差,方差橢圓,圓周邊比率和正方形區(qū)域是用來描述蘋果的形狀特征輪廓。這四個特征向量與形狀特征提取。對應(yīng)的特征值的計算后,他們作為每個樣本的特征向量,用于訓(xùn)練和分類。 最后,基于支持向量的分類算法并建立了辨識蘋果方法。仿真和實驗表明, 基于蘋果的顏色特征和形狀特征，支持向量機(jī)方法和徑向基函數(shù)(RBF)的內(nèi)核是對于蘋果被發(fā)現(xiàn)最好的認(rèn)可。 Chen XueFu Received in revised form Published online 6 August 2011 control strategy. The spoon-shaped end-effector with the pneumatic actuated gripper was requirements for harvesting apple. The harvesting robot autono- ent of hours, still depend on manual labour (Xu Sakai, Osuka, Maekawa, Sarig, 1993; Van Henten, Hemming, Van Tuijl, Kornet, Meuleman, 2002). In addition, several relevant studiesonagriculturalrobotsingreenhouseshavebeencarried * Corresponding author. Tel.: 86 511 82028322; fax: 86 511 82028322. Available at biosystems engineering 110 (2011) 112e122 E-mail address: (J. Wei). Harvesting is expected to be automated because the farming population is gradually decreasing in China. In addition, since the fruit trees are tall, harvesting work has to be con- ducted using step ladders, which makes manual harvesting dangerous and inefficient. Therefore, there is a strong desire Pellenc, Journeau, and Aldon (1987), developed an apple- harvesting robot. Since then, their pioneering studies were followed by many research papers covering several aspects (e.g., ;Edan, Rogozin, Flash, Foglia Hwang Kondo Muscato, and the continuous adjustment of planting structures, fruit cultivation areas, such as apple, citrus and pear, have reached8-9millionhasince1993,accountingforone-quarter of the total fruit cultivation area in the world. However, fruit harvesting tasks, which take 50%e70%ofthetotalworking instance, the machine needs to detect fruit, calculate the position of the fruit and then pick it without damaging the pericarp or the fruit tree. Research on fruit harvesting robots took place in the 1980s. Kawamura, Namikawa, Fujiura, and Ura (1984) first developed 1. Introduction In China, with the rapid developm 1537-5110/$ e see front matter Crown Copyright doi:10.1016/j.biosystemseng.2011.07.005 machine with radial basis function, the fruit recognition algorithm was developed to detect and locate the apple in the trees automatically. The control system, including industrial computer and AC servo driver, conducted the manipulator and the end-effector as it approached and picked the apples. The effectiveness of the prototype robot device was confirmed by laboratory tests and field experiments in an open field. The success rate of appleharvestingwas77%,andtheaverageharvestingtimewasapproximately15sperapple. Crown Copyright 2011 Published by Elsevier Ltd on behalf of IAgrE. All rights reserved. the rural economy harvesting, which is commonly used, requires sophisticated robotic technology. In short, it is necessary to design an intelligent robot withhuman-likeperceptive capabilities. For Accepted 17 July 2011 mouslyperformeditsharvestingtaskusingavision-basedmodule.Byusingasupportvector 4 July 2011 designed to satisfy the Article history: Received 9 February 2011 structurewasgeometricallyoptimisedtoprovidequasi-linearbehaviourandtosimplifythe Research Paper Design and control of an apple Zhao De-An, Lv Jidong, Ji Wei*, Zhang Ying, School of Electrical and Information Engineering, Jiangsu University, article info A robotic device consisting control system was developed journal homepage: www.elsevi 2011 Published by harvesting robot Yu Road No.301, Zhenjiang, Jiangsu Province 212013, PR China of a manipulator, end-effector and image-based vision servo for harvesting apple. The manipulator with 5 DOF PRRRP Elsevier Ltd on behalf of IAgrE. All rights reserved. biosystems engineering 110 (2011) 112e122 113 out; for instance, tomato harvesting (Monta et al., 1998), cucumber harvesting (Van Henten, Van Tuijl, Hemming, Kornet, Bontsema q 2 ;q 3 Joint angles of waist, major arm and minor arm. u, v Image plane coordinates horizontal and vertical axes u o , v o Image centre coordinate x g , y g Projection centre coordinate of target fruit ex, ey The difference of target fruit image feature between x g , y g and u o , v o M C2 N Image plane pixels of video camera jex max j;jey max j Maximum of ex and ey Dq 1 ; Dq 2 ; Dq 3 Jointdeviationanglesofwaist,majorarmand minor arm k 1 , k 2 Control parameters of arms furtherresearchanddevelopmenttoimprovetheperformance and reduce the initial set-up costs of these robots. Based on the concepts above, this study intends to develop and evaluate a competitive low price device for automatic harvesting, i.e., an apple-harvesting robot. Firstly, a detailed description on the components of the robot including the manipulator,theend-effectorandtheimage-basedvisionservo control system is described. Secondly, the geometrically opti- misation of the manipulator to gain a quasi-linear behaviour andsimplifythecontrol strategy isdescribed. Thirdly, theend- effectorwiththepneumaticactuatedgripperdesignedtosatisfy the requirements for harvesting apple is described. Based on this design, the harvesting robot autonomously performs its harvesting task using a vision-based module to detect and locate the apple in the trees, and control system conducts the manipulator and the end-effector to approach and pick apple. To verify the validity of the developed harvesting robot, the laboratory tests and field experiments in an open field were performed. The experimental results are the important contri- bution of this paper. The paper is organised as follows: in section 2 the main components of the robot are presented in detail, i.e., the manipulator, the end-effector and the image-based vision servo control system, respectively; in section 3 the experi- mental results are discussed to show the feasibility of the robot system proposed; finally, in section 4 conclusions are drawn and suggestions for future research are made. 2. Material and methods 2.1. Mechanical structure of apple harvesting robot A prototype model of the apple harvesting robot is designed forbothefficiencyandcosteffectiveness. Itmainlyconsistsof an autonomous vehicle, a 5 degree of freedom (DOF) manip- Dd The angle to adjust for the movement of a pixel with unit of degree per pixel. Abbreviations AC Alternating Current A/D Analog, Digital CCD Charge Coupled Devices D/A Digital, Analog DC Direct Current. DOF Degree of Freedom GPS Global Position System HIS Hue, Intensity, Saturation IBVS Image-Based Vision Servo PBVS Position-Based Vision Servo PRRRP Prismatic Revolute Revolute Revolute Prismatic RBF Radial Basis Function RST Rotation Scale, Translation SVM Support Vector Machine USB Universal Serial Bus VFW Video for Windows ulator, an end-effector, the sensors, the vision system and control system. The mechanical structure of fruit harvesting robot self-developed in this paper is shown in Fig. 1. 2.1.1. The autonomous mobile vehicle A crawler type mobile platform was selected as the mobile vehicle. It carried the power supplies, pneumatic pump, electronic hardware for data acquisition and control, and the manipulator with the end-effector for cutting the fruit. Global position system (GPS) technology was used for autonomous navigation of the mobile vehicle, whose typical speed was 1.5 ms C01 . 2.1.2. The manipulator Compared with other structures, as described in Sakai, Michihisa, Osuka, and Umeda (2008), joint structure is effec- tive for any position and orientation in three-dimensional space. The operation of a harvesting robot is a random large space distribution, where a lot of obstacles may exist around the robot. A joint manipulator with multi-degrees of freedom has an arbitrary curve fitting function. It is therefore easy to avoid obstacles by operating the corresponding joints when the end-effector reaches the object position. Therefore, a harvesting robot manipulator with 5 DOF prismatic- revolute-revolute-revolute-prismatic (PRRRP) structure to be mounted on autonomous mobile vehicle was designed. The first DOF was used for uplifting the whole manipulator. The of biosystems engineering 110 (2011) 112e122114 Fig. 1 e Schematic diagram middle three DOF were for rotation, among which, the second driving arm was designed to rotate around the waist, and the thirdandfourthoneswererotationaxesto movetheterminal operator up and down. This DOF allowed the end-effector to move towards an arbitrary direction in the work space. The fifth,andlast,DOFwasflexibleandusedforelongation,which made the end-effector reach the target location according to the robot control commands, thus achieving the harvesting of fruit (Zhao, Zhao, Zhao, Zhao, Position Sensor Pressure Sensor Vision Sensor Collision Sensor Table 1 e Motion parameters of manipulator mechanical structure. Joint Motion parameters Lift platform 0 me0.8 m Rotation joint of waist C0180 C14 e180 C14 Rotation joint of major arm C080 C14 e80 C14 Rotation joint of minor arm C080 C14 e80 C14 Flexible joint 0 me0.8 m biosystems engineering 110 (2011) 112e122 115 2.2. The sensors The non-structural and uncertain features of the operating environment, and the individual differences and random nature of the operating objects, determines that fruit har- vesting robots should have intelligent sensibility to their complexenvironment(Edanetal.,2000;Zhao,Zhao, Zhao, Zhao, Qiao, Wu, Liu, Zhang, Plebe Zhao, Yang, Mariottini, Oriolo Dd is the angle to be adjusted for the movement of a pixel with unit of degree per pixel. Then, the host computer sent instructions to the flexible jointtospread.Aftertheobjectfruitenteredintothegripperof end-effector, the flexible joint stopped spreading. The gripper wasthenclosedandtheelectricalcuttercutofftheapplestalk. Finally, the flexible joint backed to its initial position. Thereafter, the gripper was opened and fruit slid along the flexible tube into the basket. To achieve continuous picking the above steps were repeated. 2.4.4. System software design AWindowsXPsystemwasemployedasanoperatingplatform for its good stability and security. Visual C 6.0 was selected as programming development tool for the host computer. In the system, multiply tasks needed to be processed simulta- neously. Noting that a single-thread might lead to data communication jams and not guarantee real-time control, a multi-threading event-driven approach was adopted for the program control system software. The main thread was responsible for the management of visualisation control interface, system initialisation; the sub-thread was respon- sibleforcommunicationandsynchronisation.Thesub-thread biosystems engineering 110 (2011) 112e122 119 flexible joint contracted in the minor arm during the process of searching for target fruit. Therefore, the harvesting robot manipulator can be regarded as a three-joint robot manipu- lator,andtherelationshipbetweencameracoordinatesystem and robot coordinate system can be obtained according to geometrical relation shown in Fig. 9. The camera coordinates axes (X c ,Y c ,Z c ) parallel to corresponding axes in robot coor- dinates (X o ,Y o ,Z o ). L 1 , L 2 , L 3 are the lengths of the waist, major arm and minor arm respectively, and q 1 ;q 2 ;q 3 are the joint angles of the second, third and fourth DOF. Apples with radius of 40 mm (average radius of the apples) were considered as research objectives. Their projection was a circle on the image captured by video camera. Perspective projection of a fruit in 3-D space is shown in Fig. 10, and formed in the video camera. Feature information of target fruitin imageplaneis shownin Fig.11. Fora two-dimensional image captured by a video camera, the origin is a point in the upper right corner. Symbols of u and v denote horizontal and vertical axes respectively. The image feature of target fruit is characterize as ex and ey, which are the errors between projection centre coordinate (x g , y g ) and image centre coordi- nate (u o , v o ). During joint control of harvesting robot manip- ulator, image feature of ex varies along with the change of waist joint angle q 1 , and image feature of ey varies along with the change of major arm joint angles q 2 and minor arm joint angles q 3 . It can be seen that the manipulator with 5 DOF PRRRP mechanicalstructurewasgeometricallyoptimisedtosimplify the control strategy, and the control algorithm designed to avoid complicated jacobian operations. At the same time, the vision systems software gave only planar information of the target fruit in our robotic system. The distance information between target fruit and camera was unknown. Hence the manipulatorjacobiancouldnotbedirectlyusedinoursystem. The process of picking target fruit can be presented as follows. Firstly, each module of harvesting robot was ini- tialised, and the manipulator made to approachthe fruit trees at a proper location. Then the video obtained image infor- mation of target fruit, and the recognition and location were obtained by image processing software such that the centroid coordinate x g , y g of target in image and the errors ex and ey obtained by comparison with the image centre coordinate u o and v o . Secondly,therobotwascontrolledtomovewithsmallstep according to the calculated deviations ex and ey, and eventu- ally it drove them to be zero. Assuming that image plane pixels of video camera are M C2 N, then jex max jM=2 and jey max jN=2. The flowchart of the small step transformation algorithm is shown in Fig. 12. When the deviations of the smallstepmovementsofthewaist,majorarmandminorarm were zero, then the centroid of target fruit was coincident with image centre. During the process of eliminating devia- tions ex and ey, each joint angle was required to move. This was calculated according to Eq. (1) Dq 1 exC2Dd Dq 2 k 1 C2eyC2Dd Dq 3 k 2 C2eyC2Dd (1) where Dq 1 ; Dq 2 ; Dq 3 are joint angles of waist, major arm and minor arm respectively; k 1 ,k 2 are the control parameters of Fig. 13 e Main program flowchart of robot harvesting task. 3. Experiment results 3.1. Laboratory tests biosystems engineering 110 (2011) 112e122120 3.1.1. Recognition and Location experiment For the control system of the fruit harvesting robot, live video windowsonthecontrolsoftwareinterfacewasusedtodisplay the real-time process of picking. Target recognition windows In this section, the results of a feasibility study of the system performed throughlaboratorytestsalongwithfieldvalidation arepresented.Thelaboratoryexperimentswereperformedon the prototype operating in simulation working conditions. This stage was helpfulto set up and optimisethe components of our system. Finally, the performance of the harvesting robot was verified in field tests. system involved video capture, motion control, elongation test of flexible joint and extraction test of prism sub-threads. The main program flowchart for the fruit harvesting robot is shown in Fig. 13. Fig. 14 e Recognition and location showed the accuracy of target recognition, where red “” implied image centre and blue “” implied the centroid of the object fruit. The position of object fruit could be easily shown in the images. In the target location windows, the track the centroid of target fruit with regard to image centre during the location process was marked with a blue line. Table 2 e Dynamic images recognition time. Image frame 123456789101 Recognition time(ms) 235 235 315 390 315 310 390 390 310 390 390 Image frame 26 27 28 29 30 31 32 33 34 35 36 Recognition time(ms) 310 390 390 310 315 390 390 315 310 390 390 Image frame 51 52 53 54 55 56 57 58 59 60 61 Recognition time(ms) 315 390 395 310 390 390 310 315 390 390 315 Image frame 76 77 78 79 80 81 82 83 84 85 86 Recognition time(ms) 315 390 310 315 390 395 310 390 390 310 390 Recognition and location test results of object fruit can be seen in Fig. 14. It is obvious that in the figure, accurate recognition and smooth location track made following fruit grabbing possible, which verified that the designed robot has good tracking performance to meet the requirements of accurate real-time recognition and location. During the process of picking operations, video image signals needed to be acquired dynamically and continuously, and handled frame by frame. In the video, the size of one frame of dynamic images was 320 C2 240 pixels. The recogni- tion time for 100 continuous and dynamic images is shown in Table 2. From Table 2, the average recognition time of 100 frame images was 352 ms. From these results, it was concluded that the developed recognition algorithm met the requirements of real-timeoperationandthatthesystemcouldbeusedtoguide a robot manipulator as it approached an apple in real-time. 3.1.2. Harvesting experiments A photograph of the fruit harvesting robot operating during laboratory simulation harvesting tests is shown in Fig. 15. Underlaboratoryconditions,appleswithradiusabout40mm, were hung on fresh branches in different directions. The results in laboratory tests. periodofimageacquisitionwas100ms.100pickingtestswere carried out in 10 different positions. The test results were as follows: successful picking occa- sions 86, and failed occasions 14. Therefore the success rate was 86%. Without regard to the set-up time, the average time of picking one apple is 14.3 s. This was high enough to meet 121314151617181920212232425 310 390 310 235 315 390 390 310 390 390 390 315 390 395 37 38 39 40 41 42 43 44 45 46 47 48 49 50 310 390 390 310 390 390 310 315 390 315 310 390 390 390 62 63 64 65 66 67 68 69 70 71 72 73 74 75 310 390 390 310 390 390 310 390 390 310 315 390 395 310 87 88 89 90 91 92 93 94 95 96 97 98 99 100 390 310 315 390 390 310 315 390 390 315 390 390 315 390 not clamping tightly. After calculation, the mean recognition time for picking was 15.4 s and the picking success rate was biosystems engineering 110 (2011) 112e122 121 requirements of continuous harvesting operations. The main reasons for failure could be attributed to the experimental environment, where the soft foliage and apple vibration during operation resulted in a decrease in precision posi- tioning.Inaddition,occasionallythecuttingknifefailedtocut the apple stalk. 3.2. Field tests To further verify the reliability and adaptability of harvesting robot system, field tests were carried out in the Beijing Changping orchard during October 2009. The recognition result in the orchard is shown in Fig. 16. There 7 apples were well recognised, which indicated that the recognition algorithm could identify apples efficiently. Where apples are behind branches and leaves and apples cover each other, the apples cannot be picked directly. Those without label “”inFig. 16, wouldbe recognised after pickinga certain number of apples. Fig. 15 e Harvesting experiments in laboratory tests. In practice, once an image such as that in Fig. 16 was acquired, the vision system of robot located and picked the apple which had the minimum distance from the image centre of the visible-field of the camera. The vision system of robot located the next target fruit. Continuously picking Fig. 16 e Apple recognition results in an orchard. 77%, which indicates that the prototype machine and control system could be used to carry out the picking operation outdoors. 4. Conclusions and future research A self-developed fruit harvesting robot and its control system was developed. The main components of the robot, i.e., the manipulator, the end-effector and the image-based vision servo control system, have been described in detail. The experiments (shown in Fig. 17) were carried out in an orchard with a complex environment. In 10 min, 39 apples were rec- ognised, of which 30 apples were picked and put into the container successfully. Six apples failed to be picked since their image was blocked by branches. Three were picked but fell down to the ground due to their small size and the gripper Fig. 17 e Harvesting experiments in an orchard. manipulator was geometrically optimised to gain a quasi- linear behaviour and simplify the control strategy, and the end-effector with the pneumatic actuated gripper was designed to satisfy the requirements for the harvesting of apples. The harvesting robot autonomously performed its harvesting task using a vision-based module to detect and localise the apple in the trees, and c