Machine Vision for a Micro Weeding Robot in a

机器人、控制、系统

BiosystemsEngineering(2003)85(4),393–404

doi:10.1016/S1537-5110(03)00078-3

AE}AutomationandEmergingTechnologies

Available online at http://wendang.chazidian.com

MachineVisionforaMicroWeedingRobotinaPaddyField

B.Chen1;S.Tojo2;K.Watanabe2

1

DepartmentofEngineeringCollege,ChinaAgriculturalUniversity,P.O.Box50,QinghuaDonglu#17,HaidianDistrict,Beijing100083,China;

e-mailofcorrespondingauthor:chenbingqi@http://wendang.chazidian.com

2

DepartmentofEnvironmentalandAgriculturalEngineering,TokyoUniversityofAgricultureandTechnology,3-5-8,Saiwai-cho,Fuchu,

Tokyo,183-8509Japan;e-mail:tojo@cc.tuat.ac.jp

(Received24December2001;acceptedinrevisedform9April2003;publishedonline7June2003)

Thestudydevelopedanimageprocessingmethodtodeterminethetraveldirectrixforamicroweedingrobotinapaddy eld.Theprocessingsamples,whichincluded24imagesineachsample,wereselectedfromthevideotapeswhichwereobtainedonaweeklybasis,whenthecameramanuallyrevolved3608eighttimesateachheightandanglepositioninthespacebetweenthericerowsfromthethirddayaftertransplantingto7weeks.Thetargetimagewasabstractedfromthe24imagesbyanalysingthedistributionofbluepixelsinthecolourimageandthedistributionofblackpixelsinitsbinaryimage.Thecandidatepointsforthedirectrixlineinthetargetimagewereobtainedbyanalysingeachhorizontallinepro leofitsbinaryimageortheblueimageaccordingtothelevelofgrowthinthepaddy eld.Thetraveldirectrixlinewasdetectedbypassingaknown-pointHoughtransform.Thedirectrixlinesofall69samplesusedinthestudywerecorrectlydetected.Theprocessingtimesrequiredtodrawthedirectrixlineafterreadingthe24imageswereabout0Á4–0Á8s.

#2003SilsoeResearchInstitute.AllrightsreservedPublishedbyElsevierScienceLtd

1.Introduction

RiceisthemainfoodoftheJapanese.Soricefarminghasanimportantsigni canceinJapan.Theagriculturalmachinesforricefarming,including eldcultivating,ricetransplanting,ricegrowthmanagementandriceharvesting,areapproachingcurrentdesignoptimums.Futureresearchisnowaimedatimprovedautomation,e.g.theuseofrobots.Developmentprojectshaverecentlyproducedvariousinnovations:Yukumotoetal.(1998a,1998b,1998c)developedatillingrobotguidedbyapositionsensingsystem(PSS)andageomagneticdirectrixsensor(GDS),Nagasakaetal.(1999)developedanautonomoustransplanterusingareal-timekinematicglobalpositionsystem(RTKGPS),Chosaetal.(2000)developedanautomaticpaddy eldmanagementmachine,whichtravelsalongapreparedguidancecable,Iidaetal.(1999a,1999b)andMatsuuraetal.(2001)developedanautonomousfollow-upvehicleguidancesystemsothatoneortwoharvesterswereabletofollowamanuallydrivencombineharvester,auto-matically.

Inpreviousstudies,ariceseedlingrowandpaddy eldridgedetectionsystem(RRDS),whichguidedan

1537-5110/03/$30.00

automaticricetransplanter(Chenetal.,1997,1998,1999a,1999b;Watanabeetal.,1997),andatravelroutedetectionmethod(TRDM),whichguidedanautomatedmanagementmachineinpaddy elds(Chenetal.,2002),weredevelopedusingimageprocessing.Forreducingtheenvironmentalimpactfromherbicideutilisation,theresearchaboutweeddetectionbyimageprocessinginthe eldincreasedinrecentyears(Burksetal.,2000;Ei-Fakietal.,2000;Tianetal.,1999).Thesestudieswereaimedatsprayingtheherbicideondetectedweedorcropareastoreducetheherbicideutilisation.However,ifamicrorobotcouldbedevelopedtotravelalongthecroprows,thenweedingcouldbecompletedbymechanicalmeansratherthanwithherbicides.Somestudiesonamicroweedingrobotinapaddy eldhavebeenreported(Sekietal.,1998,1999,2000;Hayashietal.,2000).However,thesestudiesfocusedontheeffectofweedinganddidnotreportaneffectiveguidancemethod.

Otherthanimageprocessing,alaserlightsensor,ultrasonicsensor,orinfraredsensorcanbeusedtosensethedistancefromthesensortoanobject.Amongthesesensors,theinfraredsensoristheleastexpensive(aboutUS$2persensor)andcanbeusedforneardistance

393

#2003SilsoeResearchInstitute.Allrightsreserved

PublishedbyElsevierScienceLtd

机器人、控制、系统

394

B.CHENETAL.

sensing,therebymakingitsuitableforamicroweedingrobottravellingbetweenthericerows,withaninter-rowspacingof30cm,butitispoorinsunlight.Theothersensors,effectiveoveradistanceof20cm,areunsuitableforusebetweenrows.Further,theultrasonicsensorisnoteffectiveforusewithsmallriceseedlings.Therefore,thisstudyisaimedatdevelopinganimageprocessingmethodtoguideamicroweedingrobotinapaddy eld.Furthermore,the eldtrialsforthisstudyarecarriedoutwithplantsatlatergrowthstagesthanintheRRDSandTRDMstudies.Therobottravelsintothetunnelbetweenthericerows,andtheimageswerecompletelydifferentfromthoseofpreviousstudieswhentherobotsensorrevolved.Owingtotheagilityofthesmallbodyoftherobot,theorientationoftherobotcouldbechangedeasilytoreceivedifferentimagesintheroboteye.Therefore,itnecessarytoredeterminethetraveldirectrixafterstoppingforweedingwork.

Withtheseconditionsinmind,theobjectivesofthepresentstudyweretodevelopanimageprocessingdetectionalgorithmfordeterminingthetraveldirectrix

whenthemicroweedingrobotisrevolvinginthepaddy eld.Thealgorithmhadtobeeffectiveinvariousorientationsofthemicroweedingrobotapplicabletodrivingduringtheentiregrowingperiodfromtheplantingofriceseedlingstotheharvesting.

2.Equipmentandsampling2.1.Equipment

Todevelopamachinevisionsystemthatdoesnotneedcostlyequipment,aninexpensivemacrocameraset,aTR-89CwirelesssurveillancecameraandaTR-801Creceiver,withatotalcostofaboutUS$350inJapan,wereusedtosamplethevideoimagesinthepaddy eld.Atelevisionsetwasusedtoreceivethesignalviathereceiver,andstorethevideoimagesinvideotape.Thecamerahada63mmmetricchargecoupleddevice(CCD)imagesensorwith250000pixelsineffect.TheimagesignalinNationalTelevisionSystemsCommittee(NTSC)formatoutputtedbyaterminalwithavoltage

机器人、控制、系统

MACHINEVISIONFORMICROWEEDINGROBOT

395

of1Vandaresistanceof75O.Thefocallengthofthecameralenswas3Á7mmandtheaperturesettingwasf2Á0.Thelowestintensityoftheimagedobjectwas2l.Thefrequencyofthetransmittedimagewas2Á4GHz.Thecamerasizeswere32mminwidth,27mminheight,and68mminlength(CoronaElectronic,Inc.).

TheimagesforprocessingwerecapturedusingaFDM-PCIIIimagedigitiserboardprovidedbyPho-tron,Inc.Imageprocessingwasperformedusingapersonalcomputerwithapentium400MHzcentralprocessingunit(CPU).Processingsoftwarewasdevel-opedusingMicrosoftVisualC++6.0.2.2.Imageprocessingsampling

Transplantingofthericeseedlingswasperformedon13June2001,attheexperimentalfarmofTokyoUniversityofAgricultureandTechnology.Samplingoperationswereperformedfrom16Juneto4August2001onaweeklybasis,givingatotalofeightsampleperiods.Thecamerawaspositionedapproximately10,20,and30cmabovethesurfaceofthewaterandwaspositionedapproximatelyhorizontal,facing10and208downwardfromthehorizontalateachheightposition.Videotapesampleswereobtainedwhenthecamerawasmanuallyrevolved3608ateachheightandangleposition(Fig.1).Duringthesamplingperiod,thericeheightwasapproximately10–65cmabovethewatersurface.

Atotalof69processingsampleswereobtained.Eachprocessingsamplehad24images,whichwerecapturedfromthevideotapesamplesapproximatelyatevenintervalsandsavedinvideo leformat.Twoparametersoftheimageinputboard,brightnessandcontrast,werebothestablishedasthemediumvaluewhilesamplingthe

Fig.1.Paddy eldandcamera

images.Theimagesizewas512by480pixels(seeFig.4).Theprocessingsamplesweregivenanidenti cationcode:Smn(S,samplenumber;m,weeknumber(0–7);n,videotapenumber(1–9))(Table1).

3.Detectingalgorithm

Avideo le,whichincluded24images,wasreadinto24framesofthecomputermemorytopreparethedetectingprocessing.Whentheprocessingwasstarted,theparameters,suchasprocessingframeF,maximumblueparametervalueBpm,max,maximumobjectnumberNO,max,maximumarearate1AR1,max,maximumarearate2AR2,max,judgedframeFJandcandidateframe1FC1,wereinitialisedasF51,Bpm,max50,NO,max50,AR1,max50,AR2,max50,FJ5À1andFC15À1,respec-tively(Fig.2).Theimageswerethenprocessedfromframe1toframe24,framebyframe,todeterminetheimagewherethespacebetweenthericerowswasnearestthecentreoftheimage(calledthetargetimage).Afterprocessingtodeterminethetargetimage,candidatepointsoftraveldirectrixweredetectedoneachhorizontallineinthetargetimage.Thetraveldirectrixwasthendetectedbythemethodcalled‘passingaknownpointHoughtransform(PKPHT)’(Chenetal.,1997),whichestablishedaknownpoint,andthetraveldirectrixisassumedfromthemaximumnumberofcollineardetectedcandidatepointspassingtheknownpoint.3.1.Determiningtargetimage

3.1.1.Makingthebluevaluedistributionandcalculatingtheblueparameter

Thetargetimagewasthatofthespacebetweenthericerowsnearestthecentreoftheimageinthefullimagescapturedwhenthecamerawasrevolving.Thus,thecentreofthebluedistributioninthetargetimagewasnearestthecentreoftheimageinthefullimages.Therefore,ablueparameterBpmcouldbeestablishedtojudgethetargetimage.

Whenanimagewasprocessed, rst,foreachverticalline,eachpixelbluevaluewassummedtoobtainthebluedistributiongraphb(x)wherexistheco-ordinate

ontheabscissaoftheimage.Themeanb

%,andstandarddeviationSdbofb(x)werecalculated,respectively,usingthefollowingequations

511b%51X512bðxÞð1Þ

x50

r S1db5X511À512

x50b%

ÀbðxÞÁ2

ð2Þ

机器人、控制、系统

396

B.CHENETAL.

Table1

Samplingconditions

Week

Date(2001)

Riceheight,cm

Weather

Cameraheight,cm

Sampleimageidenti ers*

Cameraangle,deg0

16Jun

10

Cloudy

102030102030102030102030102030102030102030102030

S01}}S11S14S17S21S24S27S31S34S37S41S44S47S51S54S57S61S64S67S71S74S77

10S02S05S08S12S15S18S22S25S28S32S35S38S42S45S48S52S55S58S62S65S68S72S75S78

20}S06S09S13S16S19S23S26S29S33S36S39S43S46S49S53S56S59S63S66S69S73S76S79

122Jun12Cloudy

229Jun20Sunny

306Jul25Cloudy

413Jul30Sunny

519Jul40Sunny

627Jul55Sunny

704Aug65Cloudy

Note:transplantingwasperformedon13Jun2001.*

Sampleimageidenti ers:Smn–S,samplenumber;m,weeknumber(0–7);n,videotapenumber(1–9);‘}’,nosample.

Next,thedataweresmoothedbyaveragingoveragroupof20pixelswidth,movingacrosstheoriginaldatapixelbypixel,andthedistributionb(x)wasevaluatedasbelowtoobtaintheblueparameterBpm.ThehighestpointP,whoseco-ordinatewasxP,wassoughtonthedistributionb(x)(Fig.3).Ifb(xP)>bu,wherebuistheupperboundoftotalblueintensityequal%þSdb,theminimumpointsonbothsidesofPweretob

searchedabovethatvaluebu,andthepoint,whichwasthehighestoftheminimumpoints,wasnamedp1,whoseco-ordinatewasxp1.Theintersectionbetweenthedistributionb(x)andthebluevalueatp1ontheoppositesideofPwasnamedp2,whoseco-ordinatewasxp2.IftherewasnominimumpointoneithersideofPabovetheupperboundoftotalblueintensitybu,thetwointersectionsbetweenthedistributionb(x)andbuonbothsidesofPwereusedasp1andp2.Theblue

parameterBpmwasobtainedusingEqn(3)when512/2ÀxP¼0orEqn(4)when256ÀxP50.Otherwise,ifb(xP)4buwastrue,Bpmwas xedas0.ThenumberofmaximumpeakpointsNP,abovethelinebu,wasalsoobtainedwhensearchingforthehighestblueintensitypointP:

Á2Pxp2À

bðxÞÀbðxÞp1xp1

Bpm5256Àxp¼0ð3Þ

j256Àxpj

Bpm5

xp2XÀxp1

Á2

bðxÞÀbðxp1Þ;

256Àxp50ð4Þ

IfthecalculatedvalueoftheblueparameterBpmexceededthecurrentmaximumvalueBpm,max,thenitreplacedthevalue,Bpm,max5Bpm,andtheprocesswastakentothenextstep(Fig.2).

机器人、控制、系统

MACHINEVISIONFORMICROWEEDINGROBOT

397

%,meanpixelblueFig.2.Flowchartfordetectingdirection:AR,arearate;AR1,max,AR2,max,maximumarearates1and2;b

intensity;Bpm,blueparameter;Bpm,max,maximumblueparameter;F,processingframe;FC1,FC2,candidateframenumbers1–2;FJ,judgedframe;LR,linerate;NO,max,maximumobjectnumber;NO,objectnumber;NP,numberofpeaks;Sdb,standarddeviationof

bluedistribution

3.1.2.Judgingthebinaryimage

Owingtothein uenceofre ectedlight,themaximumvalueforBpm,maxmaynotbelongtothetargetimage;therefore,theimagewiththemaximumvalueforBpm,maxhastobecheckedbyotherconditionstoascertainthatitisthetargetimage.

Iftheimagepassedtheabovestep,thebinaryimagewasthenjudged.ThebinaryimagewasobtainedbycomparingtheblueintensityB,andthegreenintensityGofeachpixel.IfG>Bwastruethepixelwas xedas255(white,representingriceplants);forotherconditionsthepixelwas xedas0(black,representingwater).Atrapezium,sized64(5512/8)pixelsatthetopand256(5512/2)pixelsatthebottom(eachhorizontallinelengthwas64+y(256-64)/480pixels,whereyistheordinateoftheimagefrom0to479),centredatthepositionwithco-ordinatexP(Fig.4),wasestablishedonthebinaryimagetocountthenumberofblackpixels,whichwascalledtheobjectnumberNO.ThearearateARandthelinerateLR(Fig.2)werethencalculatedinthetrapeziumarea.ThevalueforARwasobtainedbydividingthenumberofobjectsNObythetotalpixelsinthetrapeziumarea.ThevalueforLRwasobtainedbydividingh,thenumberoflinescontaining55%blackpixelsatthetopoftheimageinthetrapeziumarea,by480,theimageheight(Fig.4).AftercountingtheobjectnumberNO,calculatingthearearateARandthelinerateLR,thesedatawerecheckedinordertodecidesubsequentprocessing.IfallofthecriteriaNO>NO,max,