Automatic 3D City Modeling Using a Digital Map and Panoramic Images from a Mobile Mapping System

Hindawi Publishing Corporation

Mathematical Problems in EngineeringVolume 2014, Article ID 383270, 10 pageshttp://wendang.chazidian.com/10.1155/2014/383270

ResearchArticle

Automatic3DCityModelingUsingaDigitalMapandPanoramicImagesfromaMobileMappingSystem

HyungkiKim,1YunaKang,2andSoonhungHan1

12

KoreaAdvancedInstituteofScienceandTechnology,291Daehak-ro,Yuseong-gu,Daejeon305-701,RepublicofKorea

1stR&DInstitute,AgencyforDefenseDevelopment,488Bugyuseong-daero,Yuseong-gu,Daejeon305-152,RepublicofKorea

CorrespondenceshouldbeaddressedtoSoonhungHan;shhan@kaist.ac.kr

Received20May2014;Revised16July2014;Accepted16July2014;Published13August2014AcademicEditor:Jong-HyukPark

Copyright©2014HyungkiKimetal.ThisisanopenaccessarticledistributedundertheCreativeCommonsAttributionLicense,whichpermitsunrestricteduse,distribution,andreproductioninanymedium,providedtheoriginalworkisproperlycited.Three-dimensionalcitymodelsarebecomingavaluableresourcebecauseoftheirclosegeospatial,geometrical,andvisualrelationshipwiththephysicalworld.However,ground-orientedapplicationsinvirtualreality,3Dnavigation,andcivilengineeringrequireanovelmodelingapproach,becausetheexistinglarge-scale3Dcitymodelingmethodsdonotproviderichvisualinformationatgroundlevel.Thispaperproposesanewframeworkforgenerating3Dcitymodelsthatsatisfyboththevisualandthephysicalrequirementsforground-orientedvirtualrealityapplications.Toensureitsusability,theframeworkmustbecost-effectiveandallowforautomatedcreation.Toachievethesegoals,weleverageamobilemappingsystemthatautomaticallygathershigh-resolutionimagesandsupplementssensorinformationsuchasthepositionanddirectionofthecapturedimages.Toresolveproblemsstemmingfromsensornoiseandocclusions,wedevelopafusiontechniquetoincorporatedigitalmapdata.Thispaperdescribesthemajorprocessesoftheoverallframeworkandtheproposedtechniquesforeachstepandpresentsexperimentalresultsfromacomparisonwithanexisting3Dcitymodel.

1.Introduction

Three-dimensionalcitymodelsarewidelyusedinapplica-tionsinvariousfields.Suchmodelsrepresenteitherrealorvirtualcities.Virtual3Dcitymodelsarefrequentlyusedinmoviesorvideogames,whereageospatialcontextisnotnecessary.Real3Dcitymodelscanbeusedinvirtualreality,navigationsystems,orcivilengineering,astheyarecloselyrelatedtoourphysicalworld.GoogleEarth[1]isawell-known3Drepresentationofrealcities.Itillustratestheentireearthusingsatellite/aerialimagesandmapssuperimposedonanellipsoid,providinghigh-resolution3Dcitymodels.

Intheprocessof3Dcitymodeling,boththecostandthequalityrequirementsmustbeconsidered.Thecostcanbeestimatedasthetimeandresourceconsumptionofmodelingthetargetarea.Thequalityfactorconsidersbothvisualqual-ityandphysicalreliability.Thevisualqualityisproportionaltothedegreeofvisualsatisfaction,whichaffectsthelevelofpresenceandreality.Thephysicalreliabilityisthegeospa-tialandgeometricalsimilaritybetweentheobjects—inourcase,mainlybuildings—inthemodeledandphysicalworlds.

Generally,accomplishingasatisfactorylevelforbothrequire-mentsisdifficult.

Numeroustechniquescanbeusedfor3Dcitymodeling.Forinstance,colorandgeometrydatafromLiDARaremainlyusediftheapplicationrequiresdetailedbuildingmodelsforasmallarea.Ifthecitymodelcoversalargeareaanddoesnotneeddetailedfeatures,reconstructionfromsatellite/aerialimagesismoreefficient[2].Thismeansthattheeffectiveapproachcandifferaccordingtothetargetapplicationusingthe3Dcitymodel.Thegoalofourresearchistoproposea3Dcitymodelingmethodthatcanbeappliedinground-orientedandinteractivevirtualrealityapplications,includingdrivingsimulatorsand3Dnavigationsystems,whichrequireeffective3Dcitymodelingmethodsfordiverseareas.

2.RelatedWork

Existing3Dcitymodelingmethodscanbedividedintoman-ualmodelingmethods,BIM(BuildingInformationModel)data-basedmethods,LiDARdata-basedmethods,andimage-basedmethods.

2

Themanualmodelingmethodishighlydependentonthemodelingexperts.OlderversionsofGoogleEarthandTerraVistaemployedthismethodintheirmodelingsystems.Althoughcurrentapplicationsemploymanualmodelingbecauseofitshighquality,themethodisalsoahigh-cost,labor-intensiveprocess.Hence,itisnotefficientforurbanenvironmentsthatincludenumerousbuildings.TheBIMdata-basedmethodfacilitatestheuseofbuildingdesigndatafromtheconstructionstageandisappliedincityplanning[3]andfireandrescuescenario[4].However,thismethodisonlyefficientforapplicationsinwhichtheactivityareasarestrictlyconstrained,asgatheringBIMdataisproblematicorevenimpossiblegiventhelargesizeofurbanenvironments.More-over,theBIMdatashouldbepostprocessedfortheuseofvirtualrealityapplications.ThisisbecausetheinformationinBIMdoesnotcontaintheas-built3Dmodel;sothepropertiesforthevisualvariablesshouldbemapped.

Toaddresstheseproblems,remotesensingtechniquesarebeingaggressivelyadoptedandstudiesofLiDARdata-basedmethodsandimage-basedmethodsareincreasinglycommon[5].LiDARisadevicethatsamplesprecisedatafromthesur-roundingenvironmentusinglaserscanningtechnology.Inseveralstudies(e.g.,[6,7]),high-quality3DcitymodelshavebeenreconstructedusingLiDARdata.However,asnotedinotherwork[6],ground-levelscanninghasalimiteddatagath-eringrange,meaningthatredundantdatacollectionisunavoidableinthemodelingofdiverseareas,whereasair-borneLiDAR[8]islimitedintermsofitscostandcolordatacollectionmethods.

Image-basedmethodsincludethosebasedonstereomatchingandinverseproceduralmodelingapproaches.Inearlierresearch[7,9],amethodbasedonstereomatchingwasusedtorecover3Ddatafromthefeaturepointmatchingbetweenaseriesofimages.Thisapproachusuallyrequiresnumerousimagestosatisfytheaccuracyandrobustnessrequirementsoffeaturepointmatching.Severalrecentinverseproceduralmodelingapproaches[10–12]havemod-eledbuildingsusingrelativelyfew(mainlyone)images.Thiscanovercomethedifficultiesofdata-collectioninstereomatching.Thisapproachemploysaplausibleassumption;thatis,thattheshapeofabuildingconsistsofasetofplanesinthreedimensions,toreconstructindividual3Dbuild-ingswithoutpixel-wise,3Dinformation.However,becauseimage-basedmethodsarenotrobustagainstinstancesofocclusion,userinputorstrongconstraintsarefrequentlynec-essary.Thisreducestheircosteffectivenessand/orphysicalreliability.

Inourresearch,theapproachwhichpreservesthecost-efficiencybyusingtheexistingimagedatabasewhileincreas-ingphysicalreliabilitywillbeproposed.TheimagedatabaseisrelativelyeasytoaccessthanLiDARdatabasesocostonthedatacollectioncanbedecreased.Ontheotherhandthemethodbasedonthestereomatchingrequiresnumerousimagesonthelarge-scalemodelingthatdecreasestheuni-versalapplicabilityofmethod.Thereforeinverseproceduralmodelingapproachispreferredonourobjective,whilethephysicalreliabilitycanbeincreasedbycombiningaccuratereferencedata[13].

MathematicalProblemsinEngineering

3.ProposedMethod

3.1.MobileMappingSystemandDigitalMap.Inthisstudy,weproposeaframeworkthatusesamassivenumberofimagesgatheredfromamobilemappingsystem(MMS).Thisaddressesmanyproblemsinexistingmethods,whichcannotsimultaneouslyprovidefeasiblelevelsofcosteffectiveness,visualquality,orphysicalreliability.AnMMScollectsandprocessesdatafromsensordevicesmountedonavehicle.ServicessuchasGoogleStreetView[14],NaverMaps[15],andBaiduMaps[16]presentinformationintheformofhigh-resolutionpanoramicimagesthatincludethegeospatialpositionanddirectionofeachimagetaken.Themainfocusoftheseservicesistooffervisualinformationaboutthesur-roundingenvironmentatagivenlocation.TheadvantagesofdatacollectedfromMMSareasfollows.

(1)Nationwideorevenworldwidecoveragefollowingthedevelopmentofremotesensingtechnologiesandmapservices.(2)Rich,visual,andomnidirectionalinformation.(3)http://wendang.chazidian.comingtheseadvantages,wecanmodeladiversecityareaforground-orientedinteractivesystemsinacost-effectivewaywiththeexistingimagedatabase.Moreover,highvisualqualityatgroundlevelcanbeprovidedbyhigh-resolutionpanoramicimages[17].However,therearecurrentlyseveraldisadvantagesinthedatacollectedfromMMS.

(1)Sensordataincludesnoise,whichlowersitsphysicalreliability.(2)Thenumberofimagesinagivenareaislimitedandisinsufficientforstereomatching-basedreconstruc-tion.(3)Inclusionofanenormousamountofunnecessaryvisualinformation,includingocclusions,cars,andpedestrians.Noiseisunavoidableinthesensingprocess.Theamountoferrorthisintroducesdiffersaccordingtothesurroundingenvironment;a±5mpositionalerrorand±6?directionalerrorhavebeenreportedinGoogleStreetViewdata[18].Sucherrorlevelscanbeproblematicintheanalysisrequiredfor3Dmodeling.Moreover,thecurrentservicehasaninter-valof?10mbetweenimages,whichlowersthepossibilityofsuccessfulreconstructionusingstereomatching.Addi-tionally,theuncontrolledcollectionenvironmentresultsinaseveredisadvantageforinverseproceduralmodeling.MMSdataalsorequiresanadditionalprocesstoclassifyindi-vidualbuildings,unliketheinverseproceduralmodelingapproaches.

ToaddresstheseproblemswithMMSdata,weproposeamethodthatincorporates2Ddigitalmapdata.Digitalmapshaveaccurategeospatialinformationaboutvariousfeaturesinthephysicalworld.Forinstance,the1:5000digitalmapsappliedinourframeworkhaveahorizontalaccuracyof1m,whichisfivetimesbetterthanthatoftheMMSpositiondata.

MathematicalProblemsinEngineering

Individual building region

3

Building texture

Textured 3D model

Figure1:Overviewoftheproposed3Dcitymodelingframework.

Therefore,bycombiningdata,theproblemsofsensorerrorscanbeovercomeandtheselectiveuseofvisualinformationispossible.Ontheotherhand,thegeometricalcharacteristicofthebuildingisrestrictedtoaquasi-Manhattanworldmodel.Thequasi-Manhattanworldmodelistheassumptionthatstructuresconsistofverticalandhorizontalplanes,andisanextensionoftheManhattanworldmodelthatassumesstruc-turesconsistofverticalandhorizontalplanesorthogonaltoeachother.

3.2.ProcessOverview.TheproposedframeworkisillustratedinFigure1.Theinputdataaretheaforementioneddigitalmaps,whichcontainbuildingfootprintinformationandpanoramicimagesfromtheMMSsystemwithsensordata.Thebase3Dmodelisgeneratedfromthefootprintinforma-tionofthebuildings;theindividualbuildingregionsareseg-mentedandreprojectedaccordingtothecombinedGPS/INS(InertialNavigationsystem)information.Thereprojectedregionisfurthersegmentedandrectifiedtoproducethetextureimage.Heightestimationispossiblebycombiningthebuildingcontourinformationfromthetextureimageandthereprojectedimage.Wecanthenobtainthetextured3Dmodelbyapplyingtheheightinformationtomodifytheheightofthebase3Dmodels.

ThedetailedprocessisillustratedinFigure2.Theentiremodelingprocedurecanbedividedintothefollowingfourstages.

(1)Image/buildinganalysis.(2)Errorcorrection/compensation.

(3)Segmentationandvalidation.(4)Texturemapping.

Theerrorcorrection/compensationandsegmentationandvalidationprocessesincludeafeedbacklooptosequen-tiallyobtainthetextureofindividualbuildings.

3.3.Image/BuildingAnalysis.Thisstageanalyzesthecorrela-tionbetweeneachimageandthedigitalmap.Todothis,abase3Dmodelisgeneratedfromthefootprintofthebuild-ingsbyextendingthemodelintheverticaldirection.Thefootprintdataconsistsofthegeospatialcoordinatesofthebuildingcontourprojectedtothegroundsurface.Thisdatashouldretainacertainlevelofaccuracyandthereforecon-tainspreciseinformationaboutthebuildings.

Theinputimagecanthenbepositionedinthe3DenvironmentaccordingtotheGPS/INSsensorinformation.Next,buildingsareclassifiedaccordingtothethreecriterialistedbelow.Theobjectiveofthisclassificationistoseparatethebuildingsintotexture-acquirableexamplesandothersatahighresolution.Theproposedcriteriaareasfollows.

(1)ThedistancebetweenthelocationfromtheGPSsensorcorrespondingtotheimageandthebuilding.(2)Theocclusionbetweenthebuildings.

(3)Theregionoccupiedbythebuildingintheimage.Thedistancecriterionisquitestraightforward:moredistantbuildingsarelesslikelytoappearintheimage.Theocclusioncriterionisalsoreasonable,asanoccludedbuilding

4MathematicalProblemsinEngineering

Image-building analysis

1

Error correction

2

ROI setting

Validation

Segmentation

Remapping

3

Model refinementTexture mapping

4

Figure2:Detailedprocessoftheproposedapproach.

cannotappearintheimage.Thethirdcriterionconsidersinformationaboutthebrieftextureresolutionofeachimagebycalculatingthefac¸adeangleandwidthoftheimage,andthenestimatingtheresolutionofthetexture.Assuming??isthewidthofthepanoramicimagethathasa360-degreefieldofview(FOV),andthenthepixelperradianinthehorizontaldirectioncanbecalculatedas??/??.Thelocationofthecapturedimagecanbeexpressedas??∈R2inthe2Ddigitalmapandbothendsofthefac¸adefootprintscanbe

2

expressedas????,????∈R.Thenthepixelpermeter,whichisthetextureresolutionforeachfac¸ade,canbecalculatedusingthelawofcosinesTextureResolution=where

??2????2????2??

)+(??)?(???????????????????????????????(??????????????)).??=arccos(????????????????????

(1)ThesecriteriaareillustratedinFigure3.

3.4.ErrorCorrection/Compensation.Thecorrection/compen-sationprocessturnstheomnidirectionalbuildingdetectionproblemintothesimpleproblemofsegmentingasingleimage.NoiseintheGPS/INSsensoristheprimarysourceofthemismatchbetweentheimage/buildinganalysisresultsandthegroundtruth.

?????

,First,correctionoftheGPS/INSsensorerrorisper-formedusingimage-basedlocalizationmethods.Image-basedlocalizationisamajorresearchissueinrobotvisionandlocation-basedservices.Thelocalizationofapanoramicimagecanbeconductedbyapplyingsemanticsegmentation[18].However,theauthorsassumedtheexistenceofadetailed3Dcadastralmodel,butthesearenotoftenavailableincityareas.Otherresearch[19]hasproposedalocalizationmethodthatusesimagesanddigitalmaps,butthisisstronglydepen-dentonuserinput.Hence,theirmethodisnotappropriateforourframework,whichdealswithalargenumberofimages.Instead,ourframeworkutilizesamethodthatlocalizesthepanoramicimagebasedontheorientationdescriptor[20].Thefootprintorientation(FPO)descriptorencodestherel-ativeanglebetweenthelinesemittedradiallyfromacertainlocationonthemapandthefootprintsofthebuildings.Inthesameway,theFPOdescriptorcanbecalculatedfromthepanoramicimagebecausethepanoramicimagehasomnidi-rectionalinformationsothatbyvanishingpointestimationwecancalculatetheanglebetweenthelocationoftheimageandthefootprintsofthevisiblebuildings.Byfindingthemin-imumdistancebetweentheFPOdescriptorcalculatedfromtheimageandthesampledlocationsonthemap,wecanestimatewherethepanoramicimagehasbeentaken.Experi-mentsshowedthattheerrorafterestimationislessthan2m,whichissufficientforourframework,andtoproceedtotheprocessingstage.Meanwhile,the360-degreeFOVpanoramicimageispreferredbecauseofthiserrorcorrection.Ascanbeseeninearlierresearch,asingleimagefromanormallenscontainsalimitedamountofvisualinformationaboutthe

MathematicalProblemsinEngineering5

50m

(a)(b)

?100(c)

Figure3:Criteriaoftheimage/buildinganalysis.(a)Distancebetweentheimageandbuildings;(b)occlusionbetweenbuildings;and(c)facingviewingangleofthebuildingfac¸ade,thatisusedtocalculatethetextureresolution.

surroundingenvironment[21],souser-inputshouldbecon-sideredinordertomoreaccuratelyestimatethelocationwheretheimagewastaken[22].

Theerrorcompensationutilizesthiserrorboundtosettheregionofinterest(ROI).Ourobjectiveinerrorcompen-sationistoprocessthesingle360-degreepanoramicimageintoseveralnormal-lensimagestobuildeachtarget,bypar-titioningandreprojecting.Aswementionedbefore,theerrorcorrectionreducesthepositionandorientationerrorbuttherestillexistmismatches,upto2minposition,betweenthebase3Dmodelfromthedigitalmapandthepanoramicimage.SowecalculatetheFOVforeachtargetedbuilding,whichisROI,witha2-mmargin.Afterthat,thepanoramicimageisreprojectedusingrectilinearprojectiontogenerateanimagewhichpreservesthestraightlinesin3Dspaceintheprojectedimage.ThentheROIcontainsthecompleteimageofthetargetbuildingandthecomplexityoftheimage-segmentationprocessisreduced.

3.5.SegmentationandValidation.Thisstepinvolvestheseg-mentationandvalidationofindividualbuildings.Asnotedinearlierstudies[23,24],http://wendang.chazidian.comually,themethodproposedin[23]givesrobustresults,becausehorizontallinesegmentsarelessaffectedbyocclusions,whereas[24],whichreliesontheverticalvanishingpoint,suffersfromocclusionscausedbypedestrians,trees,andcars.

Theouterboundarycanbeobtainedbyminimizingthe1DMarkovrandomfieldenergy,whichisdefinedas

??(L)

=∑????(????)+∑????,??+1(????,????+1)

??=1

??=1

??

???1

(see[18])

(2)

[??(????)>??]???if????=0

????(????)={

100???????(????)otherwise????,??(????,????)=[????=????]???(??????(????)+??????(????)),

(3)(4)

whereL=(??1,??2,...,????)isthelabelingoftheentirecolumn

intheimagesothatnisthepixelwidthoftheimageand????∈{1,2,...,??}isthelabelforeachcolumn????wheremisthenumberofdetectedhorizontallineorientations(e.g.,??=2inFigure4(a)).??????(????)isthenumberoflinesegmentswiththespecifichorizontallineorientationinthecolumn????,andthetotalnumberoflinesegmentsofanyorientationin????is??(????)=∑??????(????).Therefore????(????)istheunarypotential,whichhasalowerenergywhentherearemorehorizontallinesegmentscrossing????where??isthethresholdand??controlsthecostforno-fac¸aderegion.????,??+1(????,????+1)isthepairwisepotentialforthelinesegmentscorrespondingtothevanish-ingpointsofthe????and????+1,where??istheweightfactor.Thisdescribesthesmoothnessfactorwhenlabelingthedifferentpixelvaluesbyprovidinghigherenergywhenthelabelsdifferbetweenthosepixels.TheresultisillustratedinFigure4.