Model selection and model averaging

Cambridge University Press

978-0-521-85225-8 - Model Selection and Model Averaging

Gerda Claeskens and Nils Lid Hjort

ExcerptMore information

1

Modelselection:dataexamplesandintroduction

Thisbookisaboutmakingchoices.Ifthereareseveralpossibilitiesformod-

ellingdata,whichshouldwetake?Ifmultipleexplanatoryvariablesaremea-

sured,shouldtheyallbeusedwhenformingpredictions,makingclassi?cations,

orattemptingtosummariseanalysisofwhatin?uencesresponsevariables,or

willincludingonlyafewofthemworkequallywell,orbetter?Ifso,which

onescanwebestinclude?Modelselectionproblemsarriveinmanyformsand

onwidelyvaryingoccasions.Inthischapterwepresentsomedataexamples

http://wendang.chazidian.comterinthebookwecomeback

tothesedataandsuggestsomeanswers.Ashortpreviewofwhatistocomein

laterchaptersisalsoprovided.

1.1Introduction

Withthecurrenteaseofdatacollectionwhichinmany?eldsofappliedsciencehasbecomecheaperandcheaper,thereisagrowingneedformethodswhichpointtointer-esting,importantfeaturesofthedata,andwhichhelptobuildamodel.Themodelwewishtoconstructshouldberichenoughtoexplainrelationsinthedata,butontheotherhandsimpleenoughtounderstand,explaintoothers,anduse.Itiswhenwenegotiatethisbalancethatmodelselectionmethodscomeintoplay.Theyprovideformalsupporttoguidedatausersintheirsearchforgoodmodels,orfordeterminingwhichvariablestoincludewhenmakingpredictionsandclassi?cations.

Statisticalmodelselectionisanintegralpartofalmostanydataanalysis.Modelselectioncannotbeeasilyseparatedfromtherestoftheanalysis,andthequestion‘whichmodelisbest’isnotfullywell-poseduntilsupplementinginformationisgivenaboutwhatoneplanstodoorhopestoachievegiventhechoiceofamodel.Thesurveyofdataexamplesthatfollowsindicatesthebroadvarietyofapplicationsandrelevanttypesofquestionsthatarise.

Beforegoingontothissurveyweshallbrie?ydiscusssomeofthekeygeneralissuesinvolvedinmodelselectionandmodelaveraging.1© Cambridge University http://wendang.chazidian.com

Cambridge University Press

978-0-521-85225-8 - Model Selection and Model Averaging

Gerda Claeskens and Nils Lid Hjort

ExcerptMore information

2Modelselection:dataexamplesandintroduction

(i)Modelsareapproximations:Whendealingwiththeissuesofbuildingorselectingamodel,itneedstoberealisedthatinmostsituationswewillnotbeabletoguessthe‘correct’or‘true’model.Thistruemodel,whichinthebackgroundgeneratedthedatawecollected,mightbeverycomplex(andalmostalwaysunknown).Forworkingwiththedataitmightbeofmorepracticalvaluetoworkinsteadwithasimpler,butalmost-as-goodmodel:‘Allmodelsarewrong,butsomeareuseful’,asamaximformulatedbyG.E.P.Boxexpressesthisview.Severalmodelselectionmethodsstartfromthisperspective.

(ii)Thebias–variancetrade-off:Thebalanceandinterplaybetweenvarianceandbiasisfundamentalinseveralbranchesofstatistics.Intheframeworkofmodel?ttingandselectionittakestheformofbalancingsimplicity(fewerparameterstoestimate,leadingtolowervariability,butassociatedwithmodellingbias)againstcomplexity(enteringmoreparametersinamodel,e.g.regressionparametersformorecovariates,meansahigherdegreeofvariabilitybutsmallermodellingbias).Statisticalmodelselectionmethodsmustseekaproperbalancebetweenover?tting(amodelwithtoomanyparameters,morethanactuallyneeded)andunder?tting(amodelwithtoofewparameters,notcapturingtherightsignal).

(iii)Parsimony:‘Theprincipleofparsimony’takesmanyformsandhasmanyfor-mulations,inareasrangingfromphilosophy,physics,arts,communication,andindeedstatistics.TheoriginalOckham’srazoris‘entitiesshouldnotbemultipliedbeyondne-cessity’.Forstatisticalmodellingareasonabletranslationisthatonlyparametersthatreallymatteroughttobeincludedinaselectedmodel.Onemight,forexample,bewillingtoextendalinearregressionmodeltoincludeanextraquadratictermifthismanifestlyimprovespredictionquality,butnototherwise.

(iv)Thecontext:Allmodellingisrootedinanappropriatescienti?ccontextandisforacertainpurpose.AsDarwinoncewrote,‘Howodditisthatanyoneshouldnotseethatallobservationmustbefororagainstsomeviewifitistobeofanyservice’.Onemustrealisethat‘thecontext’isnotalwaysapreciselyde?nedconcept,anddifferentresearchersmightdiscoverorlearndifferentthingsfromthesamedatasets.Also,differentschoolsofsciencemighthavedifferentpreferencesforwhattheaimsandpurposesarewhenmodellingandanalysingdata.Breiman(2001)discusses‘thetwocultures’ofstatistics,broadlysortingscienti?cquestionsintorespectivelythoseofpredictionandclassi?cationononehand(whereevena‘blackbox’modelis?neaslongasitworkswell)andthoseof‘deeperlearningaboutmodels’ontheotherhand(wherethediscoveryofanon-nullparameterisimportantevenwhenitmightnothelpimproveinferenceprecision).ThusS.Karlin’sstatementthat‘Thepurposeofmodelsisnotto?tthedata,buttosharpenthequestions’(inhisR.A.Fishermemoriallecture,1983)isimportantinsomecontextsbutlessrelevantinothers.Indeedtherearedifferentlyspiritedmodelselectionmethods,gearedtowardsansweringquestionsraisedbydifferentcultures.© Cambridge University http://wendang.chazidian.com

Cambridge University Press

978-0-521-85225-8 - Model Selection and Model Averaging

Gerda Claeskens and Nils Lid Hjort

ExcerptMore information

1.2Egyptianskulldevelopment3

(v)Thefocus:Inappliedstatisticsworkitisoftenthecasethatsomequantitiesorfunctionsofparametersaremoreimportantthanothers.Itisthenfruitfultogearmodelbuildingandmodelselectioneffortstowardscriteriathatfavourgoodperformancepreciselyforthosequantitiesthataremoreimportant.Thatdifferentaimsmightleadtodifferentlyselectedmodels,forthesamedataandthesamelistofcandidatemodels,shouldnotbeconsideredaparadox,asitre?ectsdifferentpreferencesanddifferentlossfunctions.Inlaterchaptersweshallinparticularworkwithfocussedinformationcriteriathatstartfromestimatingthemeansquarederror(varianceplussquaredbias)ofcandidateestimators,foragivenfocusparameter.

(vi)Con?ictingrecommendations:Asisclearfromtheprecedingpoints,questionsabout‘whichmodelisbest’areinherentlymoredif?cultthanthoseofthetype‘foragivenmodel,howshouldwecarryoutinference’.Sometimesdifferentmodelselectionstrategiesendupofferingdifferentadvice,forthesamedataandthesamelistofcandidatemodels.Thisisnotacontradictionassuch,butstressestheimportanceoflearninghowthemostfrequentlyusedselectionschemesareconstructedandwhattheiraimsandpropertiesare.

(vii)Modelaveraging:Mostselectionstrategiesworkbyassigningacertainscoretoeachcandidatemodel.Insomecasestheremightbeaclearwinner,butsometimesthesescoresmightrevealthatthereareseveralcandidatesthatdoalmostaswellasthewinner.Insuchcasestheremaybeconsiderableadvantagesincombininginferenceoutputacrossthesebestmodels.

1.2Egyptianskulldevelopment

MeasurementsonskullsofmaleEgyptianshavebeencollectedfromdifferentarchaeo-logicaleras,withaviewtowardsestablishingbiometricaldifferences(ifany)andmoregenerallystudyingevolutionaryaspects.Changesovertimeareinterpretedanddiscussedinacontextofinterbreedingandin?uxofimmigrantpopulations.Thedataconsistoffourmeasurementsforeachof30skullsfromeachof?vetimeeras,originallypresentedbyThomsonandRandall-Maciver(1905).The?vetimeperiodsaretheearlypredy-nastic(around4000b.c.),latepredynastic(around3300b.c.),12thand13thdynasties(around1850b.c.),theptolemaicperiod(around200b.c.),andtheRomanperiod(around150a.d.).Foreachofthe150skulls,thefollowingmeasurementsaretaken(allinmil-limetres):x1=maximalbreadthoftheskull(MB),x2=basibregmaticheight(BH),x3=basialveolarlength(BL),andx4=nasalheight(NH);seeFigure1.1,adaptedfromManly(1986,page6).Figure1.2givespairwisescatterplotsofthedataforthe?rstandlasttimeperiod,respectively.Similarplotsareeasilymadefortheothertimeperiods.Wenotice,forexample,thatthelevelofthex1measurementappearstohaveincreasedwhilethatofthex3measurementmayhavedecreasedsomewhatovertime.Statisticalmodellingandanalysisarerequiredtoaccuratelyvalidatesuchclaims.© Cambridge University http://wendang.chazidian.com

Cambridge University Press

978-0-521-85225-8 - Model Selection and Model Averaging

Gerda Claeskens and Nils Lid Hjort

ExcerptMore information

4Modelselection:dataexamplesand

introduction

Fig.1.1.Thefourskullmeasurementsx1=MB,x2=BH,x3=BL,x4=NH;fromManly(1986,page6).

Thereisafour-dimensionalvectorofobservationsyt,iassociatedwithskulliandtimeperiodt,fori=1,...,30andt=1,...,5,wheret=1correspondsto4000b.c.,

¯t,?todenotethefour-dimensionalvectorandsoon,uptot=5for150a.d.Weusey

ofaveragesacrossthe30skullsfortimeperiodt.Thisyieldsthefollowingsummarymeasures:

¯1,?=(131.37,133.60,99.17,50.53),y

¯2,?=(132.37,132.70,99.07,50.23),y

¯3,?=(134.47,133.80,96.03,50.57),y

¯4,?=(135.50,132.30,94.53,51.97),y

¯5,?=(136.27,130.33,93.50,51.37).y

Standarddeviationsforthefourmeasurements,computedfromaveragingvarianceesti-matesoverthe?vetimeperiods(intheorderMB,BH,BL,NH),are4.59,4.85,4.92,

3.19.WeassumethatthevectorsYt,iareindependentandfour-dimensionalnormallydistributed,withmeanvectorξtandvariancematrix??tforerast=1,...,5.However,itisnotgiventoushowthesemeanvectorsandvariancematricescouldbestruc-tured,orhowtheymightevolveovertime.Hence,althoughwehavespeci?edthatdatastemfromfour-dimensionalnormaldistributions,themodelforthedataisnotyetfullyspeci?ed.

Wenowwishto?ndastatisticalmodelthatprovidestheclearestexplanationofthemainfeaturesofthesedata.Giventheinformationandevolutionarycontextalludedtoabove,searchingforgoodmodelswouldinvolvetheirabilitytoanswerthefollowingquestions.Dothemeanparameters(populationaveragesofthefourmeasurements)© Cambridge University http://wendang.chazidian.com

Cambridge University Press

978-0-521-85225-8 - Model Selection and Model AveragingGerda Claeskens and Nils Lid HjortExcerpt

More information

1.2Egyptianskulldevelopment

120 125 130 135 140 145

115

60

5

10580 85 90 95

NH

120

130

140

BH

BL

120130140

45

120

5055

130140

MB

115

60

MB

60

MB

105

55

NH

80 85 90 95

50

NH

120 125 130 135 140 145

BL

45

120 125 130 135 140 145

45

80 85 90 95

5055

105115

BH

120 125 130 135 140 145

115

BH

60

BL

10580 85 90 95

NH

120

130

140

BH

BL

120130140

45

120

5055

130140

MB

115

60

MB

60

MB

105

55

NH

80 85 90 95

50

NH

120 125 130 135 140 145

BL

45

120 125 130 135 140 145

45

80 85 90 95

5055

105115

BHBHBL

Fig.1.2.PairwisescatterplotsfortheEgyptianskulldata.Firsttworows:earlypredy-nasticperiod(http://wendang.chazidian.comsttworows:Romanperiod(150a.d.).

© Cambridge University http://wendang.chazidian.com