1309.0911v1

Beysian

ABAYESIANINFORMATIONCRITERIONFORSINGULAR

MODELS

MATHIASDRTONANDMARTYNPLUMMER

Abstract.WeconsiderapproximateBayesianmodelchoiceformodelselec-

tionproblemsthatinvolvemodelswhoseFisher-informationmatricesmayfail

tobeinvertiblealongothercompetingsubmodels.Suchsingularmodelsdonot

obeytheregularityconditionsunderlyingthederivationofSchwarz’sBayesian

informationcriterion(BIC)andthepenaltystructureinBICgenerallydoes

notre?ectthefrequentistlarge-samplebehavioroftheirmarginallikelihood.

Whilelarge-sampletheoryforthemarginallikelihoodofsingularmodelshas

beendevelopedrecently,theresultingapproximationsdependonthetruepa-

rametervalueandleadtoaparadoxofcircularreasoning.Guidedbyexamples

suchasdeterminingthenumberofcomponentsofmixturemodels,thenumber

offactorsinlatentfactormodelsortherankinreduced-rankregression,we

proposearesolutiontothisparadoxandgiveapracticalextensionofBICfor

singularmodelselectionproblems.

1.Introduction

Informationcriteriaareclassicaltoolsformodelselection.Atahigh-level,theyfallintotwocategories(Yang,2005).Ononehand,therearecriteriathattar-getgoodpredictivebehavioroftheselectedmodel;theinformationcriterionofAkaike(1974)andcross-validationbasedscoresareexamples.TheBayesianin-formationcriterion(BIC)ofSchwarz(1978),ontheotherhand,drawsmotivationfromBayesianapproaches.Fromthefrequentistperspective,ithasbeenshowninanumberofsettingsthattheBICisconsistent.Inotherwords,underoptimizationofBICtheprobabilityofselectinga?xedmostparsimonioustruemodeltendstooneasthesamplesizetendstoin?nity(e.g.,Nishii,1984,Haughton,1988,1989).FromaBayesianpointofview,theBICyieldsrathercrudebutcomputationallyinexpensiveapproximationstootherwisedi?culttocalculateposteriormodelprob-abilitiesinBayesianmodelselection/averaging;seeKassandWasserman(1995),Raftery(1995),DiCiccioetal.(1997)orHastieetal.(2009,Chap.7.7).

Inthispaper,weareconcernedwithBayesianinformationcriteriainthecontextofsingularmodelselectionproblems,thatis,problemsthatinvolvemodelswithFisher-informationmatricesthatmayfailtobeinvertible.Forexample,duetothebreak-downofparameteridenti?ability,theFisher-informationmatrixofamixturemodelwiththreecomponentdistributionsissingularatadistributionthatcanbeobtainedbymixingonlytwocomponents.Thisclearlypresentsafundamentalchallengeforselectionofthenumberofcomponents.Otherimportantexamplesofthistypeincludedeterminingtherankinreduced-rankregression,thenumberofKeywordsandphrases.Bayesianinformationcriterion,factoranalysis,mixturemodel,modelselection,reduced-rankregression,singularlearningtheory,Schwarzinformationcriterion.

1arXiv:1309.0911v1 [stat.ME] 4 Sep 2013

Beysian

2MATHIASDRTONANDMARTYNPLUMMER

factorsinfactoranalysisorthenumberofstatesinlatentclassorhiddenMarkovmodels.Moregenerally,alltheclassicalhidden/latentvariablemodelsaresingular.AsdemonstratedbySteeleandRaftery(2010)forGaussianmixturemodelsorLopesandWest(2004)forfactoranalysis,BICcanbeastate-of-the-artmethodforsingularmodelselection.However,whileBICisknowntobeconsistentintheseandothersingularsettings(Keribin,2000,Drtonetal.,2009,Chap.5.1),thetechnicalargumentsinitsBayesian-inspiredderivationdonotapply.Inanutshell,whentheFisher-informationissingular,thelog-likelihoodfunctiondoesnotadmitalarge-sampleapproximationbyaquadraticform.Consequently,theBICdoesnotre?ectthefrequentistlarge-samplebehavioroftheBayesianmarginallikelihoodofsingularmodels(Watanabe,2009).Incontrast,thispaperdevelopsageneralizationofBICthatisnotonlyconsistentbutalsomaintainsarigorousconnectiontoBayesianmodelchoiceinsingularsettings.ThegeneralizationishonestinthesensethatthenewcriterioncoincideswithSchwarz’swhenthemodelisregular.

Thenewcriterion,whichweabbreviatetosBIC,ispresentedinSection3.Itreliesontheoreticalknowledgeaboutthelarge-samplebehaviorofthemarginallikelihoodoftheconsideredmodels.Section2reviewsthenecessarybackgroundonthistheoryasdevelopedbyWatanabe(2009).ConsistencyofsBICisshowninSection4,andtheconnectiontoBayesianmethodsisdevelopedinSection5.InthenumericalexamplesinSection6,sBICachievesimprovedstatisticalinferenceswhilekeepingcomputationalcostlow.ConcludingremarksaregiveninSection7.

2.Background

LetYn=(Yn1,...,Ynn)denoteasampleofnindependentandidenticallydis-tributedobservations,andlet{Mi:i∈I}bea?nitesetofcandidatemodelsforthedistributionoftheseobservations.ForaBayesiantreatment,supposethatwehavepositivepriorprobabilitiesP(Mi)forthemodelsandthat,ineachmodelMi,apriordistributionP(πi|Mi)isspeci?edfortheprobabilitydistributionsπi∈Mi.WriteP(Yn|πi,Mi)forthelikelihoodofYnunderdata-generatingdistributionπifrommodelMi.Let??(2.1)L(Mi):=P(Yn|Mi)=P(Yn|πi,Mi)dP(πi|Mi).

Mi

bethemarginallikelihoodofmodelMi.Bayesianmodelchoiceisthenbasedontheposteriormodelprobabilities

P(Mi|Yn)∝P(Mi)L(Mi),i∈I.

TheprobabilitiesP(Mi|Yn)canbeapproximatedbyvariousMonteCarlopro-cedures,seeFrielandWyse(2012)forarecentreview,butpractitionersalsoof-tenturntocomputationallyinexpensiveproxiessuggestedbylarge-sampletheory.TheseproxiesarebasedontheasymptoticpropertiesofthesequenceofrandomvariablesL(Mi)obtainedwhenYnisdrawnfromadata-generatingdistributionπ0∈Mi,andweletthesamplesizengrow.

Inpractice,apriordistributionP(πi|Mi)istypicallyspeci?edbyparametrizingMiandplacingadistributionontheinvolvedparameters.Soassumethat(2.2)Mi={πi(ωi):ωi∈?i}

withdi-dimensionalparameterspace?i?Rdi,andthatP(πi|Mi)isthetrans-formationofadistributionP(ωi|Mi)on?iunderthemapωi→πi(ωi).The

Beysian

ABAYESIANINFORMATIONCRITERIONFORSINGULARMODELS3

marginallikelihoodthenbecomesthedi-dimensionalintegral

??(2.3)L(Mi)=P(Yn|πi(ωi),Mi)dP(ωi|Mi).

?i

TheobservationofSchwarzandothersubsequentworkisthat,undersuitabletechnicalconditionsonthemodelMi,theparametrizationωi→πi(ωi)andthepriordistributionP(ωi|Mi),itholdsforallπ0∈Mithat

(2.4)logL(Mi)=logP(Yn|π?i,Mi)?dilog(n)+Op(1).2

Here,P(Yn|π?i,Mi)isthemaximumofthelikelihoodfunction,andOp(1)standsforaremainderthatisboundedinprobability,i.e.,uniformlytightasthesamplesizengrows.The?rsttwotermsontheright-handsideof(2.4)arefunctionsofthedataYnandthemodelMialoneandmaythusbeusedasamodelscoreorproxyforthelogarithmofthemarginallikelihood.

De?nition2.1.TheBayesianorSchwarz’sinformationcriterionformodelMiis

diBIC(Mi)=logP(Yn|π?i,Mi)?log(n).2

Brie?yput,thelarge-samplebehaviorfrom(2.4)reliesonthefollowingpropertiesofregularproblems.First,withhighprobability,theintegrandin(2.3)isnegligi-blysmalloutsideasmallneighborhoodofthemaximumlikelihoodestimatorofωi.Second,insuchaneighborhood,thelog-likelihoodfunctionlogP(Yn|πi(ωi),Mi)canbeapproximatedbyanegativede?nitequadraticform,whileasmoothpriorP(ωi|Mi)isapproximatelyconstant.Theintegralin(2.3)maythusbeapproxi-matedbyaGaussianintegral,whosenormalizingconstantleadsto(2.4).Were-markthatthisapproachalsoallowsforestimationoftheremaindertermin(2.4),givingaLaplaceapproximationwitherrorOp(n?1/2);comparee.g.,TierneyandKadane(1986),Haughton(1988),KassandWasserman(1995),Wasserman(2000).

Alarge-samplequadraticapproximationtothelog-likelihoodfunctionisnotpossible,however,whentheFisher-informationmatrixissingular.Consequently,theclassicaltheoryalludedtoabovedoesnotapplytosingularmodels.Indeed,(2.4)isgenerallyfalseinsingularmodels.Nevertheless,asymptotictheoryforthemarginallikelihoodofsingularmodelshasbeendevelopedoverthelastdecade,culminatinginthemonographofWatanabe(2009).Theorem6.7inWatanabe(2009)showsthatawidevarietyofsingularmodelshavethepropertythat,forYndrawnfromπ0∈Mi,

(2.5)logL(Mi)=????logP(Yn|π0,Mi)?λi(π0)log(n)+mi(π0)?1loglog(n)+Op(1);seealsotheintroductiontothetopicinDrtonetal.(2009,Chap.5.1).Ifthese-quenceoflikelihoodratiosP(Yn|π?i,Mi)/P(Yn|π0,Mi)isboundedinprobability,thenwealsohavethat

(2.6)logL(Mi)=????logP(Yn|π?i,Mi)?λi(π0)log(n)+mi(π0)?1loglog(n)+Op(1).

Beysian

4MATHIASDRTONANDMARTYNPLUMMER

Forsingularsubmodelsofexponentialfamiliessuchasthereduced-rankregressionandfactoranalysismodelstreatedlater,thelikelihoodratiosconvergeindistri-butionandarethusboundedinprobability(Drton,2009).Formorecomplicatedmodels,suchasmixturemodels,likelihoodratioscanoftenbeshowntoconvergeindistributionundercompactnessassumptionsontheparameterspace;comparee.g.Aza¨?setal.(2006,2009).Suchcompactnessassumptionsalsoappearinthederivationof(2.5).Wewillnotconcernourselvesfurtherwiththedetailsoftheseissuesasthemainpurposeofthispaperistodescribeastatisticalmethodthatcanleveragemathematicalinformationintheformofequation(2.6).

Thequantityλi(π0)isknownasthelearningcoe?cient(oralsoreallog-canonicalthresholdorstochasticcomplexity)andmi(π0)isitsmultiplicity.IntheanalyticsettingsconsideredinWatanabe(2009),itholdsthatλi(π0)isarationalnumberin[0,di/2]andmi(π0)isanintegerin{1,...,di}.Weremarkthatinsingularmodelsitisverydi?culttoestimatetheOp(1)remaindertermin(2.6).Wearenotawareofanysuccessfulworkonhigher-orderapproximationsinstatisticallyrelevantsettings.

Example2.1.Reduced-rankregressionismultivariatelinearregressionsubjecttoarankconstraintonthematrixofregressioncoe?cients(ReinselandVelu,1998).Keepingonlywiththemostessentialstructure,supposeweobservenindependentcopiesofapartitionedzero-meanGaussianrandomvectorY=(Y1,Y2),withY1∈RNandY2∈RM,andwherethecovariancematrixofY2andtheconditionalcovariancematrixofY1givenY2areboththeidentitymatrix.Thereduced-rankregressionmodelMiassociatedtoanintegeri≥0postulatesthattheN×MmatrixπintheconditionalexpectationE[Y1|Y2]=πY2hasrankatmosti.

InaBayesiantreatment,considertheparametrizationπ=ω2ω1,withabsolutelycontinuouspriordistributionsforω2∈RN×iandω1∈Ri×M.Letthetruedata-generatingdistributionbegivenbythematrixπ0ofrankj≤i.AoyagiandWatanabe(2005)derivedthelearningcoe?cientsλi(π0)andtheirmultiplicitiesmi(π0)forthissetup.Inparticular,λi(π0)andmi(π0)dependonπ0onlythroughthetruerankj.Foraconcreteinstance,takeN=5andM=3.Thenthemultiplicitymi(π0)=1unlessi=3andj=0inwhichcasemi(π0)=2.Thevaluesofλi(π0)are:

j=0j=1j=2j=3

i=00

37i=19i=2369111315i=3Notethatthetableentriesforj=iareequaltodim(Mi)/2,wheredim(Mi)=i(N+M?i)isthedimensionofMi,whichcanbeidenti?edwiththesetofN×Mmatricesofrankatmosti.ThedimensionisalsothemaximalrankoftheJacobianofthemap(ω1,ω2)→ω2ω1.ThesingularitiesofMicorrespondtothepointswheretheJacobianfailstohavemaximalrank.Thesehaverank(ω2ω1)<i.Thefactthatthesingularitiescorrespondtoadropinrankpresentsachallengeformodelselection,whichhereamountstoselectionofanappropriaterank.

SimulationstudiesonrankselectionhaveshownthatthestandardBIC,withdi=dim(Mi)inDe?nition2.1,hasatendencytoselectoverlysmallranks;fora

Beysian

ABAYESIANINFORMATIONCRITERIONFORSINGULARMODELS5

recentexampleseeChengandPhillips(2012).Thequotedvaluesofλi(π0)giveatheoreticalexplanationastheuseofdimensioninBICleadstooverpenalizationofmodelsthatcontainthetruedata-generatingdistributionbutarenotminimalinthatregard.?

Determininglearningcoe?cientscanbeachallengingproblem,butprogresshasbeenmade.Forsomeoftheexamplesthathavebeentreated,wereferthereadertoAoyagi(2010a,b,2009),WatanabeandAmari(2003),WatanabeandWatanabe(2007),RusakovandGeiger(2005),YamazakiandWatanabe(2003,2005,2004),andZwiernik(2011).Theuseoftechniquesfromcomputationalalgebraandcombi-natoricsisemphasizedinLin(2011);seealsoArnol??detal.(1988),Vasil??ev(1979).

Thementionedtheoreticalprogress,however,doesnotreadilytranslateintopracticalstatisticalmethodologybecauseonefacestheobstaclethatthelearningcoe?cientsdependontheunknowndata-generatingdistributionπ0,asindicatedinournotationin(2.6).Forinstance,fortheproblemofselectingtherankinreduced-rankregression(Example2.1),theBayesianmeasureofmodelcomplexitythatisgivenbythelearningcoe?cientanditsmultiplicitydependsontherankwewishtodetermineinthe?rstplace.Itisforthisreasonthatthereiscurrentlynostatisticalmethodthattakesadvantageoftheoreticalknowledgeaboutlearningcoe?cients.Intheremainderofthispaper,weproposeasolutionforhowtoovercometheproblemofcircularreasoningandgiveapracticalextensionoftheBayesianinformationcriteriontosingularmodels.

3.NewBayesianinformationcriterionforsingularmodels

Ifthetruedata-generatingdistributionπ0wasknown,then(2.6)wouldsuggestreplacingthemarginallikelihoodL(Mi)by

(3.1)L???i,Mi)·n?λi(π0)(logn)mi(π0)?1.π0(Mi):=P(Yn|π

Thedata-generatingdistributionbeingunknown,however,weproposetofollowthestandardBayesianapproachandtoassignaprobabilitydistributionQitothedistributionsinmodelMi.Wetheneliminatetheunknowndistributionπ0bymarginalization.Inotherwords,wecomputeanapproximationtoL(Mi)as

??(3.2)L??L??Qi(Mi):=π0(Mi)dQi(π0).

Mi

ThecruxofthematternowbecomeschoosinganappropriatemeasureQi.BeforediscussingparticularchoicesforQi,westressthatanychoiceforQireducestoSchwarz’scriterionintheregularcase.

Proposition3.1.IfthemodelMiisregular,thenitholdsforallprobabilitymea-suresQionMithat

BIC(Mi)L??.Qi(Mi)=e

Proof.Inourcontext,aregularmodelwithdiparameterssatis?esλi(π0)=di/2andmi(π0)=1foralldata-generatingdistributionsπ0∈Mi.Hence,theintegrandin(3.2)isconstantandequalto

BIC(Mi)L??.π0(Mi)=e??