gem5, GPGPUSim, McPAT, GPUWattch, Your favorite simulator here

Stack Layers

AlgorithmApplicationCompilerOSIO

Mem Controler

CachesCore Microarch

CircuitsGatesTransistorsPhysics

Small FootprintMeduimFootprintLargeFootprint

gem5,GPGPUSim,McPAT,GPUWattch,"Yourfavoritesimulatorhere"

ConsideredHarmful

TonyNowatzki

Basic Energy Mathematical

tjn@cs.wisc.edumenon@cs.wisc.eduCharacterizationProofCustom First-Program Reasoned

Order ModelsAnalysisArguements

Cycle Accurate Simulation

JaikrishnanMenonChen-HanHoKarthikeyanSankaralingam

UniversityofWisconsin-Madison

ho9@wisc.edu

karu@cs.wisc.edu

MuchasDijkstra,in1968,observedthedangersofrely-Best Research Approach?How do I ingonthegotostatement,weobservethatthedominant

evaluate relianceonquantitativesimulatorsishavingadetrimentalef-fectonour eld.Overtime,simulatortoolshavebecomemoreSatisfy Program Committee?my idea?

sophisticated.FromthesimpledaysofthenowdebunkedSim-“Cycle Accurate Simulation”pleScalarwithitsRUU-basedOOOmodelwith xedDRAM

latency,tothegem5+DramSim+GPGPUSim+McPATmashupsimulator,wehavecomealongwayinwhatarchitectsareclaimingasvalidatedtools.Weargue,though,thatnewgener-ationsofsimulatorsareoftenover ttedtocertainbenchmarksorcon gurationsforvalidationandcanhavesigni cantmod-elingerrorsthatresearchersarenotawareof.Thoughtheexis-tenceoftheseerrorsareunsurprising,theycancauseunawareuserstoderiveincorrectconclusions.Simultaneously,andevenmoreproblematic,isthatreviewersdemandresearchersinappropriatelyusethesetools.Weenumerateeightcommon,butnotacknowledgedorrecognizedpitfallsofsimulatorsorsimulatoruse,consideringfourmodernsimulationinfrastruc-tures.Weproposethattheevaluationstandardsforaworkshouldmatchit’s“footprint,”thebreadthoflayerswhichthetechniqueaffects,andconcludewithouropiniononhowtoescapeoutofour eld’ssimulate-or-rejectmindset.

1.Introduction

Foranumberofyearswehavebeenfamiliarwiththeobserva-tionthatthequalityofarchitectureresearchersisadecreasingfunctionofthere-lianceonquantitativearchitecturesimulatorsinthearchitecturepaperstheyproduce.Morerecentlywediscov-eredwhytheuseofarchitecturesimulatorshassuchdisastrouseffects,andwebecameconvincedthatthearchitecturesimulatorshouldbeabolishedfromall"higherlevel"architectureresearch.Atthattimewedidnotattachtoomuchimportancetothisdiscovery;wenowsubmitourconsiderationsforpub-lication1.MuchasDijkstraobservedtheeraofrelianceonthegotostatementwashavinganegativeeffect,weobservetheeraofover-relianceonquantitativesimulatorsishavingadetrimentaleffectonthe eld,andshouldcometoanend.Weobservethatsimulation,inparticular“detailed”toolsthatprovidecycle-accurateperformanceestimates,areaesti-mates,powerandenergyestimates,asavehicleforarchitec-paragraphisreproducedandcriticismsaremodi edfromDijkstra’s

seminalACaseagainsttheGOTOStatement[10].Additionsareinitalics.

1This

Figure1:Thefootprintofatechnique(thescopeoflayersitinteractswith),andthechoiceresearchersfacebetweenap-propriateevaluationandPC-compliantevaluationpractices.

tureresearchisubiquitous.FromthesimpledaysofthenowdebunkedSimpleScalar[8]withitsRUU-basedOOOmodel+ xedmemorytogem5+DramSim+GPGPUSim+McPATmashupsimulator,wehavecomealongwayinwhatarchi-tectsareclaimingasvalidatedsimulators2.Thislevelofaddeddetailhasledtothebeliefthatwehavebettertoolsandaredoingbetterandbetterquantitativeevaluation.Ithasalsoledtothepreponderanceofpapersrelyingonsuchtoolsandhascreatedanimplicitstandardandtemplateofhowquantitativeevaluationmustbedone.Thisrelianceandbeliefinsuchdetailedtoolsishurtingthe eldandcreatingvariouspitfalls.Partoftheproblemisthatthesetoolsarecommonlyover- ttedforvalidation,meaningthattheirparametersaretunedsuchthattheyareaccurateonlyonasmallsetofbenchmarksorcon gurationparameters.Theimplicationofover ttingisthatsimulatormodelscapturethenoiseratherthanthefunda-mentalrelationshipsandtradeoffs.Inaddition,simulatortoolsoftenhavesigni cantmodelingerrorswhicharenoteasily

2We

remarkthatnotallsimulators’authorsthemselvesclaimvalidation.

accessiblebyusers.Overall,therelianceonsimulatorsarecreatingmanypitfallsbothintechnicalaspectsandinhurtingthe eldbydistortingreviewerexpectationsofwhatentailsgoodquantitativeevaluation.

Asawayforwardforresearchers,webelievethatthecorrectapproachdependsonthefootprint,orlayersofthestackwhichthetechniqueaffectsorrelieson,andthatthereisnoone-size- ts-allsolutiontoarchitectureresearch.Figure1highlightshowdifferenttechniques,representedbygrayboxes,canaf-fectdifferentstacklayers.Unfortunately,itistoooftenthecasethatresearchersmakethechoiceofresearchapproachbasedonwhatwillgettheirpaperaccepted,ratherthanwhatisthemostscienti c.Werevisitaversionofthis gurewithspeci cexamplesinSection4.Mostimportantly,webelievethatreviewersmustrecalibratetheirevaluationstandards,andappropriatelygaugethemtothefootprintoftheresearch.Webelievethisissueisimportantandvitalnowasmoreresearchinour eldismovingtowardlargerfootprints,evidencedbyrecentkeynotes[7]andfundingcalls[21].Restrictingour-selvestoanill-suitedone-size- tsallapproachcouldcurtailscienti cadvancementoftheseefforts.

Thispaperenumerateseightcommon,butnotacknowl-edgedorrecognizedpitfalls,consideringfourmodernsim-ulationinfrastructures:gem5[5],McPAT[19],GPGPUSimV2.x[3],andGPUWattch[18].Webeginthispaperwithasectiondescribingerrorsinpopularsimulators,whichweusetosubstantiatethepitfalls.Indiscussingsimulatorerrorsandpitfalls,ourgoalisnottooffendorcriticizebuttoinformandprovokethoughtfuldiscussion.Weconcludewithstrategieswhichcanallowustoescapeoutofour eld’stemplatizedsimulate-or-rejectmindset.

andwebelievetheseproblemscanbetackledwithoutdif -culty.Werevisitthebene tsofcommunitydriventoolsinSection4.2.

Conservative/ObscureDefaultforWritebackMechanism:Thegem5OOOmodelonlyschedulesinstructionsforissueifthereareguaranteedtobeenough“writebackbuffers”forthem,wherethetotalbuffersarecalculatedbywriteback-width×writeback-depth.Thedefault,acrossallISAs,isawriteback-depthof1.Thismeansthatifafewlonglatencyinstructionsholdupwriteback-bufferslots,thentheeffectiveissuewidthgoesto0.ForanOOO2-widecorewithbench-marksthathavelong-latencymemoryreferences,adding5bufferslotsincreasesperformancebymorethan5X.WedonotbelievethistradeoffisrepresentativeofrealOOOdesigns,andthisimportantparameterisnotsuf cientlyde nedinthedocumentationorsourcecode.

InconsistentPipelineReplayMechanism:gem5’sOOOmodelforspeculativeinstructionschedulingandpipelinere-playappearstobebothcontradictoryandunnecessarilycon-servative.Toexplain,adeeplypipelinedOOOcoremustspeculativelyscheduleinstructionstoenableback-to-backex-ecution.Whenanunexpectedlatencyoccurs,thescheduleforthemiss-dependentinstructionsneedstobecorrected.Ingem5,whenaloadissuestoablockedcache,gem5conser-vativelymodelsthe“correction”tothespeculativescheduleby ushingtheentirepipeline.Thelargerissueisthatafterapipeline ush,instructionsareimmediatelyrescheduled,evenifthecacheremainsblocked.Thisleadstoacycleofrepeated ushingoftheentirepipeline.Whiletheperformancedoesnottakeasigni canthit,theamountofenergycandoubleonsomebenchmarksversusadesignwithahandfulmoreMSHRstopreventthecachefromblocking.

Tobeconsistent,anarchitecturewhich ushesthepipelineonacache-blockshouldalso ushthepipelineonothervariable-latencyevents.However,gem5doesnot ushthepipelineoneventslikecachemisses,whichwouldhavevari-ablelatency.Inshort,thepipelinereplaymechanismissimul-taneouslybothhighlyconservativeandoptimistic.

Inef cient/MislabeledMicro-ops:gem5micro-opsareop-timizedmoreforcorrectnessandeconomyratherthanef -ciency.Oneexampleisthatthesamemicro-opthatperformsconditionalmovesalsoperformsregularregistermoves.Thismeansthatregularmoveswillincurthedynamicdependenceandenergycostofreadingthedestinationregister,eventhoughtheyarecompletelyoverwritingit.Also,thoughthegem5 agregisterimplementationhasgreatlyimprovedinrecentversions,afewinstructionsstillrequireextradependenciesandregisterreadsbecauseof agregistergrouping.Oneex-ampleishowlogicalinstructions(likeXOR)don’twritetheAF ag,butsinceitisgroupedwiththeother ags,itmustbereadbeforewritten.Thisisarguablyacceptable,butdif culttounderstandandaccessasauser.

Animportantyet xableproblemisthatsomemicro-opsare2.Errorsinsimulators

Webeginby rstoutliningsomeexampleinstancesoferrorsinmainstreamandpopularsimulators.Webelievetheexis-tenceoftheseerrorsshouldneitherbesurprising,noraretheyintendedasanattackonparticularsimulatorsorsimulatorauthors;anylargebodyofcodewillhaveerrors.Weonlybringattentiontoaddsomecontexttoourpitfallsandaidinsubstantiation.Ifanything,ourcriticismissquarelyaimedatusersofsuchtools,forexampleGovindarajuetal.[12].Errorreportsareavailableathttp://www.cs.wisc.edu/vertical/sim-harmful,whichhavebeenveri edbyatleastoneotherpersonnotaf liatedwithourresearchgroup.Theirpurposeistopointoutthetypeofproblemswhichcanbedetrimentalifusersarenotaware.Inthissection,foreachtool,we rstpresentobservationsaboutanissue,thengiveouropinionstheissue’simplications.

2.1.gem5

Tobeclear,theerrorsdiscussedinthissectionhaveonlybeenveri edontheX86versionofgem5,andthemicro-opis-suescanonlyapplytoX86.Also,someofthebelowerrorshavebeencommunicatedtothequiteactivegem5community,

内容需要下载文档才能查看

which(nofp

memorymultiplies,upinthedatafromgem5toperformenergyanalysison oatingpointcodewouldproduceincorrectresultsbypotentiallyintegerfactors.2.2.McPAT

Unclear/Over ttedFunctionalUnit(FU)EnergyModel-ing:IntheMcPATmodel,ifthecoreisOOO,thenasmalldynamiccomponentofenergyisaddedforeachFUregardlessofwhethertheFUisbeingused.Thisconstantiscitedas“averagenumbersfromIntel4Gand773Mhz(Wattch)”.WhythisoccursinOOObutnotInorderprocessorscouldbeduetoover ttinginvalidation.Anotherrelatedexampleisfortheper-accessenergyofanFU.Iftheprocessoris“embed-ded,”thenthispowerisdividedbytwo,citing:“AccordingtoARMdataembeddedprocessorhasmuchlowerperaccenergy”.Whetherornotthese(inouropinion)seeminglyarbitrarydecisionsarevalidornot,sincetheyarenoteas-ilyaccessibleordecipherablebytheuser,theymaycometoincorrectconclusionsaboutthequantitativeresults.

ErrorinPipelineandClockPowerMcPATcalculatesanestimateofthepipelineandclockpowerconsideringswitch-ingfactorsinpipeline ip- ops.ThispowerisnotreporteddirectlyinMcPAT,rather,itisdistributedequallyamongstthevariousprocessorstructures,makingitdif culttodeter-minewhenthereareerrors.Figure2showsthedynamicpowerwhichthepipelinecontributesforinorderandOOOprocessors(65nm),whichcanonlybeseenbyinstrumentingtheMcPATsourcecode.OurexperimentsshowthatthiscomponentofpoweriseffectivelydroppedforallOOOcoreexperimentslastinglongerthanafewcycles.Theerrorappearstobeintro-ducedwhenconvertingbetweenpowerandenergy,whereafactorofthenumberofcyclesislostfortheOOOcoreonly.ThisapparenterrorisinallversionsofMcPATthatwetested(fromv0.7tov1.1March2014).Theimplicationofthiserroristhatitcreatesuncertaintyabouttheestimationofpipelineandclockpower3.

2.3.GPGPUSimV2.x

Inthissubsection,weconsiderawidelyadoptedversionoftheGPGPUSimtool,anddescribeseveralmissingorabstractedcomponentsofitsarchitecturalmodel.WediscussGPG-PUSimV2.x,eventhoughitisnotthelatestversionofthetool,speci callybecausemanyresearchersarestillusingthisversion[16,17],andwebelievethefollowingclaimsabout

thoseawareoftheexactdetailsofMcPAT,whenitisusedina ne-grainedmode(calledeverycycle-asopposedtotheXMLinterfaceofcallingatendofmillionsofcyclesofsimulation),thisissuewilldisappear.However,theXMLbulkmodeisthemostprevalentusageofMcPATinliterature.

3For

Figure2:McPATpipelinepowerfora65nmidleprocessor.

itsmodelingfeaturescanbemadewithoutcontroversy.GPG-PUSim3.xhas xedmanyoftheseissues,(seeslide20inthetutorial[2]).

RegisterFilemicroarchitecture:Theoperandcollector(single-portedregister lebanks+arbiter+X-bar+col-lectorunits)ismodeledassuming xedlatencyaccessestotheSRAMwithsomeadditionalqueuinglatency.Itdoesnotmodellow-leveldetails,likecontention,whichimpactperformanceinhigh-computebandwidthscenarios.

Thread/warp/wavefrontschedulinganddispatch:Threadschedulingisfunctional,andwhileanumberofdifferentwarpschedulingschemesareimplemented,thesearenotmodeledinthemicroarchitecture,theyaresimplygeneratedfunctionally.

BranchdivergencestructuresandBranchUnit:Similartothreaddispatch,branchdivergencetrackingstructuresarefunctionallyemulatedaspartoftheabstracthardwaremodel,andthebranchunitmicroarchitectureisnotmodeledatthecycle-level.

Theeffectofomittingthedetailedmodelingofthesemicro-architecturalfeatures,andaccountingforthemabstractlyorfunctionally,isthatitencouragesarchitectsnottoreasonaboutthemicroarchitecturalfeasibilityoftheproposedtechnique.Forexample,considerdevelopingandevaluatinganon-trivialwarpschedulingtechniqueinGPGPUSimV2.x.Itsmodelwouldbeafunctionalone,meaningthatitwouldnotcapturetheindividualcomponentsofthehardware,theircommunica-tion,ortheirpipelinestages.ThiswouldbetantamounttoaCPUload-storequeuedesignevaluationwhichfunctionallymodelsthedependencepredictor,whileignoringcacheportcontentionetc.FortheCPUdomain,thismashupofhigh-levelmodelingandlow-levelsimulationwouldnotbeconsideredsuf cienttounderstandtheeffectivenessofatechniquequan-titatively.2.4.GPUWattch

GiventhestraightforwardreadingoftheGPUWattch[18]pa-per,itsmethodologyhasaformofmodelingerrorwhichwecall“mathematicallyirrelevant”modeling.Wede nethisasmodelwhich,whentakenasawhole,containsmathematically-irrelevantsub-components.The rstpartofthissubsection

willdescribehowthisformoferrorappliestothemethodol-ogy(aspresented)inLengetal.[18].Essentially,thedetailedmodelingusingMcPAT,empiricalmemorymodelsandsyn-thesisbasedmodelsaren’tmeaningfultothe nalobtainedprediction.However,additionalunpresenteddetailsoftheGPUWattchmethodologyhelpjustifythedetailedmodeling.Therefore,wewillsubsequentlydiscusssomeofthesedetails,andconcludewiththeimplicationsforappropriatemodelus-age.

GPUWattchPowerModeling(aspresented)GPUWattchmodelsthecycle-levelpowerofGPUarchitecturesby rstus-ingGPGPUSimtoobtainactivityfactors.Then,GPUWattchcalculatesthedynamicpowerofaparticularbenchmarkPbench,asthesumoftheactivityfactorsαbench,comp,multipliedbythemaximumpowerofthecomponent,PMAXcomp.ThePMAXcomppowervalueisobtainedthroughhighlydetailedmodelingusingacombinationofMcPAT-basedmodeling,em-piricalmodels,andsynthesis-basedmodels.TheGPUWattchauthorsstatethatsinceMcPATistunedtoCPUs,andsincetherearemanyundocumentedGPUfeatures,theyneedtocorrectforthisbyaddinganerrortermforeachcomponentxcomp,anduseleast-squaresestimation(linearregression)toestimatetheerrors.Their nalmodelfordynamicpoweris:

Pbench=

Ifthismethodologywasdirectlyapplied,thecomputationofPMAXismathematicallyirrelevant.Performingalin-eartransformationontheexplanatoryvariablesofalinearregressiondoesnotaffecttheerrororpredictionaccuracy.Infact,runningthebelowregression,whichdoesnothavePMAXcompvalues,wouldbemathematicallyequivalent,andtheresultingregressioncoef cientsaresimplyscaledasfol- lows:xcomp=PMAXcomp×xcomp.

Pbench=

comp

∑

αbench,comp×xcomp

(2)

WhatthismeansisthatthePMAXcompvariablesaremath-ematicallymeaninglesstothe nalmodel.Therefore,userswhoapplytheGPUWattchmethodologyaswrittenwillputunnecessaryeffortintodetailedpowermodeling(whichwouldincludeMcPAT,empiricalmodelandsynthesismodeldevel-opment).

GPUWattchPowerModeling(asimplemented)Themethodologywhichisimplementedactuallydoesemploythedetailedpowermodelingresultsduringthescalingparameterselectionforsomepurpose,asweexplainnext5.

First,insteadofscalingtheinternalpowervaluesofMc-PAT’soptimizationframework,whatisactuallyscaledaretheactivitycountswhicharefedasinputstoMcPAT.ThisassumesthatMcPAT’schoiceofcomponentswouldbeunaf-fectedbythedifferentpowerscalingfactorappliedtovariouscomponents.

Second,insteadofautomaticlinearregression,theycal-culatetherootmeansquareerroroftheirpredictionsandmanuallymodifythescalingcoef cientstoreducetheerror.Thisalonewouldbethemanualequivalenttolinearregression,andhencewouldstillhavethemathematicalirrelevancebug.However,theauthorsalsoboundthescalingcoef cientsbybetween10×to50×foron-coreandoff-corecomponentsrespectively(here,theauthorsexplainthattheboundischosenbasedonthecon denceintheoriginaldetailedmodel).Theauthorsobservethatwithoutboundingthescalingcoef cients,theerrorisactuallyless:amathematically“better”model.However,theper-componentbreakdownswithpurelinearre-gressiondonotmatchexpectedintuition(toobigornegativescalingfactors).Therefore,theboundsonscalingfactorsserveasaroughguidelineinattainingaplausiblepowerdistribution.Overall,webelieveitispossibletouseapurelymathematicalapproach,applyingthesametypeofroughintuition,toachievethesamequalityofresultswithoutdetailedpowermodelinglikeMcPAT.

WhatareGPUWattch’sappropriateusecases?ThemethodologybehindtheGPUWattchpowermodelhasim-plicationsforitsappropriateusage.Ourpositionisthatitthatitcanonlybeappropriatelyemployedwhenaphysicalartifactwithmeasurablepowernumbersareavailable.Forthe

authorsgraciouslyprovidedusdetailsontheirmethodology,andwe

taketheblameifwehavemadeanymistakesinreproducingithere.

5The

comp

∑

αbench,comp×PMAXcomp×xcomp(1)

Inlinearregressionterminology,αbench,comp×PMAXcomparetheexplanatoryorinputvariables,xcomparetheregressioncoef cientsandPbenchisthedependentvariable.Atthispoint,theauthors’methodologyisasfollows:

Weiterativelyre nethepowermodelonthebasisofthesourcesofthevariousinaccuraciesthatLSE[regression]identi es.Forinstance,inourinfrastructure(i.e.,McPAT)thepowerestimationforcertaincomponentsisbiasedtowardCPUimplementations.WenarrowtheresultinginaccuracygapfortheGPUpowermodelby xingourinitialassump-tionsabouttheimplementationandthenapplyingthescalingfactorsthatareobtainedfromLSE.

WecontendthatadirectinterpretationoftheirmethodologywouldbetoruntheregressioninEquation1usingmeasuredvaluesofPbenchofsomemicrobenchmarksto ndthecompo-nentwiseerrorsxcomp,thenmodifythesourcecodetomultiplytheoriginalcomponentwisepowerαbench,comp×PMAXcompbythe“scalingfactor”forthatcomponent,xcomp,toobtainthe nalpowerestimate.Notethatthisprocedureisperformedonaplatformspeci cbasis4.

theXML lesprovidedinthetoolforGTX480and

QuadroFX5600,thescalingcoef cientsaretheseriesof32paramnamesstart-ingatline31(TOT_INST,FP_INT,IC_H,etc.)Inthesourcecode,ingpgpu_sim_wrapper.c,theseareusedinmethodslikeset_inst_power,set_regfile_poweretc.toscaleuptheMcPATcomputedvalues.

4Considering

twoplatformswhichhavecon gurationsnow,theGTX480wasreleasedin2010,andthetheQuadroFX5600isevenolder.GeneratinganewGPUWattchcon gurationrequiresattainingdetailedpowermeasurements,includingphysicallyinstrumentingtheGPUpowersupplywithsensingresistors,followedbyanapplicationofthemanualerror-minimizationproceduredescribedabove.

Thereasonwhyaphysicalartifactisnecessaryisthatthescalingfactors,xcomp,areplatformspeci c.Asanexample,considerthepowerofregister leaccessintheGTX480andFX5600.TheMcPATscalingbetweenthetwodesignsdoesnotcapturetheirarchitecturaldifferences,whichshowsupintheGPUWattchmodelastheratiooftheirregister lescalingfactors,whichis1.7×.

IfwewanttoconsiderahypotheticalGPUwithdifferentcon gurationparameters,withoutchangingthescalingfactors,weshouldnotexpecttheGPUWattchpowermodeltobevalid.Toexplain,theaveragescalingfactormagnitudeis22×fortheGTX480,and8×forQuadroFX5600.Toclaimthatthehypo-theticalGPUcon gurationisvalid,theargumentthatwouldhavetobemadeisthatMcPATgetsthepowerwrongbyanor-derofmagnitude,butsomehowcangettherelativescalingofcomponentscorrect.Thisisapositionwebelieveisuntenablewithoutevidence.Thelackofvalidatedcon gurabilitywouldimpedeanaccuratedesignspaceexploration.

GoodusesforGPUWattchwouldincludeestimatingtheenergyimpactofpolicychangeswhichaffecttheactivityfac-tors,orinaddingcomponentswhichhaveexternallyvalidatedpowercharacteristics(again,ifthetargetarchitecturealreadyhasaGPUWattchpowermodel).Revisitingtheconceptoffootprint,thesearebothsmall-footprintevaluationscenarios.WeclarifyherethattheauthorsofGPUWattchnevermentiondesignspaceexplorationassomethingtheirtoolismeantfor.Soagain,ourcriticismisaimedattoolusersratherthandevel-opers,andadditionallythereviewerwhonowthinksenergyestimationforGPUresearchisalwaysdoable.

accessible,makingbugsdif cultto ndevenwithcarefuldataanalysis.Thatisbecausemanyofthefeaturesareobscuredbehindimplicitassumptions,lackofdocumentationandlackofgoodreportingofresults.OneexampleishowthepipelinepowerisreportedinMcPAT.Sinceitisimplicitlydistributedamongsttheindividualcomponentsoftheprocessor,whatappearstobeasigni canterrorisobfuscated.Errorslikethisputresearchersinadif cultposition.Shouldtheygo xthetoolwhichisalreadyvalidated?Andwhatifanothererrorintheoppositedirectioniscancelingouttheeffects?

Suggestions:Authorsshould rstvalidateandsanitycheckthesimulatorindividually.Further,whenitmakessense,theyshouldconsiderbuildingtrace-driventoolsthatmodelthe rstordereffectstheyareawareoff,insteadofusingcycle-accuratetools.Webelieveitisbettertohaveatoolwithknownabstractionerrorsthananunknownblackbox.Reviewersandthecommunityneedstochangeitsmindsetaswell–havingblindfaithin“standardtools,”whilecompletelydiscountingothertoolsisnotappropriate.Werevisittheissueofopenversusin-housetoolsinSection4.

3.2.Pitfall2:Falsecon dencefromvalidation-over-generalizationinsimulatorpapers,ortoolmisusesSimulatorwriterstypicallymakenarrowandfactuallyconsis-tentstatementsaboutvalidation,andsomeexamplesarebelow.However,thenatureofvalidationisoftenmisunderstoodbyusers,andthesetoolsareputtouseinwaysnotintendedfor,includingmakingquantitativegeneralizations.

gem5’sOOOmodeliswidelyused,butasobservedinarecentpaper[13]andourobservationsabove,ithasseveralspeci cationerrors.Thoughthegem5authorsthemselvesdonotclaimitassuch,somedoclaimitisa“validatedsimu-lator.”Clearly,thiscannotbetakenasalleffectsmodeled.Forinstance,atechniquethatworksontheinstructionfront-endmustpayattentiontogem5’sbaselineand rst xthespeci cationerrordescribedhere[13].

ConsideringMcPAT,accordingtotheirowndocumentationandcodecomments,constantsaresometimeschosentomatchthevalidationtargets.Weagreethisisareasonabledecisioninsomecases,especiallywhenhighlycustomizedlogicisemployed(e.g.functionalunitimplementations).Thedangeriswhenresearchersattempttogeneralizetheresultsoutsidethevalidatedprocessors.Theseconstantswilllikelynotbeappropriate.

ForGPUWattch,itmightbetemptingforresearcherstoper-formsensitivitystudiesbyvaryingMcPATparameters.Thepathofleastresistancewouldbetousethesamescalingfac-tors,insteadofmeasuringthepowerofaknownGPUandderivingnewscalingfactorsusingtheGPUWattchmethodol-ogy.Forreasonsdescribedintheprevioussection,wearguethatwithoutobtainingnewscalingfactors,thistypeofsensi-tivityanalysiswouldbeinappropriate.

Suggestions:Usewithcautionvalidatedsimulators.Lookfordetailsonthesimulator’sdesignandfactorthosedecisions3.Pitfalls

Thissectiondescribeseightpitfallsofmodernsimulatorsandsimulatorusage.Foreachpitfall,wedescribethehigh-levelproblemandsubstantiateourpositionwithempiricalevidence.Wethengiveouropinionsonhowbesttoavoidthepitfall.3.1.Pitfall1:Errorsinsimulatorsareinaccessibleto

users

Asoutlinedabove,simulatortoolscanhavesigni cantabstrac-tion,modeling,andspeci cationerrors.Furthermore,sincesimulatorsaredistributedasC/C++codewithlittlespeci -cation,itisdif cultforenduserstoevenbecomeawareoftheseerrors.Withoutunderstandingwhetherthesimulatoriscorrectlycapturingtheparticularphenomenonadesignerisinterestedin,off-the-shelfusagerendersthemineffectiveforeven rst-orderanalysisofeffects.

Sometimes,thefeaturesofmodelingtoolsarenoteasily