The Cochrane Collaboration’s tool for assessing risk

Cochrane Collaboration’s tool for assessing risk of bias

BMJ2011;343:d5928doi:10.1136/bmj.d5928

Page1of9

RESEARCHMETHODS&REPORTING

ofbiasinrandomised

trials

OPENACCESS

Flaws in the design, conduct, analysis, and reporting of randomised trials can cause the effect ofan intervention to be underestimated or overestimated.The Cochrane Collaboration’s tool forassessing risk of bias aims to make the process clearer and more accurate

JulianPTHigginsseniorstatistician,DouglasGAltmandirector,PeterCGøtzschedirector,4567PeterJüniheadofdivision,DavidMoherseniorscientist,AndrewDOxmanseniorresearcher,89JelenaSavovi?postdoctoralfellow,KennethFSchulzvicepresident,LauraWeeksresearch58associate,JonathanACSterneprofessorofmedicalstatisticsandepidemiology,CochraneBiasMethodsGroup,CochraneStatisticalMethodsGroup

MRCBiostatisticsUnit,InstituteofPublicHealth,CambridgeCB20SR,UK;2CentreforStatisticsinMedicine,UniversityofOxford,Oxford,UK;3TheNordicCochraneCentre,RigshospitaletandUniversityofCopenhagen,Denmark;4InstituteofSocialandPreventiveMedicine,UniversityofBern,Switzerland;5ClinicalEpidemiologyProgram,OttawaHospitalResearchInstitute,Ottawa,Ontario,Canada;6DepartmentofEpidemiologyandCommunityMedicine,FacultyofMedicine,UniversityofOttawa,Canada;7PreventiveandInternationalHealthCareUnit,NorwegianKnowledgeCentrefortheHealthServices,Oslo,Norway;8DepartmentofSocialMedicine,UniversityofBristol,Bristol,UK;9FHI,ResearchTrianglePark,NorthCarolina,USA1123

Randomisedtrials,andsystematicreviewsofsuchtrials,providethemostreliableevidenceabouttheeffectsofhealthcareinterventions.Providedthatthereareenoughparticipants,randomisationshouldensurethatparticipantsintheinterventionandcomparisongroupsaresimilarwithrespecttobothknownandunknownprognosticfactors.Differencesinoutcomesofinterestbetweenthedifferentgroupscantheninprinciplebeascribedtothecausaleffectoftheintervention.1

Causalinferencesfromrandomisedtrialscan,however,beunderminedbyflawsindesign,conduct,analyses,andreporting,leadingtounderestimationoroverestimationofthetrueinterventioneffect(bias).2However,itisusuallyimpossibletoknowtheextenttowhichbiaseshaveaffectedtheresultsofaparticulartrial.(whichassesstrialswithoutproducingascore).4-7Untilrecently,Cochranereviewsusedavarietyofthesetools,mainlychecklists.8In2005theCochraneCollaboration’smethodsgroupsembarkedonanewstrategyforassessingthequalityofrandomisedtrials.Inthispaperwedescribethecollaboration’snewriskofbiasassessmenttool,andtheprocessbywhichitwasdevelopedandevaluated.DevelopmentofriskassessmenttoolSystematicreviewsaimtocollateandsynthesiseallstudiesthatmeetprespecifiedeligibilitycriteria3usingmethodsthatattempttominimisebias.Toobtainreliableconclusions,reviewauthorsmustcarefullyconsiderthepotentiallimitationsoftheincludedstudies.Thenotionofstudy“quality”isnotwelldefinedbutrelatestotheextenttowhichitsdesign,conduct,analysis,andpresentationwereappropriatetoansweritsresearchquestion.Manytoolsforassessingthequalityofrandomisedtrialsareavailable,includingscales(whichscorethetrials)andchecklistsInMay2005,16statisticians,epidemiologists,andreviewauthorsattendedathreedaymeetingtodevelopthenewtool.Beforethemeeting,JPTHandDGAcompiledanextensivelistofpotentialsourcesofbiasinclinicaltrials.Theitemsonthelistweredividedintosevenareas:generationoftheallocationsequence;concealmentoftheallocationsequence;blinding;attritionandexclusions;othergenericsourcesofbias;biasesspecifictothetrialdesign(suchascrossoverorclusterrandomisedtrials);andbiasesthatmightbespecifictoaclinicalspecialty.Foreachofthesevenareas,anominatedmeetingparticipantpreparedareviewoftheempiricalevidence,adiscussionofspecificissuesanduncertainties,andaproposedsetofcriteriaforassessingprotectionfrombiasasadequate,inadequate,orunclear,supportedbyexamples.Correspondence to: J P T Higgins julian.higgins@mrc-bsu.cam.ac.uk.Further details on the items included in risk assessment tool (see http://wendang.chazidian.com/content/343/bmj.d5928/suppl/DC1)

No commercial reuse: See rights and reprints http://wendang.chazidian.com/permissionsSubscribe:

http://wendang.chazidian.com/subscribe

Cochrane Collaboration’s tool for assessing risk of bias

BMJ2011;343:d5928doi:10.1136/bmj.d5928Page2of9

Duringthemeetingdecisionsweremadebyinformalconsensusregardingitemsthatweretrulypotentialbiasesratherthansourcesofheterogeneityorimprecision.Potentialbiaseswerethendividedintodomains,andstrategiesfortheirassessmentwereagreed,againbyinformalconsensus,leadingtothecreationofanewtoolforassessingpotentialforbias.Meetingparticipantsalsodiscussedhowtosummariseassessmentsacrossdomains,howtoillustrateassessments,andhowtoincorporateassessmentsintoanalysesandconclusions.Minutesofthemeetingweretranscribedfromanaudiorecordinginconjunctionwithwrittennotes.Afterthemeeting,pairsofauthorsdevelopeddetailedcriteriaforeachincludediteminthetoolandguidanceforassessingthepotentialforbias.Documentsweresharedandfeedbackrequestedfromthewholeworkinggroup(includingsixwhocouldnotattendthemeeting).Severalemailiterationstookplace,whichalsoincorporatedfeedbackfrompresentationsoftheproposedguidanceatvariousmeetingsandworkshopswithintheCochraneCollaborationandfrompilotworkbyselectedreviewteamsincollaborationwithmembersoftheworkinggroup.Thematerialswereintegratedbytheco-leadsintocomprehensiveguidanceonthenewriskofbiastool.ThiswaspublishedinFebruary2008andadoptedastherecommendedmethodthroughouttheCochraneCollaboration.9facts,includingverbatimquoteswherepossible.Thesourceofthisinformationshouldbestated,andwhenthereisnoinformationonwhichtobaseajudgment,thisshouldbestated.Thesecondpartofthetoolinvolvesassigningajudgmentofhigh,low,orunclearriskofmaterialbiasforeachitem.Wedefinematerialbiasasbiasofsufficientmagnitudetohaveanotableeffectontheresultsorconclusionsofthetrial,recognisingthesubjectivityofanysuchjudgment.DetailedcriteriaformakingjudgmentsabouttheriskofbiasfromeachoftheitemsinthetoolareavailableintheCochraneHandbook.13Ifinsufficientdetailisreportedofwhathappenedinthetrial,thejudgmentwillusuallybeunclearriskofbias.Ajudgmentofunclearriskshouldalsobemadeifwhathappenedinthetrialisknownbuttheassociatedriskofbiasisunknown—forexample,ifparticipantstakeadditionaldrugsofunknowneffectivenessasaresultofthembeingawareoftheirinterventionassignment.Werecommendthatjudgmentsbemadeindependentlybyatleasttwopeople,withanydiscrepanciesresolvedbydiscussioninthefirstinstance.

Evaluationphase

Athreestageprojecttoevaluatethetoolwasinitiatedinearly2009.Aseriesoffocusgroupswasheldinwhichreviewauthorswhohadusedthetoolwereaskedtoreflectontheirexperiences.Findingsfromthefocusgroupswerethenfedintothedesignofquestionnairesforuseinthreeonlinesurveysofreviewauthorswhohadusedthetool,reviewauthorswhohadnotusedthetool(toexplorewhynot),andeditorialteamswithinthecollaboration.Weheldameetingtodiscussthefindingsfromthefocusgroupsandsurveysandtoconsiderrevisionstothefirstversionoftheriskofbiastool.Thiswasattendedbysixparticipantsfromthe2005meetingand17others,includingstatisticians,epidemiologists,coordinatingeditorsandotherstaffofCochranereviewgroups,andtheeditorinchiefoftheCochraneLibrary.Someoftheitemsinthetool,suchasmethodsforrandomisation,requireonlyasingleassessmentforeachtrialincludedinthereview.Forotheritems,suchasblindingandincompleteoutcomedata,twoormoreassessmentsmaybeusedbecausetheygenerallyneedtobemadeseparatelyfordifferentoutcomes(orforthesameoutcomeatdifferenttimepoints).However,werecommendthatreviewauthorslimitthenumberofassessmentsusedbygroupingoutcomes—forexample,assubjectiveorobjectiveforthepurposesofassessingblindingofoutcomeassessmentoras“patientreportedat6months”or“patientreportedat12months”forassessingriskofbiasduetoincompleteoutcomedata.EvaluationofinitialimplementationThefirst(2008)versionofthetoolwasslightlydifferentfromtheonewepresenthere.The2008versiondidnotcategorisebiasesbythesixdomains(selectionbias,performancebias,etc);hadasingleassessmentforblinding;andexpressedriskofbiasintheformat‘”yes,”“no,”or“unclear”(referringtolackofarisk)ratherthanaslow,high,orunclearrisk.The2010evaluationoftheinitialversionfoundwideacceptanceoftheneedfortheriskofbiastool,withaconsensusthatitrepresentsanimprovementovermethodspreviouslyrecommendedbytheCollaborationorwidelyusedinsystematicreviews.Participantsinthefocusgroupsnotedthatthetooltooklongertocompletethanpreviousmethods.Of187authorssurveyed,88%tooklongerthan10minutestocompletethenewtool,44%longerthan20minutes,and7%longerthananhour,but83%consideredthetimetakenacceptable.Therewasconsensusthatclassifyingitemsinthetoolaccordingtocategoriesofbias(selectionbias,performancebias,etc)wouldhelpusers,soweintroducedthese.Therewasalsoconsensusthatassessmentofblindingshouldbeseparatedintoblindingofparticipantsandhealthprofessionals(performancebias)andblindingofoutcomeassessment(detectionbias)andthatthephrasingofthejudgmentsaboutriskshouldbechangedtolow,high,andunclearrisk.Thedomainsreportedtobethemostdifficulttoassesswereriskofbiasduetoincompleteoutcomedataandselectivereportingofoutcomes.Therewasagreementthatimprovedtrainingmaterialsandavailabilityofworkedexampleswouldincreasethequalityandreliabilityofbiasassessments.TheriskofbiastoolAtthe2005workshoptheparticipantsagreedthesevenprinciplesonwhichthenewriskofbiasassessmenttoolwasbased(box).Theriskofbiastoolcoverssixdomainsofbias:selectionbias,performancebias,detectionbias,attritionbias,reportingbias,andotherbias.Withineachdomain,assessmentsaremadeforoneormoreitems,whichmaycoverdifferentaspectsofthedomain,ordifferentoutcomes.Table1?http://wendang.chazidian.com.Foreachiteminthetool,theassessmentofriskofbiasisintwoparts.Thesupportforjudgmentprovidesasuccinctfreetextdescriptionorsummaryoftherelevanttrialcharacteristiconwhichjudgmentsofriskofbiasarebasedandaimstoensuretransparencyinhowjudgmentsarereached.Forexample,theitemaboutconcealmentoftherandomisedallocationsequencewouldprovidedetailsofwhatmeasureswereinplace,ifany,http://wendang.chazidian.comrmationforthesedescriptionswilloftencomefromasinglepublishedtrialreportbutmaybeobtainedfromamixtureoftrialreports,protocols,publishedcommentsonthetrial,andcontactswiththeinvestigators.Thesupportforthejudgmentshouldprovideasummaryofknown

No commercial reuse: See rights and reprints http://wendang.chazidian.com/permissionsSubscribe:http://wendang.chazidian.com/subscribe

Cochrane Collaboration’s tool for assessing risk of bias

BMJ2011;343:d5928doi:10.1136/bmj.d5928Page3of9

Principlesforassessingriskofbias

1.Donotusequalityscales

Qualityscalesandresultingscoresarenotanappropriatewaytoappraiseclinicaltrials.Theytendtocombineassessmentsofaspectsofthequalityofreportingwithaspectsoftrialconduct,andtoassignweightstodifferentitemsinwaysthataredifficulttojustify.Boththeoreticalconsiderations10andempiricalevidence11suggestthatassociationsofdifferentscaleswithinterventioneffectestimatesareinconsistentandunpredictable

2.Focusoninternalvalidity

Theinternalvalidityofastudyistheextenttowhichitisfreefrombias.Itisimportanttoseparateassessmentofinternalvalidityfromthatofexternalvalidity(generalisabilityorapplicability)andprecision(theextenttowhichstudyresultsarefreefromrandomerror).Applicabilitydependsonthepurposeforwhichthestudyistobeusedandislessrelevantwithoutinternalvalidity.Precisiondependsonthenumberofparticipantsandeventsinastudy.Asmalltrialwithlowriskofbiasmayprovideveryimpreciseresults,withawideconfidenceinterval.Conversely,theresultsofalargetrialmaybeprecise(narrowconfidenceinterval)buthaveahighriskofbiasifinternalvalidityispoor

3.Assesstheriskofbiasintrialresults,notthequalityofreportingormethodologicalproblemsthatarenotdirectlyrelatedtoriskofbias

Thequalityofreporting,suchaswhetherdetailsweredescribedornot,affectstheabilityofsystematicreviewauthorsandusersofmedicalresearchtoassesstheriskofbiasbutisnotdirectlyrelatedtotheriskofbias.Similarly,someaspectsoftrialconduct,suchasobtainingethicalapprovalorcalculatingsamplesize,arenotdirectlyrelatedtotheriskofbias.Conversely,resultsofatrialthatusedthebestpossiblemethodsmaystillbeatriskofbias.Forexample,blindingmaynotbefeasibleinmanynon-drugtrials,anditwouldnotbereasonabletoconsiderthetrialaslowqualitybecauseoftheabsenceofblinding.Nonetheless,manytypesofoutcomemaybeinfluencedbyparticipants’knowledgeoftheinterventionreceived,andsothetrialresultsforsuchoutcomesmaybeconsideredtobeatriskofbiasbecauseoftheabsenceofblinding,despitethisbeingimpossibletoachieve

4.Assessmentsofriskofbiasrequirejudgment

Assessmentofwhetheraparticularaspectoftrialconductrendersitsresultsatriskofbiasrequiresbothknowledgeofthetrialmethodsandajudgmentaboutwhetherthosemethodsarelikelytohaveledtoariskofbias.Wedecidedthatthebasisforbiasassessmentsshouldbemadeexplicit,byrecordingtheaspectsofthetrialmethodsonwhichthejudgmentwasbasedandthenthejudgmentitself

5.Choosedomainstobeassessedbasedonacombinationoftheoreticalandempiricalconsiderations

Empiricalstudiesshowthatparticularaspectsoftrialconductareassociatedwithbias.212However,thesestudiesdidnotincludeallpotentialsourcesofbias.Forexample,availableevidencedoesnotdistinguishbetweendifferentaspectsofblinding(ofparticipants,healthprofessionals,andoutcomeassessment)andisverylimitedwithregardtohowauthorsdealtwithincompleteoutcomedata.Theremayalsobetopicspecificanddesignspecificissuesthatarerelevantonlytosometrialsandreviews.Forexample,inareviewcontainingcrossovertrialsitmightbeappropriatetoassesswhetherresultswereatriskofbiasbecausetherewasaninsufficient“washout”periodbetweenthetwotreatmentperiods

6.Focusonriskofbiasinthedataasrepresentedinthereviewratherthanasoriginallyreported

Somepapersmayreporttrialresultsthatareconsideredasathighriskofbias,forwhichitmaybepossibletoderivearesultatlowriskofbias.Forexample,apaperthatinappropriatelyexcludedcertainpatientsfromanalysesmightreporttheinterventiongroupsandoutcomesforthesepatients,sothattheomittedparticipantscanbereinstated

7.Reportoutcomespecificevaluationsofriskofbias

Someaspectsoftrialconduct(forexample,whethertherandomisedallocationwasconcealedatthetimetheparticipantwasrecruited)applytothetrialasawhole.Forotheraspects,however,theriskofbiasisinherentlyspecifictodifferentoutcomeswithinthetrial.Forexample,allcausemortalitymightbeascertainedthroughlinkagestodeathregistries(lowriskofbias),whilerecurrenceofcancermighthavebeenassessedbyadoctorwithknowledgeoftheallocatedintervention(highriskofbias)

Presentationofassessments

Resultsofanassessmentofriskofbiascanbepresentedinatable,inwhichjudgmentsforeachitemineachtrialarepresentedalongsidetheirdescriptivejustification.Table2?presentsanexampleofariskofbiastableforonetrialincludedinaCochranereviewoftherapeuticmonitoringofantiretroviralsforpeoplewithHIV.14Risksofbiasduetoblindingandincompleteoutcomedatawereassessedacrossalloutcomeswithineachincludedstudy,ratherthanseparatelyfordifferentoutcomesaswillbemoreappropriateinsomesituations.this,reviewauthorsmustdecidewhichdomainsaremostimportantinthecontextofthereview,ideallywhenwritingtheirprotocol.Forexample,forhighlysubjectiveoutcomessuchaspain,blindingofparticipantsiscritical.Thewaythatsummaryjudgmentsofriskofbiasarereachedshouldbeexplicitandshouldbeinformedbyempiricalevidenceofbiaswhenitexists,likelydirectionofbias,andlikelymagnitudeofbias.Table3?providesasuggestedframeworkformakingsummaryassessmentsoftheriskofbiasforimportantoutcomeswithinandacrosstrials.

Presentingriskofbiastablesforeverystudyinareviewcanbecumbersome,andwesuggestthatillustrationsareusedtosummarisethejudgmentsinthemainsystematicreviewdocument.Thefigure?providesanexample.Herethejudgmentsapplytoallmeta-analysesinthereview.Analternativewouldbetoillustratetheriskofbiasforaparticularmeta-analysis(orforaparticularoutcomeifastatisticalsynthesisisnotundertaken),showingtheproportionofinformationthatcomesfromstudiesatlow,unclear,orhighriskofbiasforeachiteminthetool,amongstudiescontributinginformationtothatoutcome.AssessmentsofriskofbiasandsynthesisofresultsSummaryassessmentsoftheriskofbiasforanoutcomewithineachtrialshouldinformthemeta-analysis.Thetwopreferableanalyticalstrategiesaretorestricttheprimarymeta-analysistostudiesatlowriskofbiasortopresentmeta-analysesstratifiedaccordingtoriskofbias.Thechoicebetweenthesestrategiesshouldbebasedonthecontextoftheparticularreviewandthebalancebetweenthepotentialforbiasandthelossofprecisionwhenstudiesathighorunclearriskofbiasareexcluded.Meta-regressioncanbeusedtocompareresultsfromstudiesathighandlowriskofbias,butsuchcomparisonslackpower,15

andlackofasignificantdifferenceshouldnotbeinterpretedasimplyingtheabsenceofbias.SummaryassessmentofriskofbiasTodrawconclusionsabouttheoverallriskofbiaswithinoracrosstrialsitisnecessarytosummariseassessmentsacrossitemsinthetoolforeachoutcomewithineachtrial.Indoing

No commercial reuse: See rights and reprints http://wendang.chazidian.com/permissionsAthirdstrategyistopresentameta-analysisofallstudieswhileprovidingasummaryoftheriskofbiasacrossstudies.However,Subscribe:http://wendang.chazidian.com/subscribe

Cochrane Collaboration’s tool for assessing risk of bias

BMJ2011;343:d5928doi:10.1136/bmj.d5928Page4of9

thisrunstheriskthatbiasisdownplayedinthediscussionandconclusionsofareview,sothatdecisionscontinuetobebased,atleastinpart,onflawedevidence.Thisriskcouldbereducedbyincorporatingsummaryassessmentsintobroader,butexplicit,measuresofthequalityofevidenceforeachimportantoutcome,forexampleusingtheGRADEsystem.16Thiscanhelptoensurethatjudgmentsabouttheriskofbias,aswellasotherfactorsaffectingthequalityofevidence(suchasimprecision,heterogeneity,andpublicationbias),areconsideredwheninterpretingtheresultsofsystematicreviews.1718Developmentmeetingparticipants(May2005):DougAltman(co-lead),GerdAntes,ChrisCates,JonDeeks,PeterGøtzsche,JulianHiggins(co-lead),SallyHopewell,PeterJüni(organisingcommittee),SteffLewis,PhilippaMiddleton,DavidMoher(organisingcommittee),AndyOxman,KenSchulz(organisingcommittee),NandiSiegfried,JonathanSterne,SimonThompson.Othercontributorstotooldevelopment:HildaBastian,RachelleBuchbinder,IainChalmers,MirandaCumpston,SallyGreen,PeterHerbison,VictorMontori,HannahRothstein,GeorgiaSalanti,GuidoSchwarzer,IanShrier,JayneTierney,IanWhiteandPaulaWilliamson.

Evaluationmeetingparticipants(March2010):DougAltman(organisingcommittee),ElaineBeller,SallyBell-Syer,ChrisCates,RachelChurchill,JuneCody,JonathanCook,ChristianGluud,JulianHiggins(organisingcommittee),SallyHopewell,HayleyJones,PeterJ?ni,MonicaKjeldstrøm,TobyLasserson,AllysonLipp,LaraMaxwell,JoanneMcKenzie,CraigRamsey,BarneyReeves,JelenaSavovi?(co-lead),JonathanSterne(co-lead),DavidTovey,LauraWeeks(organisingcommittee).

Othercontributorstotoolevaluation:IsabelleBoutron,DavidMoher(organisingcommittee),LucyTurner.

Funding:ThedevelopmentandevaluationoftheriskofbiastoolwasfundedinpartbyTheCochraneCollaboration.TheviewsexpressedinthisarticlearethoseoftheauthorsandnotnecessarilythoseofTheCochraneCollaborationoritsregisteredentities,committeesorworkinggroups..JPTHwasalsofundedbyMRCgrantnumberU.1052.00.011.DGAwasfundedbyCancerResearchUKgrantnumberC-5592.DMwasfundedbyaUniversityResearchChair(UniversityofOttawa).TheCanadianInstitutesofHealthResearchprovidesfinancialsupporttotheCochraneBiasMethodsGroup.

Competinginterests:http://wendang.chazidian.com/coi_disclosure.pdf(availableonrequestfromthecorrespondingauthor)anddeclaresupportfromtheCochraneCollaborationforthedevelopmentandevaluationofthetooldescribed;theyhavenofinancialrelationshipswithanyorganisationsthatmighthaveaninterestinthesubmittedworkinthepreviousthreeyearsandnootherrelationshipsoractivitiesthatcouldappeartohaveinfluencedthesubmittedwork.

Provenanceandpeerreview:Notcommissioned;externallypeerreviewed.

1KleijnenJ,GøtzscheP,KunzRH,OxmanAD,ChalmersI.Sowhat’ssospecialaboutrandomisation?In:MaynardA,ChalmersI,eds.Non-randomreflectionsonhealthservicesresearch:onthe25thanniversaryofArchieCochrane’sEffectivenessandEfficiency.BMJBooks,1997:93-106.WoodL,EggerM,GluudLL,SchulzK,JüniP,AltmanDG,etal.Empiricalevidenceofbiasintreatmenteffectestimatesincontrolledtrialswithdifferentinterventionsandoutcomes:meta-epidemiologicalstudy.BMJ2008;336:601-5.EggerM,DaveySmithG,AltmanDG,eds.Systematicreviewsinhealthcare:meta-analysisincontext.BMJBooks,2001.MoherD,JadadAR,NicholG,PenmanM,TugwellP,WalshS.Assessingthequalityofrandomizedcontrolledtrials—anannotatedbibliographyofscalesandchecklists.ControlledClinTrials1995;12:62-73.JüniP,AltmanDG,EggerM.Systematicreviewsinhealthcare:assessingthequalityofcontrolledclinicaltrials.BMJ2001;323:42-6.WestS,KingV,CareyTS,LohrKN,McKoyN,SuttonSF,etal.Systemstoratethestrengthofscientificevidence.Evidencereport/technologyassessmentno47.AHRQpublicationNo02-E016.AgencyforHealthcareResearchandQuality,2002.CroweM,SheppardL.Areviewofcriticalappraisaltoolsshowtheylackrigor:alternativetoolstructureisproposed.JClinEpidemiol2011;64:79-89.LundhA,GøtzschePC.RecommendationsbyCochraneReviewGroupsforassessmentoftheriskofbiasinstudies.BMCMedResMethodol2008;8:22.HigginsJPT,GreenS,eds.Cochranehandbookforsystematicreviewsofinterventions.Wiley,2008.GreenlandS,O’RourkeK.Onthebiasproducedbyqualityscoresinmeta-analysis,andahierarchicalviewofproposedsolutions.Biostatistics2001;2:463-71.JüniP,WitschiA,BlochR,EggerM.Thehazardsofscoringthequalityofclinicaltrialsformeta-analysis.JAMA1999;282:1054-60.GluudLL.Biasinclinicalinterventionresearch.AmJEpidemiol2006;163:493-501.HigginsJPT,AltmanDG.Assessingriskofbiasinincludedstudies.In:HigginsJPT,GreenS,eds.Cochranehandbookforsystematicreviewsofinterventions.Wiley,2008:187-241.KredoT,VanderWaltJ-S,SiegfriedN,CohenK.TherapeuticdrugmonitoringofantiretroviralsforpeoplewithHIV.CochraneDatabaseSystRev2009;3:CD007268.HigginsJPT,ThompsonSG.Controllingtheriskofspuriousfindingsfrommeta-regression.StatMed2004;23:1663-82.DiscussionDiscrepanciesbetweentheresultsofdifferentsystematicreviewsexaminingthesamequestion1920andbetweenmeta-analysesandsubsequentlargetrials21haveshownthattheresultsofmeta-analysescanbebiased,whichmaybepartlycausedbybiasedresultsinthetrialstheyinclude.Webelieveourriskofbiastoolisoneofthemostcomprehensiveapproachestoassessingthepotentialforbiasinrandomisedtrialsincludedinsystematicreviewsormeta-analyses.Inclusionofdetailsoftrialconduct,onwhichjudgmentsofriskofbiasarebased,providesgreatertransparencythanpreviousapproaches,allowingreaderstodecidewhethertheyagreewiththejudgmentsmade.Thereiscontinuinguncertainty,andgreatvariationinpractice,overhowtoassesspotentialforbiasinspecificdomainswithintrials,howtosummarisebiasassessmentsacrosssuchdomains,andhowtoincorporatebiasassessmentsintometa-analyses.Arecentstudyhasfoundthatthetooltakeslongertocompletethanothertools(theinvestigatorstookameanof8.8minutesperpersonforasinglepredeterminedoutcomeusingourtoolcomparedwith1.5minutesforapreviousratingscaleforqualityofreporting).22Thereliabilityofthetoolhasnotbeenextensivelystudied,althoughthesameauthorsobservedthatlargereffectsizeswereobservedonaverageinstudiesratedasathighriskofbiascomparedwithstudiesatlowriskofbias.22Byexplicitlyincorporatingjudgmentsintothetool,weacknowledgethatagreementsbetweenassessorsmaynotbeashighasforsomeothertools.However,wealsoexplicitlytargettheriskofbiasratherthanreportedcharacteristicsofthetrial.Itwouldbeeasiertoassesswhetheradrop-outrateexceeds20%thanwhetheradrop-outrateof21%introducesanimportantriskofbias,butthereisnoguaranteethatresultsfromastudywithadrop-outratelowerthan20%areatlowriskofbias.Preliminaryevidencesuggeststhatincompleteoutcomedataandselectivereportingarethemostdifficultitemstoassess;kappameasuresofagreementof0.32(fair)and0.13(slight)respectivelyhavebeenreportedforthese.22Itisimportantthatguidanceandtrainingmaterialscontinuetobedevelopedforallaspectsofthetool,butparticularlythesetwo.Wehopethatwidespreadadoptionandimplementationoftheriskofbiastool,bothwithinandoutsidetheCochraneCollaboration,willfacilitateimprovedappraisalofevidencebyhealthcaredecisionmakersandpatientsandultimatelyleadtobetterhealthcare.Improvedunderstandingofthewaysinwhichflawsintrialconductmaybiastheirresultsshouldalsoleadtobettertrialsandmorereliableevidence.Riskofbiasassessmentsshouldcontinuetoevolve,takingintoaccountanynewempiricalevidenceandthepracticalexperienceofauthorsofsystematicreviews.Contributors:Allauthorscontributedtothedraftingandeditingofthemanuscript.JPTH,DGA,PCG,PJ,DM,ADO,KFSandJACScontributedtothechapterintheCochraneHandbookforSystematicReviewsofInterventionsonwhichthepaperisbased.JPTHwillactasguarantor.

No commercial reuse: See rights and reprints http://wendang.chazidian.com/permissions23456789101112131415Subscribe:http://wendang.chazidian.com/subscribe

Cochrane Collaboration’s tool for assessing risk of bias

BMJ2011;343:d5928doi:10.1136/bmj.d5928Page5of9

Summarypoints

Systematicreviewsshouldcarefullyconsiderthepotentiallimitationsofthestudiesincluded

TheCochraneCollaborationhasdevelopedanewtoolforassessingriskofbiasinrandomisedtrials

Thetoolseparatesajudgmentaboutriskofbiasfromadescriptionofthesupportforthatjudgment,foraseriesofitemscoveringdifferentdomainsofbias

16

17

18

19

20

21GuyattGH,OxmanAD,VistGE,ZunzR,Falck-YtterY,Alonso-CoelloP,etal.GRADE:anemergingconsensusonratingqualityofevidenceandstrengthofrecommendations.BMJ2008;336:924-6.SchünemannHJ,OxmanAD,HigginsJPT,VistGE,GlasziouP,GuyattGH,etal.Presentingresultsand“Summaryoffindings”tables.In:HigginsJPT,GreenS,eds.Cochranehandbookforsystematicreviewsofinterventions.Wiley,2008:335-8.SchünemannHJ,OxmanAD,VistGE,HigginsJPT,DeeksJJ,GlasziouP,etal.Interpretingresultsanddrawingconclusions.In:HigginsJPT,GreenS,eds.Cochranehandbookforsystematicreviewsofinterventions.Wiley,2008:359-87.LeizoroviczA,HaughMC,ChapuisFR,SamamaMM,BoisselJP.Lowmolecularweightheparininpreventionofperioperativethrombosis.BMJ1992;305:913-20.NurmohamedMT,RosendaalFR,BullerHR,DekkerE,HommesDW,VandenbrouckeJP,etal.Low-molecular-weightheparinversusstandardhepariningeneralandorthopaedicsurgery:http://wendang.chazidian.comncet1992;340:152-6.LelorierJ,BenhaddadA,LapierreJ,DerderianF.Discrepanciesbetweenmeta-analysesandsubsequentlargerandomized,controlledtrials.NEnglJMed1997;337:536-42.22HartlingL,OspinaM,LiangY,DrydenDM,HootonN,KrebsSJ,etal.Riskofbiasversusqualityassessmentofrandomisedcontrolledtrials:crosssectionalstudy.BMJ2009;339:b4012.Accepted:22July2011Citethisas:BMJ2011;343:d5928Thisisanopen-accessarticledistributedunderthetermsoftheCreativeCommonsAttributionNon-commercialLicense,whichpermitsuse,distribution,andreproductioninanymedium,providedtheoriginalworkisproperlycited,theuseisnoncommercialandisotherwiseincompliancewiththelicense.See:http://wendang.chazidian.com/licenses/by-nc/2.0/andhttp://wendang.chazidian.com/licenses/by-nc/2.0/legalcode.

No commercial reuse: See rights and reprints http://wendang.chazidian.com/permissionsSubscribe:http://wendang.chazidian.com/subscribe