experimental research bias

An official website of the United States government

Official websites use .gov A .gov website belongs to an official government organization in the United States.

Secure .gov websites use HTTPS A lock ( Lock Locked padlock icon ) or https:// means you've safely connected to the .gov website. Share sensitive information only on official, secure websites.

Publications
Account settings
Advanced Search
Journal List

Moving towards less biased research

Mark yarborough.

Author information
Article notes
Copyright and License information

Correspondence to Dr Mark Yarborough; [email protected]

Corresponding author.

Collection date 2021.

Keywords: bias, research ethics, conflict of interest

This is an open access article distributed in accordance with the Creative Commons Attribution 4.0 Unported (CC BY 4.0) license, which permits others to copy, redistribute, remix, transform and build upon this work for any purpose, provided the original work is properly cited, a link to the licence is given, and indication of whether changes were made. See: https://creativecommons.org/licenses/by/4.0/ .

Introduction

Bias, perhaps best described as ‘any process at any stage of inference which tends to produce results or conclusions that differ systematically from the truth,’ can pollute the entire spectrum of research, including its design, analysis, interpretation and reporting. 1 It can taint entire bodies of research as much as it can individual studies. 2 3 Given this extensive detrimental impact, effective efforts to combat bias are critically important to biomedical research’s goal of improving healthcare. Champions for such efforts can currently be found among individual investigators, journals, research sponsors and research regulators. The central focus of this essay is assessing the effectiveness of some of the efforts currently being championed and proposing new ones.

Current efforts fall mainly into two domains, one meant to prevent bias and one meant to detect it. Much like a proverbial chain, efforts in either domain are hampered by their weakest components. Hence, it behoves us to constantly probe antibias tools so that we can identify weak components and seek ways to compensate for them. Further, given the high stakes—conclusions that align with rather than diverge from truth—it further behoves the biomedical research community to prioritise to the extent possible bias prevention over bias detection. The less likely any given study is to be tainted by bias, the fewer research publications reporting biased results there will be. The value of detected bias pales in comparison, for it extends only as far as those who are aware of that detection after the fact, meaning that biased conclusions at variance with the truth can mislead those unaware of the bias that taints them for as long as the affected publications endure.

With these preliminary considerations about bias in mind, let us first examine some current antibias efforts and probe their weaknesses. Doing so will show why we need to develop additional strategies for preventing bias in the first place, and space is set aside at the end to examine two related candidate strategies for how we could attempt to do that.

Current bias countermeasures

Table 1 reflects some current countermeasures being employed to combat various kinds of biases. Though the table is far from comprehensive, (dozens of biases have been catalogued) 1 it does include major biases of concern, representative countermeasures to combat them, whether those countermeasures prevent or detect bias, and their likely relative strength.

Sponsorship bias

The bias that probably draws the most attention is what is known as sponsorship bias, 4 5 wherein pecuniary interests undermine the disinterestedness meant to prevail in scientific investigations. 6 The most prominent countermeasure against it consists in multiple disclosure practices that flag financial relationships between scientists and private companies. For example, academic institutions may require faculty to disclose annually their financial relationships with private companies; research sponsors may require applicants to make such disclosures when submitting applications; and journals typically require authors to make such disclosures when submitting manuscripts. The right-hand column of table 1 prompts the question, ‘to what extent do such disclosures actually prevent sponsorship bias?’ There is now ample conceptual analysis 7–10 and empirical evidence produced over many years such that we can safely state that there is an over-reliance on disclosure.

This extensive prior work shows, for example, that journal disclosure policies targeting authors fail to capture many financial ties between researchers and industry. Recent studies show that consulting agreements between researchers and companies, as well as financial ties between biomedical companies and organisations that produce clinical practice guidelines, often go undisclosed. 11 12 Looking at journal disclosure policies, we see further evidence of disclosure’s limited ameliorative effect. A recent study that randomised article reviewers into one group that received financial interests disclosures along with the manuscripts to be reviewed and another group that did not found that the disclosures had no effect on reviewer assessments of the manuscripts. 13 Another recent study looked at editorial practices regarding the financial interests of authors at 30 leading medical journals and found that none had actual tools for determining whether and how disclosed financial relationships might have impacted any given research report. 14

Additional considerations help to further explain the weaknesses of journal disclosure policies. First, disclosures are usually mistimed. When financial relationships bias studies, that bias occurs long before anyone discloses the relationships in reports about the studies. 15 Second, it is those, and only those, designated as authors who are subject to them. Often those who lead the design, conduct, analysis and reporting of a study are not in fact considered authors of it. 16 Private companies that sponsor the majority of drug studies and/or contract research organisations they hire control the design, manage the conduct, and analyse the data, as well as write the articles about that analysis for studies. 17 Journal disclosure mandates leave untouched the bias that these conflicted sponsors can introduce into clinical trials because of sizeable holes in the International Committee of Medical Journal Editors (ICMJE) authorship policy. Followed by an outsized portion of biomedical research journals, it ‘support[s] practices of commercial data control, content development and attribution that run counter to science’s values of openness, objectivity and truthfulness’ because ‘the ICMJE accepts the use of commercial editorial teams to produce manuscripts, which is a potential source of bias, and accepts private company ownership and analysis of clinical trial data.’ 16 In other words, even though readers of journals assume that journals accurately attribute those, and only those, who are responsible for the design, conduct, analysis and reporting of a study, authorship practices do not in fact require such accurate attribution. Thus, we are relying on disclosure, often after the fact of conducting a study, to combat the bias that financial entanglements can cause prior to a study’s launch and the disclosure practices themselves often mistarget those who should be making the disclosures. The end result is that current disclosure practices can conceal rather than reveal the prospect of sponsorship bias.

Furthermore, even if disclosures were better targeted, this would not negate the potential that disclosures themselves have to cause unintended detrimental consequences. Commentators long ago noted that disclosing financial relationships may contribute to people having a sense of ‘moral license to (act in biased ways more) than they would without disclosure. With disclosure, (acting in a biased way) might seem like fair play. While most professionals might care about their (audience), disclosure (practices) can encourage these professionals to exhibit this concern in a merely perfunctory way.’ 18

There are two final considerations about disclosure that need to be noted. First, disclosure is not meant to actually detect bias. Rather, it is meant to alert people to its possibility. Thus, even though disclosure is our major tool for combating one of the most detrimental forms of bias, it is not clear what good it actually does, which leads us to the second consideration. Since disclosure does nothing to prevent sponsorship bias, more substantial countermeasures aimed at prevention are needed. It is beyond the scope of this essay to examine the suitability of possible countermeasures for preventing sponsorship bias, such as sequestering investigators from private companies whenever possible. 15 Referencing this one example, though, highlights the substantial difference there can be between detecting bias on the one hand and actually preventing it on the other, a topic we will return to later.

Returning for the moment, though, to detection of sponsorship bias, these collective concerns about the most prevalent safeguard against it suggest that it can facilitate rather than detect, let alone prevent, bias. By stopping at disclosure, it suggests that financial entanglements are often permissible; we just need to make sure they are relatively transparent to others. The end result is that there is a pall of uncertainty cast over a large body of published research, including a major portion of the clinical trials that society relies on to improve healthcare. 17

Additional major sources of bias

Evidence about the effectiveness of safeguards against other prominent sources of bias besides sponsorship bias is equally disconcerting. Consider, for example, biases that impact the design, conduct and reporting of preclinical animal studies. This class of studies is of particular concern for multiple reasons, not the least of which is the fact that early phase clinical trials, and the risks intrinsic to them, can launch on a single, highly prized ‘proof-of-concept finding in an animal model without wider preclinical validation.’ 19 This risk is particularly grave when we consider the interests and welfare of the patients who volunteer for the early phase clinical trials. 20

Given such high stakes, it is critical that there be effective safeguards that, once again, counter biases that undermine the rigour that studies capable of producing reliable findings require. Here too table 1 prompts investigation of how well current safeguards actually work. Evidence about excess significance bias, a publishing bias due in large part to selective publishing of results by both authors and journals, shows major limitations in their effectiveness. Looking, for example, at the neurosciences preclinical studies generally 2 and stroke studies specifically, 21 we see that excess significance bias is a major contributor to well documented failure 22 23 to successfully ‘translate preclinical animal research (results) to clinical trials.’ 24

When we look at biases resulting from poor study design, across all fields of preclinical inquiry, we find that studies that lack construct, internal and/or external validity that produce biased research reports are ubiquitous. 25 Not only have such findings contributed to ‘spectacular failures of irreproducibility’ 25 that cast concern over entire fields of research, 3 they also forecast failure for the clinical trials that seek to translate preclinical findings into clinical therapies. 26 Illustrating this is a recent study estimating that a majority of the reports of positive findings from animal studies meant to inform clinical studies of acute stroke actually report what are likely to be false positive results. 27

With this evidence in mind, we must consider anew the harm caused by, for example, toxicities, personal expenses and opportunity costs 28 that phase 1 trial participants endure in trials that launch on the basis of preclinical studies whose biased design produces unreliable research reports used to justify the clinical trials. 29 Those participants have no choice but to rely on a properly functioning research oversight system to protect their interests and welfare. Alas, that oversight system is much weaker than the research and research oversight communities likely would care to admit. 30 All the more reason, then, that our efforts to guard against bias should be as varied and robust as its many sources.

The fact of the matter, though, is that the most prominent safeguard against them is peer review. Since it occurs at the reporting stage of the research continuum, it is preceded by other safeguards, such as reporting guidelines, which are reviewed below. None of these other safeguards are as ubiquitous as peer review, however, and it is the gate that publications must ultimately navigate through. Given this level of significance, its effectiveness warrants careful scrutiny. Scrutiny begins by noting that peer review is meant to detect rather than prevent bias. One perhaps could counter that peer review actually is a hybrid countermeasure since it is capable of actually preventing bias at times, or at the least the dissemination of reports tainted by it since, when peer review works, it can prevent publication of suspect findings. However, though it is no doubt true that peer reviewers can reject manuscripts out of concern for bias, concerns about false positive findings, and the like, there is no assurance that manuscripts rejected at one journal will be rejected by all journals. Hence, even if one were to confer it a hybrid status wherein it can both prevent and detect bias, the extent of bias that has long been documented in peer-reviewed journals reveals major weaknesses in peer review. Recent high-profile COVID-19 -related retractions 31 and commentary 32 further confirms these weaknesses. Consequently, we need to be guarded in our expectations about the central antibias safeguard and its ability to assure the reliability of published research findings.

The upshot of all this is that current bias safeguards do little to alert clinical investigators, research ethics review committees, and others to the prospects of biased findings in either pivotal preclinical studies that are the precursors to clinical trials or the full spectrum of clinical trials themselves. This raises genuine concerns that far too many ill-advised clinical trials get conducted rather than avoided. It also underscores the need for conducting the individual studies that constitute any given body of preclinical or clinical research in a manner that is free of bias in the first place. Additional safeguards that prevent rather than detect bias will be needed if we are to succeed at this. No doubt multiple ones are needed. In the balance of this piece, I will focus on ones that could be used for preclinical studies, leaving clinical studies safeguards for other occasions.

Preventing bias

Examples of current bias prevention tools.

We are fortunate that there are some safeguards for combatting bias in preclinical studies already in place. Perhaps the most notable are reporting guidelines such as the ARRIVE (Animal Research: Reporting of In Vivo Experiments) guidelines. 33 Recently revised, 34 the guidelines are designed to assure transparency of critical methodological aspects of animal studies. If widely enough adopted, they should promote greater rigour in animal research and thus prevent much of the bias that currently plagues it. Unfortunately, though, uptake of the guidelines has been lacklustre to date, mainly because too many animal researchers are either unaware of them or do not follow them. 35 Not all the evidence about reporting guidelines is so discouraging though. A recent study of reporting guidelines tailored for the journal Stroke found that they substantially improved the quality of published preclinical studies when compared with reports in other journals that did not require use of the same guidelines. 36 37

Despite the mixed evidence about the effectiveness of reporting guidelines, both general and journal-tailored reporting guidelines do have value that is worth noting. Even though they target the reporting stage of research, their use can influence how researchers design and conduct their studies. This highlights the true promise of reporting guidelines: they can incline researchers toward well-designed research and robust reports about it. To the extent that this occurs, they function as true bias prevention safeguards.

Nevertheless, enthusiasm for reporting guidelines must be tempered by the mixed evidence about them to date. It suggests that reporting guidelines will have an incremental effect at best on preventing bias. This is borne out by evidence, for example, pertaining to the TREAT-NMD Advisory Committee for Therapeutics. Although this committee does not promulgate specific reporting guidelines, it does promote the kinds of research practices that reporting guidelines are meant to foster. It does this by ‘provid(ing) detailed constructive feedback on clinical proposals for neuromuscular diseases submitted by researchers in both academia and industry.’ This group provided feedback on just under 60 preclinical research programmes between 2010 and 2019. It reports having raised concerns in just under a third of their reviews about the use of control groups, blinding and randomisation with researchers whose preclinical research they reviewed. They also report raising concerns about a misalignment between preclinical data and claimed preclinical efficacy almost a third of the time as well. 19 While some may take comfort in the fact that the group’s reviews found deficiencies in basic elements of sound research in far less than half of the studies they reviewed, all likely agree that the frequency of deficiencies still remains troubling.

Two new strategies for preventing bias in preclinical studies

Experience with the ARRIVE guidelines to date suggest that systematic adoption of new research practices will be sporadic, though, rather than widespread until we find ways to systematically move towards widespread adoption of reforms aimed at preventing bias. Perhaps the first step in moving in that direction is collectively grappling with an obvious inference to be drawn from all the evidence noted above: current success metrics in research can too often reward rather than prevent biased research. People may enjoy rewards from design-deficient studies, in the form of publications and funding, as well as the prestige that follows both. This suggests that efforts to combat bias are not just hampered by ineffective and often ill-timed bias countermeasures. They are also hampered by current flawed and entrenched incentive structures and researcher performance metrics that Hardwicke and Ioannidis contend ‘preferentially valu[e] aesthetics over authenticity.’ 38 While many readers may not agree that the current incentive structures are this far askew, we nevertheless must worry, based on the assembled evidence, that research institutions and sponsors may often incentivise biases in very much the same way that private sponsors can cause sponsorship bias.

If this analysis is sound, then widespread adoption of research practices capable of preventing bias will hinge on resisting current incentive structures. The most logical opportunity for generating such resistance resides jointly, I think, with institutional leaders and individual investigators. Though systems-level incentive structures contribute to biased research, the fact of the matter is that investigative teams conduct research and their members are trained at and often employed by research institutions. Thus, the path forward seems to depend on finding ways to get both investigators and research institutions to prize ‘authenticity’ more. This, no doubt, will prove challenging given the extent to which both groups can flourish under current rewards structures.

There are at least two complimentary strategies to look at that might prove beneficial. One encourages both investigators and research institutions to recognise the extent to which they are entangled in a major conflict of interest. Their primary interest in conducting authentic science is too often at odds with the secondary interest in being successful and enjoying the individual and institutional rewards of that success. Though we typically do not label this situation as a conflict of interest, often preferring instead the nomenclature of conflicts of commitment, the situation most assuredly is just as deeply conflicted as are the financial relationships that create sponsorship bias. If it was so designated, continued indifference about it would be difficult to maintain. That prospect alone warrants us labelling the situation the conflict of interest that it is.

The other strategy might provide additional motivation. It requires research teams and research institutions, either separately or jointly, to carefully examine the extent to which they may be contributing to the production of biased research. Here is one way they could do that: identify a systematic review of a given body of research in a given field that those participating in the exercise agree employed a reliable meta-analysis plan that identified bias and/or research deficiencies, determine whether any of the published studies included in the review originated from one’s lab or institution, and determine whether that study may have been at risk for contributing to the bias/deficiencies reported in the systematic review. If no studies from a lab group or the institution were included in the systematic review, they could still determine whether there are any published studies from the lab or institution that could have been included in the systematic review and, if so, whether their studies would have contributed to the worrisome findings reported in the systematic review. With these results in hand, the next step would be to develop a prevention plan that is designed to prevent future studies from exhibiting those problems. With the prevention plan in place, one could then determine what institutional and/or lab-level changes would be required in order to implement the prevention plan.

It is likely that few, if any, prevention plans would need to start from scratch. As most readers of this journal are no doubt aware, there is already a wealth of published scholarship about how to improve the quality of biomedical research. Some of the most relevant examples from it include routine use of study preregistration 39 40 and research reports, 38 41 42 supplementing the 3Rs 43 of animal studies with the 3 Vs of scientific validity, 25 and clearly reporting whether a study is a hypothesis generating or a hypothesis confirming study. 26

We must acknowledge at the outset, though, that developing a prevention plan will likely prove much easier than fully adopting one because adoption will reveal how deeply entrenched the conflict of interest between professional success and rewards and good science often is. For example, clearly labelling research studies as exploratory ones in publications will temper claims about innovation that researchers may be accustomed to making about their work. Similarly, employing research reports will restrict study analyses and descriptions, which will often result in more constrained publications. 41 Different researchers no doubt will respond differently to these changes, but one can hope that enough of them will feel empowered by the changes to become champions of science reforms within their institutions and professional societies meant to align success metrics with good research. Supporting this expectation are recent studies reporting that researchers are eager for improved research climates at their organisations. 44 45

While research teams develop and implement prevention plans, institutional leaders will need to take responsibility for eliminating the conflicts of interest that promote bias in research. They would not need to start from scratch either, since important preliminary work that could help with this is already underway. This work includes efforts that show how to align institutional metrics of professional success with good science. 46–48 An additional resource they could fruitfully draw from is the recently published ‘Hong Kong Principles for assessing researchers.’ 49 Here too it will no doubt be easier to develop than implement plans meant to avoid the entrenched conflict of interest. But benefits may quickly materialise as soon as the work to develop prevention plans materialise. Once institutions name, and thus acknowledge, the conflict of interest that they are helping to perpetuate, maintaining the status quo should prove that much more difficult. This should help to create at least some inertia tilted toward reform and thus away from stasis.

Many readers will no doubt be less sanguine about the success prospects for either strategy. The teams and institutions that choose to adopt them would no doubt have concerns that they would be unilaterally placing themselves at a disadvantage to those that choose not to burden themselves with the demands of either of the proposed strategies. With such concerns in mind, it is helpful to ponder how we might address them. Probably the best option for doing so is to implement some pilot projects to test the use of systematic reviews to develop bias prevention plans. There are at least two options for implementing such pilot projects.

One is for either an institution or a professional society to host a competition where the team that develops the best prevention plan for their work receives some kind of institutional/professional society recognition or reward. Institutional rewards might be monetary in the form of travel stipends for graduate students or postdoctoral fellows to attend conferences. Professional society rewards might be a plenary session at a society’s annual meeting where the winning team could present its bias prevention plan.

The other option is for research institutions to work through their main research officers to sponsor audits of the work of research teams. The audits would be informed by relevant systematic reviews. The audits could either be random or limited to teams that volunteer. To ensure that the audits are not seen or experienced as punitive, the launch of the audits would need to be preceded by a communication campaign that explained the purpose and value of the audits. Others may identify additional options for implementing pilot projects. Whatever options research teams, institutions, and/or professional societies might use, such pilot projects should prove valuable. They are likely the quickest way to learn whether systematic reviews could be used to interrogate research quality at the local level and to develop prevention plans for reducing bias in research.

There is no one panacea capable of turning away all the contributors to decades of disappointing clinical translation efforts. And even if we could snap our fingers and banish overnight the biases that are among the contributors to the disappointing results, science still may not take us to the goal of improved clinical treatments that we seek. After all, we are dealing with science, not magic. But if we could muster the desire and discipline to better combat bias in research, at least we could take comfort in the fact that what we are calling science is in fact actual science, as free of bias as we can possibly make it. The two complimentary strategies described above are offered in hopes that they could help to muster that desire and discipline. If either or both were to prove beneficial, we would find ourselves in a place far preferable to the one we are in now.

Acknowledgments

The author would also like to acknowledge the support of Fondation Brocher, the thoughtful suggestions of several reviewers, and useful input from colleagues Robert Nadon and Fernando Fierro.

Correction notice: This article has been corrected since it was published Online First. In the Acknowledgments, name "Fernando Feraro" has been corrected to "Fernando Fierro".

Contributors: The author conceived the ideas for the manuscript and exclusively wrote all versions of the manuscript, including the final one.

Funding: A portion of the author’s time was supported by the National Centre for Advancing Translational Sciences, National Institutes of Health, through grant number UL1 TR001860.

Competing interests: None declared.

Provenance and peer review: Not commissioned; externally peer reviewed.

Data availability statement: Data sharing not applicable as no datasets generated and/or analysed for this study. This manuscript does not report about any original empirical research and thus there are no research data to share.

Open peer review: Prepublication and Review History is available online at http://dx.doi.org/10.1136/bmjos-2020-100116 .

1. Sackett DL. Bias In Analytic Research. In: Ibrahim MA, ed. The case-control study consensus and controversy. Pergamon, 1979: 51–63. [ Google Scholar ]
2. Tsilidis KK, Panagiotou OA, Sena ES, et al. Evaluation of excess significance bias in animal studies of neurological diseases. PLoS Biol 2013;11:e1001609. 10.1371/journal.pbio.1001609 [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ]
3. Ioannidis JPA. Why most published research findings are false. PLoS Med 2005;2:e124–701. 10.1371/journal.pmed.0020124 [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ]
4. Jefferson T. Sponsorship bias in clinical trials - growing menace or dawning realisation? JLL Bull. Available: https://www.jameslindlibrary.org/articles/sponsorship-bias-in-clinical-trials-growing-menace-or-dawning-realisation/ [Accessed 18 Apr 2020]. [ DOI ] [ PMC free article ] [ PubMed ]
5. Fabbri A, Lai A, Grundy Q, et al. The influence of industry sponsorship on the research agenda: a scoping review. Am J Public Health 2018;108:e9–16. 10.2105/AJPH.2018.304677 [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ]
6. John S. Scientific deceit. Synthese;28. 10.1007/s11229-018-02017-4 [ DOI ] [ Google Scholar ]
7. Holman B, Elliott KC. The promise and perils of industry-funded science. Philos Compass 2018;13:e12544. 10.1111/phc3.12544 [ DOI ] [ Google Scholar ]
8. Doucet M, Sismondo S. Evaluating solutions to sponsorship bias. J Med Ethics 2008;34:627–30. 10.1136/jme.2007.022467 [ DOI ] [ PubMed ] [ Google Scholar ]
9. Elliott KC. Scientific judgment and the limits of conflict-of-interest policies. Account Res 2008;15:1–29. 10.1080/08989620701783725 [ DOI ] [ PubMed ] [ Google Scholar ]
10. de Melo-Martín I, Intemann K. How do disclosure policies fail? Let us count the ways. Faseb J 2009;23:1638–42. 10.1096/fj.08-125963 [ DOI ] [ PubMed ] [ Google Scholar ]
11. Mello MM, Murtagh L, Joffe S, et al. Beyond financial conflicts of interest: institutional oversight of faculty consulting agreements at schools of medicine and public health. PLoS One 2018;13:e0203179. 10.1371/journal.pone.0203179 [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ]
12. Campsall P, Colizza K, Straus S, et al. Financial relationships between organizations that produce clinical practice guidelines and the biomedical industry: a cross-sectional study. PLoS Med 2016;13:e1002029. 10.1371/journal.pmed.1002029 [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ]
13. John LK, Loewenstein G, Marder A, et al. Effect of revealing authors’ conflicts of interests in peer review: randomized controlled trial. BMJ 2019;2:l5896. 10.1136/bmj.l5896 [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ]
14. Lundh A, Rasmussen K, Østengaard L, et al. Systematic review finds that appraisal tools for medical research studies address conflicts of interest superficially. J Clin Epidemiol 2020;120:104–15. 10.1016/j.jclinepi.2019.12.005 [ DOI ] [ PubMed ] [ Google Scholar ]
15. Goldberg DS. The shadows of sunlight: why disclosure should not be a priority in addressing conflicts of interest. Public Health Ethics 2019;12:202–12. 10.1093/phe/phy016 [ DOI ] [ Google Scholar ]
16. Matheson A. The ICMJE Recommendations and pharmaceutical marketing--strengths, weaknesses and the unsolved problem of attribution in publication ethics. BMC Med Ethics 2016;17:20. 10.1186/s12910-016-0103-7 [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ]
17. Sismondo S. Ghost-manged medicine: big pharma’s invisible hands. Mattering Press, 2018. [ Google Scholar ]
18. Cain DM, Loewenstein G, Hall P, et al. The dirt on coming clean: perverse effects of disclosing conflicts of interest;33. [ Google Scholar ]
19. Willmann R, Lee J, Turner C, et al. Improving translatability of preclinical studies for neuromuscular disorders: lessons from the TREAT-NMD Advisory Committee for therapeutics (TACT). Dis Model Mech 2020;13:dmm042903. 10.1242/dmm.042903 [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ]
20. Yarborough M, Bredenoord A, D'Abramo F, et al. The bench is closer to the bedside than we think: uncovering the ethical ties between preclinical researchers in translational neuroscience and patients in clinical trials. PLoS Biol 2018;16:e2006343. 10.1371/journal.pbio.2006343 [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ]
21. Sena ES, van der Worp HB, Bath PMW, et al. Publication bias in reports of animal stroke studies leads to major overstatement of efficacy. PLoS Biol 2010;8:e1000344. 10.1371/journal.pbio.1000344 [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ]
22. Howells DW, Sena ES, Macleod MR. Bringing rigour to translational medicine. Nat Rev Neurol 2014;10:37–43. 10.1038/nrneurol.2013.232 [ DOI ] [ PubMed ] [ Google Scholar ]
23. Kola I, Landis J. Can the pharmaceutical industry reduce attrition rates? Nat Rev Drug Discov 2004;3:711–6. 10.1038/nrd1470 [ DOI ] [ PubMed ] [ Google Scholar ]
24. Bailoo JD, Reichlin TS, Würbel H. Refinement of experimental design and conduct in laboratory animal research. Ilar J 2014;55:383–91. 10.1093/ilar/ilu037 [ DOI ] [ PubMed ] [ Google Scholar ]
25. Würbel H. More than 3Rs: the importance of scientific validity for harm-benefit analysis of animal research. Lab Anim 2017;46:164–6. 10.1038/laban.1220 [ DOI ] [ PubMed ] [ Google Scholar ]
26. Kimmelman J, Mogil JS, Dirnagl U. Distinguishing between exploratory and confirmatory preclinical research will improve translation. PLoS Biol 2014;12:e1001863. 10.1371/journal.pbio.1001863 [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ]
27. Schmidt-Pogoda A, Bonberg N, Koecke MHM, et al. Why most acute stroke studies are positive in animals but not in patients: a systematic comparison of preclinical, early phase, and phase 3 clinical trials of neuroprotective agents. Ann Neurol 2020;87:40–51. 10.1002/ana.25643 [ DOI ] [ PubMed ] [ Google Scholar ]
28. Halpern J, Paolo D, Huang A. Informed consent for early-phase clinical trials: therapeutic misestimation, unrealistic optimism and appreciation. J Med Ethics 2019;45:384–7. 10.1136/medethics-2018-105226 [ DOI ] [ PubMed ] [ Google Scholar ]
29. Kimmelman J, Federico C. Consider drug efficacy before first-in-human trials. Nature 2017;542:25–7. 10.1038/542025a [ DOI ] [ PubMed ] [ Google Scholar ]
30. Yarborough M. Do we really know how many clinical trials are conducted ethically? Why research ethics Committee review practices need to be strengthened and initial steps we could take to strengthen them. J Med Ethics 2020;36:medethics-2019-106014. 10.1136/medethics-2019-106014 [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ]
31. Joseph A. Lancet, NEJM retract Covid-19 studies that sparked backlash. STAT, 2020. Available: https://www.statnews.com/2020/06/04/lancet-retracts-major-covid-19-paper-that-raised-safety-concerns-about-malaria-drugs/ [Accessed 8 Jun 2020].
32. Peer-Reviewed Scientific Journals Don’t Really Do Their Job | WIRED. Available: https://www.wired.com/story/peer-reviewed-scientific-journals-dont-really-do-their-job/ [Accessed 26 Jun 2020].
33. Kilkenny C, Browne WJ, Cuthill IC, et al. Improving bioscience research reporting: the ARRIVE guidelines for reporting animal research. PLoS Biol 2010;8:e1000412. 10.1371/journal.pbio.1000412 [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ]
34. Percie du Sert N, Ahluwalia A, Alam S, et al. Reporting animal research: explanation and elaboration for the ARRIVE guidelines 2.0. PLoS Biol 2020;18:e3000411. 10.1371/journal.pbio.3000411 [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ]
35. Enserink M. Sloppy reporting on animal studies proves hard to change. Science 2017;357:1337–8. 10.1126/science.357.6358.1337 [ DOI ] [ PubMed ] [ Google Scholar ]
36. Minnerup J, Dirnagl U, Schäbitz W-R. Checklists for authors improve the reporting of basic science research. Stroke 2020;51:6–7. 10.1161/STROKEAHA.119.027626 [ DOI ] [ PubMed ] [ Google Scholar ]
37. Ramirez FD, Jung RG, Motazedian P, et al. Journal Initiatives to Enhance Preclinical Research: Analyses of Stroke, Nature Medicine, Science Translational Medicine. Stroke 2020;51:291–9. 10.1161/STROKEAHA.119.026564 [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ]
38. Hardwicke TE, Ioannidis JPA. Mapping the universe of registered reports. Nat Hum Behav 2018;2:793–6. 10.1038/s41562-018-0444-y [ DOI ] [ PubMed ] [ Google Scholar ]
39. Nosek BA, Ebersole CR, DeHaven AC, et al. The preregistration revolution. Proc Natl Acad Sci U S A 2018;115:2600–6. 10.1073/pnas.1708274114 [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ]
40. Dirnagl U. Preregistration of exploratory research: learning from the golden age of discovery. PLoS Biol 2020;18:e3000690. 10.1371/journal.pbio.3000690 [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ]
41. Scheel AM, Schijen M, Lakens D. An excess of positive results: comparing the standard psychology literature with registered reports.
42. Hair K. Registered reports at BMJ open science: making preclinical research match-fit for translation. BMJ Open Sci 2018. https://blogs.bmj.com/openscience/2018/02/08/registered-reports-at-bmj-open-science-making-preclinical-research-match-fit-for-translation/ [ Google Scholar ]
43. Russell WMS, Burch RL. The principles of humane experimental technique. London: Methuen, 1959. [ Google Scholar ]
44. Edwards MA, Roy S. Academic research in the 21st century: maintaining scientific integrity in a climate of perverse incentives and Hypercompetition. Environ Eng Sci 2017;34:51–61. 10.1089/ees.2016.0223 [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ]
45. Haven T, Pasman HR, Widdershoven G, et al. Researchers' perceptions of a responsible research climate: a multi focus group study. Sci Eng Ethics 2020. 10.1007/s11948-020-00256-8. [Epub ahead of print: 10 Aug 2020]. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ]
46. Benedictus R, Miedema F, Ferguson MWJ. Fewer numbers, better science. Nature 2016;538:453–5. 10.1038/538453a [ DOI ] [ PubMed ] [ Google Scholar ]
47. Rice DB, Raffoul H, Ioannidis JPA, et al. Academic criteria for promotion and tenure in biomedical sciences faculties: cross sectional analysis of international sample of universities. BMJ 2020;369:m2081. 10.1136/bmj.m2081 [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ]
48. Sox HC, Schuster MA. Criteria for academic promotion in medicine. BMJ 2020;369:m2253. 10.1136/bmj.m2253 [ DOI ] [ PubMed ] [ Google Scholar ]
49. Moher D, Bouter L, Kleinert S, et al. The Hong Kong principles for assessing researchers: fostering research integrity. PLoS Biol 2020;18:e3000737. 10.1371/journal.pbio.3000737 [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ]
50. Applbaum K. Getting to Yes: corporate power and the creation of a psychopharmaceutical blockbuster. Cult Med Psychiatry 2009;33:185–215. 10.1007/s11013-009-9129-3 [ DOI ] [ PubMed ] [ Google Scholar ]
51. Holman B, Geislar S, Drugs S. Sex drugs and corporate Ventriloquism: how to evaluate science policies intended to manage Industry-Funded bias. Philos Sci 2018;85:869–81. 10.1086/699713 [ DOI ] [ Google Scholar ]
52. Vogt L, Reichlin TS, Nathues C, et al. Authorization of animal experiments is based on confidence rather than evidence of scientific rigor. PLoS Biol 2016;14:e2000598. 10.1371/journal.pbio.2000598 [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ]
53. Smith R. Peer review: a flawed process at the heart of science and journals. J R Soc Med 2006;99:178–82. 10.1177/014107680609900414 [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ]
54. Horrobin DF. Something rotten at the core of science? Trends Pharmacol Sci 2001;22:51–2. 10.1016/S0165-6147(00)01618-7 [ DOI ] [ PubMed ] [ Google Scholar ]
55. Han S, Olonisakin TF, Pribis JP, et al. A checklist is associated with increased quality of reporting preclinical biomedical research: a systematic review. PLoS One 2017;12:e0183591. 10.1371/journal.pone.0183591 [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ]
56. Hair K, Macleod MR, Sena ES. A randomised controlled trial of an intervention to improve compliance with the ARRIVE guidelines (IICARus). Res Integr Peer Rev 2019;4:1. 10.1186/s41073-019-0069-3 [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ]
57. Viswanathan M, Patnode CD, Berkman ND, et al. Recommendations for assessing the risk of bias in systematic reviews of health-care interventions. J Clin Epidemiol 2018;97:26–34. 10.1016/j.jclinepi.2017.12.004 [ DOI ] [ PubMed ] [ Google Scholar ]
58. What is GRADE? | BMJ Best Practice. Available: https://bestpractice.bmj.com/info/us/toolkit/learn-ebm/what-is-grade/ [Accessed 15 Sep 2020].
59. Page MJ, McKenzie JE, Higgins JPT. Tools for assessing risk of reporting biases in studies and syntheses of studies: a systematic review. BMJ Open 2018;8:e019703. 10.1136/bmjopen-2017-019703 [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ]
60. Haddaway NR, Bethel A, Dicks LV, et al. Eight problems with literature reviews and how to fix them. Nat Ecol Evol 2020:1–8. [ DOI ] [ PubMed ] [ Google Scholar ]
61. Turner EH, Matthews AM, Linardatos E, et al. Selective publication of antidepressant trials and its influence on apparent efficacy. N Engl J Med 2008;358:252–60. 10.1056/NEJMsa065779 [ DOI ] [ PubMed ] [ Google Scholar ]
62. Prior M, Hibberd R, Asemota N, et al. Inadvertent P-hacking among trials and systematic reviews of the effect of progestogens in pregnancy? A systematic review and meta-analysis. BJOG 2017;124:1008–15. 10.1111/1471-0528.14506 [ DOI ] [ PubMed ] [ Google Scholar ]
View on publisher site
PDF (287.0 KB)
Collections

Add to Collections

An official website of the United States government

The .gov means it's official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you're on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Publications
Account settings
Browse Titles

NCBI Bookshelf. A service of the National Library of Medicine, National Institutes of Health.

StatPearls [Internet]. Treasure Island (FL): StatPearls Publishing; 2024 Jan-.

StatPearls [Internet].

Aleksandar Popovic ; Martin R. Huecker .

Affiliations

Last Update: June 20, 2023 .

Definition/Introduction

Bias is colloquially defined as any tendency that limits impartial consideration of a question or issue. In academic research, bias refers to a type of systematic error that can distort measurements and/or affect investigations and their results. [1] It is important to distinguish a systematic error, such as bias, from that of random error. Random error occurs due to the natural fluctuation in the accuracy of any measurement device, the innate differences between humans (both investigators and subjects), and by pure chance. Random errors can occur at any point and are more difficult to control. [2] Systematic errors, referred to as bias from here on, occur at one or multiple points during the research process, including the study design, data collection, statistical analysis, interpretation of results, and publication process. [3]

However, interpreting the presence of bias involves understanding that it is not a dichotomous variable, where the results can either be “present” or “not present.” Rather, it must be understood that bias is always present to some degree due to inherent limitations in research, its design, implementation, and ethical considerations. [4] Therefore, it is instead crucial to evaluate how much bias is present in a study and how the researchers attempted to minimize any sources of bias. [5] When evaluating for bias, it is important to note there are many types with several proposed classification schemes. However, it is easiest to view bias based on the various stages of research studies; the planning and design stage (before), data collection and analysis (during), and interpretation of results and journal submission (after).

Issues of Concern

The planning stage of any study can have bias present in both study design and recruitment of subjects. Ideally, the design of a study should include a well-defined outcome, population of interest, and collection methods before implementation and data collection. The outcome, for example, response rates to a new medication, should be precisely agreed upon. Investigators may focus on changes in laboratory parameters (such as a new statin reducing LDL and total cholesterol levels) or focus on long-term morbidity and mortality (does the new statin cause reduction in cardiovascular-related deaths?) Similarly, the investigator’s own pre-existing notion or personal beliefs can influence the question being asked and the study's methodology. [6]

For example, an investigator who works for a pharmaceutical company may address a question or collect data most likely to produce a significant finding supporting the use of the investigational medication. Thus, if possible, the question(s) being asked and the collection methods employed should be agreed upon by multiple team members in an interprofessional setting to reduce potential bias. Ethics committees also play a valuable role here.

Relatedly, the team members designing a study must define their population of interest, also referred to as the study population. Bias occurs if the study population does not closely represent a target population due to errors in study design or implementation, termed selection bias. Sampling bias is one form of selection bias and typically occurs if subjects were selected in a non-random way. It can also occur if the study requires subjects to be placed into cohorts and if those cohorts are significantly different in some way. This can lead to erroneous conclusions and significant findings. Randomization of subject selection and cohort assignment is a technique used in study design intended to reduce sampling bias. [7] [8]

However, bias can occur if subject selection occurred through limited means, such as recruiting subjects through phone landlines, thereby excluding anyone who does not own a landline. Similarly, this can occur if subjects are recruited only through email or a website. This can result in confounding or the introduction of 3 variable that influences both the independent and dependent variables. [9]

For example, if a study recruited subjects from two primary care clinics to compare diabetes screening and treatment rates but did not account for potentially different socioeconomic characteristics of the two clinics, there may be significant differences between groups not due to clinical practice but rather cohort composition.

A subtype of selection bias, admission bias (also referred to as Berkson bias), occurs when the selected study population is derived from patients within hospitals or certain specialty clinics. This group is then compared to a non-hospitalized group. This predisposes to bias as hospitalized patient populations are more likely to be ill and not represent the general population. Furthermore, there are typically other confounding variables or covariates that may skew relationships between the intended dependent and independent variables. [10]

For example, in one study that evaluated the effect of cigarette smoking and its association with bladder cancer, researchers decided to use a hospital-based case-control study design. Normally, there is a strong and well-established relationship between years of cigarette use and the likelihood of developing bladder cancer. In fact, part of screening guidelines for bladder cancer considers the total years that an individual has smoked during patient risk stratification and subsequent evaluation and follow-up. However, in one study, researchers noted no significant relationship between smoking and bladder cancer. Upon re-evaluating, they noted their cases and controls both had significant smoking histories, thereby blurring any relationships. [11]

Admission bias can be reduced by selecting appropriate controls and being cognizant of the potential introduction of this bias in any hospital-based study. If this is not possible to do, researchers must be transparent about this in their work and may try to use different methods of statistical analysis to account for any confounding variables. In an almost opposite fashion, another source of potential error is a phenomenon termed the healthy worker effect. The healthy worker effect refers to the overall improved health and decreased mortality and morbidity rates of those employed relative to the unemployed. This occurs for various reasons, including access to better health care, improved socioeconomic status, the beneficial effects of work itself, and those who are critically ill or disabled are less likely to find employment. [12] [13]

Two other important forms of selection bias are lead-time bias and length time bias. Lead-time bias occurs in the context of disease diagnosis. In general, it occurs when new diagnostic testing allows detection of a disease in an early stage, causing a false appearance of longer lifespan or improved outcomes. [14] An example of this is noted in individuals with schizophrenia with varying durations of untreated psychosis. Those with shorter durations of psychosis relative to longer durations typically had better psychosocial functioning after admission to and treatment within a hospital. However, upon further analysis, it was found that it was not the duration of psychosis that affected psychosocial functioning. Rather, the duration of psychosis was indicative of the stage of the person’s disease, and those individuals with shorter durations of psychosis were in an earlier stage of their disease. [15]

Length time bias is similar to lead-time bias; however, it refers to the overestimation of an individual’s survival time due to a large number of cases that are asymptomatic and slowly progressing with a smaller number of cases that are rapidly progressive and symptomatic. An example can be noted in patients with hepatocellular carcinoma (HCC). Those who have HCC found via asymptomatic screening typically had a tumor doubling time of 100 days. In contrast, those individuals who had HCC uncovered due to symptomatic presentation had a tumor doubling time of 42 days on average. However, overall outcomes were the same amongst these two groups. [16]

The effect of both lead time and length time bias must be taken into effect by investigators. For lead-time bias, investigators can instead look at changes in the overall mortality rate due to disease. One method involves creating a modified survival curve that considers possible lead-time bias with the new diagnostic or screening protocols. [17] This involves an estimate of the lead time bias and subsequently subtracting this from the observed survival time. Unfortunately, the consequences of length time bias are difficult to mitigate, but investigators can minimize their effects by keeping individuals in their original groups based on screening protocols (intention-to-screen) regardless of the individual required earlier diagnostic workup due to symptoms.

Channeling and procedure bias are other forms of selection bias that can be encountered and addressed during the planning stage of a study. Channeling bias is a type of selection bias noted in observational studies. It occurs most frequently when patient characteristics, such as age or severity of illness, affect cohort assignment. This can occur, for example, in surgical studies where different interventions carry different levels of risk. Surgical procedures may be more likely to be carried out on patients with lower levels of periprocedural risk who would likely tolerate the event, whereas non-surgical interventions may be reserved for patients with higher levels of risk who would not be suitable for a lengthy procedure under general anesthesia. [18] As a result, channeling bias results in an imbalance of covariates between cohorts. This is particularly important when the surgical and non-surgical interventions have significant differences in outcome, making it difficult to ascertain if the difference is due to different interventions or covariate imbalance. Channeling bias can be accounted for through the use of propensity score analysis. [19]

Propensity scores are the probability of receiving one intervention over another based on an individual's observed covariates. These scores are obtained through a variety of different methods and then accounted for in the analysis stage via statistical methods, such as logistic regression. In addition to channeling bias, procedure bias (administration bias) is a similar form of selection bias, where two cohorts receive different levels of treatment or are administered similar treatments or interviews in different formats. An example of the former would be two cohorts of patients with ACL injuries. One cohort received strictly supervised physical therapy 3 times per week, and the other cohort was taught the exercises but instructed to do them at home on their own. An example of the latter would be administering a questionnaire regarding eating disorder symptoms. One group was asked in-person in an interview format, and the other group was allowed to take the questionnaire at home in an anonymous format. [20]

Either form of procedure bias can lead to significant differences observed between groups that might not exist where they are treated the same. Therefore, both procedure and channeling bias must be considered before data collection, particularly in observational or retrospective studies, to reduce or eliminate erroneous conclusions that are derived from the study design itself and not from treatment protocols.

Bias in Data Collection & Analysis

There are also a variety of forms of bias present during data collection and analysis. One type is observer bias, which refers to any systematic difference between true and recorded values due to variation in the individual observer. This form of bias is particularly notable in studies that require investigators to record measurements or exposures, particularly if there is an element of subjectiveness present, such as evaluating the extent or color of a rash. [21] However, this has even been noted in the measurement of subjects’ blood pressures when using sphygmomanometers, where investigators may round up or down depending on their preconceived notions about the subject. Observer bias is more likely when the observer is aware of the subject’s treatment status or assignment cohort. This is related to confirmation bias, which refers to a tendency to search for or interpret information to support a pre-existing belief. [22]

In one prominent example, physicians were asked to estimate blood loss and amniotic fluid volume in pregnant patients currently in labor. By providing additional information in the form of blood pressures (hypotensive or normotensive) to the physicians, they were more likely to overestimate blood loss and underestimate amniotic fluid volume when told the patient was hypotensive. [23] Similar findings are noted in fields such as medicine, health sciences, and social sciences, illustrating the strong and misdirecting influence of confirmation bias on the results found in certain studies. [22] [24]

Investigators and data collectors need to be trained to collect data in a uniform, empirical fashion and be conscious of their own beliefs to minimize measurement variability. There should be standardization of data collection to reduce inter-observer variance. This may include training all investigators or analysts to follow a standardized protocol, use standardized devices or measurement tools, or use validated questionnaires. [21] [25]

Furthermore, the decision of whether to blind the investigators and analysts should also be made. If implemented, blinding of the investigators can reduce observer bias, which refers to the differential assessment of an outcome when subjective criteria are being assessed. Confirmation bias within investigators and data collectors can be minimized if they are informed of its potential interfering role. Furthermore, overconfidence in either the overall study’s results or the collection of accurate data from subjects can be a strong source of confirmation bias. Challenging overconfidence and encouraging multiple viewpoints is another mechanism by which to challenge this within investigators. Lastly, potential funding sources or other conflicts of interest can influence confirmation and observer bias and must be considered when evaluating for these potential sources of systematic error. [26] [27] However, subjects themselves may change their behavior, consciously or unconsciously, in response to their awareness of being observed or being assigned to a treatment group termed the Hawthorne effect. [28] The Hawthorne effect can be minimized, although not eliminated, by reducing or hiding the observation of the subject if possible. A similar phenomenon is noted with self-selection bias, which occurs when individuals sort themselves into groups or choose to enroll in studies based on pre-existing factors. For example, a study evaluating the effectiveness of a popular weight loss program that allows participants to self-enroll may have significant differences between groups. In circumstances such as this, it is more probable that individuals who experienced greater success (measured in terms of weight lost) are likely to enroll. Meanwhile, those who did not lose weight and/or gained weight would likely not enroll. Similar issues plague other studies that rely on subject self-enrollment. [20] [29]

Self-selection bias is often found in tandem with response bias, which refers to subjects inaccurately answering questions due to various influences. [30] This can be due to question-wording, the social desirability of a certain answer, the sensitiveness of a question, the order of questions, and even the survey format, such as in-person, via telephone, or online. [22] [31] [32] [33] [34] There are methods of reducing the impact of all these factors, such as the use of anonymity in surveys, the use of specialized questioning techniques to reduce the impact of wording, and even the use of nominative techniques where individuals are asked about the behavior of close friends for certain types of questions. [35] Non-response bias refers to significant differences between individuals who respond and those who do not respond to a survey or questionnaire. It is not to be confused as being the opposite of response bias. It is particularly problematic as errors can result in estimating population characteristics due to a lack of response from the non-responders. It is often noted in health surveys regarding alcohol, tobacco, or drug use, though it has been seen in many other topics targeted by surveys. [36] [37] [36] Furthermore, particularly in surveys designed to evaluate satisfaction after an intervention or treatment, individuals are much more likely to respond if they felt highly satisfied relative to the average individual. While highly dissatisfied individuals were also more likely to respond relative to average, they were less likely to respond relative to highly satisfied individuals, thus potentially skewing results toward respondents with positive viewpoints. This can be noted in product reviews or restaurant evaluations.

Several preventative steps can be taken during study design or data collection to mitigate the effects of non-response bias. Ideally, surveys should be as short and accessible as possible, and potential participants should be involved in questions design. Additionally, incentives can be provided for participation if necessary. Lastly, if necessary, surveys can be made mandatory as opposed to voluntary. For example, this could occur if school-age children were initially sent a survey via mail to their homes to complete voluntarily, but this was later changed to a survey required to be completed and handed in at school on an anonymous basis. [38] [39]

Similar to the Hawthorne effect and self-selection bias, recall bias is another potential source of systematic error stemming from the subjects of a particular study. Recall bias is any error due to differences in an individual’s recollections and what truly transpired. Recall bias is particularly prevalent in retrospective studies that use questionnaires, surveys, and/or interviews. [40]

For example, in a retrospective study evaluating the prevalence of cigarette smoking in individuals diagnosed with lung cancer vs. those without, those with lung cancer may be more likely to overestimate their use of tobacco meanwhile those without may underestimate their use. Fortunately, the impact of recall bias can be minimized by decreasing the time interval between an outcome (lung cancer) and exposure (tobacco use). The rationale for this is that individuals are more likely to be accurate when the time period assessed is of shorter duration. Other methods that can be used would be to corroborate the individual’s subjective assessments with medical records or other objective measures whenever possible. [41]

Lastly, in addition to the data collectors and the subjects, bias and subsequent systematic error can be introduced through data analysis, especially if conducted in a manner that gives preference to certain conclusions. There can be blatant data fabrication where non-existing data is reported. However, researchers are more likely to perform multiple tests with pair-wise comparisons, termed “p-hacking.” [42] This typically involves analysis of subgroups or multiple endpoints to obtain statistically significant findings, even if these findings were unrelated to the original hypothesis. P-hacking also occurs when investigators perform data analysis partway through data collection to determine if it is worth continuing or not. [43] It also occurs when covariates are excluded, if outliers are included or dropped without mention, or if treatment groups are split, combined, or otherwise modified based on the original research design. [44] [45]

Ideally, researchers should list all variables explored and all associated findings. If any observations are eliminated (outliers), they should be reported, and an explanation is given as to why they were eliminated and how their elimination affected the data.

Bias in Data Interpretation and Publication

The final stages of any study, interpretation of data and publication of results, is also susceptible to various types of bias. During data interpretation and subsequent discussion, researchers must ensure that the proper statistical tests were used and that they were used correctly. Furthermore, results discussed should be statistically significant, and discussion should be avoided with results that “approach significance.” [46] Furthermore, bias can also be introduced in this stage if researchers discuss statistically significant differences but not clinically significant if conclusions are made about causality when the experiment was purely observational if data is extrapolated beyond the range found within the study. [3]

A major form of bias found during the publication stage is appropriately named publication bias. This refers to the submission of either statistically or clinically significant results, excluding other findings. [47] Journals and publishers themselves have been found to favor studies with significant values. However, researchers themselves may, in turn, use methods of data analysis or interpretation (mentioned above) to uncover significant results. Outcome reporting bias is similar, which refers to the submission of statistically significant results only, excluding non-significant ones. These two biases have been found to affect the results of systematic analyses and even affect the clinical management of patients. [48] However, publication and outcome reporting bias can be prevented in certain cases. Any prospective trials are typically required to be registered before study commencement, meaning that all results, whether significant or not, will be visible. Furthermore, electronic registration and archiving of findings can also help reduce publication bias. [49]

Clinical Significance

Understanding basic aspects of study bias and related concepts will aid clinicians in practicing and improving evidence-based medicine. Study bias can be a major factor that detracts from the external validity of a study or the generalizability of findings to other populations or settings. [50] Clinicians who possess a strong understanding of the various biases that can plague studies will be better able to determine the external validity and, therefore, clinical applicability of a study's findings. [51] [52]

The replicability of a study with similar findings is a strong factor in determining its external validity and generalizability to the clinical setting. Whenever possible, clinicians should arm themselves with the knowledge from multiple studies or systematic reviews on a topic, as opposed to using a single study. [53] Systematic reviews allow applying strategies that limit bias through systematic assembly, appraisal, and unification of the relevant studies regarding a topic. [54]

With a critical, investigational point of view, a willingness to evaluate contrary sources, and the use of systematic reviews, clinicians can better identify sources of bias. In doing so, they can better reduce its impact in their decision-making process and thereby implement a strong form of evidence-based medicine.

Nursing, Allied Health, and Interprofessional Team Interventions

There are numerous sources of bias within the research process, ranging from the design and planning stage, data collection and analysis, interpretation of results, and the publication process. Bias in one or multiple points of this process can skew results and even lead to incorrect conclusions. This, in turn, can cause harmful medical decisions, affecting patients, their families, and the overall healthcare team. Outside of medicine, significant bias can result in erroneous conclusions in academic research, leading to future fruitless studies in the same field. [55]

When combined with the knowledge that most studies are never replicated or verified, this can lead to a deleterious cycle of biased, unverified research leading to more research. This can harm the investigators and institutions partaking in such research and discredit entire fields, even if other investigators had significant work and took extreme care to limit and explain sources of bias.

All research needs to be carried out and reported transparently and honestly. In recent years, important steps have been taken, such as increased awareness of biases present in the research process, manipulating statistics to generate significant results, and implementing a clinical trial registry system. However, all stakeholders of the research process, from investigators to data collectors, to the institutions they are a part of, and the journals that review and publish findings, must take extreme care to identify and limit sources of bias and report those transparently.

All interprofessional healthcare team members, including physicians, physician assistants, nurses, pharmacists, and therapists, need to understand the variety of biases present throughout the research process. Such knowledge will separate stronger studies from weaker ones, determine the clinical and real-world applicability of results, and optimize patient care through the appropriate use of data-driven research results considering potential biases. Failure to understand various biases and how they can skew research results can lead to suboptimal and potentially deleterious decision-making and negatively impact both patient and system outcomes.

Review Questions
Access free multiple choice questions on this topic.
Comment on this article.

Disclosure: Aleksandar Popovic declares no relevant financial relationships with ineligible companies.

Disclosure: Martin Huecker declares no relevant financial relationships with ineligible companies.

This book is distributed under the terms of the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International (CC BY-NC-ND 4.0) ( http://creativecommons.org/licenses/by-nc-nd/4.0/ ), which permits others to distribute the work, provided that the article is not altered or used commercially. You are not required to obtain permission to distribute this article, provided that you credit the author and journal.

Cite this Page Popovic A, Huecker MR. Study Bias. [Updated 2023 Jun 20]. In: StatPearls [Internet]. Treasure Island (FL): StatPearls Publishing; 2024 Jan-.

In this Page

Bulk download.

Bulk download StatPearls data from FTP

Related information

PMC PubMed Central citations
PubMed Links to PubMed

Recent Activity

Study Bias - StatPearls Study Bias - StatPearls

Your browsing activity is empty.

Activity recording is turned off.

Turn recording back on

Connect with NLM

National Library of Medicine 8600 Rockville Pike Bethesda, MD 20894

Web Policies FOIA HHS Vulnerability Disclosure

Help Accessibility Careers