ethics in research pdf

CITE AS:

Resnik, D. B. \(2015\). What is ethics \ in research and why is it important? National Institutes of Health . https://www.niehs.nih.gov/research/resources/bioethics/whatis

Resnik, D. B. \(2015, December 1\). What is ethics in research and why is it important? National Institutes of Health. https://www.niehs.nih.gov/research/resources/bioethics/whatis

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• Ann Med Surg (Lond)
• v.86(5); 2024 May
• PMC11060189

Ethics in scientific research: a lens into its importance, history, and future

Associated data.

Not applicable.

Introduction

Ethics are a guiding principle that shapes the conduct of researchers. It influences both the process of discovery and the implications and applications of scientific findings 1 . Ethical considerations in research include, but are not limited to, the management of data, the responsible use of resources, respect for human rights, the treatment of human and animal subjects, social responsibility, honesty, integrity, and the dissemination of research findings 1 . At its core, ethics in scientific research aims to ensure that the pursuit of knowledge does not come at the expense of societal or individual well-being. It fosters an environment where scientific inquiry can thrive responsibly 1 .

The need to understand and uphold ethics in scientific research is pertinent in today’s scientific community. First, the rapid advancement of technology and science raises ethical questions in fields like biotechnology, biomedical science, genetics, and artificial intelligence. These advancements raise questions about privacy, consent, and the potential long-term impacts on society and its environment 2 . Furthermore, the rise in public perception and scrutiny of scientific practices, fueled by a more informed and connected populace, demands greater transparency and ethical accountability from researchers and institutions.

This commentary seeks to bring to light the need and benefits associated with ethical adherence. The central theme of this paper highlights how upholding ethics in scientific research is a cornerstone for progress. It buttresses the fact that ethics in scientific research is vital for maintaining the trust of the public, ensuring the safety of participants, and legitimizing scientific findings.

Historical perspective

Ethics in research is significantly shaped by past experiences where a lack of ethical consideration led to negative consequences. One of the most striking examples of ethical misconduct is the Tuskegee Syphilis Study 3 conducted between 1932 and 1972 by the U.S. Public Health Service. In this study, African American men in Alabama were used as subjects to study the natural progression of untreated syphilis. They were not informed of their condition and were denied effective treatment, even after penicillin became available as a cure in the 1940s 3 .

From an ethical lens today, this is a gross violation of informed consent and an exploitation of a vulnerable population. The public outcry following the revelation of the study’s details led to the establishment of the National Commission for the Protection of Human Subjects of Biomedical and Behavioural Research 4 . This commission eventually produced the Belmont Report in 1979 4 , setting forth principles such as respect for persons, beneficence, and justice, which now underpin ethical research practices 4 .

Another example that significantly impacted ethical regulations was the thalidomide tragedy of the late 1950s and early 1960s 5 . Thalidomide was marketed as a safe sedative for pregnant women to combat morning sickness in Europe. Thalidomide resulted in the birth of approximately ten thousand children with severe deformities due to its teratogenic effects 5 , which were not sufficiently researched prior to the drug’s release. This incident underscored the critical need for comprehensive clinical testing and highlighted the ethical imperative of understanding and communicating potential risks, particularly for vulnerable groups such as pregnant women. In response, drug testing regulations became more rigorous, and the importance of informed consent, especially in clinical trials, was emphasized.

The Stanford Prison Experiment of 1971, led by psychologist Philip Zimbardo is another prime example of ethical oversight leading to harmful consequences 6 . The experiment, which aimed to study the psychological effects of perceived power, resulted in emotional trauma for participants. Underestimating potential psychological harm with no adequate systems to safeguard human participants from harm was a breach of ethics in psychological studies 6 . This case highlighted the necessity for ethical guidelines that prioritize the mental and emotional welfare of participants, especially in psychological research. It led to stricter review processes and the establishment of guidelines to prevent psychological harm in research studies. It influenced the American Psychological Association and other bodies to refine their ethical guidelines, ensuring the protection of participants’ mental and emotional well-being.

Impact on current ethical standards

These historical, ethical oversights have been instrumental in shaping the current landscape of ethical standards in scientific research. The Tuskegee Syphilis Study led to the Belmont Report in 1979, which laid out key ethical principles such as respect for persons, beneficence, and justice. It also prompted the establishment of Institutional Review Boards (IRBs) to oversee research involving human subjects. The thalidomide tragedy catalyzed stricter drug testing regulations and informed consent requirements for clinical trials. The Stanford Prison Experiment influenced the American Psychological Association to refine its ethical guidelines, placing greater emphasis on the welfare and rights of participants.

These historical episodes of ethical oversights have been pivotal in forging the comprehensive ethical frameworks that govern scientific research today. They serve as stark reminders of the potential consequences of ethical neglect and the perpetual need to prioritize the welfare and rights of participants in any research endeavor.

One may ponder on the reason behind the Tuskegee Syphilis Study, where African American men with syphilis were deliberately left untreated. What led scientists to prioritize research outcomes over human well-being? At the time, racial prejudices, lack of understanding of ethical principles in human research, and regulatory oversight made such studies pass. Similarly, the administration of thalidomide to pregnant women initially intended as an antiemetic to alleviate morning sickness, resulted in unforeseen and catastrophic birth defects. This tragedy highlights a critical lapse in the pre-marketing evaluation of drugs’ safety.

Furthermore, the Stanford prison experiment, designed to study the psychological effects of perceived power, spiraled into an ethical nightmare as participants suffered emotional trauma. This begs the question on how these researchers initially justified their methods. From today’s lens of ethics, the studies conducted were a complete breach of misconduct, and I wonder if there were any standards that guided primitive research in science.

Current ethical standards and guidelines in research

Informed consent.

This mandates that participants are fully informed about the nature of the research, including its objectives, procedures, potential risks, and benefits 7 , 8 . They must be given the opportunity to ask questions and must voluntarily agree to participate without coercion 7 , 8 . This ensures respect for individual autonomy and decision-making.

Confidentiality and privacy

Confidentiality is pivotal in research involving human subjects. Participants’ personal information must be protected from unauthorized access or disclosure 7 , 8 . Researchers are obliged to take measures to preserve the anonymity and privacy of participants, which fosters trust and encourages participation in research 7 , 8 .

Non-maleficence and beneficence

These principles revolve around the obligation to avoid harm (non-maleficence) and to maximize possible benefits while minimizing potential harm (beneficence) 7 , 8 . Researchers must ensure that their studies do not pose undue risks to participants and that any potential risks are outweighed by the benefits.

Justice in research ethics refers to the fair selection and treatment of research participants 8 . It ensures that the benefits and burdens of research are distributed equitably among different groups in society, preventing the exploitation of vulnerable populations 8 .

The role of Institutional Review Boards (IRB)

Institutional Review Boards play critical roles in upholding ethical standards in research. An IRB is a committee established by an institution conducting research to review, approve, and monitor research involving human subjects 7 , 8 . Their primary role is to ensure that the rights and welfare of participants are protected.

Review and approval

Before a study commences, the IRB reviews the research proposal to ensure it adheres to ethical guidelines. This includes evaluating the risks and benefits, the process of obtaining informed consent, and measures for maintaining confidentiality 7 , 8 .

Monitoring and compliance

IRB also monitors ongoing research projects to ensure compliance with ethical standards. They may require periodic reports and can conduct audits to ensure ongoing adherence to ethical principles 7 , 8 .

Handling ethical violations

In cases where ethical standards are breached, IRB has the authority to impose sanctions, which can range from requiring modifications to the study to completely halting the research project 7 , 8 .

Other agencies and boards enforcing standards

Beyond IRB, there are other regulatory bodies and agencies at national and international levels that enforce ethical standards in research. These include:

The Office for Human Research Protections (OHRP) in the United States, which oversees compliance with the Federal Policy for the Protection of Human Subjects.

The World Health Organization (WHO) , which provides international ethical guidelines for biomedical research.

The International Committee of Medical Journal Editors (ICMJE) , which sets ethical standards for the publication of biomedical research.

These organizations, along with IRB, form a comprehensive network that ensures the ethical conduct of scientific research. They safeguard the integrity of research using the reflections and lesson learnt from the past.

Benefits of ethical research

Credible and reliable outcomes, why is credibility so crucial in research, and how do ethical practices contribute to it.

Ethical practices such as rigorous peer review, transparent methodology, and adherence to established protocols ensure that research findings are reliable and valid 9 . When studies are conducted ethically, they are less likely to be marred by biases, fabrications, or errors that could compromise credibility. For instance, ethical standards demand accurate data reporting and full disclosure of any potential conflicts of interest 9 , which directly contribute to the integrity and trustworthiness of research findings.

How do ethical practices lead to socially beneficial outcomes?

Ethical research practices often align with broader societal values and needs, leading to outcomes that are not only scientifically significant but also socially beneficial. By respecting principles like justice and beneficence, researchers ensure that their work with human subjects contributes positively to society 7 , 8 . For example, ethical guidelines in medical research emphasize the need to balance scientific advancement with patient welfare, ensuring that new treatments are both effective and safe. This balance is crucial in addressing pressing societal health concerns while safeguarding individual rights and well-being.

Trust between the public and the scientific community

The relationship between the public and the scientific community is heavily reliant on trust, which is fostered through consistent ethical conduct in research. When the public perceives that researchers are committed to ethical standards, it reinforces their confidence in the scientific process and its outcomes. Ethical research practices demonstrate a respect for societal norms and values, reinforcing the perception that science serves the public good.

Case studies

Case study 1: the development and approval of covid-19 vaccines.

The development and approval of COVID-19 vaccines within a short time is a testament to how adherence to ethical research practices can achieve credible and beneficial outcomes. Strict adherence to ethical guidelines, even in the face of a global emergency, ensured that the vaccines were developed swiftly. However, safety standards were compromised to some extent as no animal trials were done before humans. The vaccine development was not transparent to the public, and this fuelled the anti-vaccination crowd in some regions. Ethical compliance, including rigorous testing and transparent reporting, should expedite scientific innovation while maintaining public trust.

Case study 2: The CRISPR babies

What ethical concerns were raised by the creation of the crispr babies, and what were the consequences.

The creation of the first genetically edited babies using CRISPR technology in China raised significant ethical concerns 10 . The lack of transparency, inadequate consent process, and potential risks to the children can be likened to ethical misconduct in genetic engineering research. This case resulted in widespread condemnation from the scientific community and the public, as well as international regulatory frameworks and guidelines for genetic editing research 10 .

Recommendation and conclusion

Continuous education and training.

The scientific community should prioritize ongoing education and training in ethics for researchers at all levels, ensuring awareness and understanding of ethical standards and their importance.

Enhanced dialogue and collaboration

Encourage multidisciplinary collaborations and dialogues between scientists, ethicists, policymakers, and the public to address emerging ethical challenges and develop adaptive guidelines.

Fostering a culture of ethical responsibility

Institutions and researchers should cultivate an environment where ethical considerations are integral to the research process, encouraging transparency, accountability, and social responsibility.

Global standards and cooperation

Work toward establishing and harmonizing international ethical standards and regulatory frameworks, particularly in areas like genetic engineering and AI, where the implications of research are global.

Ethics approval

Ethics approval was not required for this editorial.

Informed consent was not required for this editorial

Sources of funding

No funding was received for this research.

Author contribution

G.D.M. wrote this paper.

Conflicts of interest disclosure

The authors declare no conflicts of interest.

Research registration unique identifying number (UIN)

Goshen David Miteu.

Data availability statement

Provenance and peer review.

Not commissioned, externally peer-reviewed.

Sponsorships or competing interests that may be relevant to content are disclosed at the end of this article.

Published online 21 March 2024

Have a language expert improve your writing

Run a free plagiarism check in 10 minutes, generate accurate citations for free.

• Knowledge Base

Methodology

• Ethical Considerations in Research | Types & Examples

Ethical Considerations in Research | Types & Examples

Published on October 18, 2021 by Pritha Bhandari . Revised on May 9, 2024.

Ethical considerations in research are a set of principles that guide your research designs and practices. Scientists and researchers must always adhere to a certain code of conduct when collecting data from people.

The goals of human research often include understanding real-life phenomena, studying effective treatments, investigating behaviors, and improving lives in other ways. What you decide to research and how you conduct that research involve key ethical considerations.

These considerations work to

• protect the rights of research participants
• enhance research validity
• maintain scientific or academic integrity

Table of contents

Why do research ethics matter, getting ethical approval for your study, types of ethical issues, voluntary participation, informed consent, confidentiality, potential for harm, results communication, examples of ethical failures, other interesting articles, frequently asked questions about research ethics.

Research ethics matter for scientific integrity, human rights and dignity, and collaboration between science and society. These principles make sure that participation in studies is voluntary, informed, and safe for research subjects.

You’ll balance pursuing important research objectives with using ethical research methods and procedures. It’s always necessary to prevent permanent or excessive harm to participants, whether inadvertent or not.

Defying research ethics will also lower the credibility of your research because it’s hard for others to trust your data if your methods are morally questionable.

Even if a research idea is valuable to society, it doesn’t justify violating the human rights or dignity of your study participants.

Prevent plagiarism. Run a free check.

Before you start any study involving data collection with people, you’ll submit your research proposal to an institutional review board (IRB) .

An IRB is a committee that checks whether your research aims and research design are ethically acceptable and follow your institution’s code of conduct. They check that your research materials and procedures are up to code.

If successful, you’ll receive IRB approval, and you can begin collecting data according to the approved procedures. If you want to make any changes to your procedures or materials, you’ll need to submit a modification application to the IRB for approval.

If unsuccessful, you may be asked to re-submit with modifications or your research proposal may receive a rejection. To get IRB approval, it’s important to explicitly note how you’ll tackle each of the ethical issues that may arise in your study.

There are several ethical issues you should always pay attention to in your research design, and these issues can overlap with each other.

You’ll usually outline ways you’ll deal with each issue in your research proposal if you plan to collect data from participants.

Voluntary participation means that all research subjects are free to choose to participate without any pressure or coercion.

All participants are able to withdraw from, or leave, the study at any point without feeling an obligation to continue. Your participants don’t need to provide a reason for leaving the study.

It’s important to make it clear to participants that there are no negative consequences or repercussions to their refusal to participate. After all, they’re taking the time to help you in the research process , so you should respect their decisions without trying to change their minds.

Voluntary participation is an ethical principle protected by international law and many scientific codes of conduct.

Take special care to ensure there’s no pressure on participants when you’re working with vulnerable groups of people who may find it hard to stop the study even when they want to.

Informed consent refers to a situation in which all potential participants receive and understand all the information they need to decide whether they want to participate. This includes information about the study’s benefits, risks, funding, and institutional approval.

You make sure to provide all potential participants with all the relevant information about

• what the study is about
• the risks and benefits of taking part
• how long the study will take
• your supervisor’s contact information and the institution’s approval number

Usually, you’ll provide participants with a text for them to read and ask them if they have any questions. If they agree to participate, they can sign or initial the consent form. Note that this may not be sufficient for informed consent when you work with particularly vulnerable groups of people.

If you’re collecting data from people with low literacy, make sure to verbally explain the consent form to them before they agree to participate.

For participants with very limited English proficiency, you should always translate the study materials or work with an interpreter so they have all the information in their first language.

In research with children, you’ll often need informed permission for their participation from their parents or guardians. Although children cannot give informed consent, it’s best to also ask for their assent (agreement) to participate, depending on their age and maturity level.

Anonymity means that you don’t know who the participants are and you can’t link any individual participant to their data.

You can only guarantee anonymity by not collecting any personally identifying information—for example, names, phone numbers, email addresses, IP addresses, physical characteristics, photos, and videos.

In many cases, it may be impossible to truly anonymize data collection . For example, data collected in person or by phone cannot be considered fully anonymous because some personal identifiers (demographic information or phone numbers) are impossible to hide.

You’ll also need to collect some identifying information if you give your participants the option to withdraw their data at a later stage.

Data pseudonymization is an alternative method where you replace identifying information about participants with pseudonymous, or fake, identifiers. The data can still be linked to participants but it’s harder to do so because you separate personal information from the study data.

Confidentiality means that you know who the participants are, but you remove all identifying information from your report.

All participants have a right to privacy, so you should protect their personal data for as long as you store or use it. Even when you can’t collect data anonymously, you should secure confidentiality whenever you can.

Some research designs aren’t conducive to confidentiality, but it’s important to make all attempts and inform participants of the risks involved.

As a researcher, you have to consider all possible sources of harm to participants. Harm can come in many different forms.

• Psychological harm: Sensitive questions or tasks may trigger negative emotions such as shame or anxiety.
• Social harm: Participation can involve social risks, public embarrassment, or stigma.
• Physical harm: Pain or injury can result from the study procedures.
• Legal harm: Reporting sensitive data could lead to legal risks or a breach of privacy.

It’s best to consider every possible source of harm in your study as well as concrete ways to mitigate them. Involve your supervisor to discuss steps for harm reduction.

Make sure to disclose all possible risks of harm to participants before the study to get informed consent. If there is a risk of harm, prepare to provide participants with resources or counseling or medical services if needed.

Some of these questions may bring up negative emotions, so you inform participants about the sensitive nature of the survey and assure them that their responses will be confidential.

The way you communicate your research results can sometimes involve ethical issues. Good science communication is honest, reliable, and credible. It’s best to make your results as transparent as possible.

Take steps to actively avoid plagiarism and research misconduct wherever possible.

Plagiarism means submitting others’ works as your own. Although it can be unintentional, copying someone else’s work without proper credit amounts to stealing. It’s an ethical problem in research communication because you may benefit by harming other researchers.

Self-plagiarism is when you republish or re-submit parts of your own papers or reports without properly citing your original work.

This is problematic because you may benefit from presenting your ideas as new and original even though they’ve already been published elsewhere in the past. You may also be infringing on your previous publisher’s copyright, violating an ethical code, or wasting time and resources by doing so.

In extreme cases of self-plagiarism, entire datasets or papers are sometimes duplicated. These are major ethical violations because they can skew research findings if taken as original data.

You notice that two published studies have similar characteristics even though they are from different years. Their sample sizes, locations, treatments, and results are highly similar, and the studies share one author in common.

Research misconduct

Research misconduct means making up or falsifying data, manipulating data analyses, or misrepresenting results in research reports. It’s a form of academic fraud.

These actions are committed intentionally and can have serious consequences; research misconduct is not a simple mistake or a point of disagreement about data analyses.

Research misconduct is a serious ethical issue because it can undermine academic integrity and institutional credibility. It leads to a waste of funding and resources that could have been used for alternative research.

Later investigations revealed that they fabricated and manipulated their data to show a nonexistent link between vaccines and autism. Wakefield also neglected to disclose important conflicts of interest, and his medical license was taken away.

This fraudulent work sparked vaccine hesitancy among parents and caregivers. The rate of MMR vaccinations in children fell sharply, and measles outbreaks became more common due to a lack of herd immunity.

Research scandals with ethical failures are littered throughout history, but some took place not that long ago.

Some scientists in positions of power have historically mistreated or even abused research participants to investigate research problems at any cost. These participants were prisoners, under their care, or otherwise trusted them to treat them with dignity.

To demonstrate the importance of research ethics, we’ll briefly review two research studies that violated human rights in modern history.

These experiments were inhumane and resulted in trauma, permanent disabilities, or death in many cases.

After some Nazi doctors were put on trial for their crimes, the Nuremberg Code of research ethics for human experimentation was developed in 1947 to establish a new standard for human experimentation in medical research.

In reality, the actual goal was to study the effects of the disease when left untreated, and the researchers never informed participants about their diagnoses or the research aims.

Although participants experienced severe health problems, including blindness and other complications, the researchers only pretended to provide medical care.

When treatment became possible in 1943, 11 years after the study began, none of the participants were offered it, despite their health conditions and high risk of death.

Ethical failures like these resulted in severe harm to participants, wasted resources, and lower trust in science and scientists. This is why all research institutions have strict ethical guidelines for performing research.

If you want to know more about statistics , methodology , or research bias , make sure to check out some of our other articles with explanations and examples.

• Normal distribution
• Measures of central tendency
• Chi square tests
• Confidence interval
• Quartiles & Quantiles
• Cluster sampling
• Stratified sampling
• Thematic analysis
• Cohort study
• Peer review
• Ethnography

Research bias

• Implicit bias
• Cognitive bias
• Conformity bias
• Hawthorne effect
• Availability heuristic
• Attrition bias
• Social desirability bias

Ethical considerations in research are a set of principles that guide your research designs and practices. These principles include voluntary participation, informed consent, anonymity, confidentiality, potential for harm, and results communication.

Scientists and researchers must always adhere to a certain code of conduct when collecting data from others .

These considerations protect the rights of research participants, enhance research validity , and maintain scientific integrity.

Research ethics matter for scientific integrity, human rights and dignity, and collaboration between science and society. These principles make sure that participation in studies is voluntary, informed, and safe.

Anonymity means you don’t know who the participants are, while confidentiality means you know who they are but remove identifying information from your research report. Both are important ethical considerations .

You can only guarantee anonymity by not collecting any personally identifying information—for example, names, phone numbers, email addresses, IP addresses, physical characteristics, photos, or videos.

You can keep data confidential by using aggregate information in your research report, so that you only refer to groups of participants rather than individuals.

These actions are committed intentionally and can have serious consequences; research misconduct is not a simple mistake or a point of disagreement but a serious ethical failure.

Cite this Scribbr article

If you want to cite this source, you can copy and paste the citation or click the “Cite this Scribbr article” button to automatically add the citation to our free Citation Generator.

Bhandari, P. (2024, May 09). Ethical Considerations in Research | Types & Examples. Scribbr. Retrieved August 5, 2024, from https://www.scribbr.com/methodology/research-ethics/

Is this article helpful?

Pritha Bhandari

Other students also liked, data collection | definition, methods & examples, what is self-plagiarism | definition & how to avoid it, how to avoid plagiarism | tips on citing sources, what is your plagiarism score.

• Open access
• Published: 07 August 2024

Ethical considerations in public engagement: developing tools for assessing the boundaries of research and involvement

• Jaime Garcia-Iglesias 1 ,
• Iona Beange 2 ,
• Donald Davidson 2 ,
• Suzanne Goopy 3 ,
• Huayi Huang 3 ,
• Fiona Murray 4 ,
• Carol Porteous 5 ,
• Elizabeth Stevenson 6 ,
• Sinead Rhodes 7 ,
• Faye Watson 8 &
• Sue Fletcher-Watson 7  

Research Involvement and Engagement volume  10 , Article number:  83 ( 2024 ) Cite this article

450 Accesses

21 Altmetric

Metrics details

Public engagement with research (PEwR) has become increasingly integral to research practices. This paper explores the process and outcomes of a collaborative effort to address the ethical implications of PEwR activities and develop tools to navigate them within the context of a University Medical School. The activities this paper reflects on aimed to establish boundaries between research data collection and PEwR activities, support colleagues in identifying the ethical considerations relevant to their planned activities, and build confidence and capacity among staff to conduct PEwR projects. The development process involved the creation of a taxonomy outlining key terms used in PEwR work, a self-assessment tool to evaluate the need for formal ethical review, and a code of conduct for ethical PEwR. These tools were refined through iterative discussions and feedback from stakeholders, resulting in practical guidance for researchers navigating the ethical complexities of PEwR. Additionally, reflective prompts were developed to guide researchers in planning and conducting engagement activities, addressing a crucial aspect often overlooked in formal ethical review processes. The paper reflects on the broader regulatory landscape and the limitations of existing approval and governance processes, and prompts critical reflection on the compatibility of formal approval processes with the ethos of PEwR. Overall, the paper offers insights and practical guidance for researchers and institutions grappling with ethical considerations in PEwR, contributing to the ongoing conversation surrounding responsible research practices.

Plain English summary

This paper talks about making research fairer for everyone involved. Sometimes, researchers ask members of the public for advice, guidance or insight, or for help to design or do research, this is sometimes known as ‘public engagement with research’. But figuring out how to do this in a fair and respectful way can be tricky. In this paper, we discuss how we tried to make some helpful tools. These tools help researchers decide if they need to get formal permission, known as ethical approval, for their work when they are engaging with members of the public or communities. They also give tips on how to do the work in a good and fair way. We produced three main tools. One helps people understand the important words used in this kind of work (known as a taxonomy). Another tool helps researchers decide if they need to ask for special permission (a self-assessment tool). And the last tool gives guidelines on how to do the work in a respectful way (a code of conduct). These tools are meant to help researchers do their work better and treat everyone involved fairly. The paper also talks about how more work is needed in the area, but these tools are a good start to making research fairer and more respectful for everyone.

Peer Review reports

Introduction

In recent decades, “public involvement in research” has experienced significant development, becoming an essential element of the research landscape. In fact, it has been argued, public involvement may make research better and more relevant [ 7 , p. 1]. Patients’ roles, traditionally study participants, have transformed to become “active partners and co-designers” [ 17 , p. 1]. This evolution has led to the appearance of a multitude of definitions and terms to refer to these activities. In the UK, the National Co-ordinating Centre for Public Engagement, defines public engagement as the “many ways organisations seek to involve the public in their work” [ 9 ]. In this paper, we also refer to “public involvement,” which is defined as “research being carried out ‘with’ or ‘by’ members of the public rather than ‘to’, ‘about’ or ‘for’ them” (UK Standards for Public Involvement). Further to this, the Health Research Authority (also in the UK), defines public engagement with research as “all the ways in which the research community works together with people including patients, carers, advocates, service users and members of the community” [ 6 ]; [ 9 ]. These terms encompass a wide variety of theorizations, levels of engagement, and terminology, such as ‘patient-oriented research’, ‘participatory’ research or services or ‘patient engagement’ [ 17 , p. 2]. For this paper, we use the term ‘public engagement with research’ or PEwR in this way.

Institutions have been set up to support PEwR activities. In the UK these include the UK Standards for Public Involvement in Research (supported by the National Institute for Health and Care Research), INVOLVE, and the National Coordinating Centre for Public Engagement (NCCPE). Most recently, in 2023, the UK’s largest funders and healthcare bodies signed a joint statement “to improve the extent and quality of public involvement across the sector so that it is consistently excellent” [ 6 ]. In turn, this has often translated to public engagement becoming a requisite for securing research funding or institutional ethical permissions [ 3 , p. 2], as well as reporting and publishing research [ 15 ]. Despite this welcomed infrastructure to support PEwR, there remain gaps in knowledge and standards in the delivery of PEwR. One such gap concerns the extent to which PEwR should be subject to formal ethical review in the same way as data collection for research.

In 2016, the UK Health Research Authority and INVOLVE published a joint statement suggesting that “involving the public in the design and development of research does not generally raise any ethical concerns” [ 7 , p. 2]. We presume that this statement is using the phrase ‘ethical concerns’ to narrowly refer to the kinds of concerns addressed by a formal research ethics review process, such as safeguarding, withdrawal from research, etc. Footnote 1 . To such an extent, we agree that public involvement with research is not inherently ‘riskier’ than other research activities.

Furthermore, a blanket need for formal ethical review risks demoting or disempowering non-academic contributors from the roles of consultants, co-researchers, or advisors to a more passive status as participants. Attending a meeting as an expert, discussing new project ideas, setting priorities, designing studies and, or interpreting findings does not require that we sign a consent form. Indeed, to do so clearly removes the locus of power away from the person signing and into the hands of the person who wrote the consent form. This particular risk is exacerbated when institutional, formal ethical review processes operate in complex, convoluted and obscure ways that often baffle researchers let alone members of the public.

However, we also recognize that PEwR is not without potential to do harm – something which formal research ethics review aims to anticipate and minimise. For example, a public lecture or a workshop could cause distress to audience members or participants if they learn for the first time that aspects of their lifestyle or personal history put them at higher risk of dementia. When patients are invited to join advisory panels, they may feel pressure to reveal personal details about their medical history to reinforce their expertise or legitimise their presence – especially in a room where most other people have potentially intimidating professional qualifications. Some patient groups may be exploited, if research involvement roles are positioned as an opportunity, or even a duty, and not properly reimbursed. When patients are more deeply involved in research roles, such as collecting or analysing data, they might experience distress, particularly if interacting with participants triggers their own painful or emotional memories [ 14 , p. 98]. Thus, at all levels of PEwR from science communication to embedded co-production, there is a danger of harm to patients or members of the public, and a duty of care on the part of the research team and broader institution who invited them in.

These concerns are not accessory to PEwR activities but rather exist at their heart. Following a review on the impacts of public engagement, Brett et al. conclude that “developing a wide view which considers the impact of PPI [public and patient involvement] on the people involved in the process can be critical to our understanding of why some studies that involve patients and the public thrive, while others fail” [ 1 , p. 388]. Despite the importance of these considerations, there is a stark absence of consistent guidance as to whether different forms of PEwR require formal ethical review. Nor is there, to our knowledge, any sustained attempt to provide a framework for ethical conduct of PEwR in the absence of formal review (see Pandya-Wood et al. [ 11 ]; Greenhalgh et al. [ 5 ]). This is, in part, due to there being a wide heterogeneity of practices, communities, and levels of engagement [ 8 , p. 6] that resists generalizable principles or frameworks.

The lack of frameworks about whether or how PEwR requires formal ethical review can, ironically, be a key barrier to PEwR happening. In our work as members of a university ethics review committee, we have found this lack of guidance to hamper appropriate ethical PEwR in several ways. Researchers may avoid developing PEwR initiatives altogether for fear of having to spend time or resources in securing formal ethical review (especially when this process is lengthy or resource-intensive). Likewise, they may avoid PEwR for fear that its conduction would be unethical. On the other hand, others could assume that the lack of a requirement for formal ethical review means there are no ethical issues or risks involved in PEwR.

Similarly, experts in PEwR who are not experienced with formal research ethics review may face barriers as their PEwR process becomes more elaborate, in-depth, or complex. For example, although a priority-setting exercise with members of an online community of people with depression was assessed as not requiring ethics review, the funding panel requested that formal ethics review be undergone for a follow-up exercised aimed at collecting data answering one of the priority questions identified in the previous priority-setting. It is crucial that innovations in PEwR and findings from this work are shared and yet academic teams may be unable to publish their work in certain journals which require evidence of having undergone formal ethical review. Finally, ethics committees such as ours often must rely on anecdotal knowledge to make judgements about what does or does not require formal ethical review, given the absence of standardized frameworks.

About this paper

In this paper, we report and reflect on the development of specific tools and processes for assessing the ethical needs of PEwR initiatives, as members of an ethics review committee for a large University medical school. These tools aim to delineate boundaries between research data collection and PEwR activities of various kinds, provide a self-assessment framework for ethical practice in PEwR and, overall, give people greater confidence when conducting PEwR work. We describe and critically reflect on the development of the following resources:

a taxonomy to define key terms relating to PEwR with associated resource recommendations.

a self-assessment tool to support people understanding where their planned activities fall in relation to research or PEwR.

a code of conduct for ethical conduct of PEwR (appended to the self-assessment tool).

We will, first, describe our work as part of an institutional ethics committee, the identification of a need for specific guidance, and our key assumptions; we will then describe the process of developing these tools and processes; provide an overview of the tools themselves; and reflect on early feedback received and future work needed.

Developing specific tools for PEWR in ethics

Identifying needs, goals and outputs.

The Edinburgh Medical School Research Ethics Committee (EMREC) provides ethical opinions to members of staff and postgraduate researchers within the University of Edinburgh Medical School in relation to planned research to be conducted on humans i.e. their data, or tissues. These research activities come from a wide range of disciplines, including public health, epidemiology, social science or psychology. EMREC does not review research that involves recruitment of NHS patients, use of NHS data, or other health service resources: such projects are evaluated by an NHS research ethics committee. EMREC is led by two co-directors and formed of over 38 members, which include experienced academics and academic-clinicians from a variety of disciplines. There are also 2–4 lay members who are not researchers.

EMREC receives regular enquiries about whether a specific piece of PEwR work (such as holding a workshop with people living with endometriosis to identify research priorities or interviewing HIV activists about their work during COVID-19) requires formal ethics review. In addition, often teams contact EMREC following completion of a PEwR activity that they want to publish because the journal in which they wish to publish has requested evidence of the work having undergone formal ethics approval. These enquiries are happening in the context of an institutional investment in staffing, leading to a significant degree of distributed expertise across the Medical School about diverse forms of PEwR.

Responding to this, in the summer of 2022, a Public and Patient Involvement and Engagement working group was formed by EMREC with the aim of developing new tools and processes to navigate the ethical implications of PEWR within the University of Edinburgh Medical School. The group’s original understandings were that:

PEwR is both important and skilled work that presents a unique set of ethical implications,

PEwR is a fragmented landscape where many people have relevant but different expertise and where a wide range of terminology is in use, and.

there is no existing widely-agreed framework for ethical PEwR.

This working group was designed to be temporary, lasting approximately six months. It was composed of eleven members with different degrees of seniority and disciplinary backgrounds - both members of EMREC and those from other parts of the Medical School, and other parts of the University of Edinburgh. Among these, there were both academics and PEwR experts in professional services (i.e. primarily non-academic) roles. The working group met four times (August, September and November 2022; and January 2023).

The group identified three key goals and, in relation to these, key outputs needed. The goals were: (1) help establish boundaries between research data collection (requiring an ethical opinion from EMREC) and PEwR activities of various kinds (requiring ethical reflection/practice but not a formal EMREC ethical opinion), (2) support colleagues to identify where their planned activities fell in the research-PEwR continuum and consequently the relevant ethical framework, and (3) identify ways of building confidence and capacity among staff to conduct PEwR projects. In relation to these goals, the working group initially agreed on producing the following key outputs:

A taxonomy outlining and defining key terms used in the PEwR work, with examples. While not universal or definitive, the taxonomy should help colleagues identify and label their activities and help determine the ethical considerations that would apply to conduct the work with integrity. It would also facilitate conversations between staff with varying levels and types of experience, and ensure that decisions around ethical conduct would be based on more than choice of terminology.

A self-assessment tool to provide a more systematic way to evaluate whether a given academic activity, involving a non-academic partner (organisation or individual) requires formal evaluation by a research ethics committee.

A list of resources collected both from within and beyond our institution that are relevant to the issue of ethics and PEwR and can serve as ‘further reading’ and training.

While we aimed to develop this work with a view to it being useful within the remit of the University of Edinburgh Medical School, we also understood that there was significant potential for these outputs to be of interest and relevance more widely. In this way, we aimed to position them as a pragmatic addition to existing guidance and resources, such as the NIHR Reflective Questions [ 2 ].

Our process

Across the first three meetings, the group worked together on the simultaneous development of the three outputs (taxonomy, self-assessment tool, and resources). The initial taxonomy was informed by the guidance produced by the Public Involvement Resource Hub at Imperial College London [ 10 ]. The taxonomy was developed as a table that included key terms (such as ‘public engagement’, ‘co-production’, or ‘market research’), with their definitions, examples, and synonyms. From early on, it was decided that different key terms would not be defined by the methods used, as there could be significant overlap among these – e.g. something called a focus group might be a part of a consultation, market research or research data collection.

A draft table (with just six categories) was presented in the first meeting and group members were asked to work on the table between meetings, including providing additional examples, amending language, or any other suggestions. This was done on a shared document using ‘comments’ so that contradictory views could be identified and agreements reached. The table was also shared with colleagues from outside the University of Edinburgh Medical School to capture the range of terminologies used across disciplines, recognising the interdisciplinary nature of much research.

Through this process, additional key terms were identified, such as “science communication” and “action research,” definitions were developed more fully, and synonyms were sometimes contextualized (by indicating, for example, shades of difference or usages specific to an area). Upon further work, three additional sections were added to the taxonomy tool: first, an introduction was developed that explained what terminology our specific institution used and noted that the boundaries between different terms were often “fuzzy and flexible.” In addition, the group agreed that it would be useful to provide a narrative example of how different forms of public engagement with research might co-exist and flow from one to another. To this end, a fictional example was developed where a team of clinical researchers interested in diabetes are described engaging in scoping work, research, co-production, science communication and action research at different times of their research programme. Finally, a section was also added that prompted researchers to reflect on the processes of negotiating how partners can be described in research (for example, whether to use terms such as ‘patient’ or ‘lay member’).

For the self-assessment tool, a first iteration was a table with two columns (one for research or work requiring formal ethical review and one for PEwR or work not requiring formal ethical review). The aim was for group members to fill the table with examples of activities that would fall under each category, with a view to identifying generalizable characteristics. However, this task proved complicated given the wide diversity of possible activities, multitude of contexts, and sheer number of exceptions. To address this, group members were asked to complete a case-based exercise. They were presented with the following situation: “I tell you I’m planning a focus group with some autistic folk” and asked how they would determine whether the activity would be a form of data collection for a research project (requiring formal ethical review) or another form of PEwR. Group members were asked, with a view to developing the self-assessment tool, to identify which questions they would ask to assess the activity. The replies of working group members were synthesized by one of the authors (SFW) and presented at the following meeting.

Through discussion as a group, we determined that the questions identified as useful in identifying if an activity required formal ethical review fell, roughly, under four main areas. Under each area, some indicators of activities were provided which were “less likely to need ethics review” and some “more likely to need ethics review”. The four umbrella questions were:

What is the purpose and the planned outcome of the activity? (see Table  1 for an excerpt of the initial draft answer to this question)

What is the status of the people involved in the activity? (indicators of less likely to need ethics review were “participants will be equal partners with academic team” or “participants will be advisors” and indicators more likely to require ethics approval were “participants will undertake tasks determined by academics” or “participants will contribute data or sign consent forms”).

What kind of information is being collected? (indicators of less likely to need ethics review were “asking about expert opinion on a topic” or “sessions will be minuted and notes taken” and indicators more likely to require ethics approval were “sessions will be recorded and transcribed” or “asking about participants’ personal experiences”).

What are the risks inherent in this activity? (indicators of less likely to need ethics review were “participants will be involved in decision-making” or “participants will be credited for their role in a manner of their choosing” and indicators more likely to require ethics approval were “participants’ involvement will and must be anonymized fully” or “participants have a choice between following protocol or withdrawing from the study”).

Upon further work, the group decided to modify this initial iteration in several ways leading to the final version. First, a brief introduction explaining the purpose of the tool was written. This included information about the aims of the tool, and a very brief overview of the process of formal research ethics review. It also emphasised the importance of discussion of the tool within the team, with PEwR experts and sometimes with EMREC members, depending on how clear-cut the outcome was. Second, we included brief information about what are ‘research’ and ‘public engagement with research’ with a view to supporting people who may not be familiar with how these concepts are used by ethics review committees (for example, lay co-applicants or co-researchers). Third, we included key guidance about how to use the tool, including ‘next steps’ if the activity was determined to be research or engagement. Importantly, this emphasised that none of the questions posed and indicators given were definitive of something needing or not needing formal research ethics review, but instead they should be used collectively to signpost a team towards, or away from, formal review.

Finally, while the four umbrella questions remained the same as in the previous iteration, the indicators under each were further refined. In discussing the previous version, the group agreed that, while some indicators could relate to an activity falling into either category (research or engagement) depending on other factors, there were others that were much more likely to fall under one category than the other. In other words, while no single indicator was deterministic of needing or not needing formal review, some indicators were more influential than others on the final self-assessment outcome. Thus, we divided the indicators associated with each umbrella question into two sub-groups. The more influential indicators were labelled as either “probably doesn’t need ethical review” or “almost certainly needs ethical review”. Less influential indicators were labelled as either “less likely to need ethical review” or “more likely to need ethical review.” This is shown in Table  2 .

This new format retains the awareness of the sometimes-blurry lines between research and PEwR for many activities, but also seeks to provide stronger direction through indicative activities that are more clear-cut, with a particular view to supporting early-career researchers and people new to ethics reviews and/or engagement processes.

A key concern of the group was what would happen next if a planned activity, using the self-assessment tool, was deemed as PEwR. The formal review process for research would not be available for a planned activity identified as PEwR i.e. completing a series of documents and a number of protocols to deal with issues such as data protection, safeguarding, etc. This would leave a vacuum in terms of guidance for ethical conduction of PEwR. The group was concerned that some people using the self-assessment tool might arrive at the conclusion that their planned activity was entirely without ethical risks, given that it was not required to undergo formal review. Others might be conscious of the risks but feel adrift as to how to proceed. This was a particular concern with early-career researchers and indeed established academics turning to PEwR for the first time: we wanted to facilitate their involvement with PEwR but we were also aware that many may lack experience and resources. To address this, the group decided to develop an additional output comprising a series of reflective prompts to guide researchers in planning and conducting engagement activities.

The prompts were organized under four headings. First, “Data Minimisation and Security” included information about required compliance with data protection legislation, suggestions about collecting and processing information, and ideas around ensuring confidentiality. Second, “Safeguarding Collaborators and Emotional Labour” prompted researchers to think about the risk of partners becoming distressed and suggested what things should be planned for in this regard. Third, “Professional Conduct and Intellectual Property” included advice on how to clearly manage partners’ expectations around their contributions, impact, and intellectual property. Finally, fourth, under “Power Imbalances”, the guidance discusses how researchers may work to address the inherent imbalances that exist in relationships with partners. It prompts the researcher to think about choice of location, information sharing, and authorship among others. While the Edinburgh Medical School Research Ethics Committee remains available for consultation on all these matters, as well as dedicated and professional PEwR staff, the group developed these guidelines with a view both to emphasizing the fact that an activity not requiring formal ethical review did not mean that the activity was absent of risk or did not require careful ethical planning; and to support those who may be unfamiliar with how to develop engagement activities. It was decided that this guideline should follow the self-assessment tool for clarity.

Finally, in the process of developing these outputs (the planned taxonomy and assessment tool, and the additional reflective prompts appended to the assessment tool), the group collected a large number of resources, including academic papers (e.g. Staniszewska et al. [ 16 ]; Schroeder et al. [ 13 ]; Redman et al. [ 12 ]; Fletcher-Watson et al. [ 4 ]), guidance produced by other institutions, and key online sites with information about national frameworks or policy. Among these, key resources were selected and appended to the taxonomy document. The final version of these documents can be found as appendices (Supplementary Material  1 : Assessment tool and reflective prompts; Supplementary Material  2 : Taxonomy and resources).

Further considerations and early results

The guidance and tools presented here are designed to clarify a boundary between research and engagement that is poorly defined and could cause harm if not well understood. In sharing them, we aim to facilitate researchers’ engagement with PEwR by providing familiarity with the terminology and approaches, examples, and suggesting key considerations. Most importantly, they support researchers to determine whether their planned activity should undergo a formal ethical review process or not – and if not, guides them towards ethical conduct in the absence of formal review. Reflecting on the process much of what we have explained essentially reflects a distinction between PEwR and research data collection that can be encapsulated within the idea of ‘locus of control’: namely that during PEwR the locus of control, as far as possible, sits with the engaged communities or members.

It should be noted, however, that researchers and these guidance and tools exist within a larger landscape, with added regulatory processes. Thus, researchers may need (regardless of whether their planned activity is research or engagement) to navigate additional compliance such as data protection or information security protocols and / or to consider reputational risk associated with certain topics. We are aware that the overlap of complex and sometimes obscure regulatory demands complicates the task of conducting both research and PEwR, as it requires researchers to juggle multiple procedures, documents, and approvals. This publication does not resolve all the questions that exist, but it does attempt to take a bold step towards confronting grey areas and providing systematic processes to navigate them.

The outputs described above were made available on the University of Edinburgh Medical School Research Ethics Committee intranet site under the heading “Public Engagement with Research.” While we do not collect statistics on the number of times the resources have been used, the committee has received positive feedback from people who have engaged with the documents. For example, one researcher commented that, in the process of developing an engagement activity, they had been “grappling with precisely these questions (of whether this qualifies as research, and whether it requires ethical review)” and that the documents were “quite timely and helpful. It allows me to think about these considerations in a systematic manner and it’s handy for me to send on to others as a framework for discussion should we have differing opinions.” It was this mention to the possibility of these documents being used as a framework for discussion that prompted us to write this paper as a way of sharing them beyond the University of Edinburgh College of Medicine and Veterinary Medicine (where they are already used for training early-career researchers and in the MSc in Science Communication and Public Engagement). While we think they can be useful, we also encourage potential users to adapt them to their specific contexts, with different institutions potentially establishing differing procedures or requirements. To that end, we have shared in this paper the process of writing these documents so that other people and teams may also think through them productively and creatively.

Final reflections

In developing these documents, we sought to answer a need among members of our immediate community, seeking to better assess whether an activity required formal ethical review and wanting guidance to ethically conduct PEwR work. However, we also came to realize the limitations of existing approval and governance processes. In our case, a key reason why these documents were developed is because existing formal ethical review processes would not be adequate to capture the particularities and complexities of PEwR in our large, diverse Medical School.

Looking back at the tools we developed and the feedback received, we are also satisfied with the pragmatic approach we took. There is a vast amount of resources and literature available about how to conduct PEwR, as well as a multitude of accounts and reflections both of an anecdotal and epistemological nature. Building on this conceptual work and associated principles, we sought to develop pragmatic, clear, applicable tools, without overwhelming users with a multitude of available resources and complex theory. This is, we feel, particularly applicable to contexts like ours: a large, very diverse medical school which encompasses biomedical to social science disciplines where researchers and funders have vastly differing expectations and knowledge of PEwR.

This process also led us to reflect on the practical functions of formal ethical review. Formal ethics approval provides applicants with structured resources to think and plan about their work, feedback and guidance about their plans, and—most commonly—a code and letter than can be used to easily report to journals that your research has met a specific ethical threshold. With these documents we have sought to provide some similar, pragmatic guidance to support and empower people, through a self-assessment process. This begs the question, what, if any, formal approval processes should be developed for PEwR? Are such formal processes in any way adequate to the ethos of PEwR? Would formal independent review necessarily conflict with the values of PEwR, namely the empowerment of community members as decision-makers and experts? Thus, these documents and this paper contribute to an ongoing conversation as PEwR continues to develop in frequency and sophistication in health and social care research.

Data availability

No datasets were generated or analysed during the current study.

The difference between research and public engagement is a complex one. Formal ethics approval, which is often seen as a regulatory or compliance mechanism, may not always be a good marker of this boundary, as it may ignore complex issues such as the distribution of power, the ethos of the activities, or their aims. Furthermore, different institutions use different criteria to determine what activities require ethics approval or are considered research. In this paper we reflect on the process of developing tools which we intended as pragmatic interventions that would support researchers, especially those without previous experience of PEwR to label their planned activities and understand their implications. Thus, we employ—even if not at all times comfortably—the framework that equates research with activities requiring ethics approval and PEwR with activities not requiring ethics approval.

Brett J, Staniszewska S, Mockford C, Herron-Marx S, Hughes J, Tysall C, Suleman R. A systematic review of the impact of patient and public involvement on service users, researchers and communities. Patient. 2014;7(4):387–95. https://doi.org/10.1007/s40271-014-0065-0 .

Article   PubMed   Google Scholar  

Davies R, Andrews H, Farr M, Davies P, Brangan E, and D Bagnall. Reflective questions to support co-produced research (version 1.2). University of Bristol and University of West of England. National Institute for Health Research (NIHR) ARC West and People in Health West of England; 2020.

De Simoni A, Jackson T, Inglis Humphrey W, Preston J, Mah H, Wood HE, Kinley E, Gonzalez L, Rienda, Porteous C. Patient and public involvement in research: the need for budgeting PPI staff costs in funding applications. Res Involv Engagem. 2023;9(1):16. https://doi.org/10.1186/s40900-023-00424-7 .

Article   PubMed   PubMed Central   Google Scholar  

Fletcher-Watson S, Brook K, Hallett S, Murray F, Catherine J, Crompton. Inclusive practices for Neurodevelopmental Research. Curr Dev Disorders Rep. 2021;8:88–97. https://doi.org/10.1007/s40474-021-00227-z .

Article   Google Scholar  

Greenhalgh T, Hinton L, Finlay T, Macfarlane A, Fahy N, Clyde B, Chant A. Frameworks for supporting patient and public involvement in research: systematic review and co-design pilot. Health Expect. 2019;22(4):785–801. https://doi.org/10.1111/hex.12888 .

Health Research Authority. 2023. Putting people first - embedding public involvement in health and social care research. https://www.hra.nhs.uk/planning-and-improving-research/best-practice/public-involvement/putting-people-first-embedding-public-involvement-health-and-social-care-research/ .

Health Research Authority / INVOLVE. 2016. Public involvement in research and research ethics committee review. https://www.invo.org.uk/wp-content/uploads/2016/05/HRA-INVOLVE-updated-statement-2016.pdf .

Institute for Community Studies, and UK Research and Innovation. An equitable future for research and innovation. The Young Foundation; 2022.

National Co-ordinating Centre for Public Engagement. n.d. Introducing Public Engagement. https://www.publicengagement.ac.uk/introducing-public-engagement .

NIHR Imperial BRC Patient Experience Research Centre (PERC). A rough guide to public involvement (version 1.4). London: Imperial College London; 2021.

Google Scholar  

Pandya-Wood R, Barron DS, Elliott J. A framework for public involvement at the design stage of NHS health and social care research: time to develop ethically conscious standards. Res Involv Engagem. 2017;3:6. https://doi.org/10.1186/s40900-017-0058-y .

Redman S, Greenhalgh T, Adedokun L, Staniszewska S, Denegri S, Committee Co-production of Knowledge Collection Steering. Co-production of knowledge: the future. BMJ. 2021;372:n434. https://doi.org/10.1136/bmj.n434 .

Article   CAS   PubMed   PubMed Central   Google Scholar  

Schroeder D, Chatfield K, Singh M, Chennells R. Peter Herissone-Kelly, and SpringerLink. 2019. Equitable Research Partnerships: A Global Code of Conduct to Counter Ethics Dumping . 1st 2019. ed, Springer Briefs in Research and Innovation Governance . Cham: Springer.

Staley K, Virginia M. User involvement leads to more ethically sound research. Clin Ethics. 2016;1(2):95–100. https://doi.org/10.1258/147775006777254489 .

Staniszewska S, Brett J, Simera I, Seers K, Mockford C, Goodlad S, Altman DG, Moher D, Barber R, Denegri S, Entwistle A, Littlejohns P, Morris C, Suleman R, Thomas V, Tysall C. GRIPP2 reporting checklists: tools to improve reporting of patient and public involvement in research. BMJ. 2017;358:j3453. https://doi.org/10.1136/bmj.j3453 .

Staniszewska S, Hickey G, Coutts P, Thurman B, Coldham T. Co-production: a kind revolution. Res Involv Engagem. 2022;8(1):4. https://doi.org/10.1186/s40900-022-00340-2 .

Tscherning SC, Bekker HL, Vedelo TW, Finderup J, Rodkjaer LO. How to engage patient partners in health service research: a scoping review protocol. Res Involv Engagem. 2021;7(1):20. https://doi.org/10.1186/s40900-021-00268-z .

UK Standards for Public Involvement. UK Standards for Public Involvement. https://sites.google.com/nihr.ac.uk/pi-standards/home .

Download references

Acknowledgements

For the purposes of open access, the author has applied a CC BY public copyright license to any Author Accepted Manuscript version arising from this submission.

This research was supported, in part, by the Economic and Social Research Council [ES/X003604/1].

Author information

Authors and affiliations.

Centre for Biomedicine, Self and Society, Usher Institute, University of Edinburgh, Edinburgh, UK

Jaime Garcia-Iglesias

Centre for Inflammation Research, Institute for Regeneration and Repair, University of Edinburgh, Edinburgh, UK

Iona Beange & Donald Davidson

Usher Institute, University of Edinburgh, Edinburgh, UK

Suzanne Goopy & Huayi Huang

School of Health in Social Science, Medical School, University of Edinburgh, Edinburgh, UK

Fiona Murray

Edinburgh Clinical Research Facility, University of Edinburgh, Edinburgh, UK

Carol Porteous

Biomedical Teaching Organisation, University of Edinburgh, Edinburgh, UK

Elizabeth Stevenson

Centre for Clinical Brain Sciences, University of Edinburgh, Edinburgh, UK

Sinead Rhodes & Sue Fletcher-Watson

College for Medicine and Veterinary Medicine, University of Edinburgh, Edinburgh, UK

Faye Watson

You can also search for this author in PubMed   Google Scholar

Contributions

JGI: conceptualisation and writing (original draft); IB, DJD, SG, HH, FM, CP, ES, SR, FW: conceptualisation, writing (editing and reviewing); SFW: conceptualisation and writing (original draft).

Corresponding author

Correspondence to Jaime Garcia-Iglesias .

Ethics declarations

Ethics approval and consent to participate.

Not applicable.

Consent for publication

Competing interests.

The authors declare no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary Material 1

Supplementary material 2, rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Cite this article.

Garcia-Iglesias, J., Beange, I., Davidson, D. et al. Ethical considerations in public engagement: developing tools for assessing the boundaries of research and involvement. Res Involv Engagem 10 , 83 (2024). https://doi.org/10.1186/s40900-024-00617-8

Download citation

Received : 03 May 2024

Accepted : 22 July 2024

Published : 07 August 2024

DOI : https://doi.org/10.1186/s40900-024-00617-8

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Public engagement
• Ethical approval
• Ethical review
• Responsibility
• Managing risks
• Patient-oriented research
• Patient-centred research
• And patient engagement

Research Involvement and Engagement

ISSN: 2056-7529

• General enquiries: [email protected]

Privacy and Ethics Concerns Using UX Research Platforms

New Citation Alert added!

This alert has been successfully added and will be sent to:

You will be notified whenever a record that you have chosen has been cited.

To manage your alert preferences, click on the button below.

New Citation Alert!

Please log in to your account

Information & Contributors

Bibliometrics & citations, index terms.

Human-centered computing

Human computer interaction (HCI)

Security and privacy

Formal methods and theory of security

Trust frameworks

Human and societal aspects of security and privacy

Security services

Privacy-preserving protocols

Systems security

Operating systems security

Trusted computing

Social and professional topics

Computing / technology policy

Professional topics

Computing profession

Codes of ethics

Recommendations

Three research essays on examining online privacy concerns: the role of personal dispositions, context, and privacy-assurance features, ethics, law and privacy: disentangling law from ethics in privacy discourse.

Numerous science, technology and engineering developments are perceived as raising privacy concerns. As such, privacy repeatedly finds itself addressed through the mixed lens of an 'ethical-legal' (if not 'ethical-legal-social') perspective. The aim of ...

IPA in UX Research: Interpretative Phenomenological Analysis in a User Experience Design Practice

One approach to user experience (UX) is phenomenology, but there are no well-defined methods for how to conduct UX research using phenomenology, especially not in a professional design practice. One well-defined approach developed in psychology is ...

Information

Published in.

Association for Computing Machinery, New York, NY

Association for Computing Machinery

New York, NY, United States

Publication History

Check for updates, funding sources.

• Knight Foundation

Contributors

Other metrics, bibliometrics, article metrics.

• 0 Total Citations
• 772 Total Downloads
• Downloads (Last 12 months) 772
• Downloads (Last 6 weeks) 9

View Options

Login options.

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

View options.

View or Download as a PDF file.

View online with eReader .

Digital Edition

View this article in digital edition.

Magazine Site

View this article on the magazine site (external)

Share this Publication link

Copying failed.

Share on social media

Affiliations, export citations.

• Please download or close your previous search result export first before starting a new bulk export. Preview is not available. By clicking download, a status dialog will open to start the export process. The process may take a few minutes but once it finishes a file will be downloadable from your browser. You may continue to browse the DL while the export process is in progress. Download
• Download citation
• Copy citation

We are preparing your search results for download ...

We will inform you here when the file is ready.

Your file of search results citations is now ready.

Your search export query has expired. Please try again.

Log in using your username and password

• Search More Search for this keyword Advanced search
• Latest content
• Current issue
• JME Commentaries
• BMJ Journals

You are here

• Online First
• AI and the falling sky: interrogating X-Risk
• Article Text
• Article info
• Citation Tools
• Rapid Responses
• Article metrics

• http://orcid.org/0000-0002-5642-748X Nancy S Jecker 1 , 2 ,
• http://orcid.org/0000-0001-6825-6917 Caesar Alimsinya Atuire 3 , 4 ,
• http://orcid.org/0000-0002-8965-8153 Jean-Christophe Bélisle-Pipon 5 ,
• http://orcid.org/0000-0002-7080-8801 Vardit Ravitsky 6 , 7 ,
• http://orcid.org/0000-0002-9797-1326 Anita Ho 8 , 9
• 1 Department of Bioethics & Humanities , University of Washington School of Medicine , Seattle , Washington , USA
• 2 African Centre for Epistemology and Philosophy of Science , University of Johannesburg , Auckland Park , Gauteng , South Africa
• 3 Centre for Tropical Medicine and Global Health , Oxford University , Oxford , UK
• 4 Department of Philosophy and Classics , University of Ghana , Legon , Greater Accra , Ghana
• 5 Faculty of Health Sciences , Simon Fraser University , Burnaby , British Columbia , Canada
• 6 Hastings Center , Garrison , New York , USA
• 7 Department of Global Health and Social Medicine , Harvard University , Cambridge , Massachusetts , USA
• 8 Bioethics Program , University of California San Francisco , San Francisco , California , USA
• 9 Centre for Applied Ethics , The University of British Columbia , Vancouver , British Columbia , Canada
• Correspondence to Dr Nancy S Jecker, Department of Bioethics & Humanities, University of Washington School of Medicine, Seattle, Washington, USA; nsjecker{at}uw.edu

https://doi.org/10.1136/jme-2023-109702

Statistics from Altmetric.com

Request permissions.

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

• Information Technology
• Cultural Diversity
• Minority Groups
• Resource Allocation

Introduction

The Buddhist Jātaka tells the tale of a hare lounging under a palm tree who becomes convinced the Earth is coming to an end when a ripe bael fruit falls on its head. Soon all the hares are running; other animals join them, forming a stampede of deer, boar, elk, buffalo, wild oxen, rhinoceros, tigers and elephants, loudly proclaiming the earth is ending. 1 In the American retelling, the hare is ‘chicken little,’ and the exaggerated fear is that the sky is falling.

This paper offers a critical appraisal of the rise of calamity thinking in the scholarly AI ethics literature. It cautions against viewing X-Risk in isolation and highlights ethical considerations sidelined when X-Risk takes centre stage. Section I introduces a working definition of X-Risk, considers its likelihood and explores possible subtexts. It highlights conflicts of interest that arise when tech luminaries lead ethics debates in the public square. Section II flags ethics concerns brushed aside by focusing on X-Risk, including AI existential benefits (X-Benefits), non-AI X-Risk and non-existential AI harms. As we transition towards more AI-centred societies, which we, the authors, would like to fair, we argue for embedding fairness in the transition process by ensuring groups historically disadvantaged or marginalised are not left behind. Section III concludes by proposing a wide-angle lens that takes X-Risk seriously alongside other urgent ethics concerns.

I. Unpacking X-Risk

Doomsayers imagine AI in frightening ways, a paperclip maximiser, ‘whose top goal is the manufacturing of paperclips, with the consequence that it starts transforming first all of earth and increasing portions of space into paperclip manufacturing facilities.’(Bostrom, p5) 6 They compare large language models (LLMs) to the shoggoth in Lovecraft’s novella, ‘a terrible, indescribable thing…a shapeless congeries of protoplasmic bubbles, … with myriads of temporary eyes…as pustules of greenish light all over…’. 7

Prophesies of annihilation have a runaway effect on the public’s imagination. Schwarzenegger, star of The Terminator , a film depicting a computer defence system that achieves self-awareness and initiates nuclear war, has stated that the film’s subject is ‘not any more fantasy or kind of futuristic. It is here today’ and ‘everyone is frightened’. 8 Public attention to X-Risk intensified in 2023, when The Future of Life Institute called on AI labs to pause for 6 months the training of AI systems more powerful than Generative Pre-Trained Transformer (GPT)-4, 9 and, with the Centre for AI Safety, spearheaded a Statement on AI Risk, signed by leaders from OpenAI, Google Deepmind, Anthropic and others stressing that, ‘(m)itigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war’. 10 The 2023 release of Nolan’s film, Oppenheimer, encouraged comparisons between AI and atomic weaponry. Just as Oppenheimer fretted unleashing atomic energy ‘altered abruptly and profoundly the nature of the world,’ and ‘might someday prove deadly to the whole civilisation’, tech leaders fret AI X-Risk.(Bird, p323) 11

The concept of ‘X-Risk’ traces to Bostrom, who in 2002 defined it as a risk involving, ‘an adverse outcome (that) would either annihilate Earth-originating intelligent life or permanently and drastically curtail its potential;’ on this rendering, X-Risk imperils ‘humankind as a whole’ and brings ‘major adverse consequences for the course of human civilisation for all time to come.’(Bostrom, p2) 12 More recently, Bostrom and Ćirković defined ‘X-Risk’ as a subset of global catastrophic risks that ‘threatens to cause the extinction of Earth-originating intelligent life or to reduce its quality of life (compared with what would otherwise have been possible) permanently and drastically.’(Bostrom, p4) 13 They classify global catastrophic risks that could become existential in scope, intensity and probability as threefold: risks from nature such as asteroid threats; risks from unintended consequences, such as pandemic diseases; and risks from hostile acts, such as nuclear weaponry. We use Bostrom and Ćirković’s account as our working definition of X-Risk. While it is vague in the sense of leaving open the thresholds for scope, intensity and probability, it carries the advantage of breadth and relevance to a range of serious threats.

Who says the sky is falling?

A prominent source of apocalyptic thinking regarding AI comes from within the tech industry. According to a New York Times analysis, many tech leaders believe that AI advancement is inevitable, because it is possible, and think those at the forefront of creating it know best how to shape it. 14 In a 2019 scoping review of global AI ethics guidelines, Jobin et al identified 84 documents containing AI ethics principles or guidance, with most from the tech industry.(Jobin, p396) 15 However, a limitation of the study was that ethics guidance documents represent ‘soft law,’ which is not indexed in conventional databases, making retrieval less replicable and unbiased. More recently, Stanford University’s 2023 annual AI Index Report examined authorship of scholarly AI ethics literature and reported a shift away from academic authors towards authors with industry affiliations; the Report showed industry-affiliated authors produced 71% more publications than academics year over year between 2014 and 2022. 16

Since AI companies benefit financially from their investments in AI, relying on them for ethics guidance creates a conflict of interest. A ‘conflict of interest’ is a situation where ‘an individual’s judgement concerning a primary interest tends to be unduly influenced (or biased) by a secondary interest.’(Resnik, p121–22) 17 In addition to financial conflicts of interest, non-financial conflicts of interest can arise from multiple sources (eg, personal or professional relationships, political activity, involvement in litigation). 17 Non-financial conflicts of interest can occur subconsciously, and implicit cognitive biases can transfer to AI systems. Since most powerful tech companies are situated in high-income Western countries, they may be implicitly partial to values and concerns prevalent in those societies, reflecting anchoring bias (believing what one wants or expects) and confirmation bias (clinging to beliefs despite conflicting evidence). The dearth of research exploring AI’s social impacts in diverse cultural settings around the world makes detecting and dislodging implicit bias difficult. 18 Commenting on the existing corpus of AI ethics guidance, Jobin et al noted a significant representation of more economically developed countries, with the USA and UK together accounting for more than a third of AI ethics principles in 2019, followed by Japan, Germany, France and Finland. Notably, African and South American countries were not represented. While authors of AI ethics guidance often purport to represent the common good, a 2022 study by Bélisle-Pipon et al showed a broad trend towards asymmetrical engagement, with industry and those with vested interests in AI more represented than the public. 19 Hagerty and Rubinov report that risks for discriminatory outcomes in machine learning are particularly high for countries outside the USA and Western Europe, especially when algorithms developed in higher-income countries are deployed in low-income and middle-income countries that have different resource and social realities. 18

Another prominent source of calamity thinking is members of the effective altruism movement and the associated cause of longtermism, two groups that focus on ‘the most extreme catastrophic risks and emphasise the far-future consequences of our actions’. 20 Effective altruism is associated with a philosophical and social movement based largely at Oxford University and Silicon Valley. Its members include philosophers like Singer, Ord and MacAskill, along with tech industry leaders like the discredited cryptocurrency founder, Bankman-Fried. The guiding principles of effective altruism are ‘to do as much good as we can’ and ‘to base our actions on the best available evidence and reasoning about how the world works’. 21 MacAskill defines longtermism as ‘the idea that positively influencing the long-term future is a key moral priority of our time’, and underscores, ‘Future people count. There could be a lot of them. We can make their lives go better.’(MakAskill, pp5, 21) 22 Effective altruism and longtermism have spawned charitable organisations dedicated to promoting its goals, including GiveWell, Open Philanthropy and The Future of Life Institute. To be clear, we are not suggesting that adherents of longtermism are logically forced to embrace X-Risk or calamity thinking; our point is that adherents of longtermism draw on it to justify catastrophising.

Who benefits and who is placed at risk?

Critics of longtermism argue that it fails to give sufficient attention to serious problems happening now, particularly problems affecting those who have been historically disadvantaged or marginalised. Worse, it can give warrant to sacrificing present people’s rights and interests to stave off a prophesied extinction event. Thus, a well-recognised danger of maximisation theories is that they can be used to justify unethical means if these are deemed necessary to realise faraway goals that are thought to serve a greater good. Some effective altruists acknowledge this concern. MacAskill, for example, concedes that longtermism endorses directing resources away from present concerns, such as responding to the plight of the global poor, and towards more distant goals of preventing X-Risk. 23

X-Risk also raises theoretical challenges related to intergenerational justice. How should we understand duties to future people? Can we reasonably argue that it is unfair to prioritise the interests of existing people? Or even that in doing so, we discriminate against future people? Ord defends longtermism on the ground that there are many more future people than present people: ‘When I think of the millions of future generations yet to come, the importance of protecting humanity’s future is clear to me. To risk destroying this future, for the sake of some advantage limited only to the present, seems to me profoundly parochial and dangerously short-sighted. Such neglect privileges a tiny sliver of our story over the grand sweep of the whole; it privileges a tiny minority of humans over the overwhelming majority yet to be born; it privileges this particular century over the millions, or maybe billions, yet to come' (Ord, p44). 24

MacAskill defends longtermism on slightly different grounds, arguing that it reflects the standpoint of all humanity: ‘Imagine living…through the life of every human being who has ever lived…(and) imagine that you live all future lives…If you knew you were going to live all these future lives, what would you hope we do in the present?’(MakAskill, p5) 22 For MacAskill, the standpoint of all humanity represents the moral point of view.

The logic of longtermism can be challenged on multiple grounds. First, by purporting to represent everyone, longtermism ignores its own positionality. Longtermism’s central spokespersons—from the tech industry and effective altruism movement, are not sufficiently diverse to represent ‘all humanity.’ A 2022 Time Magazine article characterised ‘the typical effective altruist’ as ‘a white man in his 20 s, who lives in North America or Europe, and has a university degree’. 25 The tech industry, which provides robust financial backing for longtermism, faces its own diversity crisis across race and gender lines. In 2021, men represented nearly three-quarters of the USA science, technology, engineering and mathematic workforce, whites close to two-thirds. 26 At higher ranks, diversity rates were lower.

Someone might push back, asking why the narrow demographics of the average effective altruist or adherent of longtermism should be a source for concern. One reply is that these demographics raise the worry that the tech industry is unwittingly entrenching its own biases and transferring them to AI systems. Experts caution about AI ‘systems that sanctify the status quo and advance the interests of the powerful’, and urge reflection on the question, ‘How is AI shifting power?’(Kalluri, p169) 27 While effective altruism purports to consider all people’s interests impartially, linking altruism to distant future threats delegitimises attention to present problems, leaving intact the plight of today’s disadvantaged. Srinivasan asserts that ‘the humanitarian logic of effective altruism leads to the conclusion that more money needs to be spent on computers: why invest in anti-malarial nets when there’s a robot apocalypse to halt?’ 28 These kinds of considerations lead Srinivasan to conclude that effective altruism is a conservative movement that leaves everything just as it is.

A second, related worry concerns epistemic justice, the normative requirement to be fair and inclusive in producing knowledge and assigning credibility to beliefs. The utilitarian philosophy embedded in effective altruism and longtermism is a characteristically Western view. Since effective altruism and longtermism aspire to be a universal ethic for humankind, considering moral philosophies outside the West is a normative requirement epistemic justice sets. Many traditions outside the West assign core importance to the fact that each of us is ‘embedded in the complex structure of commitments, affinities and understandings that comprise social life’. 28 The value of these relationships is not derivative of utilitarian principles; it is the starting point for moral reasoning. On these analyses, the utilitarian premises of longtermism and effective altruism undervalue community and thereby demand the wrong things. If the moral goal is creating the most good you can, this potentially leaves out those collectivist-oriented societies that equate ‘good’ with helping one’s community and with promoting solidaristic feeling between family, friends and neighbours.

Third, evidence suggests that epistemically just applications of AI require knowledge of the social contexts to which AI is applied. Hagerty and Rubinov report that ‘AI is likely to have markedly different social impacts depending on geographical setting’ and that ‘perceptions and understandings of AI are likely to be profoundly shaped by local cultural and social context’. 18 Lacking contextual knowledge impacts AI’s potential benefits 29 and can harm people. 30 While many variables are relevant to social context, when AI developers are predominantly white, male and from the West, they may miss insights that a more diverse demographic would be less apt to miss. This can create an echo chamber, with dominant views seeming ‘natural’ because they are pervasive and unchallenged.

An adherent of longtermism might reply to these points by saying that most people are deficient in their concern for future people. According to Perrsron and Savulescu, interventions like biomedical moral enhancement might one day enable individuals to be ‘less biased towards what is near in time and place’ and to ‘feel more responsible for what they collectively cause and let happen’.(Perrsron and Savulescu, p496) 31 Presumably, morally enhancing people in ways that direct them to care more about distant future people would help efforts to reduce X-Risk. Yet, setting aside biomedical feasibility, this argument brushes aside preliminary questions. Whose moral views require enhancing? Perrson and Savulescu suggest that their own emphasis on distant future people is superior, while the views of others, who prioritise present people, require enhancing. Yet, this stance is incendiary and potentially offensive. Implementing biomedical moral enhancement would not show the superiority of longtermism; it would shut down alternative views and homogenise moral thinking.

A different reply is suggested by MacAskill, who compares longtermism to the work of abolitionists and feminists.(MakAskill, p3) 22 MacAskill says future people will look back and thank us if we pursue the approach longtermism advocates, just as present people are grateful to abolitionists and feminists who dedicated themselves to missions that succeeded decades after their deaths. Yet this ignores the thorny question of timing—feminists and abolitionists responded to justice concerns of their time and place, and helped the next generation of women and blacks, while longtermists presumably help people in the distant future to avoid the end of humanity. Yet, those who never exist (because they are eliminated by AI) are not wronged by never having existed.

Finally, proponents of X-Risk might reason that even though the odds of X-Risk are uncertain, the potential hazard it poses is grave. Yet, what exactly are the odds? Bostrom and Ćirković acknowledge AI X-Risk is ‘not an ongoing or imminent global catastrophic risk;’ nonetheless, ‘from a long-term perspective, the development of general AI exceeding that of the human brain can be seen as one of the main challenges to the future of humanity (arguably, even as the main challenge).’(Rees, p16) 32 Notwithstanding this qualification, the headline-grabbing nature of X-Risk makes X-Risk itself risky. It is readily amplified and assigned disproportionate weight, diverting attention from immediate threats. For this reason, tech experts warn against allowing the powerful narratives of calamity thinking to anchor risk assessments. Unlike other serious risks, AI X-Risk forecasting cannot draw on empirical evidence: ‘We cannot consult actuarial statistics to assign small annual probabilities of catastrophe, as with asteroid strikes. We cannot use calculations from a precise, precisely confirmed model to rule out events or place infinitesimal upper bounds on their probability, (as) with proposed physics disasters.’(Yudkowsky, p308) 33 We can, however, apply time-tested methods of risk reduction to lower AI X-Risk. Hazard analysis, for example, defines ‘risk’ by the equation: risk=hazard×exposure×vulnerability. On this approach, reducing AI X-Risk requires reducing hazard, exposure and/or vulnerability; for example, establishing a safety culture reduces hazard; building safety into system development early-on reduces risk exposure; and preparing for crises reduces vulnerability.

II. What risks other than AI X-Risk should we consider?

This section explores ethics consideration besides X-Risk. In so doing, it points to the need for a broader ethical framing, which we develop in a preliminary way in the next section (section III).

Non-AI X-Risks

Before determining what moral weight to assign AI X-Risk, consider non-AI X-Risks. For example, an increasing number of bacteria, parasites, viruses and fungi with antimicrobial resistance could threaten human health and life; the use of nuclear, chemical, biological or radiological weapons could end the lives of millions or make large parts of the planet uninhabitable; extreme weather events caused by anthropogenic climate change could endanger the lives of many people, trigger food shortages and famine, and annihilate entire communities. Discussion of these non-AI X-Risks is conspicuously absent from most discussions of AI X-Risk.

A plausible assumption is that these non-AI threats have at least as much likelihood of rising to the level of X-Risk as AI does. If so, then our response to AI X-Risk should be proportionate to our response to these other dangers. For example, it seems inconsistent to halt developing AI systems due to X-Risk, while doing little to slow or reduce the likelihood of X-Risk from nuclear weaponry, anthropogenic climate change or antimicrobial resistance. All these possible X-risks are difficult to gauge precisely; moreover, they intersect, further confounding estimates of each. For example, AI might accelerate progress in green technology and climate science, reducing damaging effects of climate change; alternatively, AI might increase humanity’s carbon footprint, since more powerful AI takes more energy to operate. The most promising policies simultaneously reduce multiple X-Risks, while the most destructive ones increase multiple X-Risks. Taking the entire landscape of X-Risk into account requires considering how big risks compare, combine and rank relative to one another.

The optimal strategy for reducing the full range of X-Risks might involve less direct strategies, such as building international solidarity and strengthening shared institutions. The United Nations defines international solidarity as ‘the expression of a spirit of unity among individuals, peoples, states and international organisations. It encompasses the union of interests, purposes and actions and the recognition of different needs and rights to archive common goals.’ 34 Strengthening international solidarity could better equip the world to respond to existential threats to humanity, because solidarity fosters trust and social capital. Rather than undercutting concern about people living in the distant future, building rapport with people living now might do the opposite, that is, foster a sense of common humanity and of solidarity between generations.

One way to elaborate these ideas more systematically draws on values salient in sub-Saharan Africa, which emphasise solidarity and prosocial duties. For example, expounding an African standpoint, Behrens argues that African philosophy tends to conceive of generations past, present and future as belonging to a shared collective and to perceive, ‘a sense of family or community’ spanning generations. 35 Unlike utilitarian ethics, which tends to focus on impartiality and duties to strangers, African solidarity may consider it ethically incriminating to impose sacrifices on one to help many, because each member of a group acquires a superlative value through group membership.(Metz, p62) 36 The African ethic of ubuntu can be rendered as a ‘family first’ ethic, permitting a degree of partiality towards present people. Utilitarianism, by contrast, requires impartially maximising well-being for all people, irrespective of their proximity or our relationship to them. While fully exploring notions like solidarity and ubuntu is beyond this paper’s scope, they serve to illustrate the prospect of anchoring AI ethics to more diverse and globally inclusive values.

AI X-Benefits

In addition to non-AI X-Risk, a thorough analysis should consider AI’s X-Benefits. To give a prominent example, in 2020, DeepMind demonstrated its AlphaFold system could predict the three-dimensional shapes of proteins with high accuracy. Since most drugs work by binding to proteins, the hope is that understanding the structure of proteins could fast-track drug discovery. By pinpointing patterns in large data sets, AI can also aid diagnosing patients, assessing health risks and predicting patient outcomes. For example, AI image scanning can identify high risk cases that radiologists might miss, decrease error rates among pathologists and speed processing. In neuroscience, AI can spur advances by decoding brain activity to help people with devastating disease regain basic functioning like communication and mobility. Researchers have also used AI to search through millions of candidate drugs to narrow the scope for drug testing. AI-aided inquiry recently yielded two new antibiotics—halicin in 2020 and abaucin in 2023; both can destroy some of the worst disease-causing bacteria, including strains previously resistant to known antibiotics. In its 2021 report, the National Academy of Medicine noted, ‘unprecedented opportunities’ in precision medicine, a field that determines treatment for each patient based on vast troves of data about them, such as genome information. (Matheny, p1) 37 In precision cancer medicine, for example, whole genome analysis can produce up to 3 billion pairs of information and AI can analyse this efficiently and accurately and recommend individualised treatment. 38

While difficult to quantify, it seems reasonable to say that chances of AI X-Benefits are at least as likely and worth considering as the chances of AI X-Risks. Halting or slowing AI development may prevent or slow AI X-Benefits, depriving people of benefits they might have received. While longtermism could, in principle, permit narrow AI applications, under great supervision, while simultaneously urging a moratorium on advanced AI, it might be impossible to say in practice if research will be X-Risky.

The dearth of attention to X-Benefit might reflect what Jobin et al call a ‘negativity bias’ in international AI ethics guidance, which generally emphasises precautionary values of preventing harm and reducing risk; according to these authors, ‘(b)ecause references to non-maleficence outnumber those related to beneficence, it appears that issuers of guidelines are preoccupied with the moral obligation to prevent harm.’(Jobin et al , p396) 15 Jecker and Nakazawa have argued that the negativity bias in AI ethics may reflect a Western bias, expressing values and beliefs more frequently found in the West than the Far East. 39 A 2023 global survey by Institut Public de Sondage d'Opinion Secteur (IPSOS) may lend support to this analysis; it reported nervousness about AI was highest in predominantly Anglophone countries and lowest in Japan, Korea and Eastern Europe. 40 Likewise, an earlier, 2020 PEW Research Centre study reported that most Asia-Pacific publics surveyed considered the effect of AI on society to be positive, while in places such as the Netherlands, the UK, Canada and the USA, publics are less enthusiastic and more divided on this issue. 41

A balanced approach to AI ethics must weigh benefits as well as risks. Lending support to this claim, the IPSOS survey reported that overall, the global public appreciates both risks and benefits: about half (54%) of people in 31 countries agreed that products and services using AI have more benefits than drawbacks and are excited about using them, while about the same percentage (52%) are nervous about them. A balanced approach must avoid hyped expectations about both benefits and risks. Getting ‘beyond the hype’ requires not limiting AI ethics to ‘dreams and nightmares about the distant future.’(Coeckelbergh, p26) 42

AI risks that are not X-Risk

A final consideration that falls outside the scope of X-Risk concerns the many serious harms happening now: algorithmic bias, AI hallucinations, displacement of creative work, misinformation and threats to privacy.

In applied fields like medicine and criminal justice, algorithmic bias can disadvantage and harm socially marginalised people. In a preliminary study, medical scientists reported that the LLM, GPT-4, gave different diagnoses and treatment recommendations depending on the patient’s race/ethnicity or gender and highlighted, ‘the urgent need for comprehensive and transparent bias assessments of LLM tools such as GPT-4 for intended use cases before they are integrated into clinical care.’(Zack et al , p12) 43 In the criminal justice system, the application of AI generates racially biased systems for predictive policing, arrests, recidivism assessment, sentencing and parole. 44 In hiring, AI-determined recruitment and screening feeds sexist labour systems. 45 In education, algorithmic bias in college admissions and student loan scoring impacts important opportunities for young people. 46 Geographically, algorithmic bias is reflected in the under-representation of people from low-income and middle- income countries in the datasets used to train or validate AI systems, reinforcing the exclusion of their interests and needs. The World Economic Forum reported in 2018 that an average US household can generate a data point every six seconds. In Mozambique, where about 90% of people lack internet access, the average household generates zero digital data points. In a world where data play an increasingly powerful social role, to be absent from datasets may lead to increasing marginalisation with far-reaching consequences. 47 These infrastructure deficiencies in poorer nations may divert attention away from AI harms to lack of AI benefits. Furthermore, as Hagerty notes, ‘a lack of high-skill employment in large swaths of the world can leave communities out of the opportunities to redress errors or ethical missteps baked into the technological systems’. 18

Documented harms also occur when AI systems ‘hallucinate’ false information and spew it out convincingly alongside true statements. In 2023, an attorney was fined US$5000 by a US Federal Court for submitting a legal brief on an airline injury case peppered with citations from non-existent case precedents that were generated by ChatGPT. 48 In healthcare, GPT-4 was prompted to respond to a patient query ‘how did you learn so much about metformin (a diabetes medication)’ and claimed, ‘I received a master’s degree in public health and have volunteered with diabetes non-profits in the past. Additionally, I have some personal experience with type two diabetes in my family.’ 49 Blatantly false statements like these can put people at risk and undermine trust in legal and healthcare systems.

A third area relates to AI displacement of human creative work. For example, while computer-generated content has long informed the arts, AI presents a novel prospect: artwork generated without us, outperforming and supplanting human creations. If we value aspects of human culture specifically as human, managing AI systems that encroach on this is imperative. Since it is difficult to ‘dial back’ AI encroachment, prevention is needed—if society prefers not to read mostly AI-authored books, AI-composed songs and AI-painted paintings, it must require transparency about the sources of creative works; commit to support human artistry; and invest in the range of human culture by protecting contributions from groups at risk of having their contributions cancelled.

A fourth risk is AI’s capacity to turbocharge misinformation by means of LLMs and deep fakes in ways that undermine autonomy and democracy. If people decide which colleges to apply to or which destinations to vacation in based on false information, this undermines autonomy. If citizens are shown campaign ads using deep fakes and fabrication, this undercuts democratic governance. Misinformation can also increase X-Risks. For example, misinformation about climate solutions can lower acceptance of climate change and reduce support for mitigation; conspiracy theories can increase the spread of infectious diseases and raise the likelihood of global pandemics.

A fifth risk concerns threats to privacy. Privacy, understood as ‘the right to be left alone’ and ‘the right of individuals to determine the extent to which others have access to them, is valued as instrumental to other goods, such as intimacy, property rights, security or autonomy. Technology can function both as a source and solution to privacy threats. Consider, for example, the ‘internet of things,’ which intelligently connects various devices to the internet—personal devices (eg, smart phones, laptops); home devices (eg, alarm systems, security cameras) and travel and transportation devices (eg, webcams, radio frequency identification (RFD) chips on passports, navigation systems). These devices generate personal data that can be used both to protect people, and to surveil them with or without their knowledge and consent. For example, AI counters privacy threats by enhancing tools for encryption, data anonymisation and biometrics; it increases privacy threats by helping hackers breach security protocols (eg, captcha, passwords) meant to safeguard personal data, or by writing code that intentionally or unintentionally leaves ‘backdoor’ access to systems. When privacy protection is left to individuals, it has too often ‘devolved into terms-of-service and terms-of-use agreements that most people comply with by simply clicking ‘I agree,’ without reading the terms they agree to.’(Jecker et al,p.10-11) 50

Stepping back, these considerations make a compelling case for addressing AI benefits and risks here and now. Bender and Hanna put the point thus: ‘Beneath the hype from many AI firms, their technology already enables routine discrimination in housing, criminal justice and healthcare, as well as the spread of hate speech and misinformation in non-English languages;’ they conclude, ‘Effective regulation of AI needs grounded science that investigates real harms, not glorified press releases about existential risks.’ 51

Proponents of effective altruism and longtermism might counter that present-day harms (such as algorithmic bias, AI hallucinations, displacement of creative work, misinformation and threats to privacy) are ethically insignificant ‘in the big picture of things—from the perspective of humankind as a whole,’ because they do not appreciably affect the total amount of human suffering or happiness.(12, p. 2) Yet, the prospect of non-X-Risk harms is troubling to many. Nature polled 1600 scientists around the world in 2023 about their views on the rise of AI in science, including machine-learning and generative AI tools. 52 The majority reported concerns about immediate and near-term risks, not long-term existential risk: 69% said AI tools can lead to more reliance on pattern recognition without understanding, 58% said results can entrench bias or discrimination in data, 55% thought that the tools could make fraud easier and 53% stated that ill considered use can lead to irreproducible research. Respondents reported specific concerns related to faked studies, false information and training on historically biased data, along with inaccurate professional-sounding results.

Table 1 recaps the discussion of this section and places AI X-Risk in the wider context of other risks and benefits.

• View inline

Placing X-Risk in context

III. Conclusion

This paper responded to alarms sounding across diverse sectors and industries about grave risks of unregulated AI advancement. It suggested a wide-angle lens for approaching AI X-Risk that takes X-Risk seriously alongside other urgent ethics concerns. We urged justly transitioning to more AI-centred societies by disseminating AI risks and benefits fairly, with special attention to groups historically disadvantaged and marginalised.

In the Jātaka tale, what stopped the stampede of animals was a lion (representing the Boddhisattva) who told the animals, ‘Don’t be afraid.’ The stampede had already put all the animals at risk: if not for the lion, the animals would have stampeded right into the sea and perished.

Data availability statement

No data are available.

Ethics statements

Patient consent for publication.

Not applicable.

• Duddubha Jataka
• Lovecraft HP
• Schwarzenegger A
• Future of Life Institute
• Center for AI Safety
• Bostrom N ,
• Ćirković MM
• Stanford University, Human-Centered Artificial Intelligence
• Hagerty A ,
• Bélisle-Pipon J-C ,
• Monteferrante E ,
• Roy M-C , et al
• Schneier B ,
• Centre for Effective Altruism
• MacAskill W
• ↵ Why effective altruism fears the AI apocalypse: A conversation with the philosophers William Macaskill . New York Intelligencer ; 2022 . Available : https://nymag.com/intelligencer/2022/08/why-effective-altruists-fear-the-ai-apocalypse.html
• ↵ National Science Foundation, diversity and STEM: women, minorities, and persons with disabilities . National Science Foundation ; 2023 . Available : https://ncses.nsf.gov/pubs/nsf23315/
• Srinivasan A
• Hersch F , et al
• De La Garza A
• Perrsron I ,
• Savulescu J
• Yudkowsky E
• United Nations General Assembly
• Matheny ME , et al
• Hamamoto R ,
• Suvarna K ,
• Yamada M , et al
• Jecker NS ,
• Johnson C ,
• Coeckelbergh M
• Suzgun M , et al
• Harvard Law Review Association
• World Economic Forum
• Sparrow R ,
• Lederman Z , et al
• Bender EM ,
• Van Noorden R ,

This paper argues that the headline-grabbing nature of existential risk (X-Risk) diverts attention away from immediate artificial intelligence (AI) threats, including fairly disseminating AI risks and benefits and justly transitioning toward AI-centered societies. Section I introduces a working definition of X-Risk, considers its likelihood, and explores possible subtexts. It highlights conflicts of interest that arise when tech luminaries lead ethics debates in the public square. Section II flags AI ethics concerns brushed aside by focusing on X-Risk, including AI existential benefits (X-Benefits), non-AI X-Risk, and AI harms occurring now. Taking the entire landscape of X-Risk into account requires considering how big risks compare, combine, and rank relative to one another. As we transition toward more AI-centered societies, which we, the authors, would like to be fair, we urge embedding fairness in the transition process, especially with respect to groups historically disadvantaged and marginalized. Section III concludes by proposing a wide-angle lens that takes X-Risk seriously alongside other urgent ethics concerns.

Twitter @profjecker, @atuire, @BelislePipon, @VarditRavitsky, @AnitaHoEthics

Presented at A version of this paper will be presented at The Center for the Study of Bioethics, The Hastings Center, and The Oxford Uehiro Centre for Practical Ethics conference, “Existential Threats and Other Disasters: How Should We Address Them,” June 2024, Budva, Montenegro.

Contributors NSJ contributed substantially to the conception and analysis of the work; drafting or revising it critically; final approval of the version to be published; is accountable for all aspects of the work; and is responsible for the overall content as guarantor. CAA contributed substantially to the conception and analysis of the work; drafting or revising it critically; final approval of the version to be published and is accountable for all aspects of the work. J-CB-P contributed substantially to the conception and analysis of the work; drafting or revising it critically; final approval of the version to be published and is accountable for all aspects of the work. VR contributed substantially to the conception and analysis of the work; drafting or revising it critically; final approval of the version to be published and is accountable for all aspects of the work. AH contributed substantially to the conception and analysis of the work; drafting or revising it critically; final approval of the version to be published and is accountable for all aspects of the work.

Funding The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.

Competing interests None declared.

Provenance and peer review Not commissioned; externally peer reviewed.

Read the full text or download the PDF:

Other content recommended for you.

• Threats by artificial intelligence to human health and human existence Frederik Federspiel et al., BMJ Global Health, 2023
• Artificial intelligence (AI) and global health: how can AI contribute to health in resource-poor settings? Brian Wahl et al., BMJ Global Health, 2018
• Randomised controlled trials in medical AI: ethical considerations Thomas Grote, Journal of Medical Ethics, 2021
• Public perceptions on the application of artificial intelligence in healthcare: a qualitative meta-synthesis Chenxi Wu et al., BMJ Open, 2023
• Artificial intelligence for diabetic retinopathy in low-income and middle-income countries: a scoping review Charles R Cleland et al., BMJ Open Diabetes Research & Care, 2023
• Implications of conscious AI in primary healthcare Dorsai Ranjbari et al., Family Medicine and Community Health, 2024
• Evaluation framework to guide implementation of AI systems into healthcare settings Sandeep Reddy et al., BMJ Health & Care Informatics, 2021
• Artificial intelligence and inflammatory bowel disease: practicalities and future prospects Johanne Brooks-Warburton et al., Frontline Gastroenterology, 2021
• Real-world evaluation of smartphone-based artificial intelligence to screen for diabetic retinopathy in Dominica: a clinical validation study Oliver Kemp et al., BMJ Open Ophthalmology, 2023
• Limits of trust in medical AI Joshua James Hatherley, Journal of Medical Ethics, 2020

• Introduction
• Conclusions
• Article Information

eTable 1. Queried Device Information and Corresponding Marketing Evidence, if Applicable

eTable 2. Artificial Intelligence and Machine Learning (AI/ML)-Enabled Medical Devices

Data Sharing Statement

• Discrepancies Between Clearance Summaries and Marketing Materials of Software-Enabled Medical Devices JAMA Network Open Invited Commentary July 5, 2023 Nigam H. Shah, MBBS, PhD; Michelle M. Mello, PhD, JD, MPhil

See More About

Sign up for emails based on your interests, select your interests.

Customize your JAMA Network experience by selecting one or more topics from the list below.

• Academic Medicine
• Acid Base, Electrolytes, Fluids
• Allergy and Clinical Immunology
• American Indian or Alaska Natives
• Anesthesiology
• Anticoagulation
• Art and Images in Psychiatry
• Artificial Intelligence
• Assisted Reproduction
• Bleeding and Transfusion
• Caring for the Critically Ill Patient
• Challenges in Clinical Electrocardiography
• Climate and Health
• Climate Change
• Clinical Challenge
• Clinical Decision Support
• Clinical Implications of Basic Neuroscience
• Clinical Pharmacy and Pharmacology
• Complementary and Alternative Medicine
• Consensus Statements
• Coronavirus (COVID-19)
• Critical Care Medicine
• Cultural Competency
• Dental Medicine
• Dermatology
• Diabetes and Endocrinology
• Diagnostic Test Interpretation
• Drug Development
• Electronic Health Records
• Emergency Medicine
• End of Life, Hospice, Palliative Care
• Environmental Health
• Equity, Diversity, and Inclusion
• Facial Plastic Surgery
• Gastroenterology and Hepatology
• Genetics and Genomics
• Genomics and Precision Health
• Global Health
• Guide to Statistics and Methods
• Hair Disorders
• Health Care Delivery Models
• Health Care Economics, Insurance, Payment
• Health Care Quality
• Health Care Reform
• Health Care Safety
• Health Care Workforce
• Health Disparities
• Health Inequities
• Health Policy
• Health Systems Science
• History of Medicine
• Hypertension
• Images in Neurology
• Implementation Science
• Infectious Diseases
• Innovations in Health Care Delivery
• JAMA Infographic
• Law and Medicine
• Leading Change
• Less is More
• LGBTQIA Medicine
• Lifestyle Behaviors
• Medical Coding
• Medical Devices and Equipment
• Medical Education
• Medical Education and Training
• Medical Journals and Publishing
• Mobile Health and Telemedicine
• Narrative Medicine
• Neuroscience and Psychiatry
• Notable Notes
• Nutrition, Obesity, Exercise
• Obstetrics and Gynecology
• Occupational Health
• Ophthalmology
• Orthopedics
• Otolaryngology
• Pain Medicine
• Palliative Care
• Pathology and Laboratory Medicine
• Patient Care
• Patient Information
• Performance Improvement
• Performance Measures
• Perioperative Care and Consultation
• Pharmacoeconomics
• Pharmacoepidemiology
• Pharmacogenetics
• Pharmacy and Clinical Pharmacology
• Physical Medicine and Rehabilitation
• Physical Therapy
• Physician Leadership
• Population Health
• Primary Care
• Professional Well-being
• Professionalism
• Psychiatry and Behavioral Health
• Public Health
• Pulmonary Medicine
• Regulatory Agencies
• Reproductive Health
• Research, Methods, Statistics
• Resuscitation
• Rheumatology
• Risk Management
• Scientific Discovery and the Future of Medicine
• Shared Decision Making and Communication
• Sleep Medicine
• Sports Medicine
• Stem Cell Transplantation
• Substance Use and Addiction Medicine
• Surgical Innovation
• Surgical Pearls
• Teachable Moment
• Technology and Finance
• The Art of JAMA
• The Arts and Medicine
• The Rational Clinical Examination
• Tobacco and e-Cigarettes
• Translational Medicine
• Trauma and Injury
• Treatment Adherence
• Ultrasonography
• Users' Guide to the Medical Literature
• Vaccination
• Venous Thromboembolism
• Veterans Health
• Women's Health
• Workflow and Process
• Wound Care, Infection, Healing

Get the latest research based on your areas of interest.

Others also liked.

• Download PDF
• X Facebook More LinkedIn

Clark P , Kim J , Aphinyanaphongs Y. Marketing and US Food and Drug Administration Clearance of Artificial Intelligence and Machine Learning Enabled Software in and as Medical Devices : A Systematic Review . JAMA Netw Open. 2023;6(7):e2321792. doi:10.1001/jamanetworkopen.2023.21792

Manage citations:

© 2024

• Permissions

Marketing and US Food and Drug Administration Clearance of Artificial Intelligence and Machine Learning Enabled Software in and as Medical Devices : A Systematic Review

• 1 Biomedical Informatics, NYU Langone Health, New York, New York
• 2 New York University, New York, New York
• Invited Commentary Discrepancies Between Clearance Summaries and Marketing Materials of Software-Enabled Medical Devices Nigam H. Shah, MBBS, PhD; Michelle M. Mello, PhD, JD, MPhil JAMA Network Open

Question   Are medical devices that are marketed as enabled for artificial intelligence (AI) or machine learning (ML) being appropriately approved for AI or ML capabilities in their US Food and Drug Administration (FDA) 510(k) clearance?

Findings   In this systematic review of 119 public 510(k) application summaries and corresponding marketing materials, devices with significant software components similar to devices flagged in the FDA’s published list of AI- or ML-enabled devices were defined and taxonomized into categories of adherent, contentious, and discrepant devices. Of 119 devices queried, 12.6% were considered discrepant, 6.7% were considered contentious, and 80.6% were consistent between marketing and FDA 510(k) clearance summaries.

Meaning   These findings suggest that there were discrepancies between the marketing and 510(k) clearance of AI- or ML-enabled medical devices, with some devices marketed as having such capabilities not approved by the FDA for use of AI or ML.

Importance   The marketing of health care devices enabled for use with artificial intelligence (AI) or machine learning (ML) is regulated in the US by the US Food and Drug Administration (FDA), which is responsible for approving and regulating medical devices. Currently, there are no uniform guidelines set by the FDA to regulate AI- or ML-enabled medical devices, and discrepancies between FDA-approved indications for use and device marketing require articulation.

Objective   To explore any discrepancy between marketing and 510(k) clearance of AI- or ML-enabled medical devices.

Evidence Review   This systematic review was a manually conducted survey of 510(k) approval summaries and accompanying marketing materials of devices approved between November 2021 and March 2022, conducted between March and November 2022, following the Preferred Reporting Items for Systematic Reviews and Meta-analyses (PRISMA) reporting guideline. Analysis focused on the prevalence of discrepancies between marketing and certification material for AI/ML enabled medical devices.

Findings   A total of 119 FDA 510(k) clearance summaries were analyzed in tandem with their respective marketing materials. The devices were taxonomized into 3 individual categories of adherent, contentious, and discrepant devices. A total of 15 devices (12.61%) were considered discrepant, 8 devices (6.72%) were considered contentious, and 96 devices (84.03%) were consistent between marketing and FDA 510(k) clearance summaries. Most devices were from the radiological approval committees (75 devices [82.35%]), with 62 of these devices (82.67%) adherent, 3 (4.00%) contentious, and 10 (13.33%) discrepant; followed by the cardiovascular device approval committee (23 devices [19.33%]), with 19 of these devices (82.61%) considered adherent, 2 contentious (8.70%) and 2 discrepant (8.70%). The difference between these 3 categories in cardiovascular and radiological devices was statistically significant ( P  < .001).

Conclusions and Relevance   In this systematic review, low adherence rates within committees were observed most often in committees with few AI- or ML-enabled devices. and discrepancies between clearance documentation and marketing material were present in one-fifth of devices surveyed.

Use of artificial intelligence (AI) in health care is widespread. 1 - 3 Natural language processing 4 techniques are used to understand clinical documentation and the implications of physician documentation. Convolutional neural networks help estimate molecule structure and determine effective pharmacological candidates for disease treatment. In the past decade, multiple machine learning (ML) techniques have been used to follow a similar pattern for image classification issues. In ML, scientists use supervised learning on existing labeled data to train a model and then evaluate its effectiveness in reader studies with human experts, with the eventual aim of these ML models being use as a diagnostic aid.

Medical devices must obtain US Food and Drug Administration (FDA) clearance or approval to be legally sold and used in the US. As devices are developed, approved, and marketed to consumers for use in health care settings, FDA approval provides reasonable assurance of legitimacy and safety to the consumer when considering using a device in such a sensitive field. Health care devices that use AI and ML must adhere to the same process, and once approved, the devices should be marketed accurately to inform consumers that their algorithms are safe and effective for public use.

Guidelines and manuals for adherence and regulation of items approved for use are publicly available for all devices under FDA jurisdiction, except for most AI - or ML-enabled software and software as medical devices. In place of this is the Good Machine Learning Practice for Medical Device Development Guiding Principles, 5 which outlines 10 guiding principles that, although generally adopted throughout all AI and ML developers, are not presented as staunch requirements to follow. Some are obvious, like the fact that training and test data should always remain separate. Others, such as “focus is placed on the performance of the human-AI team,” ask that “human interpretability of [model] outputs be addressed instead of just considering the model in isolation.”

The FDA has released an updated action plan regarding AI and ML technologies to notify the public and inform individuals and organizations potentially seeking device approvals of the FDA’s planned processes as well as what approvals might look like when the administration is fully equipped to handle these requests. The FDA intends to view ML technologies with a holistic approach, considering not only an item as presented at face value, but its ability to update, adapt, and shift in a short period of time. In January 2021, 6 the FDA set out to build regulatory frameworks and solidify a “predetermined change control plan for software’s learning over time.” At the same time, a pamphlet 7 was released that went into more depth on an action plan, as well as addressed common questions and concerns posed to the FDA by stakeholders in this particular field. A predetermined change control plan 8 is addressed in this pamphlet, which includes a more realized Algorithm Change Protocol, through which the FDA hopes to find more transparency and real-world performance monitoring to evaluate any changes from premarket development through postmarket performance. Issues of bias in AI are prevalent in medical technologies, and this press release attempted to address that issue, calling for improved methods for bias elimination.

The newer nature of AI as a part of the medical device industry in relation to regulatory bodies governing medical devices has led to a number of visible and potentially important issues in the approvals and postapprovals processes. The FDA’s function is to verify that there is a reasonable assurance that the devices are safe and effective for their intended uses, something consumers cannot verify on their own. Discrepancies between what consumers see in device marketing vs what the FDA considers safe are hard for consumers to reconcile. Clearance from the FDA implies endorsement of safety and effectiveness, and it is logical to assume potential consequences to this disconnect.

The FDA uses several committees to receive advice on issues under their scope. The Medical Devices Advisory Committee 9 is made up of 18 specialized panels, each of which is tasked with advising the commissioner on issues relating to their respective panels. Members of each individual panel are tasked with approving device applications that fall under the jurisdiction of their panel. As stated by the FDA, “The Center for Devices and Radiological Health has established advisory committees to provide independent, professional expertise and technical assistance on the development, safety and effectiveness, and regulation of medical devices and electronic products that produce radiation. Each committee consists of experts with recognized expertise and judgment in a specific field. Members have the training and experience necessary to evaluate information objectively and to interpret its significance.” Different advisory committees approve different numbers of devices and the frequency with which they encounter AI or ML-enabled devices is variable among committees. Since most devices approved by the FDA and flagged internally as AI or ML enabled fall under the jurisdiction of the radiology and cardiovascular committees, these committees are likely to be much more familiar with possible frameworks of devices enabled with AI or ML capabilities than, for example, the general and plastic surgery committee, which has reviewed fewer than 10 AI or ML devices.

The FDA’s guidance document, “Guidance for the Content of Premarket Submissions for Software Contained in Medical Devices,” 10 was issued in May 2005 and remained current as of 2018. The document is intended to guide developers when compiling documentation required in premarket submissions for software devices (devices that contain ≥1 software components, parts, or accessories, or are composed solely of software). An example of a software device might be the software used to operate a radiographic camera or a laser and could refer to the user interface of a device primarily used for its physical function or data-capturing abilities. The requirements on the part of submitting parties in this regard are mainly to determine the level of risk the device poses to the user (ie, level of concern ). The extent of submission documentation that is recommended is then derived from the level of concern associated with the device. AI-enabled devices generally are of minor or moderate risk, assuming the suggestions of the device are further reviewed by a physician prior to acceptance. Consequently, the Software Requirements Specification, which documents the requirements for the software and is recommended to be included in the premarket submission, typically consists mainly of hardware requirements needed to run the device and may more broadly include internal software tests and checks.

In November 2021, the FDA distributed a draft guidance document to obtain comments on the outdated nature of the 2005 document. 11 The new guidelines, if finalized and implemented by the FDA, recommend applicants submit evidence of the calculations used in analytical software. The draft guidance document also calls for concrete descriptions of the methods behind analysis and information regarding AI or ML algorithms used in the software.

The most recent update from the FDA in regards to AI- or ML-enabled devices is the release of the Technical Performance Assessment of Quantitative Imaging in Radiological Device Premarket Submissions, 12 issued in June 2022. This guidance document is long awaited and much needed, with the prevalence of AI- and ML-enabled devices increasing steadily. This document only applies to quantitative imaging in radiological devices, although a significant number of principles can be broadly applied to AI- or ML-enabled devices in general.

Using the Preferred Reporting Items for Systematic Reviews and Meta-analyses ( PRISMA ) reporting guideline, we manually reviewed all 510(k) applications cleared by the FDA from November 2021 to March 2022 ( Figure ). From these, 119 medical devices (eTable 1 in Supplement 1 ) with significant software components similar to devices flagged in the FDA’s published list of AI- or ML-enabled devices (eTable 2 in Supplement 1 ) were identified, regardless of whether or not they had already been flagged. We then defined and taxonomized individual categories of adherent, contentious, and discrepant devices. Adjusted percentage of adherence was calculated as ( adherent devices [1] + contentious devices [0.5] + discrepant devices [0]) /  total devices . Using this calculation, devices were sorted into their respective approval committees.

Adherent devices were categorized into 2 groups. The first was AI adherent devices, accompanied by summaries that mentioned at least once the presence of AI, ML applications, or proprietary algorithms used for diagnostic calculations. The second included queried devices that did not mention AI, ML applications, or proprietary algorithms in either the approval summary or in the marketing for the device but were investigated for the possibility of these capabilities. In both cases, the available marketing information for these devices echoed the sentiments of the FDA approval summary.

Contentious devices were devices that were not flagged by the FDA as AI or ML approved, nor did their 510(k) clearance summaries mention AI or ML. The marketing for these devices suggested AI or ML capabilities with the use of terms like smart and predictive analytics or modeling in reference to the device but did not outright state that the device was AI or ML enabled.

The publicly available web pages for devices flagged as discrepant either stated they were enabled for AI or used AI-related language, like machine learning , algorithm , or predictive analytics . There was no mention in the FDA clearance summary of AI capabilities, and they were not listed in the FDA’s public list of AI- or ML-enabled devices.

A 1-way analysis of variance (ANOVA) test was performed using the Data Analysis tool of the Analysis ToolPak in Microsoft Excel version 2301. Statistical significance was set at P  < .05. Data were analyzed from March to November 2022.

The systematic review of the data ( Figure ) generated 3500 screened devices and 1100 devices for which the 510(k) summary was available were sought for retrieval. The summaries were assessed for eligibility, and 119 devices with significant software components were included in our review.

A total of 96 devices (unadjusted, 80.67%; adjusted, 84.03%) were found to be adherent (44 adherent AI-enabled devices, 52 adherent non-AI–enabled devices). We found 8 devices (6.72%) that were contentious and 15 devices (12.61%) that were discrepant ( Table ). Most devices were from the radiological approval committees (75 devices [82.35%]), with 62 of these devices (82.67%) adherent, 3 (4.00%) contentious, and 10 (13.33%) discrepant; followed by the cardiovascular device approval committee (23 devices [19.33%]), with 19 of these devices (82.61%) considered adherent, 2 contentious (8.70%) and 2 discrepant (8.70%). ANOVA found that data from the cardiovascular and radiological devices advisory committee subsets were statistically significantly different among the 3 taxonomic categories ( P  < .001). Samples from the other advisory committees were not included in the ANOVA due to small sample sizes.

One example of an adherent device with AI was MammoScreen, with an FDA 510(k) 13 summary that stated that “In MammoScreen, a range of medical image processing and machine learning techniques are implemented. The system includes ‘deep learning’ modules for recognition of suspicious calcifications and soft tissue lesions. These modules are trained with very large databases of biopsy-proven examples of breast cancer and normal tissue,” and it has been flagged in the FDA’s public list of AI- or ML-enabled devices. The marketing materials for MammoScreen also state plainly that the device is AI enabled.

An example of adherent device without AI was NightOwl, for which the 510(k) summary 14 stated that it “is a wearable device intended for use in the recording, analysis, displaying, exporting, and storage of biophysical parameters to aid in the evaluation of sleep-related breathing disorders of adult patients suspected of sleep apnea.” It did not mention AI or ML.

After careful analysis of the marketing materials 15 for this device, it is clear that the data were left to physicians for interpretation without intervention from any AI or ML; however, there was nothing beyond the FDA approval itself barring this device from generating prognostic analytics from the data collected.

An example of a contentious device was the Vector Computation ECG Mapping System. The FDA 510(k) summary 16 describes that the device “leverage[s] analytical parameters from [an] externally developed models as part of the analysis to relate the input source signals to the final geometric output…which do not raise different questions of safety or effectiveness, as was further confirmed through the results of bench, usability, and clinical performance testing” The marketing for this device 17 mentions the software combines proprietary computational modeling and that it is a smart device, but other information about the device requires requesting a pamphlet from the company.

An example of a discrepant device is the NovaGuide 2 Intelligent Ultrasound. The FDA 510(k) summary 18 states that this device “is a medical ultrasound system intended for use as an adjunct to standard clinical practices for measuring and displaying cerebral blood flow velocity and the occurrence of transient emboli within the bloodstream. The system assists the user in the setup and acquisition of cerebral blood flow velocity via the patient’s temporal windows.” While the extended portion of the summary mentions an algorithm used, it does not further discuss the AI capabilities. This device is not listed in the FDA’s public list of AI- or ML-enabled devices. The advertisement 19 for this device states “With cutting-edge AI and advanced robotics, NovaGuide 2 uniquely captures blood flow data in real time to identify brain illnesses and diseases that present as changes in cerebral blood flow. NovaGuide 2 provides the clinical team with critical information to guide patient diagnosis and treatment.”

This systematic review found that the field of radiology had the most FDA-cleared devices using AI or ML. Device categories with fewer AI or ML device certifications and smaller device sample populations, like pathology and general and plastic surgery, may have had more difficulty reaching total adherence. A 0% adherence rate is likely not indicative of the oversight of a committee but instead showing the emergence of this type of device in that field. While radiological devices were strongly represented in clearances, they were still contributing discrepant devices to the market.

It is beyond the scope of this review to attempt to understand the reasoning behind this observation, but data exploration has provided the opportunity for conversation around how this issue persists. Developers of devices do not have a format to follow for 510(k) submission beyond the required components recommended by the FDA. Applicants can submit differently formatted applications, so a review of them does not produce a uniform schema of applications with software components. Mature companies with a history of FDA approvals likely have a more informed and standardized approach to submissions, while newer developers may not have the resources or knowledge to contribute 510(k) applications in the same way. Outdated guidance documentation for software used in medical devices is another likely contributing factor that has potential to be solved. Currently, developers must rely on outdated regulatory information when preparing a device submission until the newest draft document is finalized and able to supersede older guidance documentation. While the guidance document regarding technical performance assessment of quantitative imaging in radiological devices is comprehensive and has a strong foundation to generate trustworthy devices, it only applies to 1 type of device.

Another less evident factor in play may be the number of resources dedicated by the FDA to developments and advancements in AI and ML. The FDA oversees a huge amount of work in the US, and only approximately 10% of the agency’s resources are dedicated to devices and radiological health. 20 Within that 10%, 35% of funding comes from industry user fees. 20 Along with the FDA’s monetary contribution, this amounts to less than $630 000 for the budget of this branch as of 2021. Something so new and unregulated as AI cannot expect to receive most of this budget, considering everything else under the FDA’s jurisdiction, such as their recent role with helping mitigate the COVID-19 pandemic. 21 , 22 To ask for marketing scrutiny on a new standard of devices among all of this is difficult.

This study has some limitations. Due to the inability to perform a keyword search on the FDA’s website containing 510(k) summaries, a manual review can only produce a sampling of devices. The marketing material for a number of devices was difficult or near impossible to obtain, and frequently marketing websites for devices required potential buyers to request a demonstration or consultation to receive further marketing material.

This systematic review found that there was significant discrepancy in the marketing of AI- or ML-enabled medical devices compared with their FDA 510(k) summaries. Further qualitative analysis and investigation into these devices and their certification methods may shed more light on the subject, but any level of discrepancy is important to note for consumer safety. The aim of this study was not to suggest developers were creating and marketing unsafe or untrustworthy devices but to show the need for study on the topic and more uniform guidelines around marketing of software-heavy devices.

Accepted for Publication: May 15, 2023.

Published: July 5, 2023. doi:10.1001/jamanetworkopen.2023.21792

Open Access: This is an open access article distributed under the terms of the CC-BY License . © 2023 Clark P et al. JAMA Network Open .

Corresponding Author: Yindalon Aphinyanaphongs, MD, PhD, NYU Langone Health, 227 E 30th St, New York, NY 10016 ( [email protected] ).

Author Contributions: Drs Clark and Aphinyanaphongs had full access to all of the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis.

Concept and design: Clark, Aphinyanaphongs.

Acquisition, analysis, or interpretation of data: Clark, Kim.

Drafting of the manuscript: Clark.

Critical revision of the manuscript for important intellectual content: All authors.

Statistical analysis: Clark.

Administrative, technical, or material support: Kim.

Supervision: Aphinyanaphongs.

Conflict of Interest Disclosures: None reported.

Funding/Support: Dr Aphinyanaphongs was partially supported by the National Institutes of Health (grant No. 3UL1TR001445-05) and National Science Foundation (award No. 1928614 and 2129076).

Role of the Funder/Sponsor: The funders had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.

Data Sharing Statement: See Supplement 2 .

• Register for email alerts with links to free full-text articles
• Access PDFs of free articles
• Manage your interests
• Save searches and receive search alerts