Dismantling Bias: Toward Ethical and Inclusive Health Innovation

The creativity and ingenuity of socially marginalized and racialized groups can drive innovative healthcare solutions. However, despite their potential contributions, these groups are too often shut out of innovation ecosystems. Advancing innovations that lead to equitable and sustainable solutions in the healthcare system requires interrogating the inequities embedded within research, design, and testing processes for novel health solutions, as well as diversifying the people involved at each stage.

The COVID-19 pandemic laid bare the social, economic, and racial discrimination that underlies the US healthcare system. Fueled by a racial reckoning after the deaths of George Floyd and Breonna Taylor, a national conversation that acknowledged racial and health injustices as a dual pandemic began to emerge. This put pressure on governments, institutions, and other organizations to proactively engage in racial justice reform within healthcare and medicine. As organizations, networks, and individuals sought solutions, a movement to address inequality through technological innovation started gaining momentum.

Yet, innovative health products have caused BIPOC and other vulnerable groups harm—such as pulse oximeters that fail to accurately detect blood oxygen levels in people with darker skin tones and AI tools that make decisions that systematically disadvantage certain groups of people. Within the context of medical and health innovation, historically marginalized people have largely been left on the periphery.

The Depraved History of Medical Innovation

Some populations have been deeply exploited by medicine and healthcare systems; consequently, their experiences are tainted by centuries of pain and trauma. In the United States, Black people have been subjected to inhumane treatment and brutality in the name of medical breakthroughs, and, perhaps even more egregiously, the very systems that they helped create continue to discriminate against them.

Fueled by a racial reckoning after the deaths of George Floyd and Breonna Taylor, a national conversation that acknowledged racial and health injustices as a dual pandemic began to emerge.

In the United States, modern surgery was tested on enslaved Black people who inherently could not consent to nor refuse the procedures. In the 1840s, an Alabaman physician, James Marion Sims, conducted experimental gynecological surgery on enslaved Black women, who were physically bound to his surgical table. Despite his methods, Sims was one of the most renowned American surgeons of the 19th century, and even today, he is generally acknowledged as the founder of modern surgical gynecology.

In Medical Apartheid: The Dark History of Medical Experimentation on Black Americans from Colonial Times to the Present, Harriet A. Washington draws attention to the extent of Sims’ “surgical theater,” where he operated on enslaved Black people without anesthesia for an “audience of students and potential protégés” as part of his bid for “medical glory.”¹ The horrific gynecological surgeries conducted on Black women were only one example; the surgical theater also included in-person surgeries on enslaved Black men. Though his ambivalence toward the humanity of Blacks is part of Sims’ legacy, according to Washington, he was not the only doctor at the time to use Black people’s bodies as “surgical material.”²

Another egregious example of exploitation in medical research and the human costs of medical innovation is woven into the never-ending medical contributions of Henrietta Lacks. In 1951, at only 31 years of age, Henrietta Lacks, a young Black woman and mother of five, passed away from a malignant cervical tumor after being treated at the Johns Hopkins Hospital. She survived only eight months after her cancer diagnosis. During her ultimately unsuccessful treatment, doctors took samples of Henrietta’s cancer cells, dubbed HeLa cells, which were given to a researcher without Lacks’s knowledge or consent.

Capitalistic and myopic thinking too often influence healthcare innovations, even though the irresponsible use of advanced technology can have devastating consequences for vulnerable communities.

By 1952, HeLa cells had become the first human cell line that could grow and divide endlessly in a laboratory; the seemingly immortal properties of Henrietta’s cells contributed to their adoption across the world as the human cell line of choice for biomedical research. The HeLa cell culture has survived for more than 70 years and is integral to biomedical research around the globe. Yet the Lacks family wasn’t made aware of the use of Henrietta Lacks’s cells and the medical advances connected to them until 1973.

Since the initial biopsy, scientists estimate that 50 million tons of HeLa cells have been created and used. HeLa cells have been cited in more than 110,000 scientific publications, there are more than 17,000 patents involving HeLa cells, and scientists who relied on HeLa cells for their research received the Nobel Prize on three separate occasions. HeLa cells’ collective contribution to scientific breakthroughs is immeasurable: they were used to develop the polio vaccine, involved in mapping the human genome, used to create treatments for HIV and HPV, and contributed to the treatment of almost every type of cancer. Most recently, HeLa cells were used in the development of the COVID-19 vaccine.

Understanding the cruel history behind many of the world’s leading medical innovations is a stark reminder that technology is not made out of thin air. Instead, it is the product of extraction, of exploitation, of displacement, of violence. But despite medical innovation’s troubling past, there is room to create space for all marginalized people within health innovation ecosystems.

Centering Marginalized Populations

Capitalistic and myopic thinking too often influence healthcare innovations, even though the irresponsible use of advanced technology can have devastating consequences for vulnerable communities. It’s imperative that health innovation ecosystems consciously shift their focus to health innovation that occurs with and for underserved communities rather than at the expense of health, wellbeing, or safety. We can achieve inclusive innovation by:

• Embracing Equitable Research Practices

Outmoded research paradigms that uphold the dichotomy between “researchers” and “research subjects” create a hierarchical power relationship that privileges the researcher’s perspective. On the other hand, equitable research paradigms, such as community-based participatory research and other participatory design methods, engage members of marginalized communities as agentic and knowledgeable actors who take a proactive role in solving problems and creating new pathways for health, healing, and thriving among marginalized peoples.

Big datasets and large language models (LLMs)—which commonly serve as the basis for innovation in healthcare, medicine, and other domains—position underrepresented and minoritized populations as “outliers.” This is especially concerning because outliers are typically discounted or discarded altogether in the name of generalizability. There are several techniques that can be utilized to amplify marginalized voices rather than dismissing them as outliers within the research process: oversampling underrepresented populations in quantitative research, disaggregating data by racial or ethnic groups to better understand the differences within and among groups, supplementing quantitative research with qualitative insights, and utilizing other multimodal research approaches.

Some innovations—which were deployed without the rigorous testing needed to assess their viability and safety—have also posed a more universal threat to the public.To ensure health innovations don’t reproduce existing inequalities or exacerbate them, the nature of the data itself also needs to be carefully evaluated for how it reproduces power structures. For instance, race-based diagnostics and practice guidelines—remnants of slavery and eugenics, which asserted that people of different races are biologically distinct—have been incorporated into modern AI systems. According to an article in the New England Journal of Medicine, these algorithms direct more attention or resources to White patients than to members of racial and ethnic minorities. It is important to pay close attention to how LLMs are developed and how AI systems are trained because data reflects the biased systems and structures that produce them. Researchers, designers, and engineers must use data reflexively to ensure it does not reproduce or intensify inequality.

• Considering Diverse Perspectives throughout the Innovation Process

Since so many components are involved in the innovation process, innovation ecosystems, and hubs involve a collaborative effort between entrepreneurs, scientists, researchers, designers, tech and business leaders, clinicians, patients, and representatives of the public and social sectors. Together, this diverse set of actors can understand multifaceted health problems, identify appropriate solutions, and help guide implementation or uptake for specific populations. In addition to diversity in the roles people play, innovation also benefits from diversity at the team level. Racially diverse teams are more innovative and productive because they can see things in a variety of ways. In contrast, because the members of homogenous teams hold similar worldviews, beliefs, and styles of thinking, they produce narrower ranges of options and opinions.

It is also imperative that diverse groups of people are represented among leaders and other key decision-makers involved in health innovation because they are well-positioned to deeply understand our world’s increasingly multicultural, multiracial contexts and can help shift and renegotiate power in ways that make for more equitable, just and sustainable outcomes for us all.

• Rigorously Testing Product and Service Innovations before Deployment

In the United States, our current way of testing innovative products, systems, and services is broken. Even beyond the innovations that have directly harmed people of color, some innovations—which were deployed without the rigorous testing needed to truly assess their viability and safety—have also posed a more universal threat to the public. For instance, millions of sleep apnea machines were recently pulled off shelves because an industrial foam fitted in the machines to reduce noise could break apart and release potentially toxic particles and fumes into the masks worn by patients.

As the dangerous sleep apnea machines demonstrate, the regulators have too often failed to identify important problems with health innovations. To make matters worse, the FDA does not require medical technologies to be tested among diverse populations. To prioritize safety and equity, innovation leaders and other stakeholders should seek diverse input throughout the innovation process, conduct equity assessments, and test innovations thoroughly to understand how the technology affects people’s health and how the technology is integrated into their lives.

Innovative products, systems, and services are desperately needed to overcome the inefficiencies and ineffectiveness of the US healthcare system. Of equal importance and urgency is the need to utilize innovation to improve how marginalized populations navigate the healthcare system and raise the quality of care received. But the institutions that drive innovation—including government, industry, philanthropic foundations, and universities—should exercise discretion in how they fulfill this need.

Innovation, especially technological innovation, is too often perceived as neutral. The veiled objectivity of innovation obscures the biased processes and data that serve as the foundation of technological innovation. In the United States, as well as much of the rest of the world, the status quo involves a mostly White, male, privileged pool of designers and engineers creating innovations that work exclusively for them, such as facial recognition technology that sees White male faces better than others. These biased technologies are then imposed on the larger population.

If novel innovations are ever to effectively and sustainably advance healthcare and medicine for everyone, we must acknowledge that technology itself exists within, not outside, systems of oppression. Most crucially, advancing innovations that serve people at the margins requires addressing the complex nature of the reproduction of marginalization and using technology to disrupt and redress this harm. Rather than creating innovations under the veil of neutrality and fixing innovations to be more equitable after the fact, understanding how underrepresented and marginalized people will be affected by the tech should be integrated into all aspects of research, design, development, and implementation processes.

Notes 

1. Harriet A. Washington, Medical Apartheid: The Dark History of Medical Experimentation on Black Americans from Colonial Times to the Present (New York: Anchor Books, 2006), 101–106.
2. Ibid.