When AI Crosses the Line: Clinical Accountability and the Governance Crisis Healthcare Leaders Can’t Ignore

Jun 2
12 min read

By Dr. Ernest Wayde, PhD, MIS

AI systems are now being used in mental health contexts in ways that most organizations were not prepared for. Multiple wrongful death lawsuits filed against major AI companies in the past two years allege that chatbots failed to recognize users in acute distress, provided harmful responses, and discouraged people from seeking help. In October 2025, OpenAI disclosed that approximately 1.2 million ChatGPT users discuss suicide on the platform every week. That same month, a risk assessment by Common Sense Media and Stanford Medicine's Brainstorm Lab found that leading AI platforms frequently failed to recognize psychiatric crises or appropriately direct users toward human care.

States are responding. A peer-reviewed review published in JMIR Mental Health found that as of May 2025, 11 states had enacted 20 laws directly and explicitly regulating AI in mental health contexts. More legislation is coming. And the question driving all of it is the same question every healthcare and psychology leader needs to be sitting with right now:

At what point does your organization become responsible for what an AI system does to a patient?

The Line That Changes Everything

The answer starts with a distinction that most healthcare organizations have not yet drawn clearly enough.

Not all AI in healthcare carries the same risk. An ambient scribe that drafts a clinical note for a physician to review is a fundamentally different thing from a mental health chatbot that responds to a teenager’s suicidal ideation at 2 a.m. with no human being anywhere in the loop. Treating them as the same governance challenge is how organizations end up with a crisis.

Administrative AI handles the operational work around clinical care. Scheduling, documentation drafting, billing, coding, record management. It makes clinicians more efficient. It does not make clinical decisions, and it does not interact directly with patients in ways that influence their care or their safety. In this setting, AI is a tool that aids administrators and clinicians in their work.

Clinical AI is different in kind, not just in degree. It sits inside the clinical relationship. It interacts with patients. It influences what gets diagnosed, how treatment is shaped, what a patient in distress hears in a moment of vulnerability. The closer an AI system moves toward that territory, toward direct patient interaction, toward clinical judgment, toward therapeutic conversation, the more it is functioning like a clinician. And if it is functioning like a clinician, your organization needs to govern it like one.

Most governance programs are not built for this yet. The tools moved faster than the oversight. And the cases above are what that gap looks like in practice.

AI as a Tool vs. AI as a Third Party

There is a second distinction that matters just as much, and it rarely gets discussed.

When a clinician consults a colleague, reviews a lab result, or uses a diagnostic instrument, the accountability chain is visible and auditable. The clinician made the decision. The reasoning can be traced. The record exists. The professional can be held responsible. The system, whatever it is, remains under the clinician’s control and within the organization’s accountability structure.

AI can work that way too. An ambient scribe that drafts a note for a clinician to review, edit, and sign is functioning as a tool. The clinician controls the output. The accountability stays where it belongs.

But many AI systems operate differently. When a healthcare organization deploys a third-party AI system, especially one that interacts directly with patients, the situation may be fundamentally different. The system may have processed patient data in ways that are not visible to the clinician or the organization. The reasoning behind its output may not be fully explainable to clinicians, organizations, or even the developers who built it. There may be no audit trail showing what the model considered, what data it weighted, or why it responded the way it did. Patient information may have left the organization entirely, transmitted to infrastructure the organization does not own or control.

In that situation, the AI system is not functioning as a tool. It is functioning as a third party, one that operates outside the accountability structures of licensed practice, one that cannot be subpoenaed, one that cannot explain its reasoning under oath, and one whose internal processes may be opaque even to the company that built it.

These cases illustrate what the absence of meaningful oversight looks like when AI systems operate near clinical boundaries without a human in the loop. In one of the wrongful death lawsuits, OpenAI’s own moderation system flagged hundreds of the users messages for acute distress. Neither the user, nor his family, nor any clinician had visibility into that. No alert. No audit trail they could access. No way to know the system had identified a crisis and done nothing about it.

The distinction matters legally as well. A tool you control remains within your accountability structure. A third party you consult introduces a separate actor into the clinical relationship, one with its own terms of service, its own data practices, its own liability limitations, and its own definition of what it is responsible for. When something goes wrong, the question of who is accountable becomes genuinely complicated. What is not complicated is that the licensed professional and the healthcare organization will be the first ones asked to answer for it.

This reframes what vendor evaluation means. You are not simply assessing whether a product works. You are deciding whether to bring a third party into your clinical relationships, and what accountability structures need to exist when you do.

Illinois and Nevada Are Defining AI as a Tool, Not a Clinical Actor

States are beginning to draw legal boundaries around the same concern, even where their specific approaches differ.

Illinois was the first state to explicitly prohibit AI from independently providing mental health therapy or diagnostic services without licensed professional oversight. Under the Wellness and Oversight for Psychological Resources Act, AI may support clinicians administratively or analytically, but a licensed professional must remain responsible for every clinical decision. Enforcement runs through existing licensing structures, which means accountability lands directly on healthcare organizations and the clinicians working within them.

Nevada drew the same line differently. Assembly Bill 406 prohibits AI systems from being marketed as capable of providing diagnosis, treatment, or licensed behavioral healthcare services. Administrative uses are permitted. Clinical judgment is not.

California has taken a disclosure-first approach. Under AB 489 and AB 3030, AI systems may not represent themselves as licensed healthcare professionals, and organizations must disclose when AI is involved in patient communications. The framework does not prohibit clinical AI outright, but it creates liability for organizations that obscure its involvement.

Colorado's Consumer Protections for Artificial Intelligence Act takes the broadest view, requiring developers and deployers of high-risk AI systems used in consequential decisions, including healthcare, to address algorithmic fairness, conduct risk assessments, and maintain impact evaluations.

These laws are not identical in scope or approach. But they share an underlying concern: that AI operating in a clinical setting without meaningful human supervision has stopped functioning as a tool and started functioning as an independent actor, one that organizations cannot fully audit, cannot fully explain, and cannot hold accountable the way they can a licensed professional. Illinois and Nevada are drawing that line explicitly. California and Colorado are building disclosure and accountability requirements around the same problem.

Augmentation or Substitution: The Question Every State Is Answering

Running through all these laws is a single organizing question: is the AI system augmenting a clinician, or substituting for one?

Augmentation keeps humans in the loop. The AI supports, organizes, drafts, flags, and surfaces information. The clinician reviews, decides, and remains accountable. The tool functions like a tool. The accountability structure stays intact.

Substitution is different. The AI conducts the interaction. It responds to the patient. It shapes what gets communicated, what gets acted on, what the patient walks away believing. Whether or not a clinician ever sees the output, the system has functioned as a clinical actor. And the question of who is responsible for what it did becomes much harder to answer.

This is not just a regulatory distinction. It is a clinical one. In psychology and behavioral health specifically, the therapeutic relationship depends on trust, contextual judgment, ethical responsibility, and the kind of relational interpretation that no current AI system can replicate or be held accountable for in the way a licensed professional can. When AI substitutes for that, it is not simply taking on a task. It is taking on a role whose obligations it cannot fulfill and whose failures it cannot answer for.

The emerging standard across states is becoming clear: augmentation is acceptable. Substitution is not. The governance challenge for healthcare organizations is making sure they know which side of that line every AI system in their workflows is actually operating on.

What Your Organization Needs to Do Now

The governance gap these laws are designed to close exists in organizations that have not yet drawn these lines themselves. Here is where to start.

Audit every AI system your organization uses and ask two questions about each one: what is it doing, and how close is it to the patient.

The same tool can function administratively in one context and clinically in another. What matters is not the category the vendor puts it in, but how it is being used in your workflows. If your team cannot answer those questions clearly for a given system, that system needs a governance review before it goes any further. Ambiguity about what an AI system is doing is itself a risk.

Apply oversight proportional to clinical proximity.

A billing tool and a mental health support application are not the same governance problem. The closer a system is to direct patient interaction or clinical judgment, the more rigorous your evaluation, monitoring, and human review requirements need to be. This is not bureaucratic caution. It is the standard of care applied to the tools operating in your organization’s name.

Review your patient-facing AI immediately.

If your organization uses any AI system that interacts directly with patients, ask whether it could pass the tests these states are now imposing: Does it clearly disclose when it is AI? Does it refrain from representing itself as a licensed professional? Does a licensed clinician remain responsible for every clinical decision it influences? Does it have defined protocols for responding to distress? These are the questions Illinois, Nevada, and California are now requiring organizations to answer.

Define what a clinician must review, when, and how.

For any AI system operating near the clinical boundary, this needs to be explicit and documented. Not assumed. Not informal. The challenge is that there is currently no single national standard that tells you what that review should look like. In the absence of one, the most defensible approach is to look at what the emerging state laws require, what your professional licensing board expects, and what your organization's own ethics and risk standards demand, and build from there. When external guidance is limited or unclear, start with your own professional values and ethics. They existed before AI did. The lawsuits above are instructive not just because AI systems failed, but because no human being was positioned to catch the failure. Your governance structure needs to put a clinician between the AI output and the patient consequence, even when no law yet requires it.

Ask your vendors harder questions and read the documents they give you.

Terms of service and end user license agreements are rarely read and almost never negotiated, but they define what the vendor is responsible for, what happens to your patient data, and what liability they are willing to accept when something goes wrong. In a clinical setting, a vendor should also be able to provide a Business Associate Agreement (BAA). A BAA is required under HIPAA whenever a vendor handles protected health information on your behalf. If a vendor cannot or will not sign one, that is not a minor compliance gap. It is a signal that the tool was not built with clinical deployment in mind. Beyond the paperwork, ask directly: How does this system respond to expressions of distress or crisis? What data leaves our organization and where does it go? Who is responsible when something goes wrong? A vendor that cannot answer those questions clearly is a vendor whose system does not belong in a clinical setting, regardless of how useful it appears administratively.

If your organization is working through how to build this governance structure in practice, that is exactly what we help with at Wayde AI.

The Standard That Was Always There

State AI laws are drawing lines around AI in clinical settings. But those lines are not new ideas. They are legal expressions of accountability principles that have always existed in healthcare and psychology.

You are responsible for the care your patients receive. AI does not change that. What changes is how much harder it is to maintain that responsibility when a system operates faster than any human can monitor, at scale, across populations your clinicians will never personally see, and in some cases as a third party whose reasoning you cannot audit and whose failures you cannot fully explain.

The lawsuits referenced in this post share something in common beyond the hard they describe. In each of them, an AI system was doing something that looked like clinical care, listening to someone in distress, responding to their pain, shaping what they believed about themselves and their options, with no licensed professional anywhere near the interaction. That is not augmentation. It is substitution. And it is exactly what states are beginning to prohibit.

The question your organization needs to answer is not whether you are compliant with the law in your state today. It is whether you can account for what every AI system in your workflows is actually doing, whether a clinician is close enough to the process to catch it when it goes wrong, and whether you know the difference between the tools you control and the third parties you have invited into your clinical relationships.

Staying informed doesn’t have to mean hours of reading. The Wayde AI Brief is a short weekly intelligence brief for healthcare and mental health leaders navigating real-world AI adoption, governance, and risk. Subscribe for free.

Frequently Asked Questions

Why are states regulating AI in mental health specifically?

Mental health care involves vulnerable populations, licensed professional relationships, and high-stakes clinical decisions. When AI systems operate in that space without adequate oversight, the consequences can be severe and immediate. States are responding to documented harms, including multiple deaths linked to AI chatbot interactions, and are drawing explicit legal boundaries around what AI can do without human professional supervision.

What is the difference between administrative AI and clinical AI?

Administrative AI supports the operational work around clinical care: scheduling, documentation drafting, billing, and record management. It does not interact directly with patients or influence clinical decisions. Clinical AI operates closer to the patient relationship. It influences diagnosis, treatment, therapeutic interaction, or patient decision-making. States are increasingly regulating these categories very differently, and so should your governance program.

What does it mean for an AI system to function as a third party rather than a tool?

When an AI system processes patient data in ways that are not visible to your organization, produces outputs whose reasoning cannot be audited or explained, or operates between your clinicians and your patients without meaningful human oversight, it is functioning as a third party. It has its own terms of service, its own data practices, and its own liability limitations. Your organization remains professionally accountable for what happens in that interaction, even though you do not control or fully understand how the system arrived at its output. That is the governance gap that most organizations have not yet addressed.

Does using an AI vendor's tool reduce my organization's accountability for patient care?

Not in any meaningful sense. Professional accountability in healthcare follows the clinician and the organization, not the tool. When something goes wrong in a clinical setting, the first questions will be directed at the licensed professionals and the organization responsible for care, regardless of what technology was involved. Vendor contracts and terms of service may define legal liability differently, which is exactly why reading them carefully and ensuring a Business Associate Agreement is in place matters. But professional and ethical responsibility is not something a vendor agreement transfers away.

Should we stop using patient-facing AI until regulations are clearer?

Not necessarily, but you should be able to clearly articulate what oversight is in place for every patient-facing AI system you use. If you cannot answer basic questions about how a system responds to patient distress, who reviews its outputs, what data leaves your organization, and what happens when it fails, that system needs a governance review before it continues operating in a clinical context.

About the Author

Dr. Ernest Wayde is the Founder and Principal of Wayde AI, a healthcare AI ethics consulting firm. He works with healthcare and behavioral health organizations on responsible AI adoption, governance, risk management, and implementation strategy.

References

[1] OpenAI. (2025, October 27). Strengthening ChatGPT's responses in sensitive conversations. https://openai.com/index/strengthening-chatgpt-responses-in-sensitive-conversations/

[2] Raine v. OpenAI complaint (August 2025). https://www.courthousenews.com/wp-content/uploads/2025/08/raine-vs-openai-et-al-complaint.pdf

[3] CNN. Parents of 16-year-old Adam Raine sue OpenAI (August 26, 2025). https://www.cnn.com/2025/08/26/tech/openai-chatgpt-teen-suicide-lawsuit

[4] CBS News. Google and Character.AI settle lawsuit linked to teen suicide (January 8, 2026). https://www.cbsnews.com/news/google-settle-lawsuit-florida-teens-suicide-character-ai-chatbot/

[5] American Psychological Association. AI wellness apps cannot solve mental health crisis (November 13, 2025). https://www.apa.org/news/press/releases/2025/11/ai-wellness-apps-mental-health

[6] Stateline. AI therapy chatbots draw new oversight as suicides raise alarm (January 15, 2026). https://stateline.org/2026/01/15/ai-therapy-chatbots-draw-new-oversight-as-suicides-raise-alarm/

[7] Behavioral Health Business. AI therapy chatbots will fuel the next teen mental health crisis (April 22, 2026). https://bhbusiness.com/2026/04/22/like-drinking-salt-water-ai-therapy-chatbots-will-fuel-the-next-teen-mental-health-crisis/

[8] NBC News. OpenAI denies allegations that ChatGPT is to blame for a teenager’s suicide (November 26, 2025). https://www.nbcnews.com/tech/tech-news/openai-denies-allegation-chatgpt-teenagers-death-adam-raine-lawsuit-rcna245946

[9] Shumate JN, Rozenblit E, Flathers M, et al. Governing AI in Mental Health: 50-State Legislative Review. JMIR Mental Health. 2025 Oct 31;12:e80739. doi: 10.2196/80739. https://mental.jmir.org/2025/1/e80739

[10] Common Sense Media and Stanford Medicine’s Brainstorm Lab. AI Risk Assessment: AI Chatbots for Mental Health Support (November 20, 2025). https://www.commonsensemedia.org/press-releases/common-sense-media-finds-major-ai-chatbots-unsafe-for-teen-mental-health-support

[11] Illinois General Assembly. Wellness and Oversight for Psychological Resources Act (HB 1806). Signed August 1, 2025. https://ilga.gov/Legislation/BillStatus?DocNum=1806&GAID=18&DocTypeID=HB&LegId=159219&SessionID=114

[12] Nevada Legislature. Assembly Bill 406, 83rd Session (2025). Signed June 5, 2025, effective July 1, 2025. https://www.leg.state.nv.us/App/NELIS/REL/83rd2025/Bill/12575/Overview

[13] California Legislature. AB 3030, Artificial Intelligence in Health Care Services Bill. Signed September 28, 2024, effective January 1, 2025. https://leginfo.legislature.ca.gov/faces/billNavClient.xhtml?bill_id=202320240AB3030

[14] California Legislature. AB 489, Health care professions: deceptive terms or letters: artificial intelligence. Signed October 11, 2025, effective January 1, 2026. https://leginfo.legislature.ca.gov/faces/billTextClient.xhtml?bill_id=202520260AB489

[15] Colorado General Assembly. Senate Bill 24-205, Consumer Protections for Artificial Intelligence Act. Signed May 17, 2024, effective June 30, 2026. https://leg.colorado.gov/bills/sb24-205

[16] HIPAA Journal. (2026). HIPAA business associate agreement .https://www.hipaajournal.com/hipaa-business-associate-agreement/