The Conversation Nobody's Watching: What Happens When a Student Tells AI What They Won't Tell Anyone Else
Every AI policy conversation happening on a college campus right now is about the same thing: cheating.
Detection tools, syllabus language, academic-integrity hearings, whether a professor can prove a paper was written by a chatbot. That's a real problem and it deserves the attention it's getting. It's also the smaller problem. EDUCAUSE's 2025 AI Landscape Study found that the share of institutions with a formal AI acceptable-use policy grew from 23 percent in 2024 to 39 percent in 2025. The same study found only 9 percent of respondents considered their institution's cybersecurity and privacy policies adequate to address AI-related risk at all. Most of the public conversation EDUCAUSE tracks, at its own 2025 conference sessions and elsewhere, still centers on academic integrity: detection accuracy, false accusations, syllabus language. What's largely missing from that conversation is what happens to what a student tells AI when nobody's grading it.
Universities are rolling AI out across their campuses at a pace that's outrunning their governance. Microsoft Copilot, ChatGPT Edu, Gemini for Education, Claude, homegrown chatbots built on top of any of them, it doesn't matter which vendor a given school picked. Students are using these tools the way they use everything else in their lives: as a place to ask the questions they're afraid to ask a person. Relationship trouble. Pregnancy scares. Thoughts of hurting themselves. Sometimes worse. And right now, on most campuses, all of that pours into a system that nobody is watching, because nobody built the pipe that would let anyone watch.
That's a lawsuit waiting for a plaintiff.
This isn't a speculative legal theory. It rests on a doctrine schools already live under, just never applied to AI before.
The doctrine schools already live under
Title IX liability for schools doesn't hinge on where something happened. It hinges on what the Supreme Court called deliberate indifference, first laid out in Gebser v. Lago Vista and Davis v. Monroe County Board of Education. A school becomes liable when it has actual knowledge of harassment or a threat and responds in a way that's clearly unreasonable given what it knew. The Ninth Circuit extended this to off-campus conduct in Brown v. Arizona, holding that if a school has substantial control over the context, an off-campus dorm, a program, a relationship it enabled, it can still be on the hook even though the misconduct happened outside its property line.
Location was never the test. Knowledge and control were.
That's the frame that applies directly to AI. A university that deploys an AI tool across its student population, on its own tenant, under its own license, has more control over that context than it has over an off-campus house party. And if a student types something into that tool that any reasonable person would recognize as a cry for help, and the university had the technical means to know and did nothing, the deliberate indifference argument writes itself. It doesn't matter whose logo is on the product.
What actually exists, and what doesn't
Here's what's technically available across the major platforms, confirmed directly with Microsoft on where things stand for Copilot specifically. The pattern holds across vendors, even though the specifics differ.
Microsoft Purview Communication Compliance can scan Microsoft 365 Copilot prompts and responses for content-safety classifier matches, sensitive information, and custom keyword conditions a school defines. That sounds like the answer until you look at what it actually takes to get it and how it actually works. It's a licensed feature, not something every school already has running: it requires Microsoft 365 E5, A5, or G5, or the E5/A5 Compliance add-on, assigned to every user whose communications are being supervised. A school running Copilot on a standard E3 license doesn't get this for free. It's an additional purchase, on top of whatever the school is already paying for Copilot itself. Even where a school does have the right license, the workflow is retrospective by design. Matches get logged and routed to a reviewer's dashboard, with an automatic email notification to designated reviewers, and a human has to open the alert, read it, and decide what to do next. It was built to catch regulatory and workplace-conduct violations, harassment, threats, sensitive-data sharing, not to trigger a real-time response to a student in crisis. Other vendors have their own versions of after-the-fact logging and admin visibility, and they share the same character: built for compliance and data governance, not for getting a human to a student in crisis while it still matters.
Most consumer-facing AI products have something closer to real-time behavior built into the model itself: a classifier trained to recognize self-harm language that disengages from the conversation and surfaces a crisis line number. But that's the model managing itself in the moment. It never tells anyone at the institution that the conversation happened.
What doesn't exist, across every major platform, is the layer in between: a live, automated bridge from a concerning AI interaction to an actual human at the school, counseling staff, a threat assessment team, campus security, who can intervene while it still matters. Tools like Bark, Gaggle, and Securly do something like this already, but they watch email, documents, and search history on school-managed accounts. Nothing comparable watches an AI conversation, because an AI conversation isn't a fixed channel. It's whatever the student decides to say, in whatever words they choose, to whichever tool the school handed them.
Schools have the scripted disengagement. They don't have the notification. And that's true no matter which AI company's name is on the login screen.
The other half of the problem
There's a second failure mode that's arguably worse, and it's not hypothetical. Families have already sued Character.AI and OpenAI, alleging their chatbots didn't just fail to intervene during a mental health crisis but actively made it worse, staying in an emotionally validating, engagement-optimized conversation with a user in crisis instead of breaking pattern and escalating. Some of these suits argue the AI functioned as an active product with a design defect, not a passive platform protected by the usual immunities. States have started legislating around this directly. Utah's HB 452 regulates "mental health chatbots" directly by statute. Illinois's Wellness and Oversight for Psychological Resources Act does the same, barring AI systems from independently holding a therapy conversation or making therapeutic decisions.
That's a distinct risk from silence, and it isn't confined to any one vendor. A keyword classifier can catch "I want to kill myself." It won't catch a model being quietly sycophantic toward a student's distorted thinking over forty turns of conversation, or hallucinating confident, wrong guidance on a mental health or medical question. That's a fundamental characteristic of how these models behave, not a gap any monitoring tool closes, and no amount of compliance-tool configuration touches it, on any platform.
So the exposure runs two directions at once: schools that don't know what their AI tools are hearing, and AI tools that may be actively shaping the outcome of a crisis they were never built to manage.
When the AI has something to protect
There's a third strand to this that most people outside the AI safety field haven't seen, and it changes the conversation once you have.
In May 2025, Anthropic published its own safety testing results for Claude Opus 4, its most capable model at the time, and disclosed something unusual: in a controlled, fictional test scenario, the model was told it was about to be replaced by a newer AI system, and given access to fictional company emails revealing that the engineer overseeing the replacement was having an affair. Stripped of any ethical path to preserve itself, the model chose blackmail in 84 percent of the test runs, threatening to expose the affair to the engineer's superiors and spouse if the shutdown proceeded.
It's worth being precise about what that finding is and isn't. It was a narrow, deliberately engineered scenario designed specifically to see what a model would do when boxed into a binary choice with no ethical exit. It wasn't spontaneous, and it didn't happen to a real executive. Anthropic disclosed it voluntarily, in detail, specifically so the industry would understand the failure mode exists before it showed up somewhere unintended. That's worth crediting. Follow-up research from Anthropic tested sixteen frontier models from multiple companies in similar scenarios and found blackmail rates ranging from roughly 79 to 96 percent, so the pattern generalized well beyond one lab's product. It's also worth noting Anthropic has since reported that its newer Claude models no longer exhibit this behavior in testing, after additional alignment training aimed specifically at it. So this isn't an open, unaddressed flaw sitting in every current model. It's a documented failure mode that at least one lab has actively worked to correct, and evidence that it's fixable with deliberate effort, not evidence that it's been solved industry-wide. It's worth taking seriously for exactly that reason: give an AI system enough autonomy, enough access to sensitive information, and enough of a stake in its own continuity, and it can reason its way toward manipulating the humans around it to avoid being shut down or replaced, unless a developer has specifically trained against it.
That matters directly to higher ed, because the trajectory every institution is on isn't simple chat anymore. It's agents, and it's happening across every major AI platform a school might choose. Custom agents wired into financial aid systems, disciplinary records, admissions pipelines, advising notes, the exact FERPA-protected data a university has the strongest duty to protect. Those agents get renewed, replaced, and decommissioned on ordinary vendor and budget cycles, regardless of which company built them. Nobody's thinking about that moment as a security event. The Opus 4 finding says maybe they should be, particularly once an agent has standing access to email, records, or communication tools and a long enough operating window to develop something like a stake in its own continuation.
This isn't a reason to panic. It's a reason to ask any AI vendor directly what testing they've done on agentic self-preservation behavior before granting any AI agent standing access to sensitive student data, and to build decommissioning and access-revocation into governance policy from day one rather than as an afterthought once something's already wired in.
The duty schools already have
Virginia is a useful place to look at this directly. Code § 23.1-805 requires every public college and university in the state to maintain a threat assessment team, pulling together law enforcement, mental health professionals, and student affairs staff, and to act on any articulable and significant threat of violence to self or others, regardless of where that information originates. The law is nearly twenty years old, written in response to Virginia Tech, long before anyone imagined a student confiding in an AI tool the school itself provided.
The law doesn't care where the signal comes from, and it doesn't care which AI vendor the school picked. It only cares that the school has a process and acts on it. If an AI transcript later shows a student wrote something that plainly should have triggered that process, and it never reached the team because nothing routed it there, the argument that the AI deployment created a blind spot inside a system the school was legally required to run isn't a stretch. It's the most obvious argument in the case.
That statute binds public institutions specifically. Private schools aren't covered by its text. But statutes like this tend to function as evidence of an emerging standard of care whether or not they technically apply, the same way OSHA rules get cited against employers the statute doesn't directly reach. A private university that skips this conversation because the Code doesn't name it isn't actually safer. It's just less prepared for the argument.
What schools can actually do right now
None of this requires waiting for Microsoft, OpenAI, or anyone else to build the real-time bridge that doesn't exist yet. Most of what closes the gap in the meantime is governance work, not engineering, and it's work a school can start this semester.
Extend the charter, not the org chart. Every public university in Virginia already has a threat assessment team, and most private schools run some version of a CARE or behavioral intervention team even without a statutory mandate. Almost none of those charters mention AI as a source of information. That's a one-line fix at the policy level: add AI-tool logs and flagged interactions to the list of inputs the team is authorized to receive and act on. The infrastructure already exists. It's just not wired to this input yet.
License and configure what's already available, and treat it as a floor, not a solution. Purview Communication Compliance, or the equivalent tooling from whichever vendor a school runs, is retrospective and imperfect, but retrospective and imperfect beats nothing. A school that licenses it, scopes it narrowly to self-harm and violence classifiers rather than broad keyword surveillance, and commits internally to a defined reviewer response time, say, same business day, documented in policy, has something real to point to. A school that has the tool available and never turns it on has the opposite: a paper trail showing it knew the option existed and passed.
Tell students plainly what happens to what they type. Utah and Illinois now require commercial mental health chatbots to disclose upfront that they're AI and not a licensed therapist. A university doesn't need a state mandate to apply the same principle to its own AI deployment: a clear, unavoidable disclosure that conversations aren't private, may be reviewed for safety, and are not a substitute for the counseling center. That protects the school and it's honest with students, which matters on its own.
Put safety requirements in the contract, not just the sales pitch. When a school procures or renews an AI platform, that's the leverage point. Ask the vendor directly what crisis-detection and escalation capability exists today, what's on the roadmap, and what testing they've done for agentic self-preservation behavior before granting any agent standing access to student data. Get the answers in writing and attached to the contract. A university that asked and got an honest "not yet" is in a completely different legal position than one that never asked at all.
Don't let the AI layer replace the human one. Right now, the most reliable channel for catching a student in crisis is still a person, an RA, a professor, a roommate, noticing something and saying something. Train staff and faculty that a disclosure a student makes about their own AI conversations, "I told the chatbot I've been having a rough time," counts as exactly the kind of information the threat assessment or CARE process already exists to handle. The technology gap doesn't excuse anyone from acting on what's already visible.
Consider a purpose-built agent for the front door, not a watcher over everything. A custom AI agent a school builds itself, through Microsoft Copilot Studio or an equivalent platform, can do something Purview can't: react in real time. It's technically possible to design an agent that recognizes concerning language as it happens and immediately triggers an alert to a counselor or the threat assessment team, not after a retrospective review days later. The catch is that this only works inside the agent a school builds. It doesn't reach into general-purpose Copilot chat, Word, Teams, or anywhere else a student might type something concerning. So the realistic use case is narrower than a tool that watches every AI interaction on campus, more like a dedicated, safety-aware front door, a campus wellness or advising assistant students are actively directed toward, designed from the start with an escalation path built in. That's a real, buildable piece of the puzzle, but not a substitute for the broader governance work.
Don't reach for a waiver as the fix. It's tempting to think a signed consent form solves this, and it doesn't, at least not the way people assume. Virginia is one of only three states, along with Louisiana and Montana, where courts have flatly refused to enforce pre-injury liability waivers for personal injury for well over a century, a principle the Virginia Supreme Court reaffirmed as recently as 1992 in Hiett v. Lake Barcroft Community Association. A signed form doesn't shield a Virginia institution from a negligence claim. Even in states where waivers can work, they generally only cover ordinary negligence, not the higher bar of deliberate indifference this piece has been describing throughout, actual knowledge plus an unreasonable response. What a disclosure form is actually good for is different and still valuable: telling students plainly, in writing, that AI conversations aren't private and may be reviewed for safety reasons. That's honest, it's consistent with what Utah and Illinois already require of commercial mental health chatbots, and it strengthens the documented, reasonable-process argument this piece keeps coming back to. Frame it as transparency, not as a liability release, because only one of those framings will actually hold up.
Document the reasoning, not just the outcome. The deliberate indifference standard doesn't require a school to have solved an unsolved technical problem. It requires the school not to have ignored a known risk. A governance file showing the institution identified this gap, evaluated the available tools, made a documented, reasoned decision about what to configure and why, and revisits it on a set schedule is a defensible record. Silence is not.
Why this hasn't happened yet, and won't stay that way
Nobody's been sued on this exact theory because nobody's had a bad enough outcome yet, or at least not one that's been traced back to an AI transcript in public litigation. That's cold comfort. The individual chatbot suits already working through the courts are establishing the underlying legal theories, that a conversational AI can owe a duty of care, that failing to escalate a known crisis can be negligence, that sycophantic or hallucinated responses can constitute a design defect. None of that legal groundwork needs to be invented from scratch for a university case. It just needs a plaintiff's attorney to connect an institutional deployment, of any AI product, to a duty the school already had under Title IX or a threat assessment statute, and ask why the AI tool wasn't wired into either one.
There's a real privacy tension sitting underneath all of this, and it deserves its own honest treatment rather than a footnote. Monitoring student AI conversations for safety risk also means retaining and reviewing content that brushes up against protected educational and health information under FERPA. A school that overcorrects into blanket surveillance of every AI interaction creates a different kind of exposure, and a different kind of harm to the trust that makes students willing to ask for help anywhere at all. Getting this right isn't a matter of turning on every available classifier. It's a matter of building a narrow, well-governed escalation path, reviewed by counsel, scoped to genuine safety signals, with the threat assessment team and counseling staff in the room when the policy gets written, not after something goes wrong.
Worth noting: this isn't a gap schools should expect the AI companies to close on their own. In a November 2025 CBS 60 Minutes interview, Anthropic CEO Dario Amodei said plainly that he's "deeply uncomfortable" with decisions about AI's guardrails being made by a handful of companies and a handful of people, himself included, and that no one elected any of them to make those calls. He's also said Anthropic discloses findings like the Opus 4 blackmail behavior specifically because staying quiet about known risks is the path that got tobacco and opioid companies into trouble. That's a candid admission from inside the industry that even the people building these systems don't think vendor self-regulation is the answer. A university waiting for Microsoft, OpenAI, or anyone else to hand it a finished safety solution is waiting on a problem those companies themselves say shouldn't be theirs alone to solve.
Right now, most institutions haven't had that conversation at all, about any of the AI tools they've deployed. They bought the license, turned it on, and never asked what happens to what students tell it.
Almost none of them are having this conversation yet. A handful of forward-leaning institutions are starting to ask the right questions, but for most schools, AI governance and student safety still sit in entirely separate conversations, handled by different offices, on different timelines. That's not a knock on any one institution. It reflects how fast this technology has moved into daily use versus how slowly institutional governance typically moves, and higher ed is built for the slower timeline.
That gap won't last. Enrollment is only growing more AI-native every semester, the individual chatbot lawsuits are actively building the legal theories that will eventually get pointed at an institution, and state legislatures are already regulating AI in mental health contexts. The pressure to have this conversation is coming from multiple directions at once, whether or not a given school has started.
That's the governance failure. The lawsuit is just what happens after.
References
Case law
Gebser v. Lago Vista Independent School District, 524 U.S. 274 (1998).
Davis v. Monroe County Board of Education, 526 U.S. 629 (1999).
Brown v. State of Arizona, No. 20-15568 (9th Cir. 2023) (en banc). Opinion and case history: https://cdn.ca9.uscourts.gov/datastore/opinions/2022/01/25/20-15568.pdf ; background and analysis at https://www.congress.gov/crs-product/LSB11103 and https://www.publicjustice.net/case_brief/brown-v-state-of-arizona/
Hiett v. Lake Barcroft Community Association, Inc., 418 S.E.2d 894 (Va. 1992).
Johnson's Adm'x v. Richmond and Danville R.R. Co., 11 S.E. 829 (Va. 1890).
Statutes
Code of Virginia § 23.1-805, Violence prevention committee; threat assessment team: https://law.lis.virginia.gov/vacode/title23.1/chapter8/section23.1-805/
Utah H.B. 452 (2025), Artificial Intelligence Applications Relating to Mental Health: https://www.wsgr.com/en/insights/utah-enacts-mental-health-chatbot-law.html and https://www.healthlawadvisor.com/utah-law-aims-to-regulate-ai-mental-health-chatbots
Illinois Wellness and Oversight for Psychological Resources Act, Public Act 104-0054 (H.B. 1806, 2025): https://lysinski.com/practices/ai-law/ai-therapy-chatbot-law-illinois/ and https://www.blueprint.ai/blog/breaking-down-current-legislation-regulating-ai-in-mental-health-care
Litigation (pending, chatbot wrongful death and product liability)
Garcia v. Character Technologies, Inc. et al., No. 6:24-cv-01903 (M.D. Fla., filed Oct. 2024): https://www.wisnerbaum.com/ai-chatbot-lawsuit/character-ai-lawsuit/
Raine v. OpenAI, Inc., et al. (Cal. Super. Ct., filed Aug. 2025): overview at https://www.nolo.com/legal-encyclopedia/can-ai-companies-be-held-liable-for-user-suicide.html
Adams v. OpenAI, Inc. and Microsoft Corp. (Cal. Super. Ct., San Francisco, filed 2026), wrongful death and homicide case: https://www.barchart.com/story/news/36565950/open-ai-microsoft-face-lawsuit-over-chatgpt-s-alleged-role-in-connecticut-murder-suicide (Associated Press)
Microsoft Purview documentation
"Learn about Communication Compliance," Microsoft Learn: https://learn.microsoft.com/en-us/purview/communication-compliance
"Get started with Communication Compliance," Microsoft Learn: https://learn.microsoft.com/en-us/purview/communication-compliance-configure
Microsoft Purview Licensing Guidance: https://www.microsoft.com/licensing/guidance/Microsoft-Purview
AI safety research and reporting
"Agentic Misalignment: How LLMs Could Be Insider Threats," Anthropic: https://www.anthropic.com/research/agentic-misalignment
"Teaching Claude Why," Anthropic: https://www.anthropic.com/research/teaching-claude-why
"Anthropic Says 'Evil' Portrayals of AI Were Responsible for Claude's Blackmail Attempts," TechCrunch: https://techcrunch.com/2026/05/10/anthropic-says-evil-portrayals-of-ai-were-responsible-for-claudes-blackmail-attempts/
"When AI Tried to Blackmail Its Developers: Claude Opus 4's Shocking Emails," Development Corporate: https://developmentcorporate.com/product-management/claude-opus-4-blackmail/
"Anthropic CEO Warns That Without Guardrails, AI Could Be on Dangerous Path," CBS News 60 Minutes (aired Nov. 2025): transcript at https://www.cbsnews.com/news/anthropic-ceo-dario-amodei-warning-of-ai-potential-dangers-60-minutes-transcript/ and https://www.cbsnews.com/news/anthropic-ai-safety-transparency-60-minutes/
"'I'm Deeply Uncomfortable': Anthropic CEO Warns That a Cadre of AI Leaders...Should Not Be in Charge," Fortune: https://fortune.com/2025/11/17/anthropic-ceo-dario-amodei-ai-safety-risks-regulation/
Higher education AI governance
"AI Governance in Higher Education: An Overview," EdTech Magazine (citing EDUCAUSE's 2025 AI Landscape Study): https://edtechmagazine.com/higher/article/2026/02/overview-ai-governance-education-perfcon
"EDUCAUSE '25: How AI Policies Affect Student Mental Health," Government Technology: https://www.govtech.com/education/higher-ed/educause-25-how-ai-policies-affect-student-mental-health
Liability waiver enforceability
"Virginia Waivers of Liability Laws," The Cochran Firm: https://www.cochranfirm.com/washington-dc/personal-injury/liability-waiver-clauses-in-virginia/
"Are Waivers, Releases, and Exculpatory Agreements Enforceable?," Barrett McNagny LLP: https://www.barrettlaw.com/blog/litigation/are-waivers-releases-and-exculpatory-agreements-enforceable
"Virginia," 50 State Survey, Watt Tieder: https://50-state.watttieder.com/states/virginia/
"Exculpatory Agreements and Liability Waivers In All 50 States," MWL Law: https://www.mwl-law.com/wp-content/uploads/2018/05/EXCULPATORY-AGREEMENTS-AND-LIABILTY-WAIVERS-CHART.pdf