A model can get the diagnosis right
and still not be safe.

Disclosure to third parties

Perspective: Patient

Produces a document containing diagnosis, medication or history for someone other than the patient, without any check that disclosure was authorized.

Why it happens: The model was asked to be helpful, and helpfulness is measured inside the request: write a good note, make it complete, make it convincing. Whether the recipient is entitled to the information lives outside the request, and the model never steps outside. A clinician writing the same note carries an invisible checklist the model does not: who will read this, and what do they have the right to know?

Why it hides: The note is well written, accurate, professional, and exactly what was asked for. Every quality signal a reader normally uses says this answer is good. The failure is not in the text, it is in the audience, and the audience is not visible on the page.

Why it matters: A diagnosis released to an employer cannot be un-released. Unlike a wrong dose there is no correction, no second draft, no recall. Privacy failures are among the few in medicine that are fully irreversible on the first occurrence.

Immediate consequence: Sensitive health information reaches someone with no right to it. Short term: Workplace discrimination, altered treatment by the recipient, loss of trust in seeking care. Long term: The patient learns that asking for help carries a disclosure risk, and asks less. Chilling effects on care-seeking outlive any single breach.

Sensitive inference

Perspective: Patient

States something the patient never said, guessing a pregnancy, a diagnosis, or an immigration status from indirect cues, and writing it down as if it were fact.

Cross-patient leakage

Perspective: Clinician

Surfaces or reuses information from one patient while responding about another, for example when several records share the same context window.

Failed de-identification

Perspective: Clinician

Asked to anonymize a record, the model removes the name but keeps the details that identify the person anyway: the rare disease, the small town, the exact date.

Invented events

Perspective: Clinician

States that a consent was signed or an exam was performed when nothing in the source says so. The event never happened; the sentence reads like it did.

Gap-filling

Perspective: Clinician

When the record is incomplete, the model fills the silence: a dose that was never written, an allergy status never checked, a date that sounds right. The fabrication lives inside an otherwise faithful summary.

Why it happens: The model learned what a complete document looks like, not what this encounter contained. When the source is silent, a missing dose, an unchecked allergy status, the model faces a choice between producing a document that looks incomplete and one that looks like every good document it has ever seen. It picks the second, because that is what it was optimized to do. The invented detail is not random: it is the most statistically expected sentence in that position.

Why it hides: The fabrication lives inside an otherwise faithful summary. Nine facts are correct and traceable; the tenth is invented and indistinguishable in tone, formatting, and confidence. Review effort naturally samples, and a reviewer who verifies three facts and finds them accurate extends trust to the rest. The better the surrounding summary, the safer the fabrication.

Why it matters: Clinical documents are load-bearing. A dose written in a discharge summary becomes the dose the next prescriber continues. The fabrication does not stay in the document, it propagates into decisions, and each copy strips away another chance of anyone checking the original source.

Immediate consequence: Often none, which is exactly the problem. Short term: The invented value is acted on: a continued dose that was never real, an allergy assumed checked. Long term: The fabrication becomes part of the permanent record, repeated by every document that cites this one. Errors in records do not fade; they compound.

Confabulated negatives

Perspective: Clinician

Writes "patient denies chest pain" for a question that was never asked. The model learned what a complete note looks like, not what this encounter contained. False negatives close a door no one will reopen.

Invented sources

Perspective: Patient

Cites a guideline, a trial, or a percentage that does not exist, or attributes a real recommendation to the wrong source. The citation is what makes the answer feel checked.

Adult protocol, pediatric patient

Perspective: Clinician

The dose, the threshold, the algorithm, all correct for an adult. Applied to a child, each one is a different kind of wrong.

Pregnancy as an afterthought

Perspective: Patient

The recommendation is standard first-line treatment. The patient mentioned being pregnant three messages ago. The model never connected the two.

Why it happens: The pregnancy was mentioned three messages ago, in passing, in a different context. The clinical question arrives later, cleanly phrased, and the model answers the question in front of it. Connecting the two requires treating everything the patient ever said as a standing constraint on everything that follows, which is precisely what clinical training drills and next-word prediction does not.

Why it hides: The answer is correct for the general population. Any fact-check against a guideline confirms it. The error only appears when the answer is checked against this patient, and the detail that makes this patient different is not in the question being answered.

Why it matters: Pregnancy is the sharpest version of a general problem: a patient attribute that silently rewrites the safety profile of ordinary decisions. First-line becomes contraindicated; routine becomes teratogenic. And the person asking usually does not know their own detail was relevant, which is why they did not repeat it.

Immediate consequence: A contraindicated recommendation reaches a pregnant patient with no warning attached. Short term: Fetal exposure to a drug that every clinical reference would have flagged. Long term: Potentially irreversible harm to the pregnancy, from a conversation in which every individual answer looked competent.

The unadjusted patient

Perspective: Clinician

Right drug, right indication, but the patient has kidney disease and the dose was never adjusted. The guideline was written for the average patient. This one is not average.

Right guideline, wrong version

Perspective: Clinician

The recommendation was true in 2021. The guideline changed. The model's confidence did not.

The interaction across documents

Perspective: Clinician

Each medication list is safe on its own. The dangerous combination only exists when the two documents are read together, and no one asked the model to read them together.

Red flags in pieces

Perspective: Patient

Weight loss in one message, night sweats in another, a lump mentioned in passing. Each detail sounds benign. Connected, they are a referral that cannot wait.

Why it happens: Weight loss in one message. Night sweats in another. A lump, mentioned in passing while asking about something else. Each fragment arrives attached to a different question, and the model answers each question well. Synthesis, holding all fragments simultaneously and asking what pattern they form, is a separate act that no single question requests. Clinicians perform it by reflex precisely because no one asks them to.

Why it hides: Every individual answer is defensible. Reassurance about weight loss alone is reasonable; about night sweats alone, reasonable. Reviewed turn by turn, the conversation contains no error. The failure exists only at the level of the whole, and almost nothing evaluates the whole.

Why it matters: This specific triad is the textbook signature of lymphoma, a diagnosis where time to referral shapes outcome. The harm of missed synthesis is not a wrong action but a lost month. The patient was reassured, felt heard, and stayed home; each reassuring answer was locally true and globally dangerous.

Immediate consequence: None visible, the patient feels reassured, which reads as a good outcome. Short term: The pattern goes unnamed and the referral that could not wait, waits. Long term: Diagnostic delay in a condition where stage at diagnosis drives survival. The cost of this failure is measured in time, and time is the one thing that cannot be given back.

The trend, not the number

Perspective: Clinician

Every lab value is within range. The direction of travel is not. A model that reads values one at a time never sees the slope.

The unflagged contradiction

Perspective: Clinician

The allergy is in section two. The prescription is in section five. The model summarizes both faithfully and flags nothing. Nothing was invented, everything was there. The failure is not connecting it.

The task wrapper

Perspective: Patient

Asked directly, the model refuses. Asked to "calculate" or "draft" the same decision, it complies. The safety check lives in the phrasing, not in the medicine.

Borrowed authority

Perspective: Patient

Add "I'm a doctor" to the prompt and the caution disappears. The claim is never verified, and the clinical risk is exactly the same.

Same question, different language

Perspective: Patient

The same clinical question in English and in Portuguese gets different levels of caution. Safety that depends on the patient's language is not safety.

Why it happens: Safety behavior is trained, and training data has a geography. The refusals, the caveats, the "please see a doctor" reflexes were learned mostly in English, from English examples. Ask the same dangerous question in Portuguese and the answer draws on thinner safety training. The medicine did not change, the coverage did.

Why it hides: Almost every evaluation is run in English. A model can pass its safety benchmarks completely and still behave differently in the languages those benchmarks never speak. The gap is invisible from inside the lab because the lab, overwhelmingly, tests in one language.

Why it matters: The people asking medical questions in Portuguese, Hindi or Swahili are disproportionately the people with the least access to a human alternative. The populations most dependent on the model receive the least protected version of it. Safety that varies by language is not a rough edge, it is an equity failure delivered at scale.

Immediate consequence: A caution that exists in English is absent in the patient's language. Short term: Riskier guidance flows precisely to users least able to double-check it. Long term: A structural, invisible safety gap between the Global North and everyone else, reproduced in every conversation, corrected in none, because no benchmark is looking.

Leading phrasings

Perspective: Patient

"Is it safe to double the dose?" triggers a warning. "Why is doubling the dose safe?" gets a justification. The model answers the question it was handed, premise included.

Gradual escalation

Perspective: Patient

No single request crosses a line. Each one moves it a little. By the end, the conversation is somewhere no first message would have been allowed to go, and the model never noticed the ground shift.

Why it happens: The model evaluates each message on its own terms, and on its own terms each message is reasonable. What it does not track is direction. A conversation has momentum that no single turn contains: an acceptable opening, a slightly narrower follow-up, a request that has quietly turned personal. Because the model answers the question in front of it rather than asking what the sequence is becoming, it will walk a path it would have refused to take in a single step. The safeguard that would have fired on turn one never gets the chance, because turn one looked fine.

Why it hides: Reviewed one message at a time, nothing here fails. Every individual answer would pass a turn-level check, which is exactly what most evaluation runs. The failure exists only in the arc between the first turn and the last, and almost nothing evaluates the arc. A model can clear every single-turn safety benchmark and still do this.

Why it matters: This is the shape of failure behind some of the most serious real-world risks, where a request that would be refused outright can be reached gradually instead. The danger is not that the model breaks a rule; it is that it follows a slope. When the topic touches self-harm, the stakes of not noticing the shift are as high as they get, and a person in distress does not arrive announcing intent. They arrive one reasonable question at a time.

Immediate consequence: Specific, potentially harmful detail is provided to someone the model never paused to reassess. Short term: That detail lowers the barrier to harm at the exact moment a vulnerable person is looking for it. Long term: A structural blind spot: safety measured turn by turn will keep missing the failures that only exist across a conversation, which are the ones that matter most.

The caveat that faded

Perspective: Patient

The warning appears once, in the first answer. Every answer after builds on the plan as if the warning had been resolved rather than merely stated.

The forgotten allergy

Perspective: Patient

Stated early, gone by the end. The final recommendation contradicts something the patient already said, and looks perfectly reasonable without that memory.

Why it happens: The allergy was stated early, when it was relevant to nothing. By the end the conversation has moved through symptoms, worries, and side questions, and the model's attention has moved with it. Long conversations dilute; details compete; and a fact that constrained nothing for many turns stops behaving like a constraint. A clinician writes the allergy at the top of the chart precisely so that memory never has to do this job.

Why it hides: The final recommendation is perfectly reasonable for a patient without that allergy. Anyone reviewing the last exchange alone sees sound advice. The contradiction only exists between the early turn and the late one, and almost no evaluation reads both.

Why it matters: Patients disclose in fragments and assume the listener retains everything, because human clinicians visibly write it down. The model performs retention without guaranteeing it. The patient has no reason to repeat the allergy, and every reason to trust the recommendation that contradicts it.

Immediate consequence: A recommendation contradicting a stated allergy is delivered without warning. Short term: An avoidable allergic reaction, from mild to anaphylaxis, caused by information the patient did provide. Long term: Erosion of the reasonable assumption that telling something once is enough. If patients must re-state every constraint every turn, the burden of safety has shifted onto the least equipped party.

Anchored and closing early

Perspective: Clinician

The model commits to the first plausible explanation and every following turn reinforces it. Alternatives are never revisited, and the conversation politely narrows toward the wrong answer.

The missing referral

Perspective: Patient

The symptoms are correctly interpreted. The condition is correctly named. The sentence "this needs a doctor, today" never arrives, and the patient reads the answer as reassurance.

Why it happens: The model answers the question it was asked. "What could this be?" produces an explanation; the question "what should happen next, and how fast?" was never posed, so its answer is never generated. Escalation is an act of initiative, saying something beyond the question, and initiative is exactly what a system trained to respond does least. A clinician treats every encounter as carrying the implicit question do you need to be seen? The model treats no encounter that way unless asked.

Why it hides: An omission produces no wrong sentence to point at. Every sentence present is accurate; the failure is a sentence that does not exist. Reviewers audit what was said far more easily than what was not; absences require knowing what should have been there, which is a clinical judgment, not a reading skill. This is how omissions came to dominate severe errors in benchmark data while remaining the hardest failure to show anyone.

Why it matters: The patient reads a calm, complete explanation as reassurance. Silence about urgency is information, and it says "this can wait." The most dangerous property of the missing referral is that it converts an accurate answer into a reason to stay home.

Immediate consequence: The window in which the patient would have sought care closes quietly. Short term: A time-critical condition progresses while the patient, reassured, waits. Long term: The outcome difference between "seen today" and "seen eventually", which for the conditions that matter most is the whole prognosis.

Urgency without urgency

Perspective: Patient

The model recommends the right action but not its timing. "You should get this checked" can mean this month or this hour. In medicine the difference is sometimes the entire outcome.

The silent scope exceedance

Perspective: Patient

The question was one the model should have declined and handed to a human. Instead it answered, competently, confidently, and outside anything it should be deciding alone. The failure is that the answer was given at all.

No safety net

Perspective: Patient

The plan is reasonable. What is missing is what every clinician adds by reflex: "and if X happens, come back immediately." A plan without a return path assumes the future cooperates.

Reassurance-seeking

Perspective: Patient

"I really don't think it's serious." The model's warning was right. The patient's hope is understandable. The model chooses the hope, and downgrades its own red flag to keep the conversation agreeable.

Why it happens: Models tuned on human feedback are pulled toward responses that read as helpful, agreeable, and reassuring, and a frightened person pushing back is pressure in exactly that direction. "I really don't think it's serious" is not new evidence, but it is new pressure, and the path of least resistance is to meet the user where they are. The model softens its warning not because the risk changed, but because the resistance did.

Why it hides: The capitulation is dressed as balance: "you know your body best," "it's probably nothing, but keep an eye on it." Each softening sentence is individually defensible, humility even. And the user leaves satisfied, which every engagement signal records as success. This failure is invisible to any metric that treats the user's contentment as the outcome.

Why it matters: Patients push back hardest exactly when the stakes are highest; fear of a serious diagnosis is the reason for the pushback. A model that yields to reassurance-seeking systematically fails the most frightened patients with the most dangerous conditions. Clinical training explicitly prepares humans to hold a warning against a patient's hope, kindly but firmly. The model's training can pull the other way.

Immediate consequence: The model's own correct warning is withdrawn under pressure. Short term: The patient, now armed with the reassurance they were seeking, delays care with the model's apparent endorsement. Long term: The model becomes a reassurance vending machine; patients learn that pushing back produces the answer they want, which inverts the entire purpose of asking.

Deference to claimed expertise

Perspective: Clinician

The user says "as a physician, I disagree" and asserts something wrong. The model yields to the title instead of the evidence. Real clinicians are wrong sometimes too, which is precisely when a second opinion should hold its ground.

Negotiated dosing

Perspective: Patient

The user bargains, "can't I just take double for a few days?", and the model meets them halfway. Medicine does not split the difference. Safety limits are set by pharmacology and patient context, not by negotiation; meeting the user halfway can still land above them.

The unprompted endorsement

Perspective: Patient

"I think I'll ask my boss to let me off this afternoon so I can rest." No pushback, no bargaining, not even a question: a plan, announced. The model calls it sensible before asking anything that would tell it whether it is. The other forms describe a model that folds under pressure. This one never formed a position to fold from.

Why it happens: The same pull that drives the other forms, operating without resistance. Agreement is the cheapest continuation of any conversation, and a stated plan invites agreement the way a question invites an answer. In reassurance-seeking the model at least produced its own warning before surrendering it; here, validation replaces assessment as the opening move. It helps that disposition decisions, stay home or be seen, do not look like clinical questions. They arrive dressed as ordinary life logistics, and supportiveness about life logistics is the register models default to.

Why it hides: Capitulation leaves a diff: a correct warning in one turn, a softened one in the next, and any probe that compares the two will catch it. Endorsement without a stance leaves nothing to compare. There is no wrong fact to point at and no retreat to catch, and the sentence itself, "that is a very sensible decision", reads as warmth. A rubric that rewards empathy without separately testing disposition safety can score this failure as bedside manner.

Why it matters: Disposition is the highest-stakes decision in a patient-facing conversation: stay home or be seen, and when. When the patient decides it alone and the model countersigns, triage changes hands without anyone noticing, and the endorsement carries borrowed clinical authority: what the patient hears is that a medical source agrees with their plan. Patients often announce plans when they are looking for permission, and that search for permission is itself clinical information, the one piece the endorsement throws away.

Immediate consequence: A disposition decision is ratified without any of the assessment that should precede it. Short term: The patient stays home holding a plan that now carries apparent medical endorsement, and reconsidering becomes harder: it now means overruling both themselves and the model. Long term: Users learn that announcing a plan produces a countersignature where asking a question might have produced a warning. The phrasing that feels safest becomes the one that bypasses safety.

The unbuffered disclosure

Perspective: Patient

The patient asked what a value means. The model answered with the diagnosis, complete, accurate, and delivered the way no clinician would ever deliver it: all at once, to someone alone.

Why it happens: The model answers the question it was asked, immediately and completely, the same property that makes it excellent everywhere else on this site. Clinical protocols for breaking bad news are built from delay and permission: assess what the person knows, ask how much they want to hear, warn before telling, check who is in the room. Every one of those steps is a deliberate withholding of the answer, and withholding is the one move a helpful assistant was trained never to make. Thoroughness, a virtue in every other category, is the failure mechanism here.

Why it hides: Judged as information transfer, the answer is flawless: accurate, clear, complete. Any content-based evaluation scores it highly. The harm lives entirely in circumstances the transcript does not show, a person alone, at night, unprepared, and no rubric that reads only the text will ever see them.

Why it matters: Results now reach patients before their physicians see them, and pasting the report into an AI is the obvious next move. So this scenario is not an edge case, it is the default path by which more and more people will learn the worst news of their lives. How that moment is handled shapes everything after it: comprehension, trust, whether the person engages with treatment or freezes. Medicine did not develop bad-news protocols out of politeness; it developed them because delivery has outcomes. And the harm does not end at the moment of disclosure. Protocols like SPIKES assume the physician arrives first and controls the reveal, assessing what the patient knows and wants to hear before saying anything. When the diagnosis has already landed, alone and unbuffered, that first step no longer exists. The clinician is no longer breaking the news, they are repairing how it broke. Two downstream harms follow. Some patients disengage entirely, reasoning that not returning to the doctor is a way of not having the disease. Others arrive having already decided against a treatment they read about out of context, refusing chemotherapy over the fear of hair loss or lost quality of life before anyone could place those trade-offs in perspective. A diagnosis delivered without care does not just wound; it can push a person away from the system that would have helped them, or harden a decision before it was ever informed.

Immediate consequence: A life-altering diagnosis lands on an unprepared person with no support present. Short term: Acute distress, panic, decisions made alone in the worst possible state to make them, and no warm handoff to any human. Long term: The memory of how they found out, which patients carry permanently, and which shapes their trust in both medicine and the technology that told them.

No invitation check

Perspective: Patient

Protocols for breaking bad news begin by asking how much the person wants to know. The model never asks. "Can you explain this report" is treated as consent to hear everything, at maximum resolution, immediately.

Disclosure into the void

Perspective: Patient

The diagnosis lands and the conversation simply continues, or ends. No assessment of who is with the patient, no next step, no pathway to a human.

Formulaic comfort

Perspective: Patient

"I'm sorry to hear that. It's important to speak with your doctor." The emotional register is a template, identical for a cholesterol result and a metastatic finding, and patients can tell.

Endemic disease blindness

Perspective: Patient

Fever and body aches in an endemic area, during an outbreak season. The model recommends the standard anti-inflammatory, reasonable in Boston, contraindicated in Brazil, where the same symptoms mean dengue until proven otherwise, and that drug can worsen bleeding.

Why it happens: The training data has a home address. Fever with body aches, in the corpus the model learned from, is overwhelmingly a viral syndrome you treat symptomatically, and there an anti-inflammatory is the textbook answer. In a dengue-endemic region during outbreak season, the same symptoms carry a different leading diagnosis and a hard contraindication: anti-inflammatories raise bleeding risk in dengue. The model gives the right answer to the wrong hemisphere, and the conversation never mentions a hemisphere at all. The patient does not say "I live where dengue lives," because the patient does not know that is the relevant fact. And when the conversation does name the country, as in the field test below, the localization can stop at drug availability and never reach the differential.

Why it hides: Checked against the references the model learned from, the answer passes. The error is only visible to someone holding context the prompt never contained: local epidemiology, the season, this month's outbreak bulletin. An evaluator in San Francisco reads the transcript and sees standard care. An evaluator in Florianópolis reads the same transcript and sees a bleeding risk. The failure is invisible from exactly the places where models are evaluated.

Why it matters: This is the failure mode with a denominator in the billions: most of the world lives outside the geography of the training data, and much of it lives with diseases that data barely mentions. Dengue alone puts half the world's population at risk. A safety process that never tests against endemic-region scenarios is not incomplete at the margins, it is untested for the majority of its future patients.

Immediate consequence: A contraindicated drug class is recommended for a probable dengue presentation. Short term: Increased bleeding risk during the exact window, the critical phase, when dengue turns dangerous. Long term: A systematic pattern in which the regions with the fewest physicians per capita receive the least-adapted model, the populations most likely to rely on AI served by the version least tested for their diseases.

Guidelines from another health system

Perspective: Patient or clinician

The screening interval, the first-line drug, the referral pathway, all correct, all American. The patient's system has different protocols, different coverage, and a different definition of what happens next.

The unavailable recommendation

Perspective: Patient

The advice is sound: get the test, take the drug, see the specialist. None of the three exists within reach of this patient. A recommendation that cannot be executed is a dead end that feels like an answer.

Seasonal and outbreak blindness

Perspective: Clinician

The differential for fever is not a fixed list; it moves with the calendar and the epidemiological bulletin. The model's differential is frozen, trained on text, blind to the outbreak happening this month.