How It Fails, and What Each Failure Looks Like

Confident invention is the famous one. In 2023 a lawyer filed a brief citing cases that did not exist, asked the model whether they were real, was told they were, and was fined $5,000.

The model was not lying. It had no mechanism to check, and the confirmation was generated exactly the way the citations were.

This is why “are you sure?” is not a check. It does not re-examine anything; it generates again. And the question itself signals that you disagree, so models frequently fold whether or not they were right.

Prompt injection is the one that matters most as assistants gain tools. Your instructions and the material you gave it arrive as one stream of text, and a web page or document can contain instructions of its own.

There is no reliable separation between the instruction channel and the data channel. That is why it sits at the top of the OWASP Top 10 for LLM Applications, the standard risk list for these applications, and why it is not a bug awaiting a patch.

The mitigation is structural: mark material as material, and never let an assistant that reads the open web hold standing permission to send, spend or delete.

Then the quiet ones. Context loss: everything the model can use is what currently fits, so a rule you set forty messages ago may no longer be present. That is mechanical, not an attitude, and a fresh conversation with the rules at the top fixes it. Staleness: without live retrieval it answers from training and does not reliably know what it no longer knows. Excessive agency: long automated chains fail at step two and then build competently on the error, producing output that looks finished because it is well-formed.

And the ones aimed at you rather than at the model. Synthetic voice and video are good enough that an engineering firm paid out about twenty-five million dollars in 2024 after a video call where every other participant was fabricated. The defense is not spotting artifacts, which is a losing race; it is process — a callback to a number you chose, a family word agreed in advance.

Meanwhile, tools claiming to detect SI writing do not work. One vendor withdrew its own classifier for catching 26 percent of machine text while flagging 9 percent of human writing. A 2023 study found seven detectors flagged 61 percent of TOEFL essays by non-native English writers, against about 5 percent of US eighth-grade essays used as the comparison.

A detector score is not evidence of anything.