On October 23rd, El Pais revealed that Pangram Detector attributed 99% of AI to ‘Where the Water Works’, an award-winning game whose premise prohibits the use of AI. The author denied this and suspended submissions while the publisher reviewed the case Thélyson Orélien’s work ‘C’était ça ou mourir’ (‘It was that or die’), David Silva Rosa’s novel focuses on exploitation, pain and memory. It was chosen out of 2,204 manuscripts from 22 countries “for the subtlety of its writing and masterful use of silence as a compositional resource”.
In any case, it won the 20,000 euro Almadía-Ventosa Arrufat International Short Novel Prize hoped for and is already being translated. And several of his presentations have also been cancelled. Of course, we’re not going to decide here whether the novel is the result of AI. Because, in reality, it is difficult to value. The issue is another: in a few months, a private company program has become the de facto judge of literary writers. Is Pangram the referee we need?
What it does, and what it does well
Pangram is a neural network classifier trained on human text and model-generated text, with techniques to reduce false positives. It scores by chunks and provides percentages: an estimate of how much of a text a machine would type, not a reconstruction of how it was written.
According to the company itself, Pangram 4 recognizes 99.66% of AI text and 0.0041% human error, one error per 24,000 documents. And it’s not just the company that says this: a study by the University of Chicago put it above GPTZero and Originality.ai, and El País found that it paid 0% in books ahead of ChatGPT. In the case of Goncourt, it was later confirmed that the author used AI.
The version used by El Pais, Pangram 4.1, was released on October 19, four days before the analysis was published. In 29,103 pairs of texts with minimal editing, unwanted judgment changes dropped from 1.08% to 0.15%, similar to the problem indicated by the test for adding a comma or extra space (we already talked about that here). Additionally, it provides the probability of each label (human, AI-assisted, or AI-generated) and shows less certainty on texts shorter than 150 words.
The company estimates false positives of 0.009% in English and 0.0026% in Spanish, but it measures them on web text, not novels. False negatives increased slightly, from 0.34% to 0.37%. And the list itself admits that its error rate is not zero and that a simple mistake can cause reputational damage.
Isn’t that good?
Nature reports that, when an AI substantially alters human trials, Pangram labels them as human 41% of the time. It’s not hard to fool him: Jot Down also showed how an extra space changes judgment. A recent preprint maintains that, after going through a “humanizer,” false negatives exceed 96%, just the opposite of what the company declares (98.83% detection compared to 13 humanes). Studies never agree.
There are more subtle limits. Its CEO explained that the rate of false positives is measured with millions of texts before 2022. Meanwhile, it is suspected that many authors have learned to introduce AI into their articles and hide it better. Overall, it’s a good check for beginners, but it doesn’t tell you how you score a writer in 2026, after years of reading and correcting with models. And a percentage doesn’t reveal who did what: the AI drafted, revised or just suggested the plot. Furthermore, the error is not symmetrical, as we have already said: a false positive can destroy a reputation and a false negative costs nothing.
Arbitrator in commercial interests
Pangram Labs is a company founded by two colleagues in Brooklyn, and its business grows unbelievably: according to The Decoder, which cites the New York Times, its revenue has multiplied by 35 in a year and its monthly users have grown from 2,700 to 120,000 between June 2025 and June. They have the highest error rate of 2026. As your “confidence gap” decreases, your success seems to increase. But as Tim Reckworth points out, none of his models today are capable of understanding the layers and layers that real text goes through.
The Chicago study is independent (it’s funded by the university and a Google Cloud program, not Pangram), but GPTZero claims that, with the same criteria, it now outperforms Pangram. No standards: Each provider chooses the metric, threshold, and corpus. And no protocol either. The Goncourt case began with an anonymous account on X that was broken into pieces via Pangram. The academy asked for caution and to verify the reliability of the software. Today anyone can open a case without a certain threshold or right to reply, amid the “cursed AI” noise.
What if it goes to trial?
To date, there is no Spanish ruling regarding the authentic standards of these detectors. In criminal proceedings, admissibility depends on relevance and the judge freely evaluates the evidence. A Pangram report should come as an expert opinion, able to explain an expert’s methodology, margin of error, and submit to cross-examination.
Only there are three problems we have. It’s a neural black box that doesn’t explain why it identifies a phrase; The error rate is declared by the manufacturer; And their models change (Pangram 3’s withdrawal date was September 30), which makes it difficult to reproduce a verdict months later. Most likely, this will serve as an indication to request the draft, not in itself as prosecution evidence.
What do we demand from a referee?
If we need to judge what AI is, it should be reliable, auditable, legally valid and impartial. Pangram is by far the best rate detector. Still, it’s far from infallible. Their audits belong to the academic field, they are not supported by public and recurring audits with model access. Neither standard recognizes its legal validity and there is a conflict of interest as it is the company itself that protects its tools.
Europe, on the other hand, is moving forward. Article 50 of the AI Act obliges generative AI providers to identify their results in a machine-readable format (with a deadline of December 2, 2026 for systems already on the market), and the Commission published a code of good practice on June 10. But a watermark does not prove anything by itself and does not cover the open template or templates. There is a lack of an independent, public and European auditor to measure these detectors through open and comparative tests. Not even close to what we have.
Picture | Setup created using ChatGPT
In Xataka | Pangram saw 99% AI in the award-winning novel ‘Where Water Works’. The problem is the same as always: there are no infallible detectors
