The novel ‘Where the water Works’ won the first Almadia Ventosa-Arufat International Short Novel Prize in July, ahead of more than 2,200 submissions from 22 countries and worth €20,000. At its core, a very clear condition: Participating implies that the work was not written in part or in whole with AI tools. In case of any indication or doubt, the work will be disqualified irrespective of the stage of the process.
Well, after its publication in September and the full presentation in Spain, the Peruvian writer David Silva Rojas, the publisher and the jury of the competition (who appreciated his use of silence and its construction) were scandalized: El Pais revealed that, after passing the entire book through the pangram detector, 9% of the prize-winning AI text No. 9.
What is Pangram and what does it promise?. First, the presentation: Pangram is a text classifier based on neural networks that tries to distinguish whether the text is human-generated or a language model. Under the Hood, a training technique designed to reduce false positives and, according to its authors, does not discriminate against non-native English speakers, uses mirror prompts and is understood by category.
Pangram Labs, the company behind it, and is mainly interested in its detector being reliable and trustworthy, reporting 0.0041% false positives and 0.34% false negatives. An independent study by the University of Chicago compared Pangram to other commercial detectors and came to a conclusion: Pangram is less likely to classify human text as AI.
Identification method. The El País test doesn’t come out of nowhere and doesn’t just use ‘Where the water works’: as Kiko Leneras (data engineer and one of the authors of the article) says
Their control method has three controls: They used 56 books from 2019 – 2020, i.e. before ChatGPT ( other BC), which gave 0% AI, AI-generated synthetic text identified by detectors such as Javier Cercas, Isabel Allende, Antonio Muñoz Molina, etc., by established authors, all of which gave 0%. Commitment to calibration.
shadow of doubt. To paraphrase Hitchcock’s film, it’s still a good idea to be suspicious of an AI detector. There is serious evidence that shows the limits of the detector: if a human trial is passed through a consumer AI and changes significantly, Pangram labels them as human 41% of the time. And a reality of the sector: OpenAI withdrew its detector after admitting its low success rate, and although it has another one that is theoretically very good, it is not launching in the market.
The newspaper itself admits that the pangram is not perfect and, furthermore, that anyone can get a detector and try to tickle it. Angel L. Fernández Requero explains step-by-step how a simple added space in a text changes judgment. Famous architect and populist Pedro Torrijos also tested it last month, when doubts arose over the fashionable French-language novel, leaving him at ground level based on a simple handwritten text. It is true that the text is about a page long (with a small sample, it is difficult to identify), but it is better to read his examination carefully, changing a few words.
Pedro Torrijos’ thread doesn’t get lost: from 100% human to 100% AI by changing just two sentences
underlying problem. A percentage, even 99%, is not proof of authorship. Although this obviously raised suspicions and at least initiated an investigation and requested a draft of the project. After all, these systems estimate the probability that a text was created with AI, but they do not say which role the person played and which machine: the work may have been written or edited by the AI, or the AI may have come up with the plot or characters.
There is an extreme case that is not machine-man, but man-man: Agassi’s memoirs, written by journalist JR Mohringer after 250 hours of interviews over three years and signed by the former tennis player. The question is not whether there is AI, but how much and how it is used. Of course, one thing is certain: the rules of the competition leave no room for doubt. There’s something dystopian about the subject: it’s an AI that determines whether a job is done with AI, and its judgment acts like the sword of Damocles: a false positive can destroy a reputation, but a false negative costs nothing.
In Xataka | A researcher is tired of seeing text written by AI and has developed a detector that promises 98% accuracy
In Xataka | Anthropic explains how Claude’s new watermark works: it can’t be seen, it can’t be noticed, it can’t (almost) be erased, and it doesn’t prove anything.
cover chatgpt
