AI Text Detectors: How They Work and Whether They Actually Work

⏱️ Reading time: 16 min

In July 2023, OpenAI shut down one of the first AI text detectors it offered to the public, barely six months after launching it. The company admitted the tool failed too often to be useful in production.

📑 En este artículo

1. TL;DR
2. What Are AI Text Detectors?
3. Why It Matters When They Fail
4. How They Work Under the Hood: Perplexity and Burstiness
5. Supervised Classifiers: The OpenAI and Turnitin Approach
6. Watermarking: The Mark Google DeepMind Hides in Every Token
7. Going Deeper: DetectGPT and Probability Curvature
8. Comparing the Four Approaches
9. How to Get Started: Measure a Text’s Perplexity in Python
10. What Studies Say About the Accuracy of GPT Detectors
11. Common Mistakes and How to Evade a Detector
12. Real-World Use Cases
13. Frequently Asked Questions
    1. How reliable are AI text detectors?
    2. Can a GPT detector flag text written by a person as AI?
    3. How does watermarking work for text generated by models like Gemini?
    4. Does paraphrasing work to evade an AI-generated text detector?
    5. Should universities use AI detection tools to penalize students?
    6. Is there a reliable AI text detector for Spanish?
14. References

Days ago, Martin Fowler’s essay “LLM Voice” reopened the discussion about how to recognize prose written by a model, but through human ears rather than software. That difference matters because thousands of schools, media outlets, and recruiting companies already delegate that judgment to automated tools, even though almost none of them publish their real hit and error rates.

TL;DR

• GPT detectors measure perplexity and burstiness: how predictable each chosen word is.
• OpenAI pulled its own AI classifier in July 2023 over its declared low accuracy.
• A Stanford study found that detectors flag more than half of non-native speakers’ essays as AI-generated.
• SynthID, from Google DeepMind, marks every generated token with an invisible statistical pattern.
• Paraphrasing with another model is enough to sink the accuracy of almost any trained classifier.

What Are AI Text Detectors?

AI text detectors are programs that estimate, from a piece of text, the probability that a language model wrote it rather than a person. They don’t read meaning: they measure statistical patterns such as how predictable each word is, the variation between sentences, or a hidden mark inserted when the text was generated.

There are at least four families of these tools: classifiers trained on labeled examples, training-free statistical heuristics, methods based on probability curvature, and watermarking (a digital watermark) inserted at the moment the text is generated. Each measures something different, and none actually analyzes the content: they all look at form.

Why It Matters When They Fail

An AI text detector isn’t a minor technical formality. Its result can determine whether a student has to repeat a course, whether a freelance journalist loses a contract, or whether a candidate gets dropped from a job opening before a person ever reads their resume.

In 2023, dozens of universities in the United States activated Turnitin’s AI detection feature just weeks after its launch, before any independent studies existed on its real accuracy. When an automated tool decides who cheated, any margin of error translates directly into unfair penalties for someone who never used a language model.

How They Work Under the Hood: Perplexity and Burstiness

Most statistical detectors start from a simple idea: a language model always prefers the most probable word at each position, while a person deviates from that predictability all the time. That deviation is measured with perplexity, a number that grows the more the text surprises the reference model.

Consider the sentence “the sky is clear today.” After “the sky is,” the most probable word for almost any model is “clear” or “cloudy”: both concentrate nearly all the probability. If the text does say “clear,” the model is barely surprised. A person, on the other hand, throws in quirks, typos, and personal references that no model would anticipate with the same probability, and that sustained lack of surprise sentence after sentence is exactly what a human text can hardly maintain.

Burstiness adds a second signal: the variation of that surprise between consecutive sentences. Human text alternates highly predictable sentences with completely unexpected ones. Generated text, by contrast, keeps a more even perplexity from start to finish because the model never takes too big a risk on any single word.

flowchart TD
A["Input text"] --> B["Tokenizer"]
B --> C["Reference language model"]
C --> D["Calculates perplexity and burstiness"]
D --> E["Classifier or statistical threshold"]
E --> F["Score: human or AI"]

That calculation needs no training: having access to a language model that serves as a measuring stick is enough. That’s the approach used by Binoculars, a zero-shot tool published in 2024 that compares the perplexity of the same text under two different models instead of training a separate classifier.

Supervised Classifiers: The OpenAI and Turnitin Approach

The other path is to train an AI-generated text classifier directly: take thousands of human texts and thousands generated by models, label them, and let a network learn to tell them apart. That’s how the classifier OpenAI published in January 2023 and pulled barely six months later worked.

The company itself reported the problem in its original announcement: its classifier correctly identified only 26% of AI-generated texts as “likely AI,” while flagging 9% of human-written texts as AI. That false positive rate turned out too high for serious use, and OpenAI chose to discontinue it rather than keep offering it at that accuracy.

Turnitin and GPTZero follow a similar logic, though neither publishes the full detail of its architecture. What they do share is the underlying weakness: a classifier learns the patterns of the texts it was trained on, and loses accuracy as soon as the real text strays from that set, whether because of topic, language, or the author’s own style.

There’s also a permanent maintenance problem. Every time a new model comes out (GPT, Gemini, Claude, Llama), its writing style shifts a bit, and a classifier trained on the previous generation starts losing accuracy against the new one. Keeping a classifier up to date requires retraining it constantly, something very few providers do as often as needed.

Watermarking: The Mark Google DeepMind Hides in Every Token

There’s a third path that doesn’t analyze the text’s style after it’s written, but marks it from the moment it’s generated. SynthID, from Google DeepMind, biases the choice of each token with a pseudorandom pattern derived from a cryptographic key: invisible to the reader, but reconstructible by whoever holds that key.

sequenceDiagram
participant M as Generating model
participant K as Watermarking key
participant D as Detector
M->>K: requests pattern per token
K-->>M: returns sampling bias
M->>D: delivers the generated text
D->>K: recalculates the expected pattern
D-->>D: statistically compares the received text

The advantage over a classifier is that watermarking doesn’t need to guess anything: it compares the received text against the pattern the model should have produced if it used that key during generation. The disadvantage is just as direct: it only works if the provider activated the mark when generating that specific text, and nobody can detect it in text from a model that never implemented it.

There’s no direct way to verify from outside whether a specific text carries a SynthID mark without access to the provider’s private key: Google offers its own verifier inside its products, but doesn’t publish a public method for auditing the pattern without going through that tool.

Robustness also has limits. A marked text that gets rewritten by hand or paraphrased with another model can lose enough statistical signal that the verifier no longer tells the pattern apart from noise, especially if the rewrite changes a good portion of the original words.

Going Deeper: DetectGPT and Probability Curvature

There’s a fourth approach, slower but conceptually distinct: instead of comparing perplexities, perturb the text and measure how its probability changes. DetectGPT, described by Mitchell and his team in 2023, generates dozens of minor variations of the same text, for example swapping words for synonyms, and asks the source model to calculate the probability of each variant.

The intuition is that text generated by a model usually sits at a local probability maximum under that same model. Almost any perturbation causes that probability to drop noticeably. Human text doesn’t sit at that maximum, so its perturbations raise or lower the probability more evenly. That difference in curvature is the signal DetectGPT uses to decide, without ever training a classifier.

flowchart LR
A["Suspect text"] --> B{"Is a trained classifier available?"}
B -->|"Yes"| C["Supervised classifier (OpenAI, Turnitin)"]
B -->|"No"| D["Perturbs the text with synonyms"]
D --> E["Compares original vs perturbed probability"]
E --> F["High curvature suggests AI origin"]

The cost is computational: evaluating dozens of perturbations per text multiplies inference time several times over compared to a classifier that runs a single pass. Binoculars solves part of that problem by comparing the text’s perplexity under two differently sized models in a single pass, without perturbations, though it sacrifices some of the accuracy that DetectGPT’s full curvature approach offers.

Comparing the Four Approaches

No method is superior in every scenario: each requires a different kind of access (to the source model, to a private key, or just to the text) and fails in a different way.

How to Get Started: Measure a Text’s Perplexity in Python

Before installing anything, you’ll need Python 3.10 or higher and about 2 GB of free disk space to download the reference model. Install the dependencies with:

pip install transformers torch

With that, you can already calculate the perplexity of any sentence under GPT-2, the same type of reference model several zero-shot detectors use:

import torch
from transformers import GPT2LMHeadModel, GPT2TokenizerFast

tokenizer = GPT2TokenizerFast.from_pretrained("gpt2")
model = GPT2LMHeadModel.from_pretrained("gpt2")

def perplejidad(texto):
    ids = tokenizer(texto, return_tensors="pt").input_ids
    with torch.no_grad():
        salida = model(ids, labels=ids)
    return torch.exp(salida.loss).item()

print(perplejidad("The cat climbed onto the roof to watch the sunset."))

That script tokenizes the sentence, asks the model to predict each word from the previous ones, and converts the average loss into a number: perplexity. On a laptop without a GPU, that script prints something like 42.17 for that sentence, a relatively low value because the model is barely surprised by almost any word. The exact number varies depending on the library version and the hardware, but the order of magnitude (tens, not thousands) stays stable.

The next step is to compare two sentences with different styles to see the signal perplexity-based detectors look for:

frases = {
    "informal": "idk if im gonna make it on time, the bus was super late",
    "generico": "It is important to note that, in the current context, it is essential to consider multiple factors."
}

for nombre, frase in frases.items():
    print(nombre, round(perplejidad(frase), 2))

A typical output highlights exactly the difference these detectors look for: something like informal 61.3 versus generico 24.8. The sentence loaded with formulaic connectors turns out far more predictable for the model than the informal sentence with slang and typos. That contrast is exactly what GPTZero and similar tools measure, though with larger reference models and additional adjustments that aren’t public.

💡 Tip: run the same script on a paragraph you wrote by hand and another one you asked a chatbot for, before trusting any commercial detector’s score on a text that actually matters to you.

What Studies Say About the Accuracy of GPT Detectors

Independent studies on AI text detectors are scarce, but they agree on one uncomfortable point: accuracy drops sharply outside the type of text each tool was trained or calibrated on.

The most cited case is that of Liang and his team at Stanford, published in 2023. The group ran TOEFL exam essays, written by non-native English speakers, through several commercial and open-source GPT detectors. The result: the detectors flagged more than half of those human essays as AI-generated, while almost never making mistakes with essays from native English-speaking students at the school level.

The technical explanation connects directly to perplexity. A non-native speaker tends to use simpler grammatical structures and more predictable vocabulary, exactly the pattern these detectors associate with AI text. The bias isn’t a one-off programming error, but a direct consequence of measuring predictability as a signal of origin.

Almost all of these studies were done in English. For Spanish, research is even scarcer: the few detectors with multilingual support tend to be extensions of the same classifier trained mostly in English, without recalibrating for Spanish grammar and vocabulary. That suggests the same bias against non-native English speakers could repeat, in a different direction, against Spanish-language texts these detectors analyze without any regional adjustment.

⚠️ Heads up: no AI text detector should be the only evidence used to penalize a person. Even the ones that report a low false positive rate in native English lose that edge with other writing profiles.

Common Mistakes and How to Evade a Detector

The most common mistake is treating a detector’s score as a verdict rather than a signal. A 90% “AI probability” isn’t proof: it’s a statistical measure with a margin of error that’s almost never published alongside the number.

The second mistake is underestimating how easy it is to evade a trained classifier. Running the text through a paraphraser, through another AI model, or simply rewriting a few sentences by hand breaks the statistical pattern the classifier learned to recognize. Zero-shot methods like DetectGPT are somewhat more resistant to this because they don’t depend on a fixed style, but they aren’t immune to aggressive rewriting either.

The third mistake, less talked about, is mixing human and AI text in the same document and expecting a binary result. Almost all detectors give an average score per paragraph or per full document, so a mixed text can produce an intermediate result that says nothing useful about either part.

The fourth mistake is ignoring the language of the analyzed text. A classifier calibrated almost exclusively on English and applied to Spanish, Portuguese, or any other language can produce results that don’t even fall within the accuracy range the tool originally reported.

Real-World Use Cases

In education, Turnitin offers its AI detection feature built into the same workflow universities already used to detect plagiarism, which made its mass adoption easier despite the lack of independent data on its accuracy at launch.

In journalism and editing, some outlets use these detectors as a first filter for columns from outside contributors, not as a final verdict. A high score triggers a manual editorial review, never an automatic rejection of the piece.

In hiring, some recruiting platforms have tested AI content detection systems to flag automatically generated cover letters, though the practice is controversial: it penalizes someone who used a model to polish their grammar just as much as someone who used it to invent an entire work history they never had.

In content moderation, some platforms are considering flagging comments or reviews mass-generated by AI, a use case different from the academic one because the goal there isn’t to penalize a person, but to stop automated spam campaigns or reputation manipulation.

Your next step: run this article’s perplexity script on a paragraph written by hand and another generated by a chatbot, and compare the numbers before trusting any commercial detector’s score.

Frequently Asked Questions

How reliable are AI text detectors?

They vary a lot depending on the method and the type of text. The classifier OpenAI published in 2023 correctly identified only 26% of AI texts as such and flagged 9% of human texts as AI. Independent studies show even lower accuracy outside native English.

Can a GPT detector flag text written by a person as AI?

Yes, and more often than its providers suggest. The 2023 Stanford study found that several detectors flagged more than half of real essays written by non-native English speakers as AI.

How does watermarking work for text generated by models like Gemini?

The model biases the choice of each word following a pattern derived from a cryptographic key during generation. A verifier with that same key can compare the received text against the expected pattern and estimate whether it matches.

Does paraphrasing work to evade an AI-generated text detector?

In most cases, yes. Rewriting the text, running it through another model, or editing it by hand breaks a good part of the statistical pattern a trained classifier learned to recognize, though probability curvature methods hold up somewhat better.

Should universities use AI detection tools to penalize students?

The studies on the subject themselves recommend not using them as the sole evidence. The documented false positive rate, especially against non-native English speakers, means an isolated score isn’t enough to justify a penalty.

Is there a reliable AI text detector for Spanish?

There are no broad independent studies confirming one. Most available tools were trained and calibrated primarily in English, so their accuracy in Spanish still isn’t well documented.

References

• Martin Fowler, “LLM Voice”: the essay that reopened the discussion about recognizing AI-generated prose by ear.
• OpenAI: announcement and subsequent withdrawal of its AI-generated text classifier in 2023.
• Liang et al., “GPT detectors are biased against non-native English writers”: the Stanford study on false positives in non-native speakers’ essays.
• Mitchell et al., “DetectGPT”: the paper that introduces zero-shot detection through probability curvature.
• Google DeepMind: documentation on SynthID, the statistical watermark for AI-generated text and images.
• Turnitin: official page of the academic integrity software that incorporates AI detection.

📱 Like this content? Follow @programacion on Telegram for daily tech content in Spanish: quick summaries, fresh content every day.

Featured image: Foto de Harshit Suryawanshi en Unsplash