Do AI Detectors Like Pangram, ZeroGPT, Grammarly Actually Work?

AI is everywhere now, from trip planners to fantasy football tools.

That leaves every writer I know with the same low-grade fear. You write something yourself, paste it into an AI detector, and a little gauge tells you a robot did it. So I built a test with a real control, five samples, and three popular detectors to see which ones could tell me apart from a machine.

The Experiment

I started with a review of R.F. Kuang’s Katabasis. I wrote the review in December of 2025 for The Quill To Live (and wouldn’t you know it, I have hundreds more reviews there, too!). The review involved no AI tools for any writing, proofreading, or editing. 

The original review served as my control copy. From there, I enlisted Claude to produce four additional iterations of the piece. I told Claude to ignore my history and habits so it could come to the task completely clean. I had it generate four samples:

• Light AI edit: Claude proofread my review, fixing only typos and minor phrasing issues.
• Heavy AI edit: Claude rewrote sections of my review in its own words, keeping my structure.
• Pure AI: Claude wrote its own Katabasis review from scratch and no human intervention.
• AI + human edits: I took that AI review and reworked it by hand with new subheads, jokes and my own phrasing.

I then ran each sample through Pangram 4.1, ZeroGPT, and Grammarly's AI check as identical plain text.

Pangram: Can the New Leader Get It Right?

Pangram made big promises with its latest model. The company says Pangram 4 has a false-positive rate of 0.0041% on internal benchmarks, roughly one false flag per 24,000 documents, and that it can separate AI-edited writing from mixed human-AI writing. Those are Pangram's own numbers, so I treated them as a dare.

Cue the slow clap. Pangram heard my dare and said “Hold my beer.” My control came back 100% human. The pure AI review came back 100% AI. 

Pangram really came in hot with the heavy edit. It called the piece a mixed product of human and AI writing, then flagged it sentence by sentence with remarkable accuracy.

One soft spot: Pangram called the light edit fully human, with no AI-assisted flag. Claude changed only a handful of words, so I'd call that a reasonable result, but it means a light proofread can slip past the "AI-assisted" label.

How Do ZeroGPT and Grammarly Compare?

ZeroGPT and Grammarly handily cleared my control. ZeroGPT flagged 2.1% and Grammarly 4%. Then things got weird. Both tools highlighted the same passage of my conclusion, the bit where I said I was "caught somewhere between admiration and exasperation." I wrote that line myself, long before Claude knew my name. 

The pure AI review exposed the bigger gap. ZeroGPT scored it 56.7% and Grammarly scored it 56%. Both detected AI involvement, but identified only 56% of a review that was entirely machine-generated.

The heavy edit broke them. ZeroGPT said 4% and labeled it human. Grammarly said 9%. Claude rewrote most of my sentences, and two of three detectors waved it through.

They also contradicted each other. On the AI review I edited by hand, Grammarly's score went up to 65%, higher than the untouched AI version. ZeroGPT's dropped to 38.6%. Same kind of edit, opposite directions.

Can You Fool an AI Detector?

I didn't set out to write an evasion guide, and the results wouldn't be helpful there anyway. The direction of the edit mattered more than the amount. When AI polished my writing, ZeroGPT and Grammarly mostly saw a human. When I polished AI writing, every tool still saw a machine, and Pangram called it 100% AI.

My edits added jokes, new section names and a few lines of my own. However, I admit my edits were minimal in the grand scheme. That wasn't enough to fool a detector. Perhaps a heavier hand would yield different results. The AI's skeleton stayed intact: its sentence rhythm and its tidy section-by-section structure. Detectors seem to read the skeleton.

Why False Positives Matter

A detector missing AI text is a curiosity. A detector confidently calling a human's work AI-generated can cost someone a grade, a job or a client.

The companies know it. Turnitin no longer shows a number for AI scores between 1% and 19%, citing false-positive risk, and says an AI score should not be used as the sole basis for adverse actions against a student. OpenAI went further and retired its own AI text classifier in July 2023 for low accuracy. In its own testing, the classifier caught only 26% of AI-written text and wrongly flagged human text 9% of the time.

During my test, I experienced no false accusations beyond negligible flags on single lines. Still, Grammarly's label deserves a closer look. At 4%, the tool still told me my human writing "may be AI-generated." That's a soft warning, but a nervous teacher or editor could read it as a verdict.

So Can AI Detectors Actually Tell?

Some can, some of the time. 

Pangram was the clear winner, correctly identifying the human control, the fully AI-generated review, and the heavily edited mixed-authorship piece. It missed the light AI proofreading, but still outperformed every other product. ZeroGPT and Grammarly handled the easy calls and fumbled the hard ones, especially mixed authorship. The tools disagreed often enough that running two of them could leave you more confused than running one.

This was one writer and one book review, not a lab study, so I can't confirm Pangram's 0.0041% claim. What I can say is that I wouldn't trust any single score on its own, Pangram's included. Chatbots get plenty wrong, including some truly bogus travel advice. The machines built to catch them deserve the same skepticism.