Gemini Decision 1 can't separate AI voices made after 2024
Gemini Decision 1 is a multimodal decision model from Google released on November 11, 2026. It is analogous to Jev, the System One model from TypeSafe AI.
While browsing Hacker News, I stumbled upon What music does Gemini Decision 1 like? by Liam Warfield. He ends the article with a few ideas to extend his project. The last of these suggestions, “Does it [Gemini Decision 1] like or hate AI-generated music?”, stood out to me because it implicitly suggests that the decision model may be able to recognize AI-generated audio. As it turns out, Gemini Decision 1 can separate AI voices from human voices well above chance, but only if the AI voice generator was published before 2025.
As a quick clarification, I am saying separates rather than categorizes as the model has a strong bias towards labeling all tested audio as human-made. Out of 1,020 AI clips, only 20 were given a 50% or higher chance of being AI-generated. However, the ROC-AUCchance that a random AI clip is assigned a higher P(AI) than a random human clip was significantly higher in pre-2025 models, suggesting that recent AI voices are less distinguishable than before.
Results#
Each point is an AI voice generator, measured by how well Gemini Decision 1 separates 30 of its audio clips from 200 human audiobook clips.
When pooled together, generators from 2023 and 2024 are separated with an AUC of 0.67. Six older generators, from 2017 to 2022, score the same (0.67). However, generators released in 2025 and 2026 have an AUC of only 0.52, effectively chance. All but one of the 2025-2026 generators have a 95% confidence interval that includes 0.5; the only exception, Inworld TTS-2, has a confidence interval of 0.51-0.75.
Each clip was scored with a Simple Noulbinary question with output between 0 and 1, Criteria NoulSimple Noul with provided criteria for context, and 5-Pt Rubricrescaled score from “Definitely authentic human recording” to “Definitely AI-generated or synthetic”.
| Question format | 2017–2024 | 2025–2026 |
|---|---|---|
| Simple Noul | 0.59 | 0.46 |
| Criteria Noul | 0.67 | 0.52 |
| 5-Pt Rubric | 0.66 | 0.47 |
Notably, rather than gently sloping downward, we observe that AUC drops sharply at the end of 2024, or the start of 2025.
Inhuman performance?#
The research group behind MLAAD performed a 2026 listening study with 35,532 real-or-fake guesses from 1,768 people. The generators that fooled people also tended to fool Gemini Decision 1 (r = +0.57), but they do not always align. Griffin-Lim, a reference signal-processing method introduced in 1984, was only separated with an AUC of 0.53 (near chance), despite humans recognizing it 72% of the time. Cartesia Sonic-3 was the hardest for people (32% correct) and sits right at chance for Gemini Decision 1 (AUC 0.51).
Humorously, Gemini Decision 1 only achieved an AUC of 0.58 for Gemini 3.1 Flash TTS, suggesting it is only slightly better than chance at recognizing “its own” voice. At this point, neither man nor machine can reliably distinguish recent AI voices.
Dataset and normalization#
It’s easy to imagine a few scenarios where leaked metadata could have produced these results. As with all things in AI, I want to make sure I was measuring what I intended to measure.
The AI voice clips are sourced from MLAAD, a dataset of 143 English generators reading from the same set of audiobook sentences. For testing, I took 30 clips of 2.5-10 seconds from 34 generators. Each generator’s clips were tested against the same 200 human clips from 30 speakers in LibriTTS, a dataset of LibriVox audiobook recordings. All clips are clean audiobook narration.
Before sending the audio clips to the decision API, each clip was re-encoded with ffmpeg (-map_metadata -1 -vn -ac 1 -ar 24000 -t 15 -c:a libmp3lame -b:a 128k) to prevent metadata from affecting results. In addition, each clip was band-limited to 8 kHz (aresample=16000,aresample=24000), and loudness-normalized (loudnorm=I=-20:TP=-2:LRA=11).
All 34 generators#
| Generator | Released | Type | AUC | 95% CI | People correct† |
|---|---|---|---|---|---|
| Griffin-Lim | — | Reference | 0.53 | 0.40–0.66 | 72% |
| Tacotron 2 | Dec 2017 | Open | 0.64 | 0.52–0.75 | 68% |
| Glow-TTS | May 2020 | Open | 0.80 | 0.70–0.88 | 86% |
| FastPitch | Jun 2020 | Open | 0.62 | 0.50–0.74 | 75% |
| VITS | Jun 2021 | Open | 0.69 | 0.58–0.79 | 54% |
| SpeechT5 | Oct 2021 | Open | 0.72 | 0.60–0.82 | 79% |
| Tortoise | Apr 2022 | Open | 0.55 | 0.43–0.67 | 71% |
| Bark | Apr 2023 | Open | 0.66 | 0.54–0.77 | 71% |
| MMS-TTS | May 2023 | Open | 0.69 | 0.57–0.79 | 68% |
| Matcha-TTS | Sep 2023 | Open | 0.58 | 0.46–0.71 | 75% |
| XTTS v2 | Nov 2023 | Open | 0.64 | 0.52–0.75 | 68% |
| OpenAI TTS-1 HD | Nov 2023 | Commercial | 0.63 | 0.50–0.75 | 57% |
| MetaVoice-1B | Feb 2024 | Open | 0.79 | 0.68–0.87 | 73% |
| OpenVoice V2 | Apr 2024 | Open | 0.76 | 0.66–0.85 | 78% |
| Parler-TTS Large | Aug 2024 | Open | 0.69 | 0.56–0.79 | 58% |
| F5-TTS | Oct 2024 | Open | 0.69 | 0.58–0.79 | 83% |
| Kokoro | Dec 2024 | Open | 0.57 | 0.45–0.68 | 57% |
| Llasa-3B | Jan 2025 | Open | 0.53 | 0.41–0.65 | 56% |
| Zonos | Feb 2025 | Open | 0.46 | 0.34–0.58 | 61% |
| Spark-TTS | Feb 2025 | Open | 0.50 | 0.37–0.62 | 64% |
| Sesame CSM | Mar 2025 | Open | 0.50 | 0.38–0.61 | 57% |
| Dia | Apr 2025 | Open | 0.46 | 0.34–0.59 | 54% |
| Resemble AI | Apr 2025 | Commercial | 0.54 | 0.41–0.67 | 79% |
| Chatterbox | May 2025 | Open | 0.49 | 0.36–0.61 | 58% |
| ElevenLabs v3 | Jun 2025 | Commercial | 0.57 | 0.44–0.69 | — |
| IndexTTS-2 | Jun 2025 | Open | 0.55 | 0.43–0.67 | 66% |
| VibeVoice | Aug 2025 | Open | 0.53 | 0.40–0.66 | 57% |
| Cartesia Sonic-3 | Oct 2025 | Commercial | 0.51 | 0.38–0.63 | 32% |
| MiniMax Speech 2.8 | Jan 2026 | Commercial | 0.51 | 0.39–0.63 | — |
| Qwen3-TTS | Jan 2026 | Open | 0.51 | 0.39–0.63 | 52% |
| Fish S2 Pro | Mar 2026 | Open | 0.52 | 0.39–0.65 | — |
| Gemini 3.1 Flash TTS | Apr 2026 | Commercial | 0.58 | 0.46–0.69 | — |
| Inworld TTS-2 | May 2026 | Commercial | 0.64 | 0.51–0.75 | — |
| Higgs Audio V3 | Jun 2026 | Open | 0.52 | 0.41–0.64 | — |
AUC is Criteria Noul, 30 clips per generator against 200 LibriTTS clips; the interval resamples clips and LibriTTS speakers. †Share of correct guesses for that generator in the 2026 listening study (all clips).
Data#
AI clips: MLAAD (CC BY-NC 4.0). Human clips: LibriTTS (CC BY 4.0). Human listening results: Human Audio Deepfake Perception 2026 (CC BY-NC 4.0).