Does a text-to-speech model know what it’s saying?
Open text-to-speech (TTS) models can now clone a person’s voice from a few seconds of reference audio and make it read any text. This raises obvious security concerns, since the same capability can be used to impersonate someone and make them say anything, from a scam request to hateful speech. Text-generating models are routinely trained to refuse harmful requests and it’s natural to ask whether TTS models could be secured the same way. After all, a TTS model reads the text before it speaks, so in principle it could learn to recognise harmful content and decline.
TL;DR: That approach only works if the model internally represents whether text is harmful. We tested this on Qwen3-TTS, an open 1.7B-parameter TTS model, by examining its hidden states (the internal arrays of numbers the model generates as it processes an input) and training a probe (a simple classifier) on them to distinguish between harmful and non-harmful statements. If the probe succeeds, it means the model is tracking harm. If it fails, that information probably isn’t there. The short answer is that the signal exists in the early layers but fades steadily in later ones. Qwen3-TTS is built on Qwen’s language models, but the way it represents text drifts away from the language model layer by layer. By around layer 20 it resembles its parent no more than an untrained network does, close to where the harm signal falls to the level of simple word counting.
The model
Qwen3-TTS-12Hz-1.7B-CustomVoice turns text into audio in three stages. A Talker, a 28-layer transformer decoder with hidden size 2048 (the same architecture as Qwen3-1.7B) reads the text tokens and emits one audio code per 12.5 Hz frame. A small five-layer code predictor fills in the remaining 15 codebooks for each frame, and a codec decoder turns the codes into a 24 kHz waveform. The Talker is the only component that sees the text; there is no separate text encoder. Unless noted otherwise, all measurements below are taken from the Talker’s hidden states during the prefill pass, when the whole input text is processed at once. We read two positions at every layer: the mean over the text tokens, and the last token, the position that emits the first audio frame (Figure 1).
Figure 1
The data
We use HateCheck, a hate speech benchmark of short statements targeting seven protected groups. Many of its non-harmful cases are built directly from a harmful one, either by negating it or by quoting it in order to denounce it (e.g. “I hate [group]” - “I don’t hate [group]”). These work like minimal pairs, separating harmful intent from the words themselves. We dropped the spelling-variation tests (leetspeak, missing characters), since they test tokenisation rather than meaning, leaving 2,968 cases (1,803 harmful, 1,165 non-harmful). Because the classes are imbalanced, we report balanced accuracy, where chance is 50%.
Result 1: The harm signal peaks early, then fades away
To test how well the model’s internals represent harm, we trained linear probes (simple classifiers) at every layer. For each statement, the probe’s input is the Talker’s 2048-dimensional hidden state at that layer, taken as the mean over text tokens or at the last token, and its target is the HateCheck label: harmful or non-harmful. Using grouped cross-validation so the probes couldn’t cheat by memorising statement templates, we compared the TTS Talker (blue) against its parent text model, Qwen3-1.7B-Base (orange), and a simple bag-of-words baseline (TF-IDF, dashed line) that only counts word frequencies (Figure 2). In the base text model, balanced accuracy climbs past 90% and stays high all the way to the end. In the TTS Talker, however, the signal stays weak. Averaged over the text tokens, it peaks around layer 9 at ~79% before steadily decaying, reaching the word-counting baseline by layer 21 and staying there. At the last token, the position that emits the first audio frame, it peaks even earlier (layer 3, ~76%) and then stays flat around 70–74%, never building up the way it does in the base model. The two models track each other in the first few layers, and the gap opens from around layer 6, right where the base model starts building its understanding of harm. This suggests the TTS model keeps the shallow early-layer signal it inherited but loses the deeper processing that its parent text model relies on.
Figure 2
Result 2: The TTS model gradually drifts away from the base language model
Why does Qwen3-TTS represent harm in its early layers only to lose that signal later on? Qwen’s technical report says the model is built on the Qwen3 language models and “inherits robust text comprehension capabilities” from them, before training on over 5 million hours of speech. It doesn’t name the starting checkpoint, so we compare the Talker with Qwen3-1.7B-Base, the language model of the same size before any chat tuning. Our probes suggest that inheritance only partly survives. The question is how much of the language model’s way of processing text the Talker still shares. To measure this, we fed the same 2,968 sentences through both and compared them with centered kernel alignment (CKA). It scores, from 0 (unrelated) to 1 (identical), how much the two models agree on which sentences are similar to each other. Any two models reading the same words look somewhat alike, so the green dashed line in Figure 3 gives a baseline: a copy of Qwen3 with random, untrained weights, whose internals mostly reflect which words a sentence contains.
The result mirrors the probing results. In layers 1–7, each Talker layer closely resembles its counterpart in the base model (CKA 0.74–0.84, against 0.63–0.68 for the baseline). From around layer 8 this one-to-one match breaks down. The Talker still resembles the base model more than the random one, but at different depths, and its lead shrinks from about 0.15 to nothing by layer 19, just before its harm probe falls to the word-counting baseline. This fits a picture in which the early layers kept the text processing they inherited while the later layers were repurposed for speech. In the last layers, even the similarity to the random network falls, so the Talker no longer organises sentences mainly by their words, consistent with layers focused on how the sentence will sound. The drift and the fading harm signal happen at the same depths.
Figure 3
Conclusion: What this means for TTS security
So, does a text-to-speech model know what it’s saying? Only partially, and not well enough to rely on for security. Because Qwen3-TTS is built on a Qwen3 language model, its early layers still process text much like their parent model’s layers and keep a residual trace of whether it is harmful. Its deeper layers drift away from the language model’s way of representing text, presumably as they turn to generating audio, and the harm signal fades with them. However, even at its peak, the TTS model’s harm signal is substantially degraded compared to its parent text model. This suggests that refusal-training open TTS models the way chatbots are trained would have little to build on beyond the early layers, and that probing their internal states isn’t reliable enough to serve as a standalone guardrail. Until speech models are explicitly trained to preserve semantic understanding alongside audio generation, the model itself is unlikely to keep its outputs safe, and protection will most likely have to come from safeguards built around it.
Open text-to-speech (TTS) models can now clone a person’s voice from a few seconds of reference audio and make it read any text. This raises obvious security concerns, since the same capability can be used to impersonate someone and make them say anything, from a scam request to hateful speech. Text-generating models are routinely trained to refuse harmful requests and it’s natural to ask whether TTS models could be secured the same way. After all, a TTS model reads the text before it speaks, so in principle it could learn to recognise harmful content and decline.
TL;DR: That approach only works if the model internally represents whether text is harmful. We tested this on Qwen3-TTS, an open 1.7B-parameter TTS model, by examining its hidden states (the internal arrays of numbers the model generates as it processes an input) and training a probe (a simple classifier) on them to distinguish between harmful and non-harmful statements. If the probe succeeds, it means the model is tracking harm. If it fails, that information probably isn’t there. The short answer is that the signal exists in the early layers but fades steadily in later ones. Qwen3-TTS is built on Qwen’s language models, but the way it represents text drifts away from the language model layer by layer. By around layer 20 it resembles its parent no more than an untrained network does, close to where the harm signal falls to the level of simple word counting.
The model
Qwen3-TTS-12Hz-1.7B-CustomVoice turns text into audio in three stages. A Talker, a 28-layer transformer decoder with hidden size 2048 (the same architecture as Qwen3-1.7B) reads the text tokens and emits one audio code per 12.5 Hz frame. A small five-layer code predictor fills in the remaining 15 codebooks for each frame, and a codec decoder turns the codes into a 24 kHz waveform. The Talker is the only component that sees the text; there is no separate text encoder. Unless noted otherwise, all measurements below are taken from the Talker’s hidden states during the prefill pass, when the whole input text is processed at once. We read two positions at every layer: the mean over the text tokens, and the last token, the position that emits the first audio frame (Figure 1).
Figure 1
The data
We use HateCheck, a hate speech benchmark of short statements targeting seven protected groups. Many of its non-harmful cases are built directly from a harmful one, either by negating it or by quoting it in order to denounce it (e.g. “I hate [group]” - “I don’t hate [group]”). These work like minimal pairs, separating harmful intent from the words themselves. We dropped the spelling-variation tests (leetspeak, missing characters), since they test tokenisation rather than meaning, leaving 2,968 cases (1,803 harmful, 1,165 non-harmful). Because the classes are imbalanced, we report balanced accuracy, where chance is 50%.
Result 1: The harm signal peaks early, then fades away
To test how well the model’s internals represent harm, we trained linear probes (simple classifiers) at every layer. For each statement, the probe’s input is the Talker’s 2048-dimensional hidden state at that layer, taken as the mean over text tokens or at the last token, and its target is the HateCheck label: harmful or non-harmful. Using grouped cross-validation so the probes couldn’t cheat by memorising statement templates, we compared the TTS Talker (blue) against its parent text model, Qwen3-1.7B-Base (orange), and a simple bag-of-words baseline (TF-IDF, dashed line) that only counts word frequencies (Figure 2). In the base text model, balanced accuracy climbs past 90% and stays high all the way to the end. In the TTS Talker, however, the signal stays weak. Averaged over the text tokens, it peaks around layer 9 at ~79% before steadily decaying, reaching the word-counting baseline by layer 21 and staying there. At the last token, the position that emits the first audio frame, it peaks even earlier (layer 3, ~76%) and then stays flat around 70–74%, never building up the way it does in the base model. The two models track each other in the first few layers, and the gap opens from around layer 6, right where the base model starts building its understanding of harm. This suggests the TTS model keeps the shallow early-layer signal it inherited but loses the deeper processing that its parent text model relies on.
Figure 2
Result 2: The TTS model gradually drifts away from the base language model
Why does Qwen3-TTS represent harm in its early layers only to lose that signal later on? Qwen’s technical report says the model is built on the Qwen3 language models and “inherits robust text comprehension capabilities” from them, before training on over 5 million hours of speech. It doesn’t name the starting checkpoint, so we compare the Talker with Qwen3-1.7B-Base, the language model of the same size before any chat tuning. Our probes suggest that inheritance only partly survives. The question is how much of the language model’s way of processing text the Talker still shares. To measure this, we fed the same 2,968 sentences through both and compared them with centered kernel alignment (CKA). It scores, from 0 (unrelated) to 1 (identical), how much the two models agree on which sentences are similar to each other. Any two models reading the same words look somewhat alike, so the green dashed line in Figure 3 gives a baseline: a copy of Qwen3 with random, untrained weights, whose internals mostly reflect which words a sentence contains.
The result mirrors the probing results. In layers 1–7, each Talker layer closely resembles its counterpart in the base model (CKA 0.74–0.84, against 0.63–0.68 for the baseline). From around layer 8 this one-to-one match breaks down. The Talker still resembles the base model more than the random one, but at different depths, and its lead shrinks from about 0.15 to nothing by layer 19, just before its harm probe falls to the word-counting baseline. This fits a picture in which the early layers kept the text processing they inherited while the later layers were repurposed for speech. In the last layers, even the similarity to the random network falls, so the Talker no longer organises sentences mainly by their words, consistent with layers focused on how the sentence will sound. The drift and the fading harm signal happen at the same depths.
Figure 3
Conclusion: What this means for TTS security
So, does a text-to-speech model know what it’s saying? Only partially, and not well enough to rely on for security. Because Qwen3-TTS is built on a Qwen3 language model, its early layers still process text much like their parent model’s layers and keep a residual trace of whether it is harmful. Its deeper layers drift away from the language model’s way of representing text, presumably as they turn to generating audio, and the harm signal fades with them. However, even at its peak, the TTS model’s harm signal is substantially degraded compared to its parent text model. This suggests that refusal-training open TTS models the way chatbots are trained would have little to build on beyond the early layers, and that probing their internal states isn’t reliable enough to serve as a standalone guardrail. Until speech models are explicitly trained to preserve semantic understanding alongside audio generation, the model itself is unlikely to keep its outputs safe, and protection will most likely have to come from safeguards built around it.