Most discrete diffusion language models treat the text you give them as gospel. Feed one a sentence with garbage words in it and the model copies the garbage, then builds its answer around it. A new paper, “Trust but Verify”, shows how to teach these models a little healthy suspicion, and the fix is cheap enough to run on a laptop.
The paper, by Gunmay Jhingran, was posted to arXiv on 2 October 2026 and has been accepted to the NeurIPS 2026 workshop on AI for Stochastic Dynamics. Below: what it changes, what the numbers show, and where it falls short.
Key takeaways
- Standard masked diffusion models put about 50% probability on context tokens even when those tokens are random junk.
- The paper mixes two kinds of noise and gives each region of the text its own rate. Blanks get masked, and context gets occasional random swaps.
- After fine-tuning, the model flagged uniformly random corrupted tokens with 0.96 recall and 0.91 precision.
- Infill accuracy rose by 3.8 points at 10% corruption and by 6.4 points at 20%, with no measurable loss on clean input.
- Against realistic, plausible-looking errors, the gain was only about 0.5 points and not statistically significant.
What is discrete diffusion, and why does it trust its context?
Diffusion models learn to reverse a corruption process. Image models add Gaussian noise, which is static, step by step and learn to remove it. Text is made of discrete tokens (words or word pieces), so text diffusion needs a different kind of corruption.
The most popular option is absorbing corruption. Tokens are gradually replaced with a special [MASK] symbol, and the model learns to fill the blanks back in. The 2024 NeurIPS paper on masked diffusion language models (MDLM) from Cornell Tech showed this simple recipe comes close to autoregressive models on perplexity.
The catch is that, in absorbing diffusion, a token that isn’t masked has never been corrupted. During training, every visible token is correct. So the model learns a hard rule: if you can see it, it’s true. Trust is binary.
Think of a proofreader who’s been told the manuscript has no typos and only the blank lines need writing. Hand them a page with errors on it and they’ll write new sentences that agree with the errors. That’s the failure the paper measures. When corrupted context goes in, the errors spread into the generated text, and accuracy drops right next to the bad spans.
How does “Trust but Verify” change the forward process?
The “forward process” is the corruption recipe used during training. In image diffusion, researchers tune it all the time. In text diffusion, most work simply uses absorbing noise. This paper treats the forward process as something you can design again.
The recipe is region-inhomogeneous, which just means different parts of the input get different noise:
- Missing positions (the blanks to fill) follow normal absorbing dynamics and turn into [MASK].
- Context positions (the text you supplied) get low-rate uniform mixing: now and then a token is swapped for a random one from the vocabulary.
Because the context is now sometimes wrong during training, the model learns to treat observed tokens as evidence rather than fact. Training also used a mix of “hard negatives”: random tokens, frequency-weighted tokens, embedding neighbours and the base model’s own guesses.
What happens at generation time?
Inference uses a detect, remask and refill loop over 64 reverse steps. The model scores every context token by how likely it thinks that token is. In the first half of generation, tokens below a threshold are sent back to [MASK] and regenerated. The threshold starts at 0.5 and is annealed, meaning it is gradually lowered, down to 0.1.
Edits are capped at 15% of the context, and the context is frozen halfway through so the model can’t keep second-guessing itself. The author openly calls this remasking step a heuristic, not a derived rule.
The cost is the striking part. The author fine-tuned a pretrained 170-million-parameter MDLM for 8,000 steps on FineWeb-Edu text, in under five hours on a laptop GPU with 6 GB of memory.
How do the three approaches compare?
This paper isn’t the first to mix noise types. The Self-Correcting Discrete Diffusion (SCDD) paper from February 2026 also combined absorbing and uniform transitions so a model could revise visible tokens. What’s new here is targeting the noise by region and aiming it at the user’s input.
| Approach | How tokens are corrupted | Trust in context |
|---|---|---|
| Absorbing (MDLM-style) | Tokens become [MASK] | Total: visible tokens are always treated as correct |
| Uniform mixing | Tokens swap for random vocabulary items | Every token can be revised |
| Trust but Verify (hybrid) | Blanks absorb, context gets rare random swaps | Graded: context is trusted but checked |
The paper frames absorbing diffusion as one end of a “trust axis”, the case where the gap between known and unknown regions is infinite. Image diffusion has a similar dial in inpainting, the gap in noise level between the kept pixels and the filled ones. The author presents this as a motivating view, not a proven result.
Where does the method fall short?
Random junk is easy to spot. Realistic errors are not. When the corruptions were GPT-2 samples, plausible words that fit the sentence, detection recall fell to 0.35 and the infill gain shrank to roughly half a point, which wasn’t statistically significant. The paper reports that its own pre-registered primary test failed on these cases.
The author suggests a tidy rule behind this: how detectable an error is and how much damage it does both depend on how far it sits from real data. Wild errors are easy to catch and hurt a lot. Subtle ones slip through and hurt less. That idea hasn’t been tested yet either.
The scope is also small: one 170M model, one seed per training run, one text domain and a fixed edit budget that caps recall at about 0.76 when 20% of the context is corrupted. In my view, credit belongs to the author for listing all of this up front. Plenty of bigger papers don’t.
What happens next
The paper lists a clear roadmap. Derive the exact training bound for the combined noise kernels. Treat per-position noise rates as a design choice in their own right. Replace the hand-tuned remasking rules with principled ones. Scale up to LLaDA-class diffusion models.
The harder tests are still to come: swapped names and numbers, code with injected bugs, and human preference ratings. Those are the realistic errors where the method currently struggles, and the ones that matter for anyone feeding retrieved documents or user text into a diffusion model.
FAQ
What problem does the Trust but Verify paper solve?
Masked discrete diffusion models treat every visible token as correct, so corrupted input gets copied into the output. The paper trains them to doubt their context and fix suspicious tokens.
Does the fix hurt performance on clean text?
Not measurably. On clean context, the false-edit rate stayed below 0.4%, according to the paper.
Can it catch realistic errors?
Not reliably yet. Recall on plausible, GPT-2-generated errors was 0.35, and the accuracy gain of about 0.5 points was not statistically significant.
If you work with diffusion language models, read the full paper on arXiv. Then try feeding your own model some corrupted context to see how much it trusts what it reads.
Leave a comment