Where Quality Breaks in Compressed Short-Text Generation: Staged Bottleneck Localization

@inproceedings{Gavrilov2026WhereQuality,
  title      = {Where Quality Breaks in Compressed Short-Text Generation: Staged Bottleneck Localization},
  author     = {Gavrilov, Alexey and Gazzaev, Alan-Barsag and Muravyov, Sergey},
  booktitle  = {2026 39th Conference of Open Innovations Association (FRUCT)},
  publisher  = {IEEE},
  year       = {2026},
  pages      = {69--76},
  doi        = {10.23919/FRUCT70069.2026.11506553},
  url        = {https://doi.org/10.23919/FRUCT70069.2026.11506553},
  isbn       = {978-952-65246-5-8},
}

Open a question for a concise answer grounded in the paper. Detailed evidence boundaries are listed in Limitations.

1. Where does quality break in compressed short-text generation?

   In the tested TinyStories 64-to-16 pipeline, the dominant degradation appears at codec reconstruction, before latent generation begins. Median external GPT-2 perplexity increases from 15.17 for original text to 27.36 after reconstruction, while p95 increases from 25.10 to 98.91. This is a result for one configuration rather than a universal codec ranking.

2. How can codec loss be separated from latent-generation loss?

   The staged protocol evaluates originals, paired codec reconstructions, and final generated text with one shared external scorer. The original-to-reconstruction gap estimates the codec contribution, while the reconstruction-to-generated-output comparison helps localize the additional generation-stage loss. Latent health metrics are reported separately from decoded-text evidence.

3. How should discrete-latent text generation be evaluated?

   The paper recommends checking reconstruction fidelity before comparing generators, applying the same decoded-text scorer at every stage, and reporting central and tail statistics. It also treats codebook usage, geometry, and other latent proxies as diagnostics rather than substitutes for the quality of decoded text.

4. Does code-space diffusion outperform token-space diffusion?

   Under the shared scorer and tested setup, code-space MDLM reduces mean, median, and p95 perplexity by 32.9%, 30.9%, and 36.6% relative to token-space MDLM. The comparison is descriptive and configuration-specific: it does not establish a universal ranking across datasets, compression ratios, codecs, or diffusion architectures.

5. Do better latent geometry metrics guarantee better generated text?

   No. In the available matched runs, geometry-aware regularization improved local latent-space proxies but did not improve decoded-text metrics. The result supports auditing every proxy gain at the final decoded output and treating the absence of transfer as a useful negative result rather than evidence of generation improvement.

The comparison distinguishes evaluation scope and evidence; it is not a universal ranking of the approaches.

• The empirical study uses TinyStories only.
• The main analysis covers one aggressive 64-to-16 compression regime.
• The evaluated system combines one hierarchical VQ-VAE-2 codec family with one MDLM generator.
• Comparisons are based on single runs and are descriptive rather than multi-seed statistical estimates.
• External GPT-2 perplexity is a shared diagnostic, not a universal semantic-quality metric.
• The conclusions should not be transferred directly to all datasets, codec architectures, or compression ratios.

These entries correspond to the numbered References section in the paper PDF.

1. Subham Sekhar Sahoo, Marianne Arriola, Yair Schiff, Aaron Gokaslan, Edgar Marroquin, Justin T Chiu, Alexander Rush, Volodymyr Kuleshov. 2024. Simple and Effective Masked Diffusion Language Models. Advances in Neural Information Processing Systems.
2. Aaron van den Oord, Oriol Vinyals, Koray Kavukcuoglu. 2017. Neural Discrete Representation Learning. Advances in Neural Information Processing Systems.
3. Ali Razavi, Aaron van den Oord, Oriol Vinyals. 2019. Generating Diverse High-Fidelity Images with VQ-VAE-2. Advances in Neural Information Processing Systems.
4. Jacob Austin, Daniel D. Johnson, Jonathan Ho, Daniel Tarlow, Rianne van den Berg. 2021. Structured Denoising Diffusion Models in Discrete State-Spaces. Advances in Neural Information Processing Systems.
5. Aaron Lou, Chenlin Meng, Stefano Ermon. 2024. Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution. Proceedings of the 41st International Conference on Machine Learning.
6. Xiaochuang Han, Sachin Kumar, Yulia Tsvetkov. 2023. SSD-LM: Semi-autoregressive Simplex-based Diffusion Language Model for Text Generation and Modular Control. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics.
7. Xiang Lisa Li, John Thickstun, Ishaan Gulrajani, Percy Liang, Tatsunori B. Hashimoto. 2022. Diffusion-LM Improves Controllable Text Generation. Advances in Neural Information Processing Systems.
8. Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, William T. Freeman. 2022. MaskGIT: Masked Generative Image Transformer. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.
9. Marjan Ghazvininejad, Omer Levy, Yinhan Liu, Luke Zettlemoyer. 2019. Mask-Predict: Parallel Decoding of Conditional Masked Language Models. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing.
10. Emiel Hoogeboom, Didrik Nielsen, Priyank Jaini, Patrick Forré, Max Welling. 2021. Argmax Flows and Multinomial Diffusion: Learning Categorical Distributions. Advances in Neural Information Processing Systems.
11. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever. 2019. Language Models are Unsupervised Multitask Learners. OpenAI Technical Report.
12. Nils Reimers, Iryna Gurevych. 2019. Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing.
13. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, Yoav Artzi. 2020. BERTScore: Evaluating Text Generation with BERT. International Conference on Learning Representations.
14. Krishna Pillutla, Swabha Swayamdipta, Rowan Zellers, John Thickstun, Sean Welleck, Yejin Choi, Zaid Harchaoui. 2021. MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontiers. Advances in Neural Information Processing Systems.
15. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, Ion Stoica. 2023. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. Advances in Neural Information Processing Systems, Datasets and Benchmarks Track.

Publisher

IEEE

Local text PDF

Final accepted manuscript (author-posted version with DOI)

Publication resources

The public manuscript, results tables, explanatory figure, and citation files are available here. Implementation code and checkpoints are not publicly released.

The published DOI is the primary bibliographic identifier. This page remains the single canonical project URL across versions.