Title: [2502.14831] Improving the Diffusability of Autoencoders
Description: Abstract page for arXiv paper 2502.14831: Improving the Diffusability of Autoencoders
Keywords: No keywords
Text content: [2502.14831] Improving the Diffusability of Autoencoders                                         Skip to main content      We gratefully acknowledge support from the Simons Foundation, member institutions, and all contributors. Donate       > cs > arXiv:2502.14831         Help | Advanced Search     All fields Title Author Abstract Comments Journal reference ACM classification MSC classification Report number arXiv identifier DOI ORCID arXiv author ID Help pages Full text     Search                open search       GO    open navigation menu   quick links  Login Help Pages About             Computer Science > Computer Vision and Pattern Recognition   arXiv:2502.14831 (cs)            [Submitted on 20 Feb 2025] Title:Improving the Diffusability of Autoencoders Authors:Ivan Skorokhodov, Sharath Girish, Benran Hu, Willi Menapace, Yanyu Li, Rameen Abdal, Sergey Tulyakov, Aliaksandr Siarohin View a PDF of the paper titled Improving the Diffusability of Autoencoders, by Ivan Skorokhodov and 7 other authors View PDF HTML (experimental)  Abstract:Latent diffusion models have emerged as the leading approach for generating high-quality images and videos, utilizing compressed latent representations to reduce the computational burden of the diffusion process. While recent advancements have primarily focused on scaling diffusion backbones and improving autoencoder reconstruction quality, the interaction between these components has received comparatively less attention. In this work, we perform a spectral analysis of modern autoencoders and identify inordinate high-frequency components in their latent spaces, which are especially pronounced in the autoencoders with a large bottleneck channel size. We hypothesize that this high-frequency component interferes with the coarse-to-fine nature of the diffusion synthesis process and hinders the generation quality. To mitigate the issue, we propose scale equivariance: a simple regularization strategy that aligns latent and RGB spaces across frequencies by enforcing scale equivariance in the decoder. It requires minimal code changes and only up to 20K autoencoder fine-tuning steps, yet significantly improves generation quality, reducing FID by 19% for image generation on ImageNet-1K 256x256 and FVD by at least 44% for video generation on Kinetics-700 17x256x256.          Comments: 26 pages, 22 figures, 9 tables   Subjects:  Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI); Machine Learning (cs.LG)  Cite as: arXiv:2502.14831 [cs.CV]     (or  arXiv:2502.14831v1 [cs.CV] for this version)                 https://doi.org/10.48550/arXiv.2502.14831   Focus to learn more                      arXiv-issued DOI via DataCite (pending registration)        Submission history From: Ivan Skorokhodov [view email]       [v1]         Thu, 20 Feb 2025 18:45:44 UTC (47,923 KB)       Full-text links: Access Paper:   View a PDF of the paper titled Improving the Diffusability of Autoencoders, by Ivan Skorokhodov and 7 other authorsView PDFHTML (experimental)TeX SourceOther Formats view license        Current browse context: cs.CV   < prev    |    next >   new  |  recent  | 2025-02      Change to browse by:      cs cs.AI cs.LG     References & Citations  NASA ADSGoogle Scholar Semantic Scholar     a export BibTeX citation Loading...     BibTeX formatted citation ×   loading...   Data provided by:      Bookmark            Bibliographic Tools  Bibliographic and Citation Tools       Bibliographic Explorer Toggle    Bibliographic Explorer (What is the Explorer?)        Connected Papers Toggle    Connected Papers (What is Connected Papers?)       Litmaps Toggle    Litmaps (What is Litmaps?)        scite.ai Toggle    scite Smart Citations (What are Smart Citations?)          Code, Data, Media  Code, Data and Media Associated with this Article       alphaXiv Toggle    alphaXiv (What is alphaXiv?)        Links to Code Toggle    CatalyzeX Code Finder for Papers (What is CatalyzeX?)        DagsHub Toggle    DagsHub (What is DagsHub?)        GotitPub Toggle    Gotit.pub (What is GotitPub?)        Huggingface Toggle    Hugging Face (What is Huggingface?)        Links to Code Toggle    Papers with Code (What is Papers with Code?)        ScienceCast Toggle    ScienceCast (What is ScienceCast?)              Demos  Demos       Replicate Toggle    Replicate (What is Replicate?)        Spaces Toggle    Hugging Face Spaces (What is Spaces?)        Spaces Toggle    TXYZ.AI (What is TXYZ.AI?)         Related Papers  Recommenders and Search Tools       Link to Influence Flower    Influence Flower (What are Influence Flowers?)        Core recommender toggle    CORE Recommender (What is CORE?)      Author Venue Institution Topic                      About arXivLabs           arXivLabs: experimental projects with community collaborators arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website. Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them. Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.           Which authors of this paper are endorsers? |     Disable MathJax (What is MathJax?)                  About Help      contact arXivClick here to contact arXiv  Contact   subscribe to arXiv mailingsClick here to subscribe  Subscribe            Copyright Privacy Policy     Web Accessibility Assistance   arXiv Operational Status                      Get status notifications via                     email                     or slack