Title: [2502.14458] Llamba: Scaling Distilled Recurrent Models for Efficient Language Processing
Description: Abstract page for arXiv paper 2502.14458: Llamba: Scaling Distilled Recurrent Models for Efficient Language Processing
Keywords: No keywords
Text content: [2502.14458] Llamba: Scaling Distilled Recurrent Models for Efficient Language Processing                                         Skip to main content      We gratefully acknowledge support from the Simons Foundation, member institutions, and all contributors. Donate       > cs > arXiv:2502.14458         Help | Advanced Search     All fields Title Author Abstract Comments Journal reference ACM classification MSC classification Report number arXiv identifier DOI ORCID arXiv author ID Help pages Full text     Search                open search       GO    open navigation menu   quick links  Login Help Pages About             Computer Science > Machine Learning   arXiv:2502.14458 (cs)            [Submitted on 20 Feb 2025] Title:Llamba: Scaling Distilled Recurrent Models for Efficient Language Processing Authors:Aviv Bick, Tobias Katsch, Nimit Sohoni, Arjun Desai, Albert Gu View a PDF of the paper titled Llamba: Scaling Distilled Recurrent Models for Efficient Language Processing, by Aviv Bick and 4 other authors View PDF HTML (experimental)  Abstract:We introduce Llamba, a family of efficient recurrent language models distilled from Llama-3.x into the Mamba architecture. The series includes Llamba-1B, Llamba-3B, and Llamba-8B, which achieve higher inference throughput and handle significantly larger batch sizes than Transformer-based models while maintaining comparable benchmark performance. Furthermore, Llamba demonstrates the effectiveness of cross-architecture distillation using MOHAWK (Bick et al., 2024), achieving these results with less than 0.1% of the training data typically used for models of similar size. To take full advantage of their efficiency, we provide an optimized implementation of Llamba for resource-constrained devices such as smartphones and edge platforms, offering a practical and memory-efficient alternative to Transformers. Overall, Llamba improves the tradeoff between speed, memory efficiency, and performance, making high-quality language models more accessible.         Subjects:  Machine Learning (cs.LG); Artificial Intelligence (cs.AI)  Cite as: arXiv:2502.14458 [cs.LG]     (or  arXiv:2502.14458v1 [cs.LG] for this version)                 https://doi.org/10.48550/arXiv.2502.14458   Focus to learn more                      arXiv-issued DOI via DataCite (pending registration)        Submission history From: Aviv Bick [view email]       [v1]         Thu, 20 Feb 2025 11:18:39 UTC (928 KB)       Full-text links: Access Paper:   View a PDF of the paper titled Llamba: Scaling Distilled Recurrent Models for Efficient Language Processing, by Aviv Bick and 4 other authorsView PDFHTML (experimental)TeX SourceOther Formats   view license         Current browse context: cs.LG   < prev    |    next >   new  |  recent  | 2025-02      Change to browse by:      cs cs.AI     References & Citations  NASA ADSGoogle Scholar Semantic Scholar     a export BibTeX citation Loading...     BibTeX formatted citation ×   loading...   Data provided by:      Bookmark            Bibliographic Tools  Bibliographic and Citation Tools       Bibliographic Explorer Toggle    Bibliographic Explorer (What is the Explorer?)        Connected Papers Toggle    Connected Papers (What is Connected Papers?)       Litmaps Toggle    Litmaps (What is Litmaps?)        scite.ai Toggle    scite Smart Citations (What are Smart Citations?)          Code, Data, Media  Code, Data and Media Associated with this Article       alphaXiv Toggle    alphaXiv (What is alphaXiv?)        Links to Code Toggle    CatalyzeX Code Finder for Papers (What is CatalyzeX?)        DagsHub Toggle    DagsHub (What is DagsHub?)        GotitPub Toggle    Gotit.pub (What is GotitPub?)        Huggingface Toggle    Hugging Face (What is Huggingface?)        Links to Code Toggle    Papers with Code (What is Papers with Code?)        ScienceCast Toggle    ScienceCast (What is ScienceCast?)              Demos  Demos       Replicate Toggle    Replicate (What is Replicate?)        Spaces Toggle    Hugging Face Spaces (What is Spaces?)        Spaces Toggle    TXYZ.AI (What is TXYZ.AI?)         Related Papers  Recommenders and Search Tools       Link to Influence Flower    Influence Flower (What are Influence Flowers?)        Core recommender toggle    CORE Recommender (What is CORE?)        IArxiv recommender toggle    IArxiv Recommender (What is IArxiv?)       Author Venue Institution Topic                      About arXivLabs           arXivLabs: experimental projects with community collaborators arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website. Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them. Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.           Which authors of this paper are endorsers? |     Disable MathJax (What is MathJax?)                  About Help      contact arXivClick here to contact arXiv  Contact   subscribe to arXiv mailingsClick here to subscribe  Subscribe            Copyright Privacy Policy     Web Accessibility Assistance   arXiv Operational Status                      Get status notifications via                     email                     or slack