SMELT: Scaling Laws for Compute-Matched MoE Looped Transformers (Sep 2026)

Title: SMELT: Scaling Laws for Compute-Matched MoE Looped Transformers (Sep 2026) Link: http://arxiv.org/abs/2609.01343v1 Date: September 2026 Summary: This paper evaluates looped Mixture-of-Experts Transformers while closely matching per-token FLOPs, total non-embedding parameters, and KV-cache size against unlooped baselines. It introduces SMELT, which loops the middle half of layers twice, and finds consistent improvements across four model scales up to 54B parameters. Separate scaling-law fits show 6.8–18.0% training-compute savings at equal validation loss, with especially strong gains on code, long contexts, and in-context learning. Mechanistic analyses suggest the second pass amplifies aligned residual updates and reduces attention-sink mass, redirecting attention toward relevant content. Key Topics: - Looped and recurrent Transformers - Mixture-of-Experts language models - Compute-matched scaling laws - FLOPs, parameter, and KV-cache budget matching - Chinchilla-style compute-optimal training - Downstream benchmark transfer - In-context learning and long-context behavior - Attention sinks and mechanistic interpretability Chapters: 00:00 - SMELT’s Core Insight 01:00 - Why Looped Models Mislead 02:42 - Compute-Matched Model Design 04:52 - Optimal Loop Configuration 06:01 - Scaling Laws And Savings 07:40 - Data Allocation In MoE 09:02 - Benchmarking Structured Reasoning 10:44 - In-Context Learning Gains 12:02 - Mechanistic Loop Analysis 14:53 - Eliminating Attention Sinks 16:51 - Why The Second Pass Works 17:21 - SMELT Engineering Blueprint 17:50 - Toward Adaptive Recursion 18:49 - Final Takeaways Stock video credits: - MrColo - https://www.pexels.com/@mrcolo-12653218 - Jakub Zerdzicki - https://www.pexels.com/@jakubzerdzicki - Nicola Narracci - https://www.pexels.com/@nicola-narracci-157460431 - ALL IZ Well - https://www.pexels.com/@all-iz-well-3182153 - Media Hopper Studio - https://www.pexels.com/@media-hopper-studio-328305910 - Steve B - https://www.pexels.com/@steve-b-473063 - Pavel Danilyuk - https://www.pexels.com/@pavel-danilyuk - Pressmaster - https://www.pexels.com/@pressmaster - jorguez - https://www.pexels.com/@jorguez-191283684 - Kiwi and Camera - https://www.pexels.com/@kiwi-and-camera-2158354883 - Usman AbdulrasheedGambo - https://www.pexels.com/@theonlyabdulla - Coverr - https://www.pexels.com/@coverr - Кирилл Левченко - https://www.pexels.com/@2156561057 - Colin Jones - https://www.pexels.com/@larchmedia - Adis Resic - https://www.pexels.com/@adis-resic-297996969 - Tima Miroshnichenko - https://www.pexels.com/@tima-miroshnichenko - Chandresh Uike - https://www.pexels.com/@chandresh-uike-754623426 - Oleg Gamulinskii - https://www.pexels.com/@oleg-gamulinskii-755060 - crazy motions - https://www.pexels.com/@crazy-motions-80195021 - BRoll.io - https://www.pexels.com/@brollio - Magda Ehlers - https://www.pexels.com/@magda-ehlers-pexels - Pon Balaji - https://www.pexels.com/@pon-balaji-881701 - cottonbro studio - https://www.pexels.com/@cottonbro - Cyriac von Czapiewski - https://www.pexels.com/@cyriac-von-czapiewski-1601520 - Pachon in Motion - https://www.pexels.com/@pachon-in-motion-426015731 - Anete Lusina - https://www.pexels.com/@anete-lusina - Max Fischer - https://www.pexels.com/@max-fischer - Vlada Karpovich - https://www.pexels.com/@vlada-karpovich - ROMAN ODINTSOV - https://www.pexels.com/@roman-odintsov - Monstera Production - https://www.pexels.com/@gabby-k - Ketut Subiyanto - https://www.pexels.com/@ketut-subiyanto - Shutter Break - https://www.pexels.com/@shutterbreak - Bedrijfsfilmspecialist.nl - https://www.pexels.com/@bedrijfsfilmspecialist-nl-1284006 - Cheng Shi Song - https://www.pexels.com/@cheng-shi-song-427082720 - K - https://www.pexels.com/@kelly - Philippe WEICKMANN - https://www.pexels.com/@weickmann - Kindel Media - https://www.pexels.com/@kindelmedia - Colors Motion Graphics - https://www.pexels.com/@colors-motion-graphics-183847699 - Google DeepMind - https://www.pexels.com/@googledeepmind - olia danilevich - https://www.pexels.com/@olia-danilevich - José Alfredo Munguía Lira - https://www.pexels.com/@rectorretro - Nino Souza - https://www.pexels.com/@ninosouza - Mikhail Nilov - https://www.pexels.com/@mikhail-nilov