X postAYiDiscussion
Seven years of AI scaling reflect systematic bug-fixing and architectural refinements rather than merely adding parameters. Open access to model implementations enables engineers to inspect how modern architectures tackle critical memory and cache bottlenecks.
The analysis underscores a fundamental shift in model design away from unconstrained KV caching toward bounded, recurrent memory states. By tracing the lineage from GPT-2's decoder architecture through linear attention feature maps to DeltaNet's delta-rule updates, the breakdown demonstrates how modern large-scale models manage finite memory states without catastrophic associative interference.
AYi
@AYi_AInotes · 28 July 2026
这是@Kimi_Moonshot K3开源后我见过最狠的帖子。 一个Baseten的工程师，@waterloo_intern 老哥花48小时喝了40罐气泡水，把K3的建模代码扒了个底朝天。 128万浏览，6000多收藏， 最后得出一个结论， 一个K3里能塞下22580个2019年的GPT-2， 七年时间参数涨了两万两千多倍， 但这根本不是堆参数堆出来的。 t.co/x6c4RidO8z t.co/zOO0KonyjD
2.3k likes58 replies1.9k bookmarks402k views
View on XSummary generated by AI (Gemini 3.8 Flash, 27 September 2026). It may contain errors – the original source is authoritative.
Similar in content, found by the AI search.
Article:Kimi Team (Yu Zhang et al.)
The Kimi Team presents Kimi Linear, a hybrid linear attention architecture that outperforms standard full attention across short-context, long-context, and RL scaling scenarios. It cuts KV cache consumption by up to 75% and boosts decoding throughput up to sixfold on 1M token contexts.
Link:Moonshot AI
Moonshot AI has released Kimi K3, an open-weight native multimodal agentic model featuring 2.8 trillion total parameters and a 1-million-token context window. The architecture uses a sparse MoE setup that activates 104 billion parameters per token.
Repository:PipeNetwork/kimi-k3-mlx
kimi-k3-mlx is an Apple MLX port of Moonshot AI's native multimodal MoE model Kimi-K3 (2.78T total parameters). Because the model exceeds the memory capacity of any single Apple Silicon machine, the repository includes a custom streaming converter, vision components, and expert-pruning tools.
337starsPython
VideoVideo:CURT
In the opening episode of an educational series, software engineer Colin breaks down the end-to-end development lifecycle of modern large language models using Moonshot AI's Kimi K3 technical report as a blueprint. The lifecycle covers nine distinct stages: defining the task, cleaning data, provisioning compute, model architecture, pre-training, post-training, benchmarks, safety and release, and production serving.
Article:Ju-yeong Ji, Ravin Kumar
Google outlines the architectural changes in Gemma 3, highlighting vision-language capabilities, longer context windows, an expanded tokenizer, and memory-saving attention mechanisms.
X post:wast3
An engineer showcases a low-cost memory architecture for Kimi K3 that replaces vector stores and context bloat with a four-node lifecycle based on expiration. Rather than persisting everything, memories must earn retention through active model reuse.
697bookmarks67.1kviews
Parts of this content were created with the assistance of AI.
