13 entries
Keyword: mixture-of-experts
VideoVideo:CURT
In the opening episode of an educational series, software engineer Colin breaks down the end-to-end development lifecycle of modern large language models using Moonshot AI's Kimi K3 technical report as a blueprint. The lifecycle covers nine distinct stages: defining the task, cleaning data, provisioning compute, model architecture, pre-training, post-training, benchmarks, safety and release, and production serving.
Repository:JustVugg/colibri
Colibrì is a lightweight inference engine written in pure C with zero dependencies, designed to run frontier Mixture-of-Experts (MoE) models ranging from 744B to 2.8T parameters on consumer and heterogeneous hardware. It treats VRAM, RAM, and storage as a unified memory hierarchy, streaming experts on demand.
37.7kstarsC
From X post: Ten Open-Source Repositories Powering AI Agent Infrastructure
Article:Opher Lieber, Barak Lenz, Hofit Bata et al.
Researchers introduce Jamba, a base large language model utilizing a hybrid Transformer-Mamba mixture-of-experts (MoE) architecture. By interleaving Transformer and Mamba layers with MoE modules, Jamba fits into a single 80GB GPU while supporting context lengths of up to 256K tokens.
From Video: Building an LLM from Scratch: Neural Networks, Tokenization, and Transformers
Article:Zhidx
Ant Group's Bailian team has released Ling-3.0-tiny, an open-source lightweight mixture-of-experts model. It leverages DeepSeek's MLA and Kimi's KDA architectures with 7.9B total parameters, activating just 1.3B parameters per token for local edge deployment on hardware like Apple Silicon Macs and DGX systems.
From Video: Weekly AI Updates: Model Releases, Agent Harnesses, and Developer Tooling (August 2026)
Article:Simon Willison
DeepSeek has introduced DeepSeek-V4-Pro and DeepSeek-V4-Flash under an MIT licence. Simon Willison reviews their Mixture of Experts architecture, KV cache efficiency gains, and pricing compared to OpenAI, Google, and Anthropic.
From Video: Weekly AI Updates: Model Releases, Agent Harnesses, and Developer Tooling (August 2026)
Article:Thinking Machines Lab
Thinking Machines Lab has introduced Inkling-Small, an open-weights Mixture-of-Experts model (276B total, 12B active parameters) designed to deliver performance comparable to Inkling at a fraction of its size and compute cost.
From Video: Lev Selector's Weekly AI Updates (7 August 2026)
Link:drumih
TurboFieldfare is an open-source inference engine written in Swift and Metal that executes 4-bit Gemma 4 26B-A4B-IT on Apple M-series Macs using roughly 2 GB of RAM. While the model weights require 14 GB of disk space, the engine keeps only the shared model base and KV cache in memory and streams routed expert layers on demand directly from the SSD.
From Video: AI Weekly Updates: Open-Source Model Surge, Agent Harnesses, and Meta's AI Reality Check
Link:ornith-ai
ornith-ai has released Ornith-1.5-397B, an open-source multimodal mixture-of-experts model under the MIT licence. With 397 billion parameters and an end-to-end self-improvement loop, it delivers coding and reasoning benchmark performance comparable to top proprietary models.
From Video: AI Weekly Updates: Open-Source Model Surge, Agent Harnesses, and Meta's AI Reality Check
Article:OpenAI
OpenAI has introduced gpt-oss-120b and gpt-oss-20b, two open-weight Mixture-of-Experts reasoning models released under the Apache 2.0 license.
From Video: AI Weekly Updates: Open-Source Model Surge, Agent Harnesses, and Meta's AI Reality Check
Article:Ant Group
Ant Group has introduced Ring-2.6-1T, an open-source MoE thinking model with one trillion parameters designed specifically for coding agents, tool use, and long-horizon tasks.
Link:Xiaomi MiMo Team
Xiaomi has unveiled MiMo-V2.5-Pro, an open-source Mixture-of-Experts model featuring 1.02 trillion total parameters with 42 billion active. The model supports a 1M-token context window and is specifically designed for complex agentic workflows and long-horizon software engineering.
From Video: Weekly AI Updates: Model Releases, Prompt Tricks, and Agent Architectures
Link:Poolside
Poolside has released Laguna-S-2.1, a Mixture-of-Experts language model featuring 118B total parameters with approximately 8B activated parameters per token. Targeted at software engineering and long-horizon agentic workflows, the model is available under the permissive OpenMDW-1.1 license.
From Video: Weekly AI Update: New Models, Agent Workflows, and Infrastructure Shifts
VideoVideo:EliteTestEngineering
EliteTestEngineering demonstrates running DeepSeek V4 Flash across a two-node cluster consisting of a DGX Spark and an Acer GN100. The MoE model achieves inference speeds of 10 to 16.5 tokens per second but quickly encounters memory limitations with longer context windows.
13 of 13 shown
Do you have an interesting article, a video, or a tweet that belongs in our knowledge base? Write to us at office@webconsulting.at.
Parts of this content were created with the assistance of AI.
