Fine-Tuning a 4B Model for $80 Outperforms GPT 5.6 Terra on Narrow Task

X postMaxime RivestNews

Maxime Rivest reports fine-tuning a 4B model using 5,000 synthetic examples, four hours of training, and $80, claiming it outperforms GPT 5.6 Terra on a specific science communication task.

Key points

  1. Resources: 5,000 examples, four hours of compute, and $80 were sufficient to train the 4B parameter model.
  2. Target task: Rewriting scientific papers in ecology and evolution so that 14-year-olds can understand them.
  3. Data generation: Synthetic training data was produced using frontier models, including Opus 5.5 rewrites.
  4. Core takeaway: Rivest argues fine-tuning is now affordable and practical, avoiding the generalisation overhead of large frontier models for narrow tasks.

Why it matters

This experiment highlights how accessible synthetic data and fine-tuning make smaller models viable alternatives to expensive frontier systems when tackling specific, well-defined problems.

Analysis

Rivest emphasises that coding agents and frontier models make acquiring high-quality synthetic datasets inexpensive. While the model is heavily specialised for a single task rather than general reasoning, he argues that targeted fine-tuning provides superior efficiency. Models and a write-up are slated for release on Hugging Face.

Original post

Maxime Rivest 🧙‍♂️🦙🐧

@MaximeRivest · 5 October 2026

I never thought that with only 5000 training examples, 4 hours of training and 80$ I could make a 4b model beat gpt 5.6 terra. training data is now cheap and plentiful, coding agent are now capable enough to do the work. you just need to figure out th ask you want to teach you t.co/gyASTc5n9H

427 likes49 replies380 bookmarks32.6k views

View on X
More posts in the thread (11)
  1. @imevgenii its not an agent
  2. @eragoaugustus that is exactly my point: finetuning is accessible now and should be more often considered.
  3. @mahi17939311 probably! and i wouldnt care :) it is specialized on one particular task. why paying the overhead of generalization when you can specialize that easily?
  4. @mahi17939311 sample output of the 4b below. the task is: rewrite a scientific article so a 14 year old can understand it. the dataset is 5000 of those rewrites from frontier models. All in ecology, environment, evolution type of fields. There are about 100 million scientific articles to t.co/v9yNpVoPVU
  5. @_rchaves_ writing something up but: t.co/KpYWXQtqe6
  6. @_NicholasFong t.co/KpYWXQtqe6
  7. @chirag @madmaxbr5 t.co/KpYWXQtqe6
  8. @martino_broggi synthetic was not just opus, and whatever bias opus 5.5 had, matched my taste. but indeed astra would score these differently, but worst, in my opinion.
  9. @mahi17939311 > Also 100 million is ambitious asf just 3 years on a rtx 6000 pro. :D
  10. @rakeshsahni_ working on it! i'll upload the models on hf and write something up.
  11. @jeetendrayd working on that
Selected replies (5)
  • @imevgenii @MaximeRivest Benchmaxxing. Test on real agentic tasks.
  • @_NicholasFong @MaximeRivest on what?
  • @madmaxbr5 @MaximeRivest *on a narrow task
  • @mahi17939311 @MaximeRivest Aaaah this makes so much more sense. I thought the previous was coding related. On writing type tasks, specialized models are much better than frontiers. Also 100 million is ambitious asf. Good luck finding enough GPU time lmao. Maybe try Ling 3.0 tiny or similar MOE?
  • @kennethlau12393 @MaximeRivest the hard part is the automatic scorer. 5000 examples only works if outputs can be graded without a human

Mentioned links

Summary generated by AI (Gemini 3.8 Flash, 6 October 2026). It may contain errors – the original source is authoritative.

Related entries

Similar in content, found by the AI search.

  • Link:oumi.ai

    Oumi: AI Factory Platform for Building and Deploying Custom Models

    Oumi positions itself as an AI Factory production line designed to train, deploy, and continuously improve domain-specific AI models. Rather than relying solely on generic frontier API providers, it automates the creation of specialized models using proprietary organizational data.

    AI & LLMs· Tool

  • Video

    Video:Lev Selector

    Weekly AI Updates: Local Models, GPT-6 Astra, and Agent Architectures

    Lev Selector summarises key AI industry updates for early September 2026. Highlights cover the launch of GPT-6 Astra, dramatic API cost reductions via local routine execution, major updates to open-source agent frameworks, and architectural principles for large-scale enterprise RAG deployments.

    AI & LLMs· News

  • X post:Astraia

    Hypothesis on Qwen 3.8 27B Training: GRPO Replaces Traditional SFT

    According to an analysis by X account Astraia, the strong performance of Qwen 3.8 27B is not driven by classic supervised fine-tuning (SFT). Instead, it relies on Group Relative Policy Optimization (GRPO) paired with liberal reasoning context limits and outputs from Qwen 3.8 Max as reinforcement learning targets.

    995bookmarks122kviews

    AI & LLMs· Opinion

  • X post:Udi Wertheimer

    PrismML Announces Ternary Bonsai 2 27B: A 6 GB Quantised Model

    Udi Wertheimer highlights PrismML's release of Ternary Bonsai 2 27B, a model based on Qwen3.8 27B compressed to 5.9 GB while reportedly retaining 98.2% of its full-precision benchmark performance.

    1.5kbookmarks260.1kviews

    AI & LLMs· Opinion

Back to the knowledge base

Let's talk about your project

Get in touch

Locations

  • Mattersburg
    Johann Nepomuk Bergerstraße 7/2/14
    7210 Mattersburg, Austria
  • Vienna
    Ungargasse 64-66/3/404
    1030 Wien, Austria

Parts of this content were created with the assistance of AI.