Measuring Code Sloppiness: Why Functionally Correct AI Code Degrades Quality

X postPi (@pidotdev) / Sebastian BayeResearch

In an Earendil blog post, engineer Sebastian Baye explores methods to quantify 'code sloppiness' produced by LLMs. While models generate syntactically and functionally correct code that passes automated tests, empirical metrics demonstrate that agent-generated code has roughly twice the verbosity and structural erosion of human codebases.

Key points

  1. LLMs write code that passes hidden tests, but they introduce excessive abstractions, duplicate logic, and rapid LOC inflation.
  2. Using LLMs to evaluate code ('AI as a judge') is unreliable: scoring on 1-10 scales resembles random noise, and pairwise comparisons frequently change preference based on superficial label changes.
  3. Human evaluation ensures taste and readability, but cannot scale to automated model training or cross-provider evaluations.
  4. The SlopCodeBench benchmark tracks two quantitative metrics: Verbosity (measuring cloned and AST-Grep flagged lines against LOC) and Erosion (the proportion of codebase mass inside functions with cyclomatic complexity exceeding 10).
  5. Benchmark measurements reveal that agent-produced code has double the verbosity (0.33 vs. 0.15) and double the erosion (0.68 vs. 0.31) of standard human repositories.

Why it matters

As code production costs drop to near zero, the bottleneck shifts from generating working code to maintaining human comprehension and architectural integrity. When projects expand by millions of lines of sloppy code per month, both human engineers and subsequent automated agents lose the ability to manage the repository.

Analysis

Baye points out that the tooling ecosystem is dominated by marketing claims and 'vibes' rather than rigorous metrics. Simple proxies like delta-LOC fail once optimized directly under Goodhart's Law. By examining AST-level repetition (Verbosity) and function-level complexity weighting (Erosion), SlopCodeBench offers a concrete framework to assess how iterative multi-round agent interactions compound architectural debt, contradicting assumptions that future agents can easily maintain messy legacy output.

Original post

Pi

@pidotdev · 11 September 2026

Read “If code is solved, what now?: Measuring the sloppiness of code” t.co/dJpIRripSQ

104 likes3 replies136 bookmarks6.2k views

View on X
More posts in the thread (1)
  1. When code is abundant quality becomes the differentiator, but what is good code? New blog post from Earendil engineer @SebastianBaye on why quantifying the ‘sloppiness’ of code is difficult, which eval methods don’t work well, and what metrics matter. Link to the post below t.co/Zx2DwK2KQ6
Selected replies (5)
  • @Everlier @pidotdev @SebastianBaye We need to talk more about this. More is not better. Simple is not easy.
  • @justynclark @pidotdev @SebastianBaye What's more cringe is the amount of AI generated comments on this thread. Your em dashes and non-ASCII characters are a dead giveaway. You are perpetuating slop, unrelated to code.
  • @mindfultime_ @pidotdev @SebastianBaye One indicator I use is CRAP , and also AST to see the depth of the code being written. I feel crap is a good way to capture the cc with unit test . It’s a good idea on the unnecessary complexity that ai provides
  • @geminixiang @Everlier @pidotdev @SebastianBaye Simple useful is hard.
  • @yandt888 @pidotdev @SebastianBaye agent 一天塞几十个 PR 时，「好代码」很难打分。改动面、失败复现、审不过的次数，比抽象草率分更管用。

Mentioned links

Links and tools from this entry

Summary generated by AI (Gemini 3.8 Flash, 27 September 2026). It may contain errors – the original source is authoritative.

Related entries

Similar in content, found by the AI search.

  • Article:Sebastian

    Measuring Code Sloppiness in AI-Generated Software

    Sebastian from Earendil examines quantitative approaches to measuring 'code slop' generated by language models. While LLMs deliver syntactically correct code, they frequently double verbosity and architectural erosion compared to human code, leading even advanced agents to fail during iterative development benchmarks.

    AI & LLMs· Research

  • Article:The Cognition Team

    Measuring Autonomous AI Engineer Productivity: Cognition's Framework for Devin

    Cognition has introduced an automated system to quantify the output of autonomous AI software engineers in terms of productive engineering hours. Because token consumption and diff sizes are poor proxies for actual value, a dedicated estimator agent inspects Devin session traces, removes unproductive runs, and benchmarks the result against the time a human developer would have required.

    AI & LLMs· Research

  • Repository:TinyFrontier/anti-slop-py

    anti-slop-py: Python Linter Blocking AI Agent Type Escapes

    anti-slop-py is a zero-dependency Python linter targeting low-evidence escape hatches generated by AI coding agents. It prevents patterns like excessive Any types, cast cascades, and unexplained type: ignore comments that agents use to satisfy type checkers without defining proper contracts.

    5starsPython

    AI & LLMs· Tool

  • X post:Tech with Mak

    Ponytail: Prompting AI Coding Agents to Write Less Code

    Ponytail is an open-source tool designed to teach AI coding agents to reuse existing code, standard libraries, and dependencies instead of writing new code from scratch. According to Tech with Mak, the tool reduces generated code by an average of 54 percent.

    1.9kbookmarks149.3kviews

    AI & LLMs· Tool

  • Link:The Cognition Team

    FrontierCode: Benchmarking AI Code by Maintainer Mergeability

    Cognition has published FrontierCode, a benchmark and leaderboard evaluating whether pull requests generated by AI models would realistically be merged by open-source maintainers. Scoring relies on unit tests, verifiers, and maintainer-defined rubrics.

    AI & LLMs· Resource

Back to the knowledge base

Let's talk about your project

Get in touch

Locations

  • Mattersburg
    Johann Nepomuk Bergerstraße 7/2/14
    7210 Mattersburg, Austria
  • Vienna
    Ungargasse 64-66/3/404
    1030 Wien, Austria

Parts of this content were created with the assistance of AI.