Programmatic Tool Calling vs Native JSON: When Should an Agent Write Code?

LinkMasato OkuwakiResearch

Masato Okuwaki reviews a PwC study comparing native JSON tool calling against programmatic tool calling (PTC) using Python scripts across 14 LLMs on BFCL v4. Letting models emit executable code matched or beat JSON baselines in eleven of the fourteen tested models.

Key points

  1. Under programmatic tool calling (PTC), the model writes a Python script importing typed stubs instead of emitting structured JSON objects.
  2. To ensure a fair single-turn comparison, custom middleware halts the agent loop after executing the script, preventing extra inference turns.
  3. 11 of 14 models matched or outperformed their JSON baseline with PTC, including gains of up to 10.6 points for GPT-5.6 variants and 6.5 points for Claude Sonnet 4.6.
  4. Three models (GPT-4o, GPT-4.1, and GPT-5.4-mini) suffered severe 20 to 27 point drops solely due to emitting literal '\n' sequences, causing fatal syntax errors in deterministic execution.
  5. Performance disparities tracked model generations rather than vendor families, reflecting general code generation maturity over prompt engineering.

Why it matters

Structured JSON tool calling is currently the default pattern for autonomous AI agents. Demonstrating that modern models excel when simply writing executable Python challenges standard API designs and could shift agent architectures towards programmatic execution.

Analysis

Okuwaki connects the findings to Rich Sutton's 'Bitter Lesson', arguing that general code generation capabilities ultimately outscale hand-engineered interface standards such as structured JSON. While the headline drop in older OpenAI models stemmed from trivial string formatting bugs that real-world retry loops could fix, running model-generated code in a subprocess changes an application's threat model significantly and requires isolated execution environments.

Mentioned links

Found in

Summary generated by AI (Gemini 3.8 Flash, 27 September 2026). It may contain errors – the original source is authoritative.

Related entries

Similar in content, found by the AI search.

  • Article:Ishan Patel, Sahil Sen, Elias Lumer, Vamse Kumar Subbiah

    Research Compares Programmatic Tool Calling to JSON Tool Calling

    A research paper evaluates Programmatic Tool Calling (PTC) against native JSON tool calling across 14 language models using the BFCL v4 benchmark. In PTC, tools are represented as typed Python stubs invoked via code in a single agent turn. The approach matches or exceeds native JSON tool calling in 11 out of 14 models.

    AI & LLMs· Research

  • Link:Shishir G. Patil, Huanzhi Mao, Charlie Cheng-Jie Ji, Fanjia Yan, Vishnu Suresh, Ion Stoica, Joseph E. Gonzalez

    Berkeley Function Calling Leaderboard (BFCL) V4

    The Berkeley Function Calling Leaderboard (BFCL) V4 evaluates the ability of large language models to call functions and tools accurately. Version 4 expands previous iterations by introducing holistic agentic evaluation.

    AI & LLMs· Tool

  • Article:DeepInfra

    Top LLMs for OpenClaw Agentic Workloads

    DeepInfra evaluates open-weight models for the OpenClaw autonomous agent framework, focusing on tool-calling reliability, long-session instruction adherence, and cost per completed task.

    AI & LLMs· Resource

  • Article:Anthropic

    Anthropic Introduces Claude Sonnet 4.5 and Claude Agent SDK

    Anthropic has released Claude Sonnet 4.5, delivering significant improvements in software engineering, multi-step agent reasoning, and autonomous computer use. The release is accompanied by developer tools including the Claude Agent SDK and checkpoints for Claude Code.

    AI & LLMs· Announcement

  • Link:deep-reinforce.com

    Ornith-1.0: Self-Scaffolding Open-Source Models for Agentic Coding

    Ornith-1.0 is an open-source model family tailored for agentic coding, spanning parameter sizes from 9B to 397B. Built on Gemma 4 and Qwen 3.5, the architecture trains models to jointly generate both code solutions and the task-specific scaffolds directing them.

    AI & LLMs· Announcement

Back to the knowledge base

Let's talk about your project

Get in touch

Locations

  • Mattersburg
    Johann Nepomuk Bergerstraße 7/2/14
    7210 Mattersburg, Austria
  • Vienna
    Ungargasse 64-66/3/404
    1030 Wien, Austria

Parts of this content were created with the assistance of AI.