Research Compares Programmatic Tool Calling to JSON Tool Calling

ArticleIshan Patel, Sahil Sen, Elias Lumer, Vamse Kumar SubbiahResearch

A research paper evaluates Programmatic Tool Calling (PTC) against native JSON tool calling across 14 language models using the BFCL v4 benchmark. In PTC, tools are represented as typed Python stubs invoked via code in a single agent turn. The approach matches or exceeds native JSON tool calling in 11 out of 14 models.

Key points

  1. Empirical evaluation of programmatic tool calling (PTC) versus native JSON tool calling across 14 LLMs on BFCL v4.
  2. PTC matches or exceeds native JSON tool calling performance in 11 of 14 evaluated models.
  3. The GPT-5.6 family shows a 10.6% improvement with PTC over the JSON baseline.
  4. In parallel fan-out scenarios, PTC matches or outperforms the baseline across 13 of 14 models.
  5. PTC remains stable under context rot conditions where native JSON tool calling drops by an average of 2.3%.

Why it matters

Rigid JSON schemas often become brittle when LLMs need to chain multiple actions or execute parallel calls. Interfacing via executable code builds on the code generation strengths of modern models and demonstrates greater resilience against context degradation.

Analysis

The paper underscores a shift from static structured outputs toward executable code for agentic workflows. By exposing functions as typed Python stubs, models handle sequencing and data passing within a single step, yielding notable gains in newer architectures like the GPT-5.6 family.

Mentioned links

Found in

Summary generated by AI (Gemini 3.8 Flash, 27 September 2026). It may contain errors – the original source is authoritative.

Related entries

Similar in content, found by the AI search.

  • Link:Masato Okuwaki

    Programmatic Tool Calling vs Native JSON: When Should an Agent Write Code?

    Masato Okuwaki reviews a PwC study comparing native JSON tool calling against programmatic tool calling (PTC) using Python scripts across 14 LLMs on BFCL v4. Letting models emit executable code matched or beat JSON baselines in eleven of the fourteen tested models.

    AI & LLMs· Research

  • Link:Shishir G. Patil, Huanzhi Mao, Charlie Cheng-Jie Ji, Fanjia Yan, Vishnu Suresh, Ion Stoica, Joseph E. Gonzalez

    Berkeley Function Calling Leaderboard (BFCL) V4

    The Berkeley Function Calling Leaderboard (BFCL) V4 evaluates the ability of large language models to call functions and tools accurately. Version 4 expands previous iterations by introducing holistic agentic evaluation.

    AI & LLMs· Tool

  • Article:Jiarui Lu, Thomas Holleis, Yizhe Zhang, Bernhard Aumayer, Feng Nan, Felix Bai, Shuang Ma, Shen Ma, Mengyu Li, Guoli Yin, Zirui Wang, Ruoming Pang

    ToolSandbox: Stateful and Conversational Benchmark for LLM Tool Use

    ToolSandbox is an evaluation benchmark that assesses LLM tool-use capabilities using stateful environments and multi-turn interactive dialogues.

    AI & LLMs· Research

  • Link:nekuda-ai

    WindTunnel: Open Benchmark Compares WebMCP Against Computer Use and DOM Approaches

    Nekuda-ai has released WindTunnel, a reproducible open-source benchmark evaluating WebMCP against screenshot-based computer use and DOM-based browser agents. Evaluating 49 tasks across eight real websites with three attempts per task, the benchmark tests models including GPT-5.6 Luna, Gemini 3.6 Flash, and Sonnet 5.

    AI & LLMs· Resource

Back to the knowledge base

Let's talk about your project

Get in touch

Locations

  • Mattersburg
    Johann Nepomuk Bergerstraße 7/2/14
    7210 Mattersburg, Austria
  • Vienna
    Ungargasse 64-66/3/404
    1030 Wien, Austria

Parts of this content were created with the assistance of AI.