DeepsecBench: Vercel Publishes AI Leaderboard for Vulnerability Detection

LinkVercelResource

Vercel's AI Gateway features the DeepsecBench leaderboard, evaluating AI models on their ability to detect security vulnerabilities in application code using the Deepsec cyber harness. The benchmark calculates an overall score balancing recall and precision.

Key points

  1. DeepsecBench tests AI models on code vulnerability scanning, tracking score, recall, precision, false positives, cost, runtime, and token counts.
  2. GPT-6 Astra xhigh ranks first with a 37.79 score, 32.8% recall, 97.9% precision, and 2 false positives using the Codex harness ($63.70 cost).
  3. GPT-5.6 Sol xhigh holds second place (35.44 score, 96.3% precision), followed by Claude Opus 5 max in third (32.44 score, 88.0% precision, $127.93 cost via Claude Code).
  4. Grok 4.6 xhigh is the only model achieving 100.0% precision with zero false positives, though its 13.4% recall caps its score at 16.16.
  5. Budget options like Deepseek V4 Flash xhigh and Qwen3.8 Max xhigh reach scores around 16.47 at run costs below $7.

Why it matters

The benchmark provides software engineering and security teams with clear comparative metrics on how reliably specific LLMs uncover code flaws, their false-positive rates, and the financial cost per scan.

Analysis

The benchmark highlights a persistent trade-off between recall and precision in AI security audits. Even the top performer, GPT-6 Astra xhigh, catches only 32.8% of vulnerabilities, though it maintains a high precision of 97.9%. High-end agent setups such as Claude Opus 5 in Claude Code consume significant token budgets and costs ($127+), whereas alternatives like Deepseek V4 Flash offer competitive mid-tier detection rates at a fraction of the price.

Mentioned links

Found in

Summary generated by AI (Gemini 3.8 Flash, 27 September 2026). It may contain errors – the original source is authoritative.

Related entries

Similar in content, found by the AI search.

  • X post:Guillermo Rauch

    DeepsecBench: GPT-6 Astra Tops Vercel's Cybersecurity Leaderboard

    According to Vercel CEO Guillermo Rauch, OpenAI's GPT-6 Astra has taken the top spot on Vercel's DeepsecBench. The model completes benchmark evaluations in under 50 minutes, compared to nearly four hours for its predecessor.

    180bookmarks81.1kviews

    AI & LLMs· News

  • Link:StackAI

    StackAI LLM Leaderboard: Comparing Models by Benchmarks and Cost

    StackAI provides an interactive dashboard comparing large language models across multiple benchmarks, throughput, context window sizes, and pricing. It compiles evaluation metrics such as MMLU, HumanEval+, GPQA, SWE benchmarks, and GSM8K for side-by-side model comparison.

    AI & LLMs· Tool

  • Link:clickrank.ai

    LLM Leaderboards in 2026: Benchmarks, Methodologies, and Model Rankings

    A guide by ClickRank AI examines the state of LLM evaluations in 2026. With legacy benchmarks like MMLU and HumanEval largely saturated, evaluation platforms now focus on harder reasoning tests, human pairwise blind comparisons, and composite indices covering cost and latency.

    AI & LLMs· Resource

  • Link:The Cognition Team

    FrontierCode: Benchmarking AI Code by Maintainer Mergeability

    Cognition has published FrontierCode, a benchmark and leaderboard evaluating whether pull requests generated by AI models would realistically be merged by open-source maintainers. Scoring relies on unit tests, verifiers, and maintainer-defined rubrics.

    AI & LLMs· Resource

Back to the knowledge base

Let's talk about your project

Get in touch

Locations

  • Mattersburg
    Johann Nepomuk Bergerstraße 7/2/14
    7210 Mattersburg, Austria
  • Vienna
    Ungargasse 64-66/3/404
    1030 Wien, Austria

Parts of this content were created with the assistance of AI.