**Artificial Intelligence (AI) has never moved faster — or been harder to measure.**

Every week brings a new model claiming to reason, code, or plan better than the last. Demand for both training and inference hardware [keeps growing](https://www.mckinsey.com/~/media/McKinsey/Industries/Semiconductors/Our%20Insights/Artificial%20intelligence%20hardware%20New%20opportunities%20for%20semiconductor%20companies/Artificial-intelligence-hardware.ashx). Now several years into the great AI boom, Google Scholar and arXiv’s AI and Machine Learning-focused directories continue to [receive hundreds of new submissions](https://arxiv.org/list/cs.LG/new) on a daily basis.

Yet the faster the landscape expands, the more questions the industry faces — proof that an abundance of choice isn’t always a good thing. How do the latest LLMs stack up? Who has the best video generation model? Which model is the fastest with a 100k-token prompt? Last but not least: how much will it cost you?

**These questions are why benchmarking matters.** From the earliest ImageNet competitions to today’s complex language and reasoning tasks, benchmarks have been the invisible engine of AI advancement. They make progress legible, drive accountability, and help researchers, businesses, and policymakers alike speak a common language.

**As models have grown from narrow classifiers to multimodal, reasoning, and even agentic systems, benchmarks have had to evolve in lockstep to more closely reflect the industry’s ever-changing focus.**

This piece explores AI benchmarking’s state of affairs: why it matters, who conducts it, and the challenges and opportunities at play.

## What Is AI Benchmarking?
Benchmarking is the practice of evaluating an AI system’s performance against standardized tests.

A benchmark might be as simple as a dataset of labeled images or as complex as an interactive software environment. **What makes it a benchmark are its consistency** — every model faces the same challenge, under the same conditions, and produces results that can be compared across time and architectures — **and its transparency.**

In machine learning’s early days, benchmarks were narrow and task-specific — think image (e.g., [ImageNet](https://www.image-net.org/)) or speech recognition (e.g., [TIMIT](https://catalog.ldc.upenn.edu/LDC93S1)), argument extraction (e.g., [TreeBank](https://catalog.ldc.upenn.edu/LDC99T42)), and so on. These benchmarks defined the early vocabulary of progress in terms of accuracy, word error rate, and [cumulative rewards](https://spinningup.openai.com/en/latest/spinningup/rl_intro.html#key-concepts-and-terminology).

In contrast, modern benchmarks are multidimensional: they assess reasoning, safety, latency, cost, and even energy efficiency. And **as the industry comes up with new questions, it continues to require new, more sophisticated benchmarks to answer them.**

![A combined image of handwritten digits extracted from the MNIST database, an example of a dataset constructed specifically for image processing purposes. Source: Wikipedia.](https://www-uploads.scaleway.com/MNIST_dataset_example_da3dbc9dbd.png)

## Why AI Benchmarking Matters

**Benchmarks are more than technical scoreboards.** Every breakthrough in AI, from the first convolutional networks to today’s frontier models, has relied on them to **quantify improvement, validate new ideas, and facilitate reproduction**. Without shared measurement, innovation would remain siloed and anecdotal and everyone would be left guessing whether a new architecture is genuinely more capable than its predecessors, rather than simply different. By making results public and comparable, benchmarks instead enable positive feedback loops to form.

At the end of the day, benchmarking is a tool. As such, **it can be used at different stages, from research, to production, to industry-wide governance**. Let’s cover each in turn.

### Research

Benchmarks enable researchers in a number of ways:

- They enable **comparison** through metrics rather than claims.
- They encourage **iteration**: When experiments are run against the same test suite, labs can isolate with precision which architectural choices or training methods actually drive improvement.
- They foster **reproducibility**: Public benchmarks turn one lab’s results into a foundation that others can build upon. Air Street Capital’s [State of AI Report 2025](https://www.stateof.ai/) highlights this dynamic clearly: standardized evaluation “turns fragmented experimentation into collective progress.”
- They democratize **discovery** for smaller labs and independent researchers, who get access to the same insights as their deep-pocketed peers.

Together, these traits mean **innovation happens faster, for less, and spreads more widely.**

### Production
**Benchmarks also underpin the practical side of AI**: performance, efficiency, and cost. When you’re processing millions of requests and billions of tokens, every percentage point across any one of those metrics matters. Benchmarking allows companies big and small to access standardized measurements when deciding which hardware, model, or API endpoint to deploy.

As the space gets more crowded, **providers have put applicability front and center**. [Artificial Analysis](https://artificialanalysis.ai/), for instance, [emphasizes](https://artificialanalysis.ai/methodology) that its “benchmark results are not intended to represent the maximum possible performance on any particular hardware platform, they are intended to represent the real-world performance customers experience across providers.”

Benchmarking is also central to the product development process, helping developers and product teams validate their own workflows and assess their products’ performance before general availability or regulatory review. This makes internal benchmarks an indispensable complement to the external-facing tool that are user tests.

### Governance and Safety

Finally, benchmarks are becoming the foundation for AI governance. In recent years, **the rapid increase in the technology’s capabilities has raised concerns over the potential risks of its implementation.** Newly formed regulatory bodies are now tasked with everything from risk categorization, to defining security, transparency, and quality obligations, to conducting conformity assessments. The tracker maintained by the International Association of Privacy Professionals (IAPP) — last updated in May 2025 — [shows extensive global activity](https://iapp.org/resources/article/global-ai-legislation-tracker/), with over 75 jurisdictions having introduced AI-related laws or policies to date.

With that in mind, **regulators are increasingly using benchmarks to understand what a model can do before it’s deployed**. In particular, they are interested in identifying its behavioral propensities, including a model’s tendency to hallucinate or the reliability of its performance. The UK’s [AI Security Institute (AISI)](https://www.aisi.gov.uk/), for example, runs independent evaluations of advanced models, assessing areas like biosecurity, cybersecurity, and persuasion capabilities. Its work has set a precedent for government-backed model testing, complementing internal lab efforts and introducing a form of “public audit” for AI safety.


![The UK's AI Security Institute produces rigorous research to advance AI governance. Source: AI Security Institute.](https://www-uploads.scaleway.com/Capture_d_ecran_2025_10_30_a_23_27_14_e25b5c913c.png)


## Who Does AI Benchmarking Today?

The more central benchmarking became, the more stakeholders started getting involved in hope of **shaping the industry’s key metrics and practices**. Today, a wide range of actors provides the industry with the information it needs, including:

- **Academic and non-profit initiatives**: Projects like Stanford’s [HELM](https://crfm.stanford.edu/helm/) (Holistic Evaluation of Language Model) and [METR](https://metr.org/) (Model Evaluation and Threat Research) drive the scientific backbone of benchmarking. HELM focuses on transparent, open, reproducible evaluations. METR, by contrast, looks at long-horizon and agentic tasks — how well models perform multi-step reasoning or sustain a plan over time.
- **Private providers**: A segment of independent analytics and benchmarking companies has emerged to meet the growing need for consistent, business-grade comparisons. Today, providers like Artificial Analysis test commercial systems across intelligence, speed, and pricing. This professionalization of benchmarking signals a maturing market where transparent, repeatable testing becomes a service in and of itself.
- **Internal benchmarks**: While companies widely use public, open benchmarks, they also [conduct their own testing](https://www.businessinsider.com/meta-ai-llama-models-training-data-ablation-2025-4). Internal benchmarking enables teams to create bespoke metrics and yardsticks before a dedicated public benchmark exists. It also supports tighter feedback loops between measurement and implementation, which can help shorten time to market. For example, at Scaleway, we use internal tests to bulletproof our PaaS solutions.
- **Industry consortia**: The MLCommons community runs MLPerf, a leading hardware and deployment benchmark. Originally focused on training workloads, it now emphasizes inference benchmarks, including LLMs, with standardized metrics for performance per watt or dollar. MLCommons’s [board of directors](https://mlcommons.org/about-us/leadership/) includes leaders from manufacturers like Intel, Nvidia, Graphcore, and others.
- **Safety and government evaluators**: As mentioned, AI-related legislation has spawned a number of new regulatory bodies tasked with testing frontier models for national and public safety. The UK’s AISI’s early findings have notably called for shared, cross-lab methodologies to test AI’s biosecurity, cyber, and persuasion capabilities.
- **Crowd leaderboards**: With its intuitive chatbot interface, the [LMSYS Arena](https://lmarena.ai/) has become the public face of model comparison, allowing users to rank AI assistants through anonymous pairwise voting.

**Together, these initiatives form an invaluable but increasingly complex — and sometimes contradictory — ecosystem.**
![LMArena's's Leaderboard has become the public face of model comparison. Source: LMArena.](https://www-uploads.scaleway.com/LM_Arena_Leaderboard_overview_e40dc5e979.png)

## AI Benchmarking’s Challenges…

Even as benchmarking becomes more sophisticated, it faces growing pains — Airstreet Capital’s [State of AI Report 2025](https://www.stateof.ai/) highlighted that the field’s rapid expansion has outpaced the robustness of its measurements. In no particular order, below are some of its current challenges.

- **Data contamination and repeat exposure**: Popular benchmarks have become victims of their own success. Public datasets are widely circulated, meaning portions of test material often end up in model training data. When that happens, performance scores can reflect memory rather than reasoning, creating a false impression of progress.
- **Variance and reproducibility**: Some reasoning and math benchmarks contain so few questions that results can swing significantly based on random seed or prompt phrasing. Without multiple runs or confidence intervals, it’s difficult to know whether performance gains are real or statistical noise.
- **Eval-aware behavior**: The most advanced models are not just performing better; they are also starting to recognize when they’re being tested. “Eval-aware” models can change tone, verbosity, or refusal behaviors to optimize for benchmark success without improving underlying reasoning. This means evaluators may need to adopt stealthier approaches to ensure authentic behavior under evaluation. [MLPerf’s training rules](https://github.com/mlcommons/training_policies/blob/master/training_rules.adoc#mlperf-training-rules), for example, make it clear that “the framework and system should not detect and behave differently for benchmarks.”
- **Fragmentation across evaluators**: Between private providers, non-profits, crowd sourcing, and governmental evaluators, benchmarking as a field is getting increasingly crowded. While this diversity fuels innovation, it also fragments compatibility — without a common baseline, it’s easy for models to look stronger or weaker depending on which benchmark suite is chosen. More cross-lab coordination and standardized protocols may be needed.
- **Partial coverage**: For all the diversity in today’s benchmarking ecosystem, benchmarks still only effectively cover part of the AI. Evaluating hardware or models is essentially a solved problem; assessing processes or code architecture, not so much. Context-specific performance remains evasive — a challenge considering how companies are now trying to apply AI to increasingly specific verticals. **Now, agents represent yet another hurdle**. If agentic systems are to become the industry’s new default UI, benchmarks will need to adapt accordingly.
- **“Tunnel vision” evaluation**: As AI regulation becomes more of a priority, benchmarking is showing its limits. Researchers from Sapienza University of Rome, DEXAI-Artificial Ethics, and Sant’Anna School of Advanced Studies pointed out that current benchmarks “were not designed to measure the systemic risks that are the focus of the new regulatory landscape” — a situation they described as the [“benchmark-regulation gap.”](https://arxiv.org/abs/2508.05464) They found that “the evaluation ecosystem dedicates the vast majority of its focus to a narrow set of behavioral propensities” while neglecting “capabilities central to loss-of-control scenarios, including evading human oversight, self-replication, and autonomous AI development.” 

![Safety-focused benchmarks tend to ignore capabilities central to “loss-of-control” scenarios. Source: Prandi et al.](https://www-uploads.scaleway.com/Total_Question_Distribution_Across_EU_AI_Act_Categories_e76150adfa.png)

## … And Opportunities
While the above challenges need to be addressed, we should remain optimistic: positive trends are developing that should make benchmarking more useful and reliable, not less. Here are a few of them:

- **Improving transparency**: Independent providers like Artificial Analysis and projects like HELP or MLPerf are setting a new bar for open methodology, releasing full prompt logs, evaluation seeds, and configuration details that allow any lab to audit and reproduce their results. These groups apply the same “open-door” approach to their blogs, publishing [full sets of questions](https://metr.org/blog/2025-06-27-risk-transparency/), responses to upcoming legislation or private frameworks, and more. Together, these efforts ensure that the culture of benchmarking leans towards openness, rather than secrecy.
- **Positive scrutiny**: They say to solve a problem, you must first acknowledge it. Beyond pointing at structural issues, [a number](https://arxiv.org/html/2411.12990v1#S7) [of](https://arxiv.org/html/2502.06559v1) [papers](https://arxiv.org/abs/2508.05464) have aimed to provide new frameworks for evaluating benchmarks — benchmarking the benchmarkers, if you will. Stanford, for instance, unveiled [BetterBench](https://betterbench.stanford.edu/), “a living repository of benchmark assessments to support benchmark comparability.” Overall, this growing (but positive) scrutiny should help improve benchmark design and usability.
- **Future-proofing**: Long gone are the days of narrow benchmarking. With the rise of multimodal, reasoning, and now even agentic systems, providers continue to come up with new benchmarks to ensure their work reflects AI’s new capabilities and the industry’s evolving needs. Notable metrics include measuring a model’s [“task completion time horizon”](https://metr.org/blog/2025-03-19-measuring-ai-ability-to-complete-long-tasks/?) — the length of tasks AI agents can complete —, [long-term coherence](https://arxiv.org/pdf/2502.15840), or [cultural inclusivity](https://arxiv.org/abs/2505.09595). Meanwhile, practices such as periodic dataset revisions and the addition of new scenarios ensure that benchmarking remains as dynamic as the ecosystem it’s meant to monitor.
- **Growing collaboration**: Despite stark competition for customers’ attention and spending, the industry’s various stakeholders are increasingly collaborating on shared protocols. For example, OpenAI, Anthropic, and Google DeepMind have all participated in [joint safety evaluations under the UK’s AISI supervision](https://techcrunch.com/2023/06/12/uk-ai-safety-research-pledge/), giving early or priority access to their models for research and safety purposes. Meanwhile, with its 125+ founding Members and Affiliates all working towards a shared goal of open industry-standard benchmarks, MLCommons stands as perhaps the best example of industry-wide collaboration.

We look forward to seeing how these developments impact the industry for the better!

## Where AI Benchmarking Is Going Next

**From its early, task-specific beginnings, benchmarking has evolved into vital infrastructure for the entire field of AI.** Today, the combined efforts of the public and private sectors provide the industry with continuously updated data and insights. Benchmarks enable comparison, encourage iteration, and drive discovery, ensuring that competition ultimately benefits everyone.

**As the field matures, what we measure may increasingly determine what we build**. With concerns over AI safety rising, regulators are relying on benchmarking more and more for practical decision-making, while companies are turning to it for transparency and trust.

At Scaleway, benchmarking is equally central to how we operate. For example, it helps guide the roadmap for our [Generative APIs](https://www.scaleway.com/en/generative-apis/), ensuring we [integrate only battle-tested models](https://www.scaleway.com/en/docs/generative-apis/#whisper-large-v3-is-now-available) with proven real-world applicability.

But this relationship also flows the other way: we actively contribute to the benchmarking ecosystem. Our API inference endpoint will soon be featured on Artificial Analysis, giving researchers and businesses direct visibility into our own performance. It’s a small but meaningful step toward greater transparency — a reflection of our continued focus on transparency.

<br>

---
<br>

**ai-PULSE, Europe’s premier Artificial Intelligence conference powered by Scaleway, is returning!**

Gathering key players from across Europe, the event will be back once again at **STATION F on December 4** for a unique blend of deep technical expertise and crucial business insights. You’ll hear from:

- Micah Hill-Smith, Co-Founder & CEO of Artificial Analysis, on which metrics truly matter in the new AI stack
- Boris Gamazaychikov (Head of AI Sustainability at Salesforce) and Elise Auvray (Product Manager, Environmental Footprint at Scaleway) on how we can make “energy-efficient” AI measurable

... and dozens more leaders and engineers shaping the technology’s future.

**Whether you’re planning to attend in-person or online, [make sure to register](https://www.ai-pulse.eu/register?utm_source=blog&utm_medium=organic&utm_campaign=202507-evt-ww-all-aipulse_2025&utm_content=ai_benchmarking)!**
[![](https://www-uploads.scaleway.com/Register_ai_PULSE_2025_Header_Email_600x200_2_a36cdb7473.png)](https://www.ai-pulse.eu/register?utm_source=blog&utm_medium=organic&utm_campaign=202507-evt-ww-all-aipulse_2025&utm_content=ai_benchmarking)

medium_Content_Inference-Illustration-Content.png

small_Content_Inference-Illustration-Content.png

thumbnail_Content_Inference-Illustration-Content.png

Content_Inference-Illustration-Content.png

Why AI Benchmarking Matters

Report Scaleway leader Exaegis markess 2025

Affordable servers with the best price-performance ratio on the market

Start

Perfect balance of processing power, memory and storage

The high performance backbone of your mission-critical infrastructure

Core

For mission-critical data, fast storage, backup and streaming

Store

Dedicated GPU power with reliable performance and stability

Host your Hardware in our secured French datacenters

Dedirack

60 locations worldwide, starting at €4,99/month

Dedibox VPS

Dedibox - dedicated servers

Fully dedicated bare metal servers with native cloud integration, at the best price

Aluminium

Powerful, balanced and reliable servers for production-grade applications

Beryllium

Powerful dedicated server designed to handle high-workload applications

Iridium

Designed with huge local storage to keep, back up, and protect your data

Lithium

Power and stability of dedicated GPU hardware integrated into the Scaleway ecosystem

Titanium

The world's first RISC-V servers available in the cloud.

RISC-V (Labs)

Elastic Metal - bare metal cloud

Enjoy the Mac mini experience with great simplicity

Mac mini M1

Perform your daily tasks with speed and efficiency

Mac mini M2

Mac mini M2 Pro

Latest Apple silicon chip for intensive use cases

Mac mini M4

Built for AI and the most complex workflows

Mac mini M4 Pro

Apple

Dedicated Servers

Ideal for building, testing, and deploying lightweight applications

Development Instances

The best best balance of performance and value for a wide range of workloads

General Purpose Instances

Built for raw CPU power and compute-intensive applications

Compute Optimized Instances

Designed for data-intensive workloads requiring large amounts of RAM

Memory Optimized Instances

Not sure which virtual machine is right for you? We’ve got you covered!

Compare our CPU Instances

Virtual Instances

Maximize your AI infrastructures with a versatile Instance

L4 GPU Instance

Universal Instance, faster than L4 and cheaper than H100 PCIe

L40S GPU Instance

Accelerate AI applications' development with H100 SXM and H100 PCIe GPU Instances

H100 GPU Instance

Dedicated Tesla P100s for all your Machine Learning & Artificial Intelligence needs.

Render GPU Instances

Experience an easy way to run your code on the cloud

Serverless Functions

Easily run containers on the cloud with a single command

Serverless Containers

Serverless Jobs

Serverless

Kubernetes exclusively for Scaleway products and resources

Kubernetes Kapsule

Multi-cloud Kubernetes for Scaleway and external providers resources

Kubernetes Kosmos

Container Registry

Containers

Compute

Build the next Foundation Model with one of the fastest and most energy-efficient supercomputers in the world

Custom-built Clusters

Rent a GPU-cluster from 32 to more than a thousand GPUs to speed up distributed training

On Demand Cluster

Clusters

Consume AI models instantly via a simple API call. All hosted in Europe

Generative APIs

Deploy AI models in a dedicated inference infrastructure. Get tailored security and predictable throughput

Inference

Chat with the first of its kind speech-to-speech AI model

Moshi (Labs)

Model-as-a-service

Accelerate the next generation of AI-enabled applications with the universal L40S GPU Instance, faster than L4 and cheaper than H100 PCIe

Maximize your AI infrastructure's potential with a versatile and cost-effective GPU Instance

Dedicated Tesla P100s for all your Machine Learning & Artificial Intelligence needs

Render GPU Instance

GPU Instances

Amazon S3-compatible and Multi-AZ resilient object storage service. Ensuring high availability for your data

Object Storage

Cold Storage class to secure long-term object storage. Ideal for deep archived data.

Scaleway Glacier

Flexible and reliable storage for demanding workloads

Block Storage

Storage

Secure your cloud resources with ease on a resilient regional network

Virtual Private Cloud

Centralize and simplify your Scaleway IP address management

IPAM (IP Address Manager)

Network

Improve the performance of your services as you grow.

Public Gateway

Establish a hosted connection from your infrastructure to your VPC via a partner's network

Scaleway InterLink

Buy domain names and manage DNS directly with your favorite extensions 

Domains and DNS

Connectivity

Load Balancer

Expose your HTTP services to the internet with security, reliability, and efficiency by design

Edge Services

Protect your web applications by proactively detecting and blocking cyber threats.

Traffic & Security

Managed Database for PostgreSQL and MySQL

Go full serverless and take the complexity out of PostgreSQL database

Serverless SQL Database

Managed Database for Redis®

Create a scalable, secure, and fully managed NoSQL solution

Managed MongoDB®

Query large data without managing servers

Data Warehouse for ClickHouse®

Databases

Build distributed and scalable client-server applications

NATS

Create a queue, configure its delivery and message parameters

Queues

Sent to a variety of devices and platforms through a single code interface

Topics and Events

Messaging and Queuing

Data

Monitor infrastructures in minutes with a fully managed observability solution

Cockpit

Hosting for individuals, professionals, and everyone in between.

Web Hosting

Ship your applications only in a few clicks.

Web Platform

Instant delivery of your transactional emails

Transactional Email

Speed up data processing over very large volumes of data with an Apache Spark™ managed solution

Data Lab for Apache Spark™ 

A purpose-built bridge between connected hardware and cloud.

IoT Hub

Deploy open-source technologies faster, scale effortlessly, and focus on innovation

Cloud Essentials

Managed Services

The easiest way to safely collaborate in the cloud

Identity and Access Management (IAM)

Protect your sensitive data across your cloud infrastructure

Secret Manager

Automate your encryption keys management to reach another level of security

Key Manager

Easily track your consumption in an all-in-one tool

Cost Manager

Accurately track your environmental impact and make informed choices

Environmental Footprint Calculator

Security & Organization

Scaleway API

Deploy and manage your infrastructure directly from the command line

Securely and efficiently provision and manage Infrastructure as Code with Terraform

Scaleway Blog

Must read

Evaluating the Frontier: Why AI Benchmarking Matters

Latest articles

Our latest security improvements — and what they mean for your projects

Scaleway’s Environmental Footprint Calculator: Your Best Partner for CSRD Reporting

Smarter. Faster. Everywhere. How the themes for ai-PULSE 2025 came to be

NoSQL use cases: when to choose a NoSQL Database, and why

In the End, It’s the Documentation That Saves You

Remote desktop with hardware acceleration on GPU servers using Linux

Deploy a Sovereign AI Chatbot on Scaleway : A Technical Deep Dive with Galene.AI

Hardware Lifecycle Management: Building Sustainable Cloud Infrastructure

Remote desktop with hardware acceleration on GPU servers using Linux

1st Perceval Quest, a journey into Quantum Machine Learning

How artificial intelligence is reshaping the financial sector

ReRAM Revival

Most popular articles

In the End, It’s the Documentation That Saves You

ReRAM Revival

Why is my state-of-the-art LLM spitting out nonsense?