#benchmark

Specialization Beats Scale: A Strategic Variable Most AI Procurement Decisions Overlook

Specialization Beats Scale: A Strategic Variable Most AI Procurement Decisions Overlook When a model’s training history …

Multimodal evaluators: MLLM-as-a-judge for image-to-text tasks in Strands Evals

Artificial Intelligence Multimodal evaluators: MLLM-as-a-judge for image-to-text tasks in Strands Evals If you’re buildi…

Mastering Agentic Techniques: AI Agent Evaluation

Evaluating an AI model and evaluating an AI agent are related—but they answer fundamentally different questions. A model…

The Open Agent Leaderboard

The Open Agent Leaderboard How good are general purpose AI agents? We built an open evaluation framework to find out. Mo…

Prompting Amazon Nova 2 for content moderation

Artificial Intelligence Prompting Amazon Nova 2 for content moderation If you moderate user-generated content at scale, …

vLLM Tops the Artificial Analysis Leaderboard May 11, 2026 · 15 min read How vLLM built the leading deployments of DeepSeek V3.2, MiniMax-M2.5, and Qwen 3.5 397B.

vLLM Tops the Artificial Analysis Leaderboard How vLLM built the leading deployments of DeepSeek V3.2, MiniMax-M2.5, and…

# benchmarking ( 1 )

vLLM Tops the Artificial Analysis LeaderboardMay 11, 2026·15 min readHow vLLM built the leading deployments of DeepSeek …

vLLM Tops the Artificial Analysis Leaderboard May 11, 2026 · 15 min read How vLLM built the leading deployments of DeepSeek V3.2, MiniMax-M2.5, and Qwen 3.5 397B.

vLLM Tops the Artificial Analysis Leaderboard How vLLM built the leading deployments of DeepSeek V3.2, MiniMax-M2.5, and…

Adding Benchmaxxer Repellant to the Open ASR Leaderboard

Adding Benchmaxxer Repellant to the Open ASR Leaderboard TLDR: Appen Inc. and DataoceanAI have provided high-quality Eng…

AI evals are becoming the new compute bottleneck

AI evals are becoming the new compute bottleneck Summary. AI evaluation has crossed a cost threshold that changes who ca…

WHY ARE YOU LIKE THIS

25th April 2026 @scottjla on Twitter in reply to my pelican riding a bicycle benchmark: I feel like we need to stack the…

Qwen3.6-27B: Flagship-Level Coding in a 27B Dense Model

22nd April 2026 - Link Blog Qwen3.6-27B: Flagship-Level Coding in a 27B Dense Model (via) Big claims from Qwen about the…

QIMMA قِمّة ⛰: A Quality-First Arabic LLM Leaderboard

QIMMA قِمّة ⛰: A Quality-First Arabic LLM Leaderboard QIMMA validates benchmarks before evaluating models, ensuring repo…

ToolSimulator: scalable tool testing for AI agents

Artificial Intelligence ToolSimulator: scalable tool testing for AI agents You can use ToolSimulator, an LLM-powered too…

Import AI 453: Breaking AI agents; MirrorCode; and ten views on gradual disempowerment

Import AI 453: Breaking AI agents; MirrorCode; and ten views on gradual disempowerment Was fire equivalent to a singular…

ADeLe: Predicting and explaining AI performance across tasks

At a glance - AI benchmarks report performance on specific tasks but provide limited insight into underlying capabilitie…

AsgardBench: A benchmark for visually grounded interactive planning

At a glance - To successfully complete tasks, embodied AI agents must ground and update their plans based on visual feed…

Ulysses Sequence Parallelism: Training with Million-Token Contexts

Ulysses Sequence Parallelism: Training with Million-Token Contexts Ulysses Sequence Parallelism (part of the Arctic Long…

2/3/2026 The Benchmark Gap: What It Takes to Ship Kimi K2.5

The Benchmark Gap: What It Takes to Ship Kimi K2.5 Kimi K2.5 is live on Fireworks at ~1/10 the cost and 2-3x the speed o…

Pacific Northwest National Laboratory and OpenAI partner to accelerate federal permitting

Pacific Northwest National Laboratory and OpenAI partner to accelerate federal permitting New benchmark shows potential …

Shaping the future of financial services

Morgan Stanley uses AI evals to shape the future of financial services Morgan Stanley(opens in a new window) collaborate…

ExomeBench: A Benchmark for Clinical Variant Interpretation in Exome Regions February 23, 2026

Feb 23 2026 ExomeBench: A Benchmark for Clinical Variant Interpretation in Exome Regions 1. What is ExomeBench? We are e…

Import AI 446: Nuclear LLMs; China's big AI benchmark; measurement and AI policy

Import AI 446: Nuclear LLMs; China's big AI benchmark; measurement and AI policy Will AIs be jealous of one another? Wel…

Introducing EVMbench

Introducing EVMbench Making smart contracts safer by evaluating AI agents’ ability to detect, patch, and exploit vulnera…

Topping the GPU MODE Kernel Leaderboard with NVIDIA cuda.compute

Python dominates machine learning for its ergonomics, but writing truly fast GPU code has historically meant dropping in…

IBM and UC Berkeley Diagnose Why Enterprise Agents Fail Using IT-Bench and MAST

IBM and UC Berkeley Diagnose Why Enterprise Agents Fail Using IT-Bench and MAST ITBench HF Space ITBench HF Dataset MAST…

Import AI 445: Timing superintelligence; AIs solve frontier math proofs; a new ML research benchmark

Import AI 445: Timing superintelligence; AIs solve frontier math proofs; a new ML research benchmark Will 2026 be looked…

OpenEnv in Practice: Evaluating Tool-Using Agents in Real-World Environments

OpenEnv in Practice: Evaluating Tool-Using Agents in Real-World Environments OpenEnv is an open-source framework from Me…

Community Evals: Because we're done trusting black-box leaderboards over the community

Community Evals: Because we're done trusting black-box leaderboards over the community TL;DR: Benchmark datasets on Hugg…

3/2/2026 Best LLMs for coding in 2026

TL;DR The best LLM for coding in 2026 depends on your workload: The short answer to "which LLM is best for coding" depen…

Advancing AI benchmarking with Game Arena

Advancing AI benchmarking with Game Arena Chess is a game of perfect information. The real world is not. Last year, Goog…

1/27/2026 Build powerful agents on OSS models with Blazing Fast Inference on Fireworks

Kimi K2.5 just dropped yesterday and is available Day 0 on Fireworks! As open models get more powerful and agentic, low …

AssetOpsBench: Bridging the Gap Between AI Agent Benchmarks and Industrial Reality

AssetOpsBench: Bridging the Gap Between AI Agent Benchmarks and Industrial Reality Introduction While existing AI benchm…

1/13/2026 Best Open Source LLMs in 2026: We Reviewed 7 Models

With new open source LLMs launching nearly every week, figuring out which model actually fits your use case has become i…

Import AI 440: Red queen AI; AI regulating AI; o-ring automation

Import AI 440: Red queen AI; AI regulating AI; o-ring automation How many of your are LLMs? Welcome to Import AI, a news…

NVIDIA Cosmos Reason 2 Brings Advanced Reasoning To Physical AI

NVIDIA Cosmos Reason 2 Brings Advanced Reasoning To Physical AI NVIDIA today released Cosmos Reason 2, the latest advanc…

The State Of LLMs 2025: Progress, Problems, and Predictions

The State Of LLMs 2025: Progress, Problems, and Predictions As 2025 comes to a close, I want to look back at some of the…

The Open Evaluation Standard: Benchmarking NVIDIA Nemotron 3 Nano with NeMo Evaluator

The Open Evaluation Standard: Benchmarking NVIDIA Nemotron 3 Nano with NeMo Evaluator NVIDIA released Nemotron 3 Nano 30…

12/17/2025 Self-Improving Agents, Powered by Your Evals

TL;DR: Eval Protocol is a unified eval interface that powers both prompt optimization and RL on the same evaluation func…

Evaluating AI’s ability to perform scientific research tasks

Evaluating AI’s ability to perform scientific research tasks We introduce FrontierScience, a new benchmark that evaluate…

CUGA on Hugging Face: Democratizing Configurable AI Agents

CUGA on Hugging Face: Democratizing Configurable AI Agents Introduction AI agents are rapidly becoming essential for bui…

Advancing science and math with GPT-5.2

Advancing science and math with GPT‑5.2 GPT‑5.2 is our strongest model yet for math and science work. One of our hopes f…

Open ASR Leaderboard: Trends and Insights with New Multilingual & Long-Form Tracks

Open ASR Leaderboard: Trends and Insights with New Multilingual & Long-Form Tracks Most benchmarks focus on short-form E…

Signal-Decision Driven Architecture: Reshaping Semantic Routing at Scale Nov 19, 2025 · 14 min read The earlier versions of vLLM Semantic Router relied on classification-based routing, a straightforward approach where user queries are classified into one of 14 MMLU domain categories, and then...

Signal-Decision Driven Architecture: Reshaping Semantic Routing at Scale The earlier versions of vLLM Semantic Router re…

How evals drive the next chapter in AI for businesses

How evals drive the next chapter in AI for businesses This primer teaches business leaders how evaluation frameworks (“e…

OpenAI GPT-OSS 120B Benchmarked – NVIDIA Blackwell vs. Cerebras November 06, 2025

A year ago, Cerebras launched its inference API—setting a new benchmark for AI performance. While GPU-based providers we…

6/11/2025 Building AI agents with the Fireworks Experimentation Platform (GA) and Build SDK (Beta)

When building AI agents, the best AI companies are jointly developing their product and models in a process of rapid, co…

Introducing IndQA

Our mission is to make AGI benefit all of humanity. If AI is going to be useful for everyone, it needs to work well acro…

Addendum to GPT-5 System Card: Sensitive conversations

Addendum to GPT‑5 System Card: Sensitive conversations When we launched GPT‑5, we noted in the system card that we were …

Google Cloud C4 Brings a 70% TCO improvement on GPT OSS with Intel and Hugging Face

Google Cloud C4 Brings a 70% TCO improvement on GPT OSS with Intel and Hugging Face C4 Virtual Machine (VM) running on I…

BigCodeArena: Judging code generations end to end with code executions

BigCodeArena: Judging code generations end to end with code executions Inspired by LMArena for LLMs, we've built a platf…

Introducing AgentKit, new Evals, and RFT for agents

Today we’re launching AgentKit, a complete set of tools for developers and enterprises to build, deploy, and optimize ag…

Understanding the 4 Main Approaches to LLM Evaluation (From Scratch)

Understanding the 4 Main Approaches to LLM Evaluation (From Scratch) Multiple-Choice Benchmarks, Verifiers, Leaderboards…

Welcome EmbeddingGemma, Google's new efficient embedding model

Welcome EmbeddingGemma, Google's new efficient embedding model TL;DR Today, Google releases EmbeddingGemma, a state-of-t…

8/15/2025 Your AI Benchmark is Lying to You. Here's How We Caught It

Your AI Benchmark is Lying to You. Here's How We Caught It Would you give GPT-4.1 an A grade for this image? We sure wou…

Kimina-Prover-RL

Kimina-Prover-RL We are happy to introduce kimina-prover-rl, an open-source training pipeline for formal theorem proving…

📚 3LM: A Benchmark for Arabic LLMs in STEM and Code

📚 3LM: A Benchmark for Arabic LLMs in STEM and Code Why 3LM? Arabic Large Language Models (LLMs) have seen notable prog…

OpenBench: Reproducible LLM Evals Made Easy

OpenBench: Reproducible LLM Evals Made Easy Evaluating large language models (LLMs) today is fundamentally broken. If yo…

7/30/2025 Fireworks Real-World Benchmarks: Find the Best OSS Model for the Job

The open-source model landscape is exploding, making it hard to choose the right model. To help you cut through the nois…

Back to The Future: Evaluating AI Agents on Predicting Future Events

Back to The Future: Evaluating AI Agents on Predicting Future Events Future of AI Most current AI benchmarks focus on an…

Kimina-Prover: Applying Test-time RL Search on Large Formal Reasoning Models

Kimina-Prover: Applying Test-time RL Search on Large Formal Reasoning Models Numina & Kimi Team We're excited to announc…

5/28/2025 FireAttention V4: Industry-Leading Latency and Cost Efficiency with FP4

Today, we’re announcing we've achieved industry-leading speeds of >250 tokens/second on NVIDIA B200 GPUs using our lates…

Introducing HealthBench

Introducing HealthBench An evaluation for AI systems and human health. Improving human health will be one of the definin…

4/28/2025 Optimizing Llama 4 Maverick on Fireworks AI

Meta's Llama 4 Maverick is their initial natively-multimodal, Mixture-of-Experts (MoE) model. This model processes both …

BrowseComp: a benchmark for browsing agents

BrowseComp: a benchmark for browsing agents A simple and challenging benchmark that measures the ability of AI agents to…

OpenAI Pioneers Program

Announcing OpenAI Pioneers Program Advancing model performance and real world evaluation in applied domains. Today, we’r…

Arabic Leaderboards: Introducing Arabic Instruction Following, Updating AraGen, and More

Arabic Leaderboards: Introducing Arabic Instruction Following, Updating AraGen, and More As part of our ongoing efforts,…

PaperBench: Evaluating AI’s Ability to Replicate AI Research

PaperBench Evaluating AI’s Ability to Replicate AI Research. We introduce PaperBench, a benchmark evaluating the ability…

Introducing the SWE-Lancer benchmark

Introducing the SWE-Lancer benchmark Can frontier LLMs earn $1 million from real-world freelance software engineering? W…

Fixing Open LLM Leaderboard with Math-Verify

Fixing Open LLM Leaderboard with Math-Verify Today, we’re thrilled to share that we’ve used Math-Verify to thoroughly re…

1 Billion Classifications

1 Billion Classifications These tasks often use encoder models, which are much smaller than modern LLMs, but at the 1B+ …

From Where Things Are to What They’re For: Benchmarking Spatial–Functional Intelligence for Multimodal LLMs

True spatial intelligence for multimodal agents transcends low-level geometric perception, evolving from knowing where t…

12/2/2025 Unlock Advanced Reasoning with NVIDIA Nemotron Nano 2 Models on Fireworks AI

We're excited to collaborate with NVIDIA to bring their groundbreaking NVIDIA Nemotron Nano 2 9B models to the Fireworks…

The Open Arabic LLM Leaderboard 2

The Open Arabic LLM Leaderboard 2 Current status of Arabic LLMs leaderboards The growing availability of LLMs supporting…

DABStep: Data Agent Benchmark for Multi-step Reasoning

DABStep: Data Agent Benchmark for Multi-step Reasoning To tackle this challenge, Adyen and Hugging Face built the Data A…

Train 400x faster Static Embedding Models with Sentence Transformers

Train 400x faster Static Embedding Models with Sentence Transformers TL;DR This blog post introduces a method to train s…

AI Agents Are Here. What Now?

AI Agents Are Here. What Now? Introduction The sudden, rapid advancement of LLM capabilities – such as writing fluent se…

CO₂ Emissions and Models Performance: Insights from the Open LLM Leaderboard

CO₂ Emissions and Models Performance: Insights from the Open LLM Leaderboard In the last year, people have become more a…

8/1/2025 Kimi K2: Deep Dive into model performance and use-cases

Kimi K2 excels in specialized real-world software engineering tasks, achieving a 65.8% score on the SWE-Bench Verified b…

8/1/2025 Qwen3 Decoded: Choosing the Right Model For Your Task

With Thinking, Instruct, and Coder released simultaneously, confusion spiked. We stress-tested all three on your real wo…

Evaluating Audio Reasoning with Big Bench Audio

Evaluating Audio Reasoning with Big Bench Audio To support analysis of this, Artificial Analysis is releasing Big Bench …

Benchmarking Language Model Performance on 5th Gen Xeon at GCP

Benchmarking Language Model Performance on 5th Gen Xeon at GCP Introduction People believe the next frontier of artifici…

Rethinking LLM Evaluation with 3C3H: AraGen Benchmark and Leaderboard

Rethinking LLM Evaluation with 3C3H: AraGen Benchmark and Leaderboard The AraGen leaderboard makes three key contributio…

Introducing the Open Leaderboard for Japanese LLMs!

Introduction to the Open Leaderboard for Japanese LLMs We'd like to announce the Open Japanese LLM Leaderboard, composed…

Judge Arena: Benchmarking LLMs as Evaluators

Judge Arena: Benchmarking LLMs as Evaluators We’re excited to launch Judge Arena - a platform that lets anyone easily co…

Introducing SimpleQA

Introducing SimpleQA A factuality benchmark called SimpleQA that measures the ability for language models to answer shor…

CinePile 2.0 - making stronger datasets with adversarial refinement

CinePile 2.0 - making stronger datasets with adversarial refinement We're excited to share both CinePile 2.0 and our adv…

MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering

MLE-bench Evaluating Machine Learning Agents on Machine Learning Engineering We introduce MLE-bench, a benchmark for mea…

Introducing the Open FinLLM Leaderboard

Introducing the Open FinLLM Leaderboard The growing complexity of financial language models (LLMs) necessitates evaluati…

🇨🇿 BenCzechMark - Can your LLM Understand Czech?

🇨🇿 BenCzechMark - Can your LLM Understand Czech? - Reason and perform complex tasks in Czech. - Generate and verify gr…

OpenAI o1-mini

We're releasing OpenAI o1‑mini, a cost-efficient reasoning model. o1‑mini excels at STEM, especially math and coding—nea…

Learning to reason with LLMs

OpenAI o1 ranks in the 89th percentile on competitive programming questions (Codeforces), places among the top 500 stude…

What's Missing From LLM Chatbots: A Sense of Purpose

LLM-based chatbots’ capabilities have been advancing every month. These improvements are mostly measured by benchmarks l…

Our Transformers Code Agent beats the GAIA benchmark 🏅

Our Transformers Code Agent beats the GAIA benchmark 🏅 TL;DR After some experiments, we were impressed by the performan…

Replicate Intelligence #6

Replicate Intelligence #6 Welcome to Replicate’s weekly bulletin! Each week, we’ll bring you updates on the latest open-…

Data Is Better Together: A Look Back and Forward

Data Is Better Together: A Look Back and Forward Now, we have decided to move forward with the same goal. To provide an …

Launching the Artificial Analysis Text to Image Leaderboard & Arena

Launching the Artificial Analysis Text to Image Leaderboard & Arena The Artificial Analysis Text to Image Leaderboard ai…

Benchmarking Text Generation Inference

Benchmarking Text Generation Inference I’ll show you how to do this in a convenient Hugging Face Space. You can take the…

Introducing the Open Arabic LLM Leaderboard

Introducing the Open Arabic LLM Leaderboard This initiative is particularly significant given that it directly serves ov…

Introducing the Open Leaderboard for Hebrew LLMs!

Introducing the Open Leaderboard for Hebrew LLMs! Hebrew is a morphologically rich language with a complex system of roo…