#inference

Synthesize Realistic 3D Medical Images at Scale to Ship Pre‑Trained Models

High‑quality 3D medical imaging data is the foundation of modern radiology AI, but access to it is often constrained by …

Specialization Beats Scale: A Strategic Variable Most AI Procurement Decisions Overlook

Specialization Beats Scale: A Strategic Variable Most AI Procurement Decisions Overlook When a model’s training history …

Build real-time voice applications with Amazon SageMaker AI and vLLM

Artificial Intelligence Build real-time voice applications with Amazon SageMaker AI and vLLM Voice agents, live captioni…

Announcing OpenAI-compatible API support for Amazon SageMaker AI endpoints

Artificial Intelligence Announcing OpenAI-compatible API support for Amazon SageMaker AI endpoints Today, Amazon SageMak…

Cerebras Brings Trillion Parameter Inference to Enterprises with Kimi K2.6 >>

Cerebras is now running Kimi K2.6 — the leading trillion parameter open-weight model — in enterprise customer trials. Wi…

# production-serving ( 1 )

vLLM x Novita AI: PegaFlow for Production-Grade External KV Cache ·13 min read TL;DR: In collaboration with Novita AI, P…

vLLM x Novita AI: PegaFlow for Production-Grade External KV Cache May 18, 2026 · 13 min read TL;DR: In collaboration with Novita AI, PegaFlow integrates with vLLM as an external KV cache service for LLM inference, implemented as a standalone Rust process and connected through the external...

vLLM x Novita AI: PegaFlow for Production-Grade External KV Cache TL;DR: In collaboration with Novita AI, PegaFlow integ…

Running PyTorch Models on Apple Silicon GPUs with the ExecuTorch MLX Delegate

Featured projects TL;DR: Introducing the ExecuTorch MLX Delegate - The new MLX delegate enables optimized, GPU-accelerat…

vLLM and PyTorch Work Together to Improve the Developer Experience on aarch64

Featured projects TLDR: PyTorch 2.11 makes it possible to install CUDA-enabled PyTorch wheels on aarch64 Linux directly …

PaddleOCR 3.5: Running OCR and Document Parsing Tasks with a Transformers Backend

PaddleOCR 3.5: Running OCR and Document Parsing Tasks with a Transformers Backend engine="transformers" PaddleOCR contin…

# expert-parallelism ( 1 )

Elastic Expert Parallelism in vLLMMay 14, 2026·11 min readExpert parallelism (EP) is a key technique for serving Mixture…

# elastic-ep ( 1 )

Elastic Expert Parallelism in vLLMMay 14, 2026·11 min readExpert parallelism (EP) is a key technique for serving Mixture…

Announcing VeRL-Omni: Easy, Fast, and Stable RL Training for Diffusion and Omni-Modality Models May 14, 2026 · 7 min read We are excited to announce the pre-release of VeRL-Omni, a general reinforcement learning (RL) post-training framework focused on multimodal generative models, built on top of verl and vllm-omni.

Announcing VeRL-Omni: Easy, Fast, and Stable RL Training for Diffusion and Omni-Modality Models We are excited to announ…

Elastic Expert Parallelism in vLLM May 14, 2026 · 11 min read Expert parallelism (EP) is a key technique for serving Mixture-of-Experts (MoE) models at high throughput. WideEP deployments (where EP spans many workers) maximize KV cache capacity, enabling...

Elastic Expert Parallelism in vLLM Expert parallelism (EP) is a key technique for serving Mixture-of-Experts (MoE) model…

How the NVIDIA Vera Rubin Platform is Solving Agentic AI’s Scale-Up Problem

Agentic inference has fundamentally changed the runtime dynamics of inference workloads by introducing non-deterministic…

Unlocking asynchronicity in continuous batching

Unlocking asynchronicity in continuous batching TL;DR: we explain how to separate CPU and GPU workloads to get a massive…

Generating Beautiful UIs May 08, 2026

With contributions from Sherif Cherfa and Halley Chang There’s an intuitive skepticism we have toward AI-generated work.…

Fine-tune LLM with Databricks Unity Catalog and Amazon SageMaker AI

Artificial Intelligence Fine-tune LLM with Databricks Unity Catalog and Amazon SageMaker AI When you fine-tune large lan…

Efficient Edge AI on Arm CPUs and NPUs: Understanding ExecuTorch through Practical Labs

Featured projects TL;DR: - ExecuTorch extends the PyTorch ecosystem to deliver local AI inference on constrained edge de…

How to Eliminate Pipeline Friction in AI Model Serving

The path from a trained AI model to production should be smooth, but rarely is. Many teams invest weeks fine-tuning mode…

vLLM Tops the Artificial Analysis Leaderboard May 11, 2026 · 15 min read How vLLM built the leading deployments of DeepSeek V3.2, MiniMax-M2.5, and Qwen 3.5 397B.

vLLM Tops the Artificial Analysis Leaderboard How vLLM built the leading deployments of DeepSeek V3.2, MiniMax-M2.5, and…

How ChatGPT adoption broadened in early 2026

How ChatGPT adoption broadened in early 2026 Q1 data shows consumer adoption growth across inferred gender, age, and geo…

# kernel-fusion ( 1 )

vLLM Tops the Artificial Analysis LeaderboardMay 11, 2026·15 min readHow vLLM built the leading deployments of DeepSeek …

# benchmarking ( 1 )

vLLM Tops the Artificial Analysis LeaderboardMay 11, 2026·15 min readHow vLLM built the leading deployments of DeepSeek …

vLLM Tops the Artificial Analysis Leaderboard May 11, 2026 · 15 min read How vLLM built the leading deployments of DeepSeek V3.2, MiniMax-M2.5, and Qwen 3.5 397B.

vLLM Tops the Artificial Analysis Leaderboard How vLLM built the leading deployments of DeepSeek V3.2, MiniMax-M2.5, and…

Building Blocks for Foundation Model Training and Inference on AWS

Building Blocks for Foundation Model Training and Inference on AWS Figure: Adapted from "AI's Three Scaling Laws, Explai…

# turboquant ( 1 )

A First Comprehensive Study of TurboQuant: Accuracy and Performance ·12 min read TurboQuant, a method for KV-cache quant…

"OncoAgent: A Dual-Tier Multi-Agent Framework for Privacy-Preserving Oncology Clinical Decision Support"

"OncoAgent: A Dual-Tier Multi-Agent Framework for Privacy-Preserving Oncology Clinical Decision Support" - user: oncoage…

Model Quantization: Post-Training Quantization Using NVIDIA Model Optimizer

Model quantization is an effective method to reduce VRAM usage and improve inference performance on consumer devices suc…

Introducing Multi-LoRA on Cerebras Inference May 06, 2026

Today, we are launching Multi-LoRA—multi-adapter support for Low-Rank Adaptation—on Cerebras Inference in private previe…

Secure short-term GPU capacity for ML workloads with EC2 Capacity Blocks for ML and SageMaker training plans

Artificial Intelligence Secure short-term GPU capacity for ML workloads with EC2 Capacity Blocks for ML and SageMaker tr…

Serving Agentic Workloads at Scale with vLLM x Mooncake May 6, 2026 · 10 min read TL;DR: Agentic workloads generate massive shared prefixes that are often recomputed across turns. By integrating Mooncake's distributed KV cache store into vLLM, we achieve 3.8x higher throughput,...

Serving Agentic Workloads at Scale with vLLM x Mooncake TL;DR: Agentic workloads generate massive shared prefixes that a…

# agentic ( 1 )

Serving Agentic Workloads at Scale with vLLM x Mooncake ·10 min read TL;DR: Agentic workloads generate massive shared pr…

Serving Agentic Workloads at Scale with vLLM x Mooncake May 6, 2026 · 10 min read TL;DR: Agentic workloads generate massive shared prefixes that are often recomputed across turns. By integrating Mooncake's distributed KV cache store into vLLM, we achieve 3.8x higher throughput,...

Serving Agentic Workloads at Scale with vLLM x Mooncake TL;DR: Agentic workloads generate massive shared prefixes that a…

vLLM V0 to V1: Correctness Before Corrections in RL

vLLM V0 to V1: Correctness Before Corrections in RL TL;DR. vLLM V1 matched our vLLM V0 reference after we fixed four thi…

Case Studies 5/5/2026 Innovative Solutions Rebuilds Enterprise Services Delivery with Fireworks AI

Innovative Solutions, a Tier 1 AWS Premier Partner delivering hundreds of AI-driven services engagements annually, hit a…

Advancing youth safety and wellbeing in EMEA

Advancing youth safety and wellbeing in EMEA Announcing our European Youth Safety Blueprint and EMEA Youth & Wellbeing G…

MoE at Scale: Making Sparse Models Fast on Real Hardware September 03, 2025

In this video we discuss scaling MoE models on modern hardware and address key optimization challenges. If you can’t ope…

MoE Math Demystified: What Does 8x7B Actually Mean? October 14, 2025

This video breaks down MoE inference arithmetic and deployment bottlenecks across different hardware setups. If you can’…

In-Kernel Broadcast Optimization: Co-Designing Kernels for RecSys Inference

Featured projects TL;DR: - Traditional RecSys inference explicitly replicates shared user embeddings/sequences for every…

OpenAI president forced to read his personal diary entries to jury

Greg Brockman never wanted to discuss his personal journal in public. But the OpenAI president has been stuck for days d…

Capacity-aware inference: Automatic instance fallback for SageMaker AI endpoints

Artificial Intelligence Capacity-aware inference: Automatic instance fallback for SageMaker AI endpoints As organization…

Case Study - Cognition x Cerebras December 10, 2025

Dec 10 2025 Case Study - Cognition x Cerebras The Dawn of Real-Time Coding Agents TL;DR Powered by Cerebras Inference, C…

Cyber-Insecurity in the AI Era

Sponsored Cyber-Insecurity in the AI Era Presented byGC Cybersecurity Cybersecurity was already under strain before AI e…

Elon Musk Seemingly Admits xAI Has Used OpenAI’s Models to Train Its Own

While testifying on Thursday in federal court, Elon Musk seemed to indicate that his AI lab may have used OpenAI’s model…

SMG: The Case for Disaggregating CPU from GPU in LLM Serving

How It Started: Hitting the GIL Wall at Scale We’ve been running production model serving for many years. When we first …

Speed Up Unreal Engine NNE Inference with NVIDIA TensorRT for RTX Runtime

Neural network techniques are increasingly used in computer graphics to boost image quality, improve performance, and st…

Cybersecurity in the Intelligence Age

Cybersecurity in the Intelligence Age An action plan for democratizing AI-powered cyber defense. Artificial intelligence…

DeepInfra on Hugging Face Inference Providers 🔥

DeepInfra on Hugging Face Inference Providers 🔥 We're thrilled to share that DeepInfra is now a supported Inference Pro…

Run Highly Efficient Multimodal Agentic AI with NVIDIA Nemotron 3 Nano Omni Using vLLM Apr 28, 2026 · 7 min read We are excited to support the newly released NVIDIA Nemotron 3 Nano Omni model on vLLM.

Run Highly Efficient Multimodal Agentic AI with NVIDIA Nemotron 3 Nano Omni Using vLLM We are excited to support the new…

Run Highly Efficient Multimodal Agentic AI with NVIDIA Nemotron 3 Nano Omni Using vLLM Apr 28, 2026 · 7 min read We are excited to support the newly released NVIDIA Nemotron 3 Nano Omni model on vLLM.

Run Highly Efficient Multimodal Agentic AI with NVIDIA Nemotron 3 Nano Omni Using vLLM We are excited to support the new…

NVIDIA Nemotron 3 Nano Omni model now available on Amazon SageMaker JumpStart

Artificial Intelligence NVIDIA Nemotron 3 Nano Omni model now available on Amazon SageMaker JumpStart Today, we are exci…

GitHub will start charging Copilot users based on their actual AI usage

GitHub has announced that it will be shifting to a usage-based billing model for its GitHub Copilot AI service starting …

4/27/2026 DeepSeek V4 Pro: Validating Frontier Models For Production

Why we chose correctness over a Day-0 launch DeepSeek V4 Pro is one of the most important open-model releases this year,…

Choco automates food distribution with AI agents

Choco automates food distribution with AI agents Using OpenAI APIs, Choco processes millions of orders, reducing manual …

Musk and Altman face off in trial that will determine OpenAI's future

A hotly anticipated trial starts this week, where Elon Musk will attempt to prove that OpenAI, under Sam Altman, has aba…

DeepSeek V4 in vLLM: Efficient Long-context Attention Apr 24, 2026 · 17 min read A first-principles walkthrough of DeepSeek V4's long-context attention, and how we implemented it in vLLM.

DeepSeek V4 in vLLM: Efficient Long-context Attention We are excited to announce that vLLM now supports the DeepSeek V4 …

IBM Research uses vLLM at the heart of its RITS Platform

Featured projects TL;DR: vLLM has been critical to democratizing access to our research community to the latest and grea…

DeepSeek V4 in vLLM: Efficient Long-context Attention Apr 24, 2026 · 17 min read A first-principles walkthrough of DeepSeek V4's long-context attention, and how we implemented it in vLLM.

DeepSeek V4 in vLLM: Efficient Long-context Attention We are excited to announce that vLLM now supports the DeepSeek V4 …

Serving the For You feed

24th April 2026 - Link Blog Serving the For You feed. One of Bluesky's most interesting features is that anyone can run …

Figma - MultiAgents April 16, 2026

Everything is easier now. I have been toying around with agent orchestration for a while now. I’m currently running 10-2…

The State of FP8 KV-Cache and Attention Quantization in vLLM Apr 22, 2026 · 21 min read Long-context LLM serving is increasingly memory-bound: for standard full-attention decoders, the KV cache often dominates GPU memory at 128k+ contexts, and each decode step must read a large...

The State of FP8 KV-Cache and Attention Quantization in vLLM Introduction Long-context LLM serving is increasingly memor…

# fp8 ( 1 )

The State of FP8 KV-Cache and Attention Quantization in vLLM ·21 min read Long-context LLM serving is increasingly memor…

# kv_cache ( 1 )

The State of FP8 KV-Cache and Attention Quantization in vLLM ·21 min read Long-context LLM serving is increasingly memor…

The State of FP8 KV-Cache and Attention Quantization in vLLM Apr 22, 2026 · 21 min read Long-context LLM serving is increasingly memory-bound: for standard full-attention decoders, the KV cache often dominates GPU memory at 128k+ contexts, and each decode step must read a large...

The State of FP8 KV-Cache and Attention Quantization in vLLM Introduction Long-context LLM serving is increasingly memor…

Advancing Emerging Optimizers for Accelerated LLM Training with NVIDIA Megatron

Higher-order optimization algorithms such as Shampoo have been effectively applied in neural network training for at lea…

Amazon SageMaker AI now supports optimized generative AI inference recommendations

Artificial Intelligence Amazon SageMaker AI now supports optimized generative AI inference recommendations Organizations…

Cost-effective multilingual audio transcription at scale with Parakeet-TDT and AWS Batch

Artificial Intelligence Cost-effective multilingual audio transcription at scale with Parakeet-TDT and AWS Batch Many or…

Google unveils two new TPUs designed for the "agentic era"

Most of the companies that have fully committed to building AI models are gobbling up every Nvidia AI accelerator they c…

Disaggregated Serving for Hybrid SSM Models in vLLM Apr 21, 2026 · 15 min read Hybrid architectures that interleave Mamba-style SSM layers with standard full-attention (FA) layers — such as NVIDIA Nemotron-H — are gaining traction as a way to combine the linear-time...

Disaggregated Serving for Hybrid SSM Models in vLLM Introduction Hybrid architectures that interleave Mamba-style SSM la…

# mamba ( 1 )

Disaggregated Serving for Hybrid SSM Models in vLLM ·15 min read Hybrid architectures that interleave Mamba-style SSM la…

Disaggregated Serving for Hybrid SSM Models in vLLM Apr 21, 2026 · 15 min read Hybrid architectures that interleave Mamba-style SSM layers with standard full-attention (FA) layers — such as NVIDIA Nemotron-H — are gaining traction as a way to combine the linear-time...

Disaggregated Serving for Hybrid SSM Models in vLLM Introduction Hybrid architectures that interleave Mamba-style SSM la…

Run High-Throughput Reinforcement Learning Training with End-to-End FP8 Precision

As LLMs transition from simple text generation to complex reasoning, reinforcement learning (RL) plays a central role. A…

Lessons learned from building multi-agent workflows April 16, 2026

I pay my upfront subscription ($200/month), write what I hope is the right prompt (prompt AND context engineer), and wai…

Accelerate Generative AI Inference on Amazon SageMaker AI with G7e Instances

Artificial Intelligence Accelerate Generative AI Inference on Amazon SageMaker AI with G7e Instances As the demand for g…

Optimizing Effective Training Time for Meta’s Internal Recommendation/Ranking Workloads

Motivation and Introduction Across the industry, teams training and serving large AI models face aggressive ROI targets …

Full-Stack Optimizations for Agentic Inference with NVIDIA Dynamo

Coding agents are starting to write production code at scale. Stripe’s agents generate 1,300+ PRs per week. Ramp attribu…

Optimize video semantic search intent with Amazon Nova Model Distillation on Amazon Bedrock

Artificial Intelligence Optimize video semantic search intent with Amazon Nova Model Distillation on Amazon Bedrock Opti…

Cost-efficient custom text-to-SQL using Amazon Nova Micro and Amazon Bedrock on-demand inference

Artificial Intelligence Cost-efficient custom text-to-SQL using Amazon Nova Micro and Amazon Bedrock on-demand inference…

vLLM Korea Meetup 2026 Wrap-Up Apr 14, 2026 · 7 min read Hosted by the vLLM KR Community, with support from Rebellions, SqueezeBits, Red Hat APAC, and PyTorch Korea, the vLLM Korea Meetup 2026 was held in Seoul on April 2nd.

vLLM Korea Meetup 2026 Wrap-Up Hosted by the vLLM KR Community, with support from Rebellions, SqueezeBits, Red Hat APAC,…

vLLM Korea Meetup 2026 Wrap-Up Apr 14, 2026 · 7 min read Hosted by the vLLM KR Community, with support from Rebellions, SqueezeBits, Red Hat APAC, and PyTorch Korea, the vLLM Korea Meetup 2026 was held in Seoul on April 2nd.

vLLM Korea Meetup 2026 Wrap-Up Hosted by the vLLM KR Community, with support from Rebellions, SqueezeBits, Red Hat APAC,…

Canopy Labs’ Orpheus TTS is live on GroqCloud

Canopy Labs’ Orpheus TTS is live on GroqCloud In December, we announced support for Canopy Labs’ Orpheus text-to-speech …

Next-Level Inference: Why Your Single-Node vLLM Setup Needs Prefill-Decode Disaggregation Apr 7, 2026 · 22 min read TL;DR: Prefill and decode fight over the same GPUs, causing ITL spikes under load. We show how to disaggregate them on a single 8-GPU MI300X node using AMD's MORI-IO connector — achieving 2.5x...

Next-Level Inference: Why Your Single-Node vLLM Setup Needs Prefill-Decode Disaggregation TL;DR: Prefill and decode figh…

# disaggregation ( 1 )

Next-Level Inference: Why Your Single-Node vLLM Setup Needs Prefill-Decode Disaggregation ·22 min read TL;DR: Prefill an…

Next-Level Inference: Why Your Single-Node vLLM Setup Needs Prefill-Decode Disaggregation Apr 7, 2026 · 22 min read TL;DR: Prefill and decode fight over the same GPUs, causing ITL spikes under load. We show how to disaggregate them on a single 8-GPU MI300X node using AMD's MORI-IO connector — achieving 2.5x...

Next-Level Inference: Why Your Single-Node vLLM Setup Needs Prefill-Decode Disaggregation TL;DR: Prefill and decode figh…

The Debate of MCP vs. CLI Centers on Speed April 06, 2026

MCP had a formative year. Then it had a turbulent week. Perplexity CTO Denis Yarats walked on stage at Ask 2026 and anno…

4/6/2026 Own Your AI: Fireworks Training Preview

Fireworks Training is now in preview: an end-to-end platform for training and deploying frontier models at scale. Three …

Why speed wins: faster inference is about more than just quicker answers–it’s the new path to accuracy February 19, 2026

Feb 19 2026 Why speed wins: faster inference is about more than just quicker answers–it’s the new path to accuracy Watch…

4/3/2026 Scaling and Optimizing Frontier Model Training

On this page How Fireworks scales frontier model training and offers the broadest set of fine-tunable MoE models on any …

Announcing Gemma 4 on vLLM: Byte for byte, the most capable open models Apr 2, 2026 · 3 min read With the debut of Gemma 4, vLLM introduces immediate support for Google's most sophisticated open model lineup, spanning multiple hardware backends, with first-ever Day 0 support on Google TPUs,...

Announcing Gemma 4 on vLLM: Byte for byte, the most capable open models Elevating Open Models with Advanced Reasoning an…

Announcing Gemma 4 on vLLM: Byte for byte, the most capable open models Apr 2, 2026 · 3 min read With the debut of Gemma 4, vLLM introduces immediate support for Google's most sophisticated open model lineup, spanning multiple hardware backends, with first-ever Day 0 support on Google TPUs,...

Announcing Gemma 4 on vLLM: Byte for byte, the most capable open models Elevating Open Models with Advanced Reasoning an…

Achieving Single-Digit Microsecond Latency Inference for Capital Markets

In algorithmic trading, reducing response times to market events is crucial. To keep pace with high-speed electronic mar…

Accelerating Vision AI Pipelines with Batch Mode VC-6 and NVIDIA Nsight

In vision AI systems, model throughput continues to improve. The surrounding pipeline stages must keep pace, including d…

NVIDIA Platform Delivers Lowest Token Cost Enabled by Extreme Co-Design

Co-designed hardware, software, and models are key to delivering the highest AI factory throughput and lowest token cost…

Extracting hidden states from vLLM Mar 30, 2026 · 8 min read PR #33736 (included in vllm>=v0.18.0) introduced a new hidden states extraction system to vLLM. This blog post explores the motivation, design, usage, and future direction of this feature, and its...

Extracting hidden states from vLLM PR #33736 (included in vllm>=v0.18.0 ) introduced a new hidden states extraction syst…

Extracting hidden states from vLLM Mar 30, 2026 · 8 min read PR #33736 (included in vllm>=v0.18.0) introduced a new hidden states extraction system to vLLM. This blog post explores the motivation, design, usage, and future direction of this feature, and its...

Extracting hidden states from vLLM PR #33736 (included in vllm>=v0.18.0 ) introduced a new hidden states extraction syst…

Liberate your OpenClaw

Liberate your OpenClaw 🦀 If you've been cut off and your OpenClaw, Pi, or Open Code agents need resuscitation, you can …

Partner Spotlight: Armis + Cerebras Enable Teams Build and Secure Software Faster March 27, 2026

Mar 27 2026 Partner Spotlight: Armis + Cerebras Enable Teams Build and Secure Software Faster At Cerebras, we’ve always …

Jais 2: A Blueprint for Sovereign AI December 09, 2025

Arabic is spoken by more than 400 million people, yet Arabic-centric Large Language Models (LLMs)still lag behind Englis…

Cerebras is coming to AWS March 13, 2026

The world’s fastest inference is coming to the world’s leading cloud. Today we're announcing that Amazon Web Services is…