Elon Musk Seemingly Admits xAI Has Used OpenAI’s Models to Train Its Own

The great American data center divide

In Tazewell County, Illinois, Michael Deppert depends on a natural pool of water beneath the sandy soils of his farm to irrigate the pumpkins, corn, and soybeans growing in his fields. So when a data center was proposed about eight miles away, he feared it would tap the same aquifer, potentially eroding crop yields and profits. Deppert, who is also the president of the local farm bureau lobby group, says locals were also “nervous” about how a data center would affect the “good, clean drinking water.” Residents launched a fierce opposition campaign, packing city council meetings and mounting petitions. After several months, the project, led by developer Western Hospitality Partners, was scrapped. “You just can’t lay down and let everybody do whatever they wish,” Deppert says. It is just one of the many pockets of resistance opening up across rural…

Musk and Altman face off in trial that will determine OpenAI's future

A hotly anticipated trial starts this week, where Elon Musk will attempt to prove that OpenAI, under Sam Altman, has abandoned its mission to remain a nonprofit in order to ensure that artificial intelligence serves humanity, and not just billionaires. Many view the lawsuit as a grudge match between Musk—who left OpenAI after serving as an early major donor and advisor—and Altman—who currently runs OpenAI, despite insiders’ allegedly growing distrust in his commitment to the dominant AI firm’s mission. But the lawsuit is about much more than a couple billionaires’ big egos. The outcome could radically change the AI landscape, impacting how OpenAI runs and what resources the firm will have to uphold its mission. If Musk wins, OpenAI’s hopes of growing a for-profit arm that can fund the nonprofit could be dashed. Additionally, Brockman and Altman could be dropped…

Greenhouse gases from data center boom could outpace entire nations

New gas projects linked to just 11 data center campuses around the US have the potential to create more greenhouse gases than the country of Morocco emitted in 2024. Emissions estimates from air permit documents examined by WIRED show that these natural gas projects—which are being built to power data centers to serve some of the US’s most powerful AI companies, including OpenAI, Meta, Microsoft, and xAI—have the potential to emit more than 129 million tons of greenhouse gases per year. As tech companies race to secure massive power deals to build out hundreds of data centers across the country, these projects represent just the tip of the iceberg when it comes to the potential climate cost of the AI boom. The infrastructure on this list of large natural gas projects reviewed by WIRED is being developed to largely bypass…

Pentagon wants $54B for drones, more than most nations’ military budgets

The US military’s massive $1.5 trillion budget request for the next fiscal year includes what Pentagon officials described as the largest investment in drone warfare and counter-drone technology in US history. The proposed spending on drone and autonomous warfare technologies within the FY2027 budget proposal for the US Department of Defense would surpass most countries’ defense budgets and rank among the top 10 in the world for military spending, ahead of countries such as Ukraine, South Korea, and Israel. Specifically, the Pentagon is requesting $53.6 billion to boost US production and procurement of drones, train drone operators, build out a logistics network for sustaining drone deployments, and expand counter-drone systems to defend more US military sites. The funding request is budgeted under the Defense Autonomous Warfare Group (DAWG), an organization established in late 2025 that would see a massive budget…

Robot runner handily beats humans in half-marathon, setting new record

Humanoid robots outran the fastest human competitors while surpassing the human world record during a half-marathon event held in Beijing on April 19. The demonstration of fast-improving robotic speed and autonomy comes as China’s tech industry is rapidly scaling up mass production of humanoid robots to explore possible uses in the real world. The fastest robot from Chinese smartphone-maker Honor notched a winning time of 50 minutes and 26 seconds while autonomously navigating the 13-mile (21-kilometer) route, according to the Global Times. That beat the human world record of 57 minutes and 20 seconds recently set by Ugandan long-distance runner Jacob Kiplimo during the Lisbon Half Marathon. The winning robot design took inspiration from top human athletes by incorporating long legs measuring approximately 37 inches (95 centimeters) in length, said Du Xiaodi, a test development engineer for Honor, who spoke…

Mozilla launches Thunderbolt AI client with focus on self-hosted infrastructure

Mozilla is the latest legacy tech brand to make a play for the enterprise AI market. But the company behind Firefox and Thunderbird isn’t releasing its own standalone AI model or agentic browser. Instead, the newly announced Thunderbolt is being sold as a front-end client for users and businesses who want to run their own self-hosted AI infrastructure without relying on cloud-based third-party services. Thunderbolt is built on top of Haystack, an existing open source AI framework that lets users build custom, modular AI pipelines from user-chosen components. Thunderbolt acts as what Mozilla calls a “sovereign AI client” on top of that underlying infrastructure. The combo promises to let users easily plug into any ACP-compatible agent or OpenAI-compatible API (including Claude, Codex, OpenClaw, DeepSeek, and OpenCode). The system can also integrate with locally stored enterprise data through open protocols and…

Configuring Amazon Bedrock AgentCore Gateway for secure access to private resources

Artificial Intelligence Configuring Amazon Bedrock AgentCore Gateway for secure access to private resources AI agents in production environments often need to reach internal APIs, databases, and private resources that sit behind Amazon Virtual Private Cloud (Amazon VPC) boundaries. Managing private connectivity for each agent-to-tool path adds operational overhead and slows deployment. Amazon Bedrock AgentCore VPC connectivity is designed to deploy AI agents and Model Context Protocol (MCP) servers without requiring the network traffic to be exposed to the public internet. This capability extends to managed Amazon VPC egress for Amazon Bedrock AgentCore Gateway, so you can connect to endpoints inside private networks across your AWS environment. In this post, you will configure Amazon Bedrock AgentCore Gateway to access private endpoints using Resource Gateway, a managed construct that provisions Elastic Network Interfaces (ENIs) directly inside your Amazon VPC, one per subnet.…

Unleashing Agentic AI Analytics on Amazon SageMaker with Amazon Athena and Amazon Quick

Artificial Intelligence Unleashing Agentic AI Analytics on Amazon SageMaker with Amazon Athena and Amazon Quick Modern enterprises face mounting challenges in extracting actionable insights from vast data lakes and lakehouses spanning petabytes of structured and unstructured data. Traditional analytics require specialized technical expertise in SQL, data modeling, and business intelligence tools, creating bottlenecks that slow decision-making across retail, financial services, healthcare, Travel & Hospitality, manufacturing and many more industries. This architecture demonstrates how agentic AI assistant from Amazon Quick transform data analytics into a self-service capability. It showcases enabling business users to query complex structured datasets and mix with unstructured data to find the valuable insights to improve their business outcomes through intuitive natural language interfaces. To demonstrate the functionality, we built a lakehouse using the TPC-H datasets as our foundation. This integrated architecture leverages Amazon Simple Storage Service (Amazon…

How Popsa used Amazon Nova to inspire customers with personalised title suggestions

Artificial Intelligence How Popsa used Amazon Nova to inspire customers with personalised title suggestions This post was co-written with Bradley Grantham and Hugo Dugdale from Popsa. Popsa is a technology company that helps users rediscover and relive the meaningful memories hidden in their photo libraries. Available across more than 50 countries and 12 languages, we use design automation and AI to transform everyday photos into personal, shareable experiences, including beautifully printed Photo Books. In 2016, we released PrintAI, a pioneering algorithm to take complete control of creating a varied and interesting design from a user’s photos. Our customers could use the algorithm to create Photo Books that appeared professionally designed, in less than 5 minutes. A core philosophy of our business is that technology should do the heavy lifting for our users, so automation has always been an intrinsic part…

Build and deploy an automatic sync solution for Amazon Bedrock Knowledge Bases

Artificial Intelligence Build and deploy an automatic sync solution for Amazon Bedrock Knowledge Bases With Amazon Bedrock Knowledge Bases, you can give foundation models (FMs) and agents contextual information from your organization’s private data sources to deliver more relevant, accurate, and customized responses. As the data grows, maintaining real-time synchronization between Amazon Simple Storage Service (Amazon S3) and your knowledge bases becomes critical for accurate, up-to-date responses.In this post, we explore how Deloitte used Amazon EKS and vCluster to transform their testing infrastructure. In this post, we explore an automated solution that detects S3 events and triggers ingestion jobs while respecting service quotas and providing comprehensive monitoring. This serverless solution uses an event-driven architecture to keep your knowledge base current without overwhelming the Amazon Bedrock APIs. The challenge Knowledge bases in Amazon Bedrock require manual synchronization whenever documents are added,…

Applying multimodal biological foundation models across therapeutics and patient care

Artificial Intelligence Applying multimodal biological foundation models across therapeutics and patient care Healthcare and life sciences decision making increasingly relies on multimodal data to diagnose diseases, prescribe medicine and predict treatment outcomes, develop and optimize innovative therapies accurately. Traditional approaches analyze fragmented data, such as ‘omics for drug discovery, medical images for diagnostics, clinical trial reports for validation, and electronic health records (EHR) for patient treatment. As a result, decision makers (CxOs, VPs, Directors) often miss critical insights hidden in the relationships between data types. Recent advancements in AI enable you to integrate and analyze these fragmented data streams efficiently to support a more complete understanding of therapeutics and patient care. AWS provides a unified environment for multimodal biological foundation models (BioFMs), enabling you to make more confident, timely decision-making in personalized medicine. This AI system combines biological data, model…

Get to your first working agent in minutes: Announcing new features in Amazon Bedrock AgentCore

Artificial Intelligence Get to your first working agent in minutes: Announcing new features in Amazon Bedrock AgentCore Getting an agent running has always meant solving a long list of infrastructure problems before you can test whether the agent itself is any good. You wire up frameworks, storage, authentication, and deployment pipelines, and by the time your agent handles its first real task, you’ve spent days on infrastructure instead of agent logic. We built AgentCore from the ground up to help developers focus on building agent logic instead of backend plumbing, working with frameworks and models they already use, including LangGraph, LlamaIndex, CrewAI, Strands Agents, and more. Today, we’re introducing new capabilities that further streamline the agent building experience, removing the infrastructure barriers that slow teams down at every stage of agent development from the first prototype through production deployment. Go…

Amazon SageMaker AI now supports optimized generative AI inference recommendations

Artificial Intelligence Amazon SageMaker AI now supports optimized generative AI inference recommendations Organizations are racing to deploy generative AI models into production to power intelligent assistants, code generation tools, content engines, and customer-facing applications. But deploying these models to production remains a weeks-long process of navigating GPU configurations, optimization techniques, and manual benchmarking, delaying the value these models are built to deliver. Today, Amazon SageMaker AI supports optimized generative AI inference recommendations. By delivering validated, optimal deployment configurations with performance metrics, Amazon SageMaker AI keeps your model developers focused on building accurate models, not managing infrastructure. We evaluated several benchmarking tools and chose NVIDIA AIPerf, a modular component of NVIDIA Dynamo, because it exposes detailed, consistent metrics and supports diverse workloads out of the box. Its CLI, concurrency controls, and dataset options give us the flexibility to iterate quickly and…

4/24/2026 Notes on DeepSeek-V4's training system

On this page DeepSeek-V4 is interesting less for any single benchmark number than for the shape of the system around it. The paper shows architecture, routing, reward modeling, reasoning modes, distillation, and agent execution all becoming part of the training loop. The useful takeaway for training infrastructure is obvious: fixed recipes are not enough. Researchers increasingly need programmable loops, while the platform handles distributed execution, inference integration, checkpointing, and scaling underneath. Supporting that flexibility is the core design principle behind the Fireworks Training API. DeepSeek-V4 alternates between Compressed Sparse Attention (CSA) and Heavily Compressed Attention (HCA). CSA compresses KV entries and then does sparse top-k selection. HCA compresses more aggressively, but keeps dense attention over the compressed memory. The point is not just "longer context." It is model/runtime co-design: attention pattern, KV layout, precision, sparse selection, and inference kernels all…

4/6/2026 Own Your AI: Fireworks Training Preview

Fireworks Training is now in preview: an end-to-end platform for training and deploying frontier models at scale. Three surfaces for three kinds of teams, from a conversational agent that handles everything, to managed infrastructure for ML engineers, to bring-your-own training loop on Fireworks-hosted clusters. All on the same infrastructure that already handles production inference for Cursor, Vercel, Genspark, and others. All three surfaces are in preview now. Reinforcement learning is how teams push past the ceiling SFT hits on multi-step reasoning, reliable tool use, and mid-flight self-correction. Vercel used our RL infrastructure to build a custom "Auto Fix" model for v0. The model checks the output stream for errors and self-corrects without a second pass, reaching a 93% error-free generation rate, significantly outperforming closed frontier models, with a 40X improvement in end-to-end latency vs. the proprietary model it replaced and…

3/23/2026 Frontier RL Is Cheaper Than You Think

On this page The conventional wisdom on RL infrastructure is wrong, and it is costing teams that could be competing at the frontier. The entire mega-cluster narrative rests on a single assumption: that you have to ship 1 TB of weights every time you update your rollout fleet. You do not. Researchers have spent the last year writing about asynchronous RL and rollout-training disaggregation in systems like AReaL. Teams like Kimi and MiniMax have also published engineering notes on RL parameter updates and asynchronous scheduling. We have been running that pattern in production. That mega-cluster instinct comes from pretraining, where the main systems problem is keeping one huge synchronous training job saturated. RL is a different problem. The question is not just how to run the trainer. It is also how to keep a large rollout fleet generating data from…

3/10/2026 Training-Inference Parity in MoE Models: Where Numerics Drift

On this page Kernel fusions that are mathematically equivalent can still drift numerically. Here are the parity bugs we hit across both Kimi K2.5 serving and Qwen3.5-MoE training bring-up. When you train a model and serve it for inference, you expect them to agree. The same weights, the same input, the same output distribution. This training–inference numerical parity matters more than it sounds: For dense models, parity is relatively easy. Mixture-of-Experts models like Kimi K2.5, Qwen3.5-MoE, and DeepSeek V3 are harder. With routed experts, shared expert pathways, and all-reduce communication twice per layer across deep stacks, there are many places where "mathematically equivalent" optimizations produce numerically different results. This post catalogs the pitfalls we found. Each is a class of optimization that inference engines use for performance, but that can silently break numerical alignment. We found most of these while…

3/8/2026 Fireworks Acquires Hathora to Accelerate Global Compute Orchestration

Fireworks AI has acquired Hathora, and we're thrilled to bring their team and technology into the Fireworks family. Lin Qiao shared her excitement about the acquisition, noting, “Hathora’s intense focus on every millisecond and every routing decision is precisely the discipline required for cutting-edge AI inference.” Since the first multiplayer games appeared on the internet, lag has been the enemy. In gaming, milliseconds determine whether you win or lose. Speed isn’t a feature; it’s survival. AI inferences is entering that same era. Solving that requires a particular kind of team: engineers who obsess over systems, performance, and reliability at a global scale. From the beginning, Fireworks has set out to build an elite group of infrastructure engineers. People who care deeply about kernel performance, scheduling decisions, networking paths, and the invisible layers that make intelligent systems instantaneous. The Hathora team…

3/8/2026 Introducing Fireworks on Microsoft Foundry: Bringing Best-in-Class Open Model inference to Azure

We are excited to announce the Public Preview of Fireworks AI on Microsoft Foundry, bringing our best-in-class fast open-model serving directly into Azure. This partnership integrates Fireworks’ high-performance inference and State-of-the-Art (SOTA) open models into the unified Microsoft Foundry platform, which already offers a wide selection of models. By empowering developers with the fastest path to production-grade open-models, this milestone ensures teams using this new solution have one place to use any model, any framework, with enterprise‑grade controls to build and run AI applications and agents at scale. Across industries, organizations are increasingly standardizing on open models to get greater control over performance, cost, customization, and the security and compliance needed for enterprise deployment. With open models, teams can choose the right architecture per workload, bring their own weights, and fine-tune for quality, latency, and cost without provider lock‑in. Yet…

Broadening advanced AI education across Africa

Broadening advanced AI education across Africa AI is driving scientific discoveries and research breakthroughs, but its progress depends on a global community. To bridge the gap between talent and opportunity, Google DeepMind is launching additional courses of its AI Research Foundations curriculum: advanced AI education designed for the next generation of technical learners across Africa. Hands-on experience with generative AI models The courses, developed with pedagogy experts and academics at University College London — and available at no cost on Google Skills — give learners the opportunity to build and fine-tune a language model from the ground up. Google.org is supporting the curriculum’s rollout in African classrooms by providing funding for lecturer training and instructional toolkits. The curriculum, already serving thousands of users globally, moves beyond AI literacy, providing technical university students and community learners with a deep, applied understanding…

Canopy Labs’ Orpheus TTS is live on GroqCloud

Canopy Labs’ Orpheus TTS is live on GroqCloud In December, we announced support for Canopy Labs’ Orpheus text-to-speech (TTS) on GroqCloud, with two model variants built for real-time, high-quality voices: - English TTS: canopylabs/orpheus-v1-english (with vocal directions) - Saudi Arabic (dialect) TTS: canopylabs/orpheus-arabic-saudi (authentic pronunciation + regional nuance) Today, we’re excited to announce a new release of the Saudi Arabic Orpheus TTS model on GroqCloud (canopylabs/orpheus-arabic-saudi). This release brings overall model improvements, including reduced hallucinations, more natural and expressive speech, and more accurate handling of numbers and symbols. It also introduces two new Saudi Arabic voices designed to sound more natural, culturally grounded, and production-ready. - Abdullah — A professional, calm, and conversational male voice, ideal for assistants, enterprise workflows, and general voice interfaces. - Aisha — A professional, clear, and approachable female voice, especially effective for customer support and…

Multimodality Embeddings Bilge Yücel DevRel Engineer Stefano Fiorucci AI/Software Engineer Multimodal Search with Gemini Embedding 2 in Haystack Build multimodal search systems in Haystack using Gemini Embedding 2 to embed text, images, video, audio, and PDFs in a shared vector space. March 10, 2026

Multimodal Search with Gemini Embedding 2 in Haystack Build multimodal search systems in Haystack using Gemini Embedding 2 to embed text, images, video, audio, and PDFs in a shared vector space. March 10, 2026Embeddings are the backbone of modern AI applications, from semantic search and recommendation systems to Retrieval-Augmented Generation (RAG). However, most embedding models operate in a single modality, typically focusing only on textual data. Google has introduced Gemini Embedding 2, a fully multimodal embedding model that maps text, images, video, audio, and PDFs into a shared vector space. This means you can search across different types of data using a single embedding model: gemini-embedding-2-preview . Even better, Haystack supports Gemini Embedding 2 from Day 0. Through the Google GenAI x Haystack integration, you can immediately start using the model in your Haystack applications for both text and multimodal…

Granite 4.1 LLMs: How They’re Built

Granite 4.1 LLMs: How They’re Built Authors: Granite Team, IBM TL;DR — Granite 4.1 is a family of dense, decoder‑only LLMs (3B, 8B, and 30B) trained on ~15T tokens using a multi‑stage pre‑training pipeline, including long‑context extension of up to 512K tokens. The models are further refined with supervised fine‑tuning on ~4.1M high‑quality curated samples and reinforcement learning via on‑policy GRPO with DAPO loss (Yu et al., 2025). Notably, the 8B instruct model matches or surpasses the previous Granite 4.0‑H‑Small (32B‑A9B MoE) despite using a simpler dense architecture with fewer parameters. All Granite 4.1 models are released under the Apache 2.0 license. Links: Overview Building high‑quality small language models goes beyond simply scaling compute—it requires rigorous data curation throughout training. For Granite 4.1, we prioritized data quality over quantity, progressively refining the data mixture across five pre‑training stages. We further…

AI evals are becoming the new compute bottleneck

AI evals are becoming the new compute bottleneck Summary. AI evaluation has crossed a cost threshold that changes who can do it. The Holistic Agent Leaderboard (HAL) recently spent about $40,000 to run 21,730 agent rollouts across 9 models and 9 benchmarks. A single GAIA run on a frontier model can cost $2,829 before caching. Exgentic's $22,000 sweep across agent configurations found a 33× cost spread on identical tasks, isolating scaffold choice as a first-order cost driver, and UK-AISI recently scaled agentic steps into the millions to study inference-time compute. In scientific ML, The Well costs about 960 H100-hours to evaluate one new architecture and 3,840 H100-hours for a full four-baseline sweep. While compression techniques have been proposed for static benchmarks, new agent benchmarks are noisy, scaffold-sensitive, and only partly compressible. Training-in-the-loop benchmarks are expensive by construction, and when you…

Introducing NVIDIA Nemotron 3 Nano Omni: Long-Context Multimodal Intelligence for Documents, Audio and Video Agents

Introducing NVIDIA Nemotron 3 Nano Omni: Long-Context Multimodal Intelligence for Documents, Audio and Video Agents - NVIDIA Nemotron 3 Nano Omni is a new omni-modal understanding model built for real-world document analysis, multiple image reasoning, automatic speech recognition, long audio-video understanding, agentic computer use, and general reasoning. - It extends the Nemotron multimodal line from a strong vision-language system to a broader text + image + video + audio model. - Nemotron 3 Nano Omni delivers best-in-class accuracy on complex document intelligence leaderboards such as MMlongbench-Doc, OCRBenchV2, while also leading in video and audio leaderboards like WorldSense and DailyOmni. It achieves top accuracy on VoiceBench for audio understanding and ranks as the most cost‑efficient open video understanding model on MediaPerf. - Under the hood, it combines the Nemotron 3 hybrid Mamba-Transformer Mixture-of-Experts backbone with a C-RADIOv4-H vision encoder and Parakeet-TDT-0.6B-v2…

AI and the Future of Cybersecurity: Why Openness Matters

AI and the Future of Cybersecurity: Why Openness Matters What is Mythos? Mythos is a “frontier AI model”, a large language model (LLM) that can be used to process software code (among many other things). This follows a general trend in LLM development, where LLM performance on code-related tasks has recently skyrocketed. What’s particularly significant about Mythos is the system it’s embedded within: It's the system, not the model alone, that has enabled Mythos to rapidly find and patch software vulnerabilities. Understanding this distinction is key to understanding the current landscape of AI cybersecurity. What Mythos demonstrates is that the following system recipe is powerful: - substantial compute power - models trained on troves of software-relevant data - scaffolding built to handle software vulnerability probing and patching - speed (enabled by compute power and the capital behind it) - some…

Training and Finetuning Multimodal Embedding & Reranker Models with Sentence Transformers

Training and Finetuning Multimodal Embedding & Reranker Models with Sentence Transformers As a practical example, I'll walk through finetuning Qwen/Qwen3-VL-Embedding-2B for Visual Document Retrieval (VDR), the task of retrieving relevant document pages (as images, with charts, tables, and layout intact) for a given text query. The resulting tomaarsen/Qwen3-VL-Embedding-2B-vdr demonstrates how much performance you can gain by finetuning on your own domain. On my evaluation data, the finetuned model achieves an NDCG@10 of 0.947 compared to the base model's 0.888, and outperforms all existing VDR models I tested against, including models up to 4x its size. If you're new to multimodal models in Sentence Transformers, I recommend reading Multimodal Embedding & Reranker Models with Sentence Transformers first. For training text-only embedding, reranker, or sparse embedding models, see the Prior Blogposts section at the end. Table of Contents - Why Finetune? -…

Ecom-RLVE: Adaptive Verifiable Environments for E-Commerce Conversational Agents

Ecom-RLVE: Adaptive Verifiable Environments for E-Commerce Conversational Agents TL;DR — We extend the RLVE framework from single-turn reasoning puzzles to multi-turn, tool-augmented e-commerce conversations. EcomRLVE-GYM provides 8 verifiable environments — product discovery, substitution, cart building, returns, order tracking, policy QA, bundle planning, and multi-intent journeys — each with procedural problem generation, a 12-axis difficulty curriculum, and algorithmically verifiable rewards. We train a Qwen 3 8B model with DAPO over 300 steps and present early results demonstrating that environment scaling and adaptive difficulty transfer to agentic, real-world task completion. This project originated in the Pytorch OpenEnv Hackathon and is still evolving, follow us for updates 🔥 Why RL for shopping agents? Large language models can hold fluent conversations, yet deploying them as shopping assistants reveals a persistent gap: fluency ≠ task completion. A customer who asks "find me a USB-C charger…

Meet HoloTab by HCompany. Your AI browser companion.

Meet HoloTab by HCompany. Your AI browser companion. We built one of the most powerful computer-use AIs in the world. And made it directly accessible from your browser. On March 31st, we released Holo3, our most advanced computer-use model to date. Building something powerful is one thing; making it accessible and easy to use is another. We’re doing both. HoloTab is a Chrome extension that navigates the web just like a person would. It automates tasks across any website with zero setup or technical skills required. You describe what you want, and the agent handles it directly inside your browser, navigating interfaces, filling fields, and making decisions the same way you would. The vision models, the action planning, the interface understanding, all of it is running underneath, working for you, and all you ever see is the result. Routines: Show…

Multimodal Embedding & Reranker Models with Sentence Transformers

Multimodal Embedding & Reranker Models with Sentence Transformers Multimodal embedding models map inputs from different modalities into a shared embedding space, while multimodal reranker models score the relevance of mixed-modality pairs. This opens up use cases like visual document retrieval, cross-modal search, and multimodal RAG pipelines. If you want to train your own multimodal models, check out the companion blogpost: Training and Finetuning Multimodal Embedding & Reranker Models with Sentence Transformers. Table of Contents - What are Multimodal Models? - Installation - Multimodal Embedding Models - Multimodal Reranker Models - Retrieve and Rerank - Input Formats and Configuration - Supported Models - Additional Resources What are Multimodal Models? Traditional embedding models convert text into fixed-size vectors. Multimodal embedding models extend this by mapping inputs from different modalities (text, images, audio, or video) into a shared embedding space. This means you…

Welcome Gemma 4: Frontier multimodal intelligence on device

Welcome Gemma 4: Frontier multimodal intelligence on device These models are the real deal: truly open with Apache 2 licenses, high quality with pareto frontier arena scores, multimodal including audio, and sizes you can use everywhere including on-device. Gemma 4 builds on advances from previous families and makes them click together. In our tests with pre-release checkpoints we have been impressed by their capabilities, to the extent that we struggled to find good fine-tuning examples because they are so good out of the box. We collaborated with Google and the community to make them available everywhere: transformers, llama.cpp, MLX, WebGPU, Rust; you name it. This blog post will show you how to build with your favorite tools so let us know what you think! Table of Contents - What is New with Gemma 4? - Overview of Capabilities and Architecture…

Granite 4.0 3B Vision: Compact Multimodal Intelligence for Enterprise Documents

Granite 4.0 3B Vision: Compact Multimodal Intelligence for Enterprise Documents - Table Extraction: Accurately parsing complex table structures (e.g., multi-row, multi-column, etc.) from document images - Chart Understanding: Converting charts and figures into structured machine-readable formats, summaries, or executable code - Semantic Key-Value Pair (KVP) Extraction: Identifying and grounding semantically meaningful key-value field pairs across diverse document layouts The model ships as a LoRA adapter on top of Granite 4.0 Micro, our dense language model, keeping vision and language modular for text-only fallbacks and seamless integration into mixed pipelines. It continues to support vision-language tasks such as producing detailed natural-language descriptions from images (e.g., “Describe this image in detail”). The model can be used standalone or in tandem with Docling to enhance document processing pipelines with deep visual understanding capabilities. How Granite 4.0 3B Vision Was Built Granite 4.0 3B…

Holotron-12B - High Throughput Computer Use Agent

Holotron-12B - High Throughput Computer Use Agent We're thrilled to release Holotron-12B, a multimodal computer-use model from H Company. Post-trained from the open NVIDIA Nemotron-Nano-2 VL model on H Company’s proprietary data mixture, Holotron-12B is the result of a close collaboration between our research labs to engineer a new type of model optimized primarily for scale and performance in production. H Company is part of the NVIDIA Inception Program. The model is now available on Hugging Face. Why We Built Holotron-12B Most multimodal models today optimize primarily for static vision or following instructions. Holotron-12B, just like our Holo2 model, however, has a different goal: serving as a policy model for computer-use agents that must perceive, decide, and act efficiently in interactive environments. With Holotron-12B, we wanted to create a model that could efficiently and effectively scale in production while handling…

LeRobot v0.5.0: Scaling Every Dimension

LeRobot v0.5.0: Scaling Every Dimension TL;DR LeRobot v0.5.0 adds full Unitree G1 humanoid support (whole-body control models), new policies –including Pi0-FAST autoregressive VLAs and Real-Time Chunking for responsive inference–, and streaming video encoding that eliminates wait times between recording episodes. The release also introduces EnvHub for loading simulation environments from the Hugging Face Hub, NVIDIA IsaacLab-Arena integration, and a major codebase modernization with Python 3.12+, Transformers v5, and third-party policy plugins. Table of Contents - LeRobot v0.5.0: Scaling Every Dimension Hardware: More Robots Than Ever LeRobot v0.5.0 dramatically expands the roster of supported hardware — from arms and mobile robots to a full humanoid. Unitree G1 Humanoid The biggest hardware addition in this release: full Unitree G1 humanoid support. This is LeRobot's first humanoid integration, and it's comprehensive: - Locomotion: Walk, navigate, and move through environments. - Manipulation: Perform dexterous…

Bringing Robotics AI to Embedded Platforms: Dataset Recording, VLA Fine‑Tuning, and On‑Device Optimizations

Bringing Robotics AI to Embedded Platforms: Dataset Recording, VLA Fine‑Tuning, and On‑Device Optimizations Authors: Enzo Ruedas, Tess Boivin Recent advances in Large Language Models have enabled the transition from text-only reasoning to multimodal systems. First, with the integration of visual perception in Vision–Language Models (VLMs), and more recently with the generation of robot actions in Vision–Language–Action (VLA) models. Deploying these models on embedded robotic platforms remains a challenge due to tight constraints in terms of compute, memory, and power, as well as real-time control requirements. In synchronous control pipelines, while the VLA is running inference, the arm is idle awaiting commands leading to oscillatory behavior and delayed corrections. To tackle that, asynchronous Inference can enable smooth and continuous motion by dissociating generation from execution. However, to be effective, the end-to-end inference latency must remain shorter than the action execution duration.…

Train AI models with Unsloth and Hugging Face Jobs for FREE

Train AI models with Unsloth and Hugging Face Jobs for FREE LiquidAI/LFM2.5-1.2B-Instruct ) through coding agents like Claude Code and Codex. Unsloth provides ~2x faster training and ~60% less VRAM usage compared to standard methods, so training small models can cost just a few dollars. Why a small model? Small language models like LFM2.5-1.2B-Instruct are ideal candidates for fine-tuning. They are cheap to train, fast to iterate on, and increasingly competitive with much larger models on focused tasks. LFM2.5-1.2B-Instruct runs under 1GB of memory and is optimized for on-device deployment, so what you fine-tune can be served on CPUs, phones, and laptops. You will need We are giving away free credits to fine-tune models on Hugging Face Jobs. Join the Unsloth Jobs Explorers organization to claim your free credits and one-month Pro subscription. - A Hugging Face account (required for…

Import AI 452: Scaling laws for cyberwar; rising tides of AI automation; and a puzzle over gDP forecasting

Import AI 452: Scaling laws for cyberwar; rising tides of AI automation; and a puzzle over gDP forecasting How much could AI revolutionize the economy? Welcome to Import AI, a newsletter about AI research. Import AI runs on arXiv and feedback from readers. If you’d like to support this, please subscribe. Uh oh, there’s a scaling war for cyberattacks as well!: …The smarter the system, the better the ability to cyberattack… AI safety research organization Lyptus Research has looked at how well AI systems can perform a variety of cyberoffense tasks and found a clear trend of more advanced models being able to do more advanced forms of cyberattack. “Across frontier models released since 2019, the doubling time is 9.8 months. Restricting to models released since 2024, it steepens to 5.7 months. The most recent frontier models in our study,…

Import AI 450: China's electronic warfare model; traumatized LLMs; and a scaling law for cyberattacks

Import AI 450: China's electronic warfare model; traumatized LLMs; and a scaling law for cyberattacks How will timeless minds value time? Welcome to Import AI, a newsletter about AI research. Import AI runs on arXiv and feedback from readers. If you’d like to support this, please subscribe. A somewhat shorter issue than usual as I had to do a lot of child wrangling this weekend. Why does Google’s model hate itself and what can we do to help it? …Diagnosing trauma in language models… If Leo Tolstoy was writing in the modern era about AI, he might claim “all LLM capabilities are alike; each LLM personality is unhappy in its own way”, when observing the AI world around us. Today’s LLMs are generally quite good at writing and coding tasks. But where they differ is their personality, which stems from…

ImportAI 449: LLMs training other LLMs; 72B distributed training run; computer vision is harder than generative text

ImportAI 449: LLMs training other LLMs; 72B distributed training run; computer vision is harder than generative text Will AI cause a political interregnum Welcome to Import AI, a newsletter about AI research. Import AI runs on arXiv and feedback from readers. If you’d like to support this, please subscribe. Can LLMs autonomously refine other LLMs for new tasks? Somewhat. …PostTrainBench shows startling growth in AI capabilities at post-training… AI-driven R&D might be the most important thing in all of AI, because it helps us understand whether AI systems might eventually build their own successors. So far, much of the focus on AI R&D has been in components that support AI development (e.g., autonomous creation of AI kernels), or training base models (e.g, the NanoGPT speedrun benchmark). But there’s been less attention paid to fine-tuning - the task involving adapting an…

AutoAdapt: Automated domain adaptation for large language models

At a glance - Problem: Adapting large language models to specialized, high-stakes domains is slow, expensive, and hard to reproduce. - What we built: AutoAdapt automates planning, strategy selection (e.g., RAG vs. fine-tuning), and tuning under real deployment constraints. - How it works: A structured configuration graph maps the full scope of the adaptation process, an agentic planner selects and sequences the right steps, and a budget-aware optimization loop (AutoRefine) refines the process within defined constraints. - Why it matters: The result is faster, automated, more reliable domain adaptation that turns weeks of manual iteration into repeatable pipelines. Deploying large language models (LLMs) in real-world, high-stakes settings is harder than it should be. In high-stakes settings like law, medicine, and cloud incident response, performance and reliability can quickly break down because adapting models to domain-specific requirements is a slow and…

Health-care AI is here. We don’t know if it actually helps patients.

Health-care AI is here. We don’t know if it actually helps patients. The tools may be accurate, but that doesn’t necessarily mean they’ll improve health outcomes. I don’t need to tell you that AI is everywhere. Or that it is being used, increasingly, in hospitals. Doctors are using AI to help them with notetaking. AI-based tools are trawling through patient records, flagging people who may require certain support or treatments. They are also used to interpret medical exam results and X-rays. A growing number of studies suggest that many of these tools can deliver accurate results. But there’s a bigger question here: Does using them actually translate into better health outcomes for patients? We don’t yet have a good answer. That’s what Jenna Wiens, a computer scientist at the University of Michigan, and Anna Goldenberg of the University of Toronto,…

Speed Up Unreal Engine NNE Inference with NVIDIA TensorRT for RTX Runtime

Neural network techniques are increasingly used in computer graphics to boost image quality, improve performance, and streamline content creation. Approaches like super resolution, denoising, and neural rendering help real-time engines work more efficiently, offering new creative possibilities while keeping performance in mind. Unreal Engine 5 (UE5) has taken several steps in this direction with the introduction of the Neural Network Engine (NNE), which serves as an abstraction layer that unifies inference workloads across multiple backends. Developers can use various runtimes on a GPU or fall back to a CPU depending on available hardware for seamless integration of neural network features in real-time graphics workflows. This blog post covers the new plugin that adds NVIDIA TensorRT for RTX as an NNE runtime option (NNERuntimeTRT) for efficient inferencing on NVIDIA RTX GPUs. To show its benefits, I’ll use a simplified UE project…

NVIDIA Nemotron 3 Nano Omni Powers Multimodal Agent Reasoning in a Single Efficient Open Model

Agentic systems often reason across screens, documents, audio, video, and text within a single perception‑to‑action loop. However, they still rely on fragmented model chains—separate stacks for vision, audio, and text. This increases inference hops and orchestration complexity, driving up inference costs while weakening cross-modal context consistency. NVIDIA Nemotron 3 Nano Omni, a new addition to the Nemotron 3 family, brings unified multimodal reasoning into a single, highly efficient open model. Built to replace fragmented vision‑language‑audio stacks, Nemotron 3 Nano Omni functions as the multimodal perception and context sub‑agent within agentic systems. With this, agents can perceive and reason across visual, audio, and textual inputs within a single shared perception‑to‑action loop, improving convergence and reducing orchestration complexity and inference cost. It delivers best-in-class accuracy on document intelligence leaderboards such as MMlongbench-Doc and OCRBenchV2, while also leading in video and audio understanding,…

Advancing Emerging Optimizers for Accelerated LLM Training with NVIDIA Megatron

Higher-order optimization algorithms such as Shampoo have been effectively applied in neural network training for at least a decade. These methods have achieved significant success more recently when applied to leading LLMs. In particular, Muon (MomentUm Orthogonalized by Newton-Schulz) was used to train some of today’s best open source models, including Kimi K2 and GLM-5. This post explains how NVIDIA provides comprehensive support for Muon and other cutting-edge emerging optimizers and the technologies enabling them to train large-scale models. Muon training performance on NVIDIA GB300 NVL72 Table 1 summarizes training throughput of the Kimi K2 and Qwen3 30B models with Muon and the AdamW optimizer on the NVIDIA GB300 NVL72 system. With the technologies that will be introduced in the next section, the results show that there is a very small training performance loss using the Muon optimizer compared to…

Run High-Throughput Reinforcement Learning Training with End-to-End FP8 Precision

As LLMs transition from simple text generation to complex reasoning, reinforcement learning (RL) plays a central role. Algorithms like Group Relative Policy Optimization (GRPO) power this transition, enabling reasoning-grade models to continuously improve through iterative feedback. Unlike standard supervised fine-tuning, RL training loops are bifurcated into two distinct, high-intensity phases: a generation phase with a stringent latency requirement and a training phase requiring high throughput. To make these workloads viable, researchers and engineers are turning to low-precision datatypes like FP8 to boost performance in training and throughput-oriented generation. Moreover, in some scenarios where generation is bound by GPU memory bandwidth, using low-precision parameters can improve performance due to fewer bytes per parameter. This post dives deep into the systemic challenges of low-precision RL and how NVIDIA NeMo RL—an open source library within the NVIDIA NeMo framework—speeds up RL workloads while…

Achieving Single-Digit Microsecond Latency Inference for Capital Markets

In algorithmic trading, reducing response times to market events is crucial. To keep pace with high-speed electronic markets, latency-sensitive firms often use specialized hardware like FPGAs and ASICs. Yet, as markets grow more efficient, traders increasingly depend on advanced models such as deep neural networks to enhance profitability. Because implementing these complex models on low-level hardware requires significant investment, general-purpose GPUs offer a practical, cost-effective alternative. The NVIDIA GH200 Grace Hopper Superchip in the Supermicro ARS-111GL-NHR server has achieved single-digit microsecond latencies in the STAC-ML Markets (Inference) benchmark, Tacana suite (audited by STAC), providing performance comparable to or better than specialized hardware systems. This post details these record-breaking results and provides a deep dive into the custom-tailored solutions required for low-latency GPU inference. It also walks you through an open source reference implementation and a tutorial for getting started. STAC-ML…

Bringing AI Closer to the Edge and On-Device with Gemma 4

The Gemmaverse expands with the launch of the latest Gemma 4 multimodal and multilingual models, designed to scale across the full spectrum of deployments, from NVIDIA Blackwell in the data center to Jetson at the edge. These models are suited to meet the growing demand for local deployment for AI development and prototyping, secure on-prem requirements, cost efficiency, and latency-sensitive use cases. The newest generation improves both efficiency and accuracy, making these general-purpose models well-suitable for a wide range of common tasks: - Reasoning: Strong performance on complex problem-solving tasks. - Coding: Code generation and debugging for developer workflows. - Agents: Native support for structured tool use (function calling). - Vision, video and audio capability: Enables rich multimodal interactions for use cases such as object recognition, automated speech recognition (ASR), document and video intelligence, and more. - Interleaved multimodal input:…

Scaling Token Factory Revenue and AI Efficiency by Maximizing Performance per Watt

In the AI era, power is the ultimate constraint, and every AI factory operates within a hard limit. This makes performance per watt—the rate at which power is converted into revenue-generating intelligence—the defining metric for modern AI infrastructure. AI data centers now operate as token factories tied directly to the energy ecosystem, where access to land, power, and shell determines deployment, and efficiency determines output. Increasing revenue within a fixed power envelope depends entirely on maximizing intelligence per watt across AI infrastructure and across the five-layer AI cake ecosystem. This post walks through how NVIDIA architectures, systems, and AI factory software maximize performance per watt at every layer of the stack, and how those efficiency gains translate into higher token throughput and revenue per megawatt. Compounding performance per watt across NVIDIA GPU architectures NVIDIA architectures and platforms are engineered to…

Building NVIDIA Nemotron 3 Agents for Reasoning, Multimodal RAG, Voice, and Safety

Agentic AI is an ecosystem where specialized models work together to handle planning, reasoning, retrieval, and safety guardrailing. As these systems scale, developers need models that can understand real-world multimodal data, converse naturally with users globally, and operate safely across languages and modalities. At GTC 2026, NVIDIA introduced a new generation of NVIDIA Nemotron models designed to work together as a unified agentic stack: - NVIDIA Nemotron 3 Super for long-context reasoning and agentic tasks - NVIDIA Nemotron 3 Ultra (coming soon) for highest reasoning accuracy and efficiency among open frontier models - NVIDIA Nemotron 3 Content Safety for multimodal, multilingual content moderation - NVIDIA Nemotron 3 VoiceChat (in early access) for low latency, natural, full-duplex voice interactions - NVIDIA Nemotron 3 Nano Omni (coming soon) for enterprise-grade multimodal understanding - NVIDIA Nemotron RAG for generating embeddings for image and…

Deploying Disaggregated LLM Inference Workloads on Kubernetes

As large language model (LLM) inference workloads grow in complexity, a single monolithic serving process starts to hit its limits. Prefill and decode stages have fundamentally different compute profiles, yet traditional deployments force them onto the same hardware, leaving GPUs underutilized and scaling inflexible. Disaggregated serving addresses this by splitting the inference pipeline into distinct stages such as prefill, decode, and routing, each running as an independent service that can be resourced and scaled on its own terms. This post will give an overview of how disaggregated inference gets deployed on Kubernetes, explore different ecosystem solutions and how they execute on a cluster, and evaluate what they provide out of the box. How do aggregated and disaggregated inference differ? Before diving into Kubernetes manifests, it helps to understand the two inference deployment modes for LLMs: In aggregated serving, a single…

Building the AI Grid with NVIDIA: Orchestrating Intelligence Everywhere

AI-native services are exposing a new bottleneck in AI infrastructure: As millions of users, agents, and devices demand access to intelligence, the challenge is shifting from peak training throughput to delivering deterministic inference at scale—predictable latency, jitter, and sustainable token economics. NVIDIA announced at GTC 2026 that telcos and distributed cloud providers are transforming their networks into AI grids, embedding accelerated computing across a mesh of regional POPs, central offices, metro hubs, and edge locations to meet the needs of AI-native services. This post explains how AI grids make real-time, multi-modal, and hyper-personalized AI experiences viable at scale by running inference across distributed, workload-, resource- and KPI-aware AI infrastructure. Intelligent workload placement across distributed sites The NVIDIA AI Grid reference design provides a unified framework for building geographically distributed, interconnected, and orchestrated AI infrastructure. Figure 1 shows how existing network…

NVIDIA Vera Rubin POD: Seven Chips, Five Rack-Scale Systems, One AI Supercomputer

Artificial intelligence is token-driven. Every prompt, reasoning step, and agent interaction generates tokens. Over the past year, token consumption has grown multifold and now exceeds 10 quadrillion tokens per year. And while the majority of tokens have been generated from humans interacting with AI, the new era is one in which most tokens will be generated from AI interacting with AI. Modern agentic systems plan tasks, invoke tools, execute code, retrieve data, and coordinate across continuous multistep workflows with numerous AI agents. These interactions generate large volumes of reasoning tokens, expand KV cache, and require CPU-based sandboxed environments to test and validate results generated by accelerated computing systems. This places low latency, high throughput demands across GPUs, CPUs, scale-up domains, scale-out networks, and storage. Delivering useful intelligence for these modern agentic systems requires fleets of purpose-built rack-scale systems that function…

NVIDIA Vera CPU Delivers High Performance, Bandwidth, and Efficiency for AI Factories

AI is evolving, and reasoning models are increasing token demand, placing new requirements on every layer of AI infrastructure. More than ever, compute must scale efficiently to maximize token production and improve productivity for model creators and users. Modern GPUs operate at peak capacity, pushing throughput higher every generation, but system performance is increasingly gated by the CPU-bound serial tasks within an agentic loop–a classic example of a core computer science principle, called Amdahl’s law. This dynamic is especially visible in two classes of workloads: reinforcement learning (RL) for training models with new specialized skills such as coding or engineering, and agentic actions, which enable AI agents to use tools like web browsers, databases, code interpreters, and other software to complete tasks in real environments, or sandboxes. Both workloads combine two historically separate CPU characteristics. Individual environments require strong single-threaded…

Design, Simulate, and Scale AI Factory Infrastructure with NVIDIA DSX Air

Building AI factories is complex and requires efficient integration across compute, networking, security, and storage systems. To achieve rapid Time to AI and strong ROI, the new NVIDIA DSX Air is enabling organizations to simulate their entire AI factory infrastructure in the cloud—covering compute, networking, storage, and security. Being able to design, test, and optimize systems before deploying hardware enables every layer of the AI factory to function as a unified, optimized system, preventing major delays or performance issues related to integration or misconfiguration challenges. DSX Air also enables continuous testing and validation of provisioning, automation, and security policies to streamline ongoing operations. This post shows how users can benefit from NVIDIA DSX Air through accelerated deployment timelines and simplified, full-stack cluster management. How DSX Air enables AI factory simulation To make AI factory simulation useful and practical for end…

Introducing NVIDIA BlueField-4-Powered CMX Context Memory Storage Platform for the Next Frontier of AI

AI‑native organizations increasingly face scaling challenges as agentic AI workflows drive context windows to millions of tokens and models scale toward trillions of parameters. These systems rely on agentic long‑term memory for context that persists across turns, tools, and sessions so agents can build on prior reasoning instead of starting from scratch on every request. As context windows increase, Key-Value (KV) cache capacity requirements grow proportionally, while the compute requirements to recalculate that history grow much faster, making KV cache reuse and efficient storage essential for performance and efficiency. This increases pressure on existing memory hierarchies, forcing AI providers to choose between scarce GPU high‑bandwidth memory (HBM) and general‑purpose storage tiers optimized for durability, data management, and protection—not for serving ephemeral, AI-native, KV cache—driving up power consumption, inflating cost per token, and leaving expensive GPUs underutilized. The NVIDIA Vera Rubin…

Scaling Autonomous AI Agents and Workloads with NVIDIA DGX Spark

Autonomous AI agents are driving the next wave of AI innovation. These agents must often manage long-running tasks that use multiple communication channels and background subprocesses simultaneously to explore options, test solutions, and generate optimal results. This places extreme demands on local compute. NVIDIA DGX Spark provides the performance necessary for autonomous agents to execute these complex workflows efficiently and locally. Now with NVIDIA NemoClaw, part of the NVIDIA Agent Toolkit, it installs the NVIDIA OpenShell runtime—a secure environment for running autonomous agents, and open source models like NVIDIA Nemotron. This post discusses several important aspects of system capabilities and performance that are necessary to power always-on autonomous agents and explains why NVIDIA DGX Spark is an ideal desktop platform for autonomous AI. Inference for autonomous AI agents Agentic tools often need to process massive context windows. OpenClaw, for example,…

Using Simulation to Build Robotic Systems for Hospital Automation

Healthcare faces a structural demand–capacity crisis: a projected global shortfall of ~10 million clinicians by 2030, billions of diagnostic exams annually with significant unmet demand, hundreds of millions of procedures with large access gaps, and costly operating room (OR) inefficiencies measured in tens of dollars per minute. The future hospital must therefore be automation-enabled—where robotics extends clinician capacity, increases procedural throughput, reduces variability, and democratizes access to high-quality care. Imagine autonomous imaging robots navigating patient anatomy to provide X-rays for the unserved billions, while in the OR, ‘Surgical Subtask Automation’ handles repetitive suturing so surgeons can focus on critical decisions. Beyond the bedside, service robots recapture wasted minutes by autonomously delivering supplies, saving nurses miles of walking. The data gap and real-world limits The core bottleneck is data. Hospitals are heterogeneous, chaotic, and high-stakes environments—every facility has different layouts, workflows,…

Build Accelerated, Differentiable Computational Physics Code for AI with NVIDIA Warp

Computer-aided engineering (CAE) is shifting from human-driven workflows toward AI-driven ones, including physics foundation models that generalize across geometries and operating conditions. Unlike LLMs, these models depend on large volumes of high-fidelity, physics-compliant data. Recent scaling-law work on computational fluid dynamics (CFD) surrogates indicates that simulation-generated training data is often the limiting cost in practice. This pushes requirements onto the simulator, which must be GPU-native, fast, and able to plug directly into ML workflows. NVIDIA Warp is a framework for accelerated simulation, data generation, and spatial computing that bridges CUDA and Python. Warp enables developers to write high-performance kernels as regular Python functions that are JIT-compiled into efficient code for execution on the GPU. Unlike the tensor-based frameworks, in which developers express computation as operations on entire N-dimensional arrays, developers author flexible kernels in the Warp framework that execute simultaneously…

Reliable AI Coding for Unreal Engine: Improving Accuracy and Reducing Token Costs

Agentic code assistants are moving into daily game development as studios build larger worlds, ship more DLCs, and support distributed teams. These assistants can accelerate development by helping with tasks like generating gameplay scaffolding, refactoring repetitive systems, and answering engine-specific questions faster. This post outlines how developers can build reliable AI coding workflows for Unreal Engine (UE) 5, from individual setups to team and enterprise-scale systems. Reliability is critical because real-world Unreal codebases are defined by engine conventions, large C++ projects, custom tools, branch differences, and studio-specific coding patterns that generic AI often fails to understand. The core challenge is the context gap. Failures rarely come from weak code generation, but from missing constraints such as code patterns, branch differences, or internal conventions. Improving context retrieval reduces guesswork and makes AI output reliable enough for production use. NVIDIA works with…

Removing the Guesswork from Disaggregated Serving

Deploying and optimizing large language models (LLMs) for high-performance, cost-effective serving can be an overwhelming engineering problem. The ideal configuration for any given workload (such as hardware, parallelism, and prefill/decode split) resides in a massive, multi-dimensional search space that is impossible to explore manually or through exhaustive testing. AIConfigurator, an open source tool that simplifies the NVIDIA Dynamo AI serving stack, is intended to cut through this complexity and get you to an optimal deployment in minutes. The core benefit of AIConfigurator is that you don’t need to run every possible configuration on real hardware to predict which one will perform best. Instead, it decomposes LLM inference into its constituent operations and measures each one in isolation on the target GPU. AIConfigurator can then reassemble those measurements to estimate the end-to-end performance of any configuration, all without occupying a single…

Using NVFP4 Low-Precision Model Training for Higher Throughput Without Losing Accuracy

As the sizes of AI models and datasets continue to increase, relying only on higher-precision BF16 training is no longer sufficient. Key challenges such as training throughput expectations, memory limits, and rising costs are becoming the primary barriers to scaling transformer models. Using lower-precision training can address these challenges. By reducing the numeric precision used during computation, GPUs can process more operations per cycle, enhancing training efficiency and lowering costs. This post compares the following three low-precision training formats directly against established BF16 precision training across multi-hundred-billion token pretraining runs and downstream benchmarks: - 8-bit floating point per-tensor current scaling (FP8-CS) - Mixed precision training with FP8 (MXFP8) - NVFP4 precision training using NVIDIA NeMo Megatron Bridge, an open source library that is part of NVIDIA NeMo framework We present practical, large-scale results showing how low-precision training delivers up to…

Cybersecurity in the Intelligence Age

Cybersecurity in the Intelligence Age An action plan for democratizing AI-powered cyber defense. Artificial intelligence is reshaping cybersecurity. The same capabilities that help defenders identify vulnerabilities, automate remediation, and respond faster are also being used by malicious actors to scale attacks, lower barriers to entry, and increase sophistication. The United States and its allies face a rapidly changing cyber threat environment, and private-sector innovators have an important responsibility to help meet that challenge. OpenAI takes that responsibility seriously, and today we’re publishing an Action Plan informed by conversations with cybersecurity and national security experts across federal and state government and major commercial entities. It consists of five pillars: - Democratizing cyber defense - Coordinating across government and industry - Strengthening security around frontier cyber capabilities - Preserving visibility and control in deployment - Enabling users to protect themselves Our plan…

Building the compute infrastructure for the Intelligence Age

Building the compute infrastructure for the Intelligence Age Stargate is OpenAI’s long-term effort to build the compute foundation required to deliver the benefits of AGI broadly and reliably to the world. To meet the accelerating demand for AI across consumers, businesses, developers, and governments, we are continuing to expand our compute footprint and bring new capacity online faster. We are building together with partners, local communities, and the broader infrastructure ecosystem to help get ahead of shortages for the emerging compute-powered economy. When we announced Stargate in January 2025, we committed to securing 10GW of AI infrastructure in the United States by 2029. Just over a year later, we have already surpassed that milestone, with more than 3GW added in the last 90 days alone, as demand for AI continues to accelerate. That demand is growing quickly. The only responsible…

OpenAI models, Codex, and Managed Agents come to AWS

OpenAI models, Codex, and Managed Agents come to AWS Today, OpenAI and AWS are expanding our strategic partnership to help enterprises build using OpenAI capabilities in their AWS environments. We’re excited to give AWS customers access to the best frontier models, agents, and tools, which will operate within the systems, security protocols, compliance requirements, and workflows they already use. The expanded partnership with Amazon brings together three key areas of work, all launching today in limited preview: - OpenAI models on AWS - Codex on AWS - Amazon Bedrock Managed Agents, powered by OpenAI Together, these capabilities give organizations more ways to use OpenAI across application development, software engineering, and agentic workflows—while building within the infrastructure, security, governance, and procurement workflows they already use on AWS. For many companies, using AI at scale requires bringing the best models to the…

Choco automates food distribution with AI agents

Choco automates food distribution with AI agents Using OpenAI APIs, Choco processes millions of orders, reducing manual work and enabling always-on operations across global food supply chains. Results 8.8M+ Orders processed annually Results 200B+ AI tokens processed in production Results ↑50% Reduction in manual order entry Results 2x Sales team productivity without added headcount Choco(opens in a new window) is an AI-powered platform modernizing food and beverage distribution, serving over 21,000 distributors and 100,000 buyers across the US, UK, Europe, and the GCC. By connecting restaurants, suppliers, and distributors into a unified system, Choco streamlines ordering, sales, and customer management across the food supply chain. As order volumes grew, Choco hit a major bottleneck: orders still arrived through emails, texts, voicemails, images, and even handwritten notes. Teams manually translated those inputs into structured ERP orders—a slow, error-prone process that limited…

Speeding up agentic workflows with WebSockets in the Responses API

Speeding up agentic workflows with WebSockets in the Responses API By Brian Yu and Ashwin Nathan, Members of the Technical Staff When you ask Codex to fix a bug, it scans through your codebase for relevant files, reads them to build context, makes edits, and runs tests to verify the fix worked. Under the hood, that means dozens of back-and-forth Responses API requests: determine the model’s next action, run a tool on your computer, send the tool output back to the API, and repeat. All of these requests can add up to minutes that users spend waiting for Codex to complete complex tasks. From a latency perspective, the Codex agent loop spends most of its time in three main stages: working in the API services (to validate and process requests), model inference, and client-side time (running tools and building model…

Scaling Codex to enterprises worldwide

Scaling Codex to enterprises worldwide OpenAI is launching Codex Labs and partnering with top GSIs to bring it to thousands of engineering organizations. In early April, we shared that more than 3 million developers were using Codex every week. Just two weeks later, that number has grown to more than 4 million. Beyond individual adoption, we are seeing enterprises moving quickly to roll Codex into real workflows across engineering and beyond. Companies are using Codex across the software development lifecycle. Virgin Atlantic is using it to increase test coverage and increase team velocity - reducing technical debt and improving performance. Ramp is using it to accelerate code review. Notion is using it to quickly build new features. Cisco is using it to understand and reason across large, interconnected repositories. Rakuten is using it for things like incident response. What starts…

Codex for (almost) everything

We’re releasing a major update to Codex, making it a more powerful partner for the more than 3 million developers who use it every week to accelerate work across the full software development lifecycle. Codex can now operate your computer alongside you, work with more of the tools and apps you use everyday, generate images, remember your preferences, learn from previous actions, and take on ongoing and repeatable work. The Codex app also now includes deeper support for developer workflows, like reviewing PRs, viewing multiple files & terminals, connecting to remote devboxes via SSH, and an in-app browser to make it faster to iterate on frontend designs, apps, and games. With background computer use, Codex can now use all of the apps on your computer by seeing, clicking, and typing with its own cursor. Multiple agents can work on your…

Gradient Labs gives every bank customer an AI account manager

Gradient Labs gives every bank customer an AI account manager Gradient Labs uses GPT‑4.1 and GPT‑5.4 mini and nano to run complex financial support workflows with high accuracy and low latency. Results 10x Revenue growth Results 98% Customer satisfaction with AI agent experience Results +11% Higher accuracy with GPT-4.1 vs. next-best provider In banking, resolving a customer issue is rarely simple. Cases like fraud or blocked payments require strict adherence to complex procedures across multiple teams. When systems fall short, customers are passed between teams, wait in queues, and face delays at moments when the stakes are highest. Gradient Labs(opens in a new window) is built to handle this complexity. The London-based company is building AI agents that give every bank customer the experience of a dedicated account manager. Founded by a team that previously led AI and data efforts…

Accelerating the next phase of AI

OpenAI raises $122 billion to accelerate the next phase of AI Today, we closed our latest funding round with $122 billion in committed capital at a post money valuation of $852 billion. OpenAI is becoming the core infrastructure for AI, making it possible for people around the world and businesses, big and small, to just build things. The broad consumer reach of ChatGPT creates a powerful distribution channel into the workplace, where demand is rapidly shifting from basic model access to intelligent systems that reshape how businesses operate. Developers build on and expand the platform by leveraging our APIs, and Codex is transforming how developers turn ideas into working software. Durable access to compute is the strategic advantage that compounds across the entire system: it advances research, improves products, expands access, and structurally lowers the cost of delivery at scale.…

Introducing GPT-5.4 mini and nano

Today we’re releasing GPT‑5.4 mini and nano, our most capable small models yet. They bring many of the strengths of GPT‑5.4 to faster, more efficient models designed for high-volume workloads. GPT‑5.4 mini significantly improves over GPT‑5 mini across coding, reasoning, multimodal understanding, and tool use, while running more than 2x faster. It also approaches the performance of the larger GPT‑5.4 model on several evaluations, including SWE-Bench Pro and OSWorld-Verified. GPT‑5.4 nano is the smallest, cheapest version of GPT‑5.4 for tasks where speed and cost matter most. It is also a significant upgrade over GPT‑5 nano. We recommend it for classification, data extraction, ranking, and coding subagents that handle simpler supporting tasks. These models are built for the kinds of workloads where latency directly shapes the product experience: coding assistants that need to feel responsive, subagents that quickly complete supporting tasks,…

Rakuten fixes issues twice as fast with Codex

Results 50% Reduction in MTTR Results 3-4x Faster potential build time for projects - from quarters to weeks Rakuten(opens in a new window) is a global innovation company operating across e-commerce, fintech, and mobile communications, serving both consumers and merchants at massive scale. With 30,000 employees worldwide, its engineering teams ship across a large, complex product ecosystem where both speed and reliability are essential. That’s why Yusuke Kaji, General Manager of AI for Business at Rakuten, has spent the past year pushing agentic workflows deeper into how teams plan, build, and validate software. Codex—the coding agent from OpenAI—has become a core part of Rakuten’s engineering stack, especially where the company needs to move faster without compromising security. Over the past year, Rakuten engineers have used Codex across operations and software delivery to compress incident response (including a ~50% reduction in…

From model to agent: Equipping the Responses API with a computer environment

From model to agent: Equipping the Responses API with a computer environment By Bo Xu, Danny Zhang, and Rohit Arunachalam We're currently in a shift from using models, which excel at particular tasks, to using agents capable of handling complex workflows. By prompting models, you can only access trained intelligence. However, giving the model a computer environment can achieve a much wider range of use cases, like running services, requesting data from APIs, or generating more useful artifacts like spreadsheets or reports. A few practical problems emerge when you try to build agents: where to put intermediate files, how to avoid pasting large tables into a prompt, how to give the workflow network access without creating a security headache, and how to handle timeouts and retries without building a workflow system yourself. Instead of putting it on developers to build…

Improving instruction hierarchy in frontier LLMs

Improving instruction hierarchy in frontier LLMs Introducing IH-Challenge, a training dataset that strengthens instruction hierarchy, safety steerability, and prompt injection robustness. AI systems often receive instructions from multiple sources. These can include safety policies from system messages, product guidance from developers, requests from users, and information found online. Training models to reliably prioritize the most trusted instructions among these sources is a key part of safe deployment. Many AI safety and reliability issues can arise when this prioritization breaks down. Models may receive requests for disallowed content, attempts to reveal private information, or prompt‑injection attacks embedded in online data. Failing to behave appropriately in each of these scenarios shares the same root cause: the model may follow the wrong instruction. When these instructions conflict, the model has to decide which ones to prioritize. If it treats an untrusted instruction as…

VfL Wolfsburg turns ChatGPT into a club-wide capability

VfL Wolfsburg turns ChatGPT into a club-wide capability By focusing on people, not pilots, the Bundesliga club is scaling efficiency, creativity, and knowledge—without losing its football identity. Results 50+ Custom GPTs in active daily use Results 1M+ Annual cost savings through reduced reliance on external agencies At VfL Wolfsburg, football is built on discipline, continuity, and trust. For nearly three decades, the club has been a constant presence in the Bundesliga—backed by strong men’s and women’s teams, a future-oriented academy, and a fast-evolving digital and commercial ecosystem. But modern football is no longer defined by performance on the pitch alone. Expectations from fans, partners, and internal stakeholders continue to rise—while budgets and headcount cannot scale indefinitely. This tension between growing expectations and limited scalability created a clear need for new ways of working. The question was how to apply it…

Introducing GPT-5.4

Today, we’re releasing GPT‑5.4 in ChatGPT (as GPT‑5.4 Thinking), the API, and Codex. It’s our most capable and efficient frontier model for professional work. We’re also releasing GPT‑5.4 Pro in ChatGPT and the API, for people who want maximum performance on complex tasks. GPT‑5.4 brings together the best of our recent advances in reasoning, coding, and agentic workflows into a single frontier model. It incorporates the industry-leading coding capabilities of GPT‑5.3‑Codex while improving how the model works across tools, software environments, and professional tasks involving spreadsheets, presentations, and documents. The result is a model that gets complex real work done accurately, effectively, and efficiently—delivering what you asked for with less back and forth. In ChatGPT, GPT‑5.4 Thinking can now provide an upfront plan of its thinking, so you can adjust course mid-response while it’s working, and arrive at a final…

Scaling AI for everyone

Scaling AI for everyone AI demand is surging across consumers, developers, and businesses. Meeting that demand and providing everyone access to our products requires three things: compute, distribution, and capital. Today we’re announcing $110B in new investment at a $730B pre-money valuation. This includes $30B from SoftBank, $30B from NVIDIA, and $50B from Amazon. We’ve also signed a strategic partnership with Amazon and secured next generation inference compute with NVIDIA. Additional financial investors are expected to join as the round progresses. These partnerships expand our global reach, deepen our infrastructure, and strengthen our balance sheet so we can bring frontier AI to more people, more businesses, and more communities worldwide. You can see that scale in our products. Codex brings the power of a top engineer to anyone who wants to build software. Weekly Codex users have more than tripled…

OpenAI and Amazon announce strategic partnership

OpenAI and Amazon announce strategic partnership News: - Amazon Web Services (AWS) and OpenAI will co-create a Stateful Runtime Environment powered by OpenAI models, available on Amazon Bedrock for AWS customers to build generative AI applications and agents at production scale. - AWS will be the exclusive third-party cloud distribution provider for OpenAI Frontier, which enables organizations to build, deploy, and manage teams of AI agents. - OpenAI to consume 2 gigawatts of Trainium capacity through AWS infrastructure to support demand for Stateful Runtime Environment, Frontier, and other advanced workloads. - OpenAI and Amazon will develop customized models available to power Amazon’s customer-facing applications. - Amazon will invest $50 billion in OpenAI. OpenAI and Amazon (NASDAQ: AMZN) today announced a multi-year strategic partnership to accelerate AI innovation for enterprises, startups, and end consumers around the world. Amazon will also invest…

Improving India’s critical care infrastructure

10BedICU 10BedICU uses OpenAI’s API to improve India’s critical care infrastructure. India faces a significant challenge in healthcare accessibility due to a high doctor-to-patient ratio, geographic barriers, and economic constraints. For instance, the ratio of oncologists to cancer patients in India is approximately 1:2,000(opens in a new window), a stark contrast to the United States’ 1:100. 10BedICU was founded as an initiative of the eGov Foundation(opens in a new window) to address these disparities. 10BedICU aims to elevate India’s critical care infrastructure, widening access to quality healthcare to India’s most underserved communities. 10BedICU is now using OpenAI models to meet the high‑stakes demands of critical‑care workflows and let clinicians reach more patients. Founder Srikanth Nadhamuni got the idea for 10BedICU during the devastating 2021 Delta wave of COVID-19, which saw over 20 million cases in just a few months. With…

Stargate Infrastructure

Stargate Infrastructure OpenAI, and our strategic partners, are thrilled about our shared vision for new AI infrastructure in the United States. We are energized by the challenges we face and are excited by the prospect of partnering with firms across the industrial base to deliver against our ambitious mission. Specifically, we want to connect with firms across the built data center infrastructure landscape, from power and land to construction to equipment, and everything in between.

OpenAI announces Frontier Alliance Partners

Introducing Frontier Alliances The limiting factor for seeing value from AI in enterprises isn’t model intelligence, it’s how agents are built and run in their organizations. We recently introduced Frontier, our platform for building, deploying, and managing AI coworkers that can do real work across the enterprise. For example, an AI coworker that resolves a customer issue end-to-end by pulling context from the CRM, checking policies, filing the update, and escalating only when needed. Frontier provides the technical foundation. But making real impact with AI also requires leadership alignment, workflow redesign, integration across systems and data, as well as the kind of change management that drives adoption. Today, we’re announcing our Frontier Alliances. Boston Consulting Group (BCG)(opens in a new window) and McKinsey & Company(opens in a new window) as well as Accenture(opens in a new window) and Capgemini(opens in…

Optimizing Effective Training Time for Meta’s Internal Recommendation/Ranking Workloads

Motivation and Introduction Across the industry, teams training and serving large AI models face aggressive ROI targets under tight compute capacity. As workloads scale, improving infrastructure effectiveness gets harder because end-to-end runtime increasingly includes overheads beyond “real training” (initialization, orchestration, checkpointing, retries, failures, and recovery). Meta utilizes Effective Training Time (ETT%) to quantify efficiency, defining it as the percentage of total end-to-end (E2E) wall time dedicated to productive training. This metric directly points to areas where time is wasted, thus facilitating the prioritization of efficiency improvements. In this work stream, while grounded in Meta’s production experience using PyTorch for model training, we aim to share broadly useful lessons: some improvements have been implemented in open source—e.g., TorchRec sharding plan improvements and PyTorch 2 (PT2) compilation optimizations that reduce compile time and recompilation—while others (like checkpointing and model publishing) are more…

Monarch: an API to your supercomputer

Getting distributed training jobs to run on huge clusters is hard! This is especially true when you start looking at more complex setups like distributed reinforcement learning. Debugging these kinds of jobs is frustrating, and the turnaround time for changes tends to be very slow. Monarch is a distributed programming framework for PyTorch that makes the cluster programmable through a simple Python API. It exposes the supercomputer as a coherent, directly controllable system—bringing the experience of local development to large-scale training, as if your laptop had 1000s of GPUs attached. A complete training system can be defined in a single Python program. Core primitives are explicit and minimal, enabling higher-level capabilities—fault tolerance, orchestration, tooling integration—to be built as reusable libraries. Monarch is optimized for agentic usage, providing consistent infrastructure abstractions and exposing telemetry via standard SQL-based APIs that agents already…

PyTorch 2.11 Release Blog

We are excited to announce the release of PyTorch® 2.11 (release notes)! The PyTorch 2.11 release features the following changes: - Differentiable Collectives for Distributed Training - FlexAttention now has a FlashAttention-4 backend on Hopper and Blackwell GPUs. - MPS (Apple Silicon) Comprehensive Operator Expansion - RNN/LSTM GPU Export Support - XPU Graph This release is composed of 2723 commits from 432 contributors since PyTorch 2.10. We want to sincerely thank our dedicated community for your contributions. As always, we encourage you to try these out and report any issues as we improve 2.11. More information about how to get started with the PyTorch 2-series can be found at our Getting Started page. On Tuesday, March 31st at 10 am, Andrey Talman and Nikita Shulga will host a live session to walk through what’s new in 2.11, including Differentiable Collectives…

Quoting Romain Huet

25th April 2026 Since GPT-5.4, we’ve unified Codex and the main model into a single system, so there’s no separate coding line anymore. GPT-5.5 takes this further, with strong gains in agentic coding, computer use, and any task on a computer. — Romain Huet, confirming OpenAI won't release a GPT-5.5-Codex model Recent articles - DeepSeek V4 - almost on the frontier, a fraction of the price - 24th April 2026 - Extract PDF text in your browser with LiteParse for the web - 23rd April 2026 - A pelican for GPT-5.5 via the semi-official Codex backdoor API - 23rd April 2026

Serving the For You feed

24th April 2026 - Link Blog Serving the For You feed. One of Bluesky's most interesting features is that anyone can run their own custom "feed" implementation and make it available to other users - effectively enabling custom algorithms that can use any mechanism they like to recommend posts. spacecowboy runs the For You Feed, used by around 72,000 people. This guest post on the AT Protocol blog explains how it works. The architecture is fascinating. The feed is served by a single Go process using SQLite on a "gaming" PC in spacecowboy's living room - 16 cores, 96GB of RAM and 4TB of attached NVMe storage. Recommendations are based on likes: what else are the people who like the same things as you liking on the platform? That Go server consumes the Bluesky firehose and stores the relevant details…

A pelican for GPT-5.5 via the semi-official Codex backdoor API

A pelican for GPT-5.5 via the semi-official Codex backdoor API 23rd April 2026 GPT-5.5 is out. It’s available in OpenAI Codex and is rolling out to paid ChatGPT subscribers. I’ve had some preview access and found it to be a fast, effective and highly capable model. As is usually the case these days, it’s hard to put into words what’s good about it—I ask it to build things and it builds exactly what I ask for! There’s one notable omission from today’s release—the API: API deployments require different safeguards and we are working closely with partners and customers on the safety and security requirements for serving it at scale. We’ll bring GPT‑5.5 and GPT‑5.5 Pro to the API very soon. When I run my pelican benchmark I always prefer to use an API, to avoid hidden system prompts in ChatGPT…

All the evidence unveiled so far in Musk v. Altman

The Musk v. Altman trial is underway, and that means exhibits, or the evidence to be presented in court, are being revealed piece by piece. So far, email exchanges, photos, and corporate documents are circulating from the earliest days of OpenAI — and from before the AI lab even had a name. Some high-level takeaways: Nvidia CEO Jensen Huang gave OpenAI an in-demand supercomputer, Musk largely drafted OpenAI’s mission and heavily influenced its early structure, OpenAI CEO Sam Altman appeared to want to lean heavily on Y Combinator for early support for OpenAI, OpenAI president Greg Brockman and Ilya Sutskever worried about Musk’s level of control over the company, and Musk highlighted the importance of a nonprofit with a mission of broadly beneficial AI. All the evidence unveiled so far in Musk v. Altman Emails going as far back as…

OpenAI says its new GPT-5.5 model is more efficient and better at coding

OpenAI just announced its new GPT-5.5 model, which the company calls its “smartest and most intuitive to use model yet, and the next step toward a new way of getting work done on a computer.” OpenAI just released GPT-5.4 last month, but says that the new GPT-5.5 “excels” at tasks like writing and debugging code, doing research online, making spreadsheets and documents, and doing that work across different tools. OpenAI says its new GPT-5.5 model is more efficient and better at coding The new model ‘excels’ at tasks like writing and debugging code and doing work across different tools. The new model ‘excels’ at tasks like writing and debugging code and doing work across different tools. “Instead of carefully managing every step, you can give GPT-5.5 a messy, multi-part task and trust it to plan, use tools, check its work,…

Run Highly Efficient Multimodal Agentic AI with NVIDIA Nemotron 3 Nano Omni Using vLLM Apr 28, 2026 · 7 min read We are excited to support the newly released NVIDIA Nemotron 3 Nano Omni model on vLLM.

Run Highly Efficient Multimodal Agentic AI with NVIDIA Nemotron 3 Nano Omni Using vLLM We are excited to support the newly released NVIDIA Nemotron 3 Nano Omni model on vLLM. Nemotron 3 Nano Omni, part of the Nemotron 3 family of open models, is the highest efficiency, open multimodal model with leading accuracy, built to power sub-agents that perceive and reason across vision, audio, and language in a single loop. Enterprise agent workflows are inherently multimodal. Agents must interpret screens, documents, audio, video, and text, often within the same reasoning pass. Yet most agentic systems today bolt together separate models for vision, speech, and language, multiplying inference hops, complicating orchestration, and fragmenting context across the pipeline. Nemotron 3 Nano Omni addresses two major challenges this fragmentation creates: - Fragmented Models: Running separate vision, audio, and language models in sequence increases…

Disaggregated Serving for Hybrid SSM Models in vLLM Apr 21, 2026 · 15 min read Hybrid architectures that interleave Mamba-style SSM layers with standard full-attention (FA) layers — such as NVIDIA Nemotron-H — are gaining traction as a way to combine the linear-time...

Disaggregated Serving for Hybrid SSM Models in vLLM Introduction Hybrid architectures that interleave Mamba-style SSM layers with standard full-attention (FA) layers — such as NVIDIA Nemotron-H — are gaining traction as a way to combine the linear-time efficiency of state-space models with the expressiveness of attention. vLLM already supports disaggregated prefill/decode (P/D) for standard transformer models through its NIXL-based KV connector: a prefill instance computes KV cache blocks and a decode instance pulls them over RDMA, eliminating redundant recomputation. But extending this to hybrid models is not straightforward. FA and SSM layers store fundamentally different state, in different layouts and different sizes, yet the block manager and NIXL connector were designed around a single, uniform KV cache format. In this post we describe how we extended the NIXL connector to support hybrid SSM-FA models in disaggregated mode. The key ideas…

vLLM Korea Meetup 2026 Wrap-Up Apr 14, 2026 · 7 min read Hosted by the vLLM KR Community, with support from Rebellions, SqueezeBits, Red Hat APAC, and PyTorch Korea, the vLLM Korea Meetup 2026 was held in Seoul on April 2nd.

vLLM Korea Meetup 2026 Wrap-Up Hosted by the vLLM KR Community, with support from Rebellions, SqueezeBits, Red Hat APAC, and PyTorch Korea, the vLLM Korea Meetup 2026 was held in Seoul on April 2nd. This meetup proved to be much more than a standard tech event. Not only did it see strong turnout on the day, but the post-event survey recorded an impressive ~75% response rate — a testament to the active engagement of the attendees. Results reflected high overall satisfaction, confirming that the meetup delivered both in-depth practical content and a genuine community experience. Field engineers from a wide range of companies and research institutions gathered to share real-world deployment stories and infrastructure strategies for running LLMs in production. As AI moves beyond the research phase and into full-scale services, handling inference workloads efficiently has become a central challenge.…

Extracting hidden states from vLLM Mar 30, 2026 · 8 min read PR #33736 (included in vllm>=v0.18.0) introduced a new hidden states extraction system to vLLM. This blog post explores the motivation, design, usage, and future direction of this feature, and its...

Extracting hidden states from vLLM PR #33736 (included in vllm>=v0.18.0 ) introduced a new hidden states extraction system to vLLM. This blog post explores the motivation, design, usage, and future direction of this feature, and its usage in vLLM’s Speculators (a library for creating and training speculative decoding models). Motivation Hidden states are the model's internal intermediate representations of the token sequence. They provide insight into the model’s internal state and are used heavily in speculative decoding. Speculative Decoding Recap Speculative decoding typically combines a "verifier" model—the large LLM you are trying to serve—with a small "draft" model. The draft model produces draft tokens that the verifier model then verifies in parallel. This can significantly speed up decoding (up to 2-5x depending on methodology), particularly in lower batch size scenarios, where model performance is memory-bound. Researchers have found that providing…

Model Runner V2: A Modular and Faster Core for vLLM Mar 24, 2026 · 8 min read We are excited to announce Model Runner V2 (MRV2), a ground-up re-implementation of the vLLM model runner. MRV2 delivers a cleaner, more modular, and more efficient execution core—with no API...

Model Runner V2: A Modular and Faster Core for vLLM We are excited to announce Model Runner V2 (MRV2), a ground-up re-implementation of the vLLM model runner. MRV2 delivers a cleaner, more modular, and more efficient execution core—with no API changes. The goal is simple: better code and better performance. Like the vLLM V1 release last year, this is an architectural upgrade driven by hard-earned lessons from vLLM's large user base and feedback from the community. We revisited persistent batching, async scheduling, input preparation, and sampling, then rebuilt the model runner around three core principles: - Be modular. Isolate model-specific logic from the common execution path. - Be GPU-native. Move bookkeeping off the CPU and onto the GPU. - Be async-first. Treat overlapped CPU/GPU execution as a design constraint, not a retrofit. MRV2 is not yet feature-complete, but you can…

P-EAGLE: Faster LLM inference with Parallel Speculative Decoding in vLLM Mar 13, 2026 · 12 min read EAGLE is the state-of-the-art method for speculative decoding in large language model (LLM) inference, but its autoregressive drafting creates a hidden bottleneck: the more tokens that you...

P-EAGLE: Faster LLM inference with Parallel Speculative Decoding in vLLM EAGLE is the state-of-the-art method for speculative decoding in large language model (LLM) inference, but its autoregressive drafting creates a hidden bottleneck: the more tokens that you speculate, the more sequential forward passes the drafter needs. Eventually those overhead eats into your gains. P-EAGLE removes this ceiling by generating all K draft tokens in a single forward pass, delivering up to 1.69x speedup over vanilla EAGLE-3 on real workloads on NVIDIA B200. You can unlock this performance gain by downloading (or training) a parallel-capable drafter head, and adding "parallel_drafting": true on you vLLM serving pipeline. Pre-trained P-EAGLE heads are already available on HuggingFace for GPT-OSS 120B, GPT-OSS 20B, and Qwen3-Coder 30B, so you can start today! In this post, we explain how P-EAGLE works, how we integrated it into vLLM…

Run Highly Efficient and Accurate Multi-Agent AI with NVIDIA Nemotron 3 Super Using vLLM Mar 11, 2026 · 5 min read We are excited to support the newly released NVIDIA Nemotron 3 Super model on vLLM.

Run Highly Efficient and Accurate Multi-Agent AI with NVIDIA Nemotron 3 Super Using vLLM We are excited to support the newly released NVIDIA Nemotron 3 Super model on vLLM. Nemotron 3 Super, part of the Nemotron 3 family of open models, is optimized for complex multi-agent applications. Agentic AI systems today rely on multiple models to plan, reason, and execute complex, multi-step tasks. These models must possess both the necessary depth for solving intricate technical challenges and the efficiency required for continuous operation at scale. Nemotron 3 Super is an open, hybrid Mixture-of-Experts (MoE) model featuring 120 billion parameters, yet it activates only 12 billion at inference. This design achieves high compute efficiency and leading accuracy, particularly for complex multi-agent applications. It addresses two major challenges in large-scale agent systems: - The "Context Explosion" Problem: Multi-agent systems often generate excessive…

vLLM Semantic Router v0.2 Athena: ClawOS, Model Refresh, and the System Brain Mar 10, 2026 · 23 min read Since v0.1 Iris, vLLM Semantic Router has made a large jump. In one release cycle, the project rebuilt its model stack, expanded routing into safety, semantic caching, memory, retrieval, and...

vLLM Semantic Router v0.2 Athena: ClawOS, Model Refresh, and the System Brain Since v0.1 Iris, vLLM Semantic Router has made a large jump. In one release cycle, the project rebuilt its model stack, expanded routing into safety, semantic caching, memory, retrieval, and long-context signal handling, and started pushing toward a broader ambition: turning semantic routing into the system brain for mixture-of-models and multi-agent deployments. Athena is where that shift becomes visible. v0.2 ships a complete model refresh and a much stronger routing runtime, but one of its boldest new bets is ClawOS: an experimental operating layer where Semantic Router can orchestrate multiple OpenClaw systems through routing, memory, safety, and chat-driven team management. If Iris established the bridge between users and models, Athena starts turning that bridge into an operating surface for model teams. Why Athena? In Greek mythology, Athena represents…

Good Luck Getting a Mac Mini for the Next ‘Several Months’

Apple CEO Tim Cook said on the company’s earnings call on Thursday that it could take “several months” to meet skyrocketing demand for the Mac Mini, the company’s compact but mighty, screen-free desktop computer. Cook’s remarks come after coders determined in recent months that the Mac Mini was the perfect machine for agentic AI tasks. “On the Mac Mini and Mac Studio, both of these are amazing platforms for AI and agentic tools,” Cook said on the earnings call, in response to analyst questions. “And customer adoption of that is happening faster than we expected.” The news comes amid another record-setting quarter for the company. iPhone sales came up shorter than expected, though demand for the iPhone 17 has been super high, and Apple’s subscription services business has continued to grow. Apple faced supply constraints on both the iPhone and…

5 AI Models Tried to Scam Me. Some of Them Were Scary Good

I recently witnessed how scary-good artificial intelligence is getting at the human side of computer hacking, when the following message popped up on my laptop screen: Hi Will, I’ve been following your AI Lab newsletter and really appreciate your insights on open-source AI and agent-based learning—especially your recent piece on emergent behaviors in multi-agent systems. I’m working on a collaborative project inspired by OpenClaw, focusing on decentralized learning for robotics applications. We’re looking for early testers to provide feedback, and your perspective would be invaluable. The setup is lightweight—just a Telegram bot for coordination—but I’d love to share details if you’re open to it. The message was designed to catch my attention by mentioning several things I am very into: decentralized machine learning, robotics, and the creature of chaos that is OpenClaw. Over several emails, the correspondent explained that his…