NVIDIA released Dynamo 1.0 on March 16, making its open-source software for coordinating large AI inference deployments available as a production release. The launch announcement focuses on a problem that grows as more agents share a data center: deciding which GPU should handle a request and where the request’s working memory should live.
Inference is the process of running a trained model to produce an answer. A busy service must handle short prompts, long conversations and bursts of tool-driven work at the same time. Buying more processors does not, by itself, settle how those requests are distributed.
Routing requests around the memory they need
Dynamo splits inference work across GPUs and moves data between processors and lower-cost storage. NVIDIA says the system can direct a request toward GPUs that already hold relevant information from earlier steps, reducing repeated work on long prompts and agent conversations.
The release also makes components available separately. KVBM handles memory management, NIXL handles data movement between GPUs, and Grove helps with scaling. NVIDIA lists integrations with inference projects including vLLM, SGLang and LMCache.
For teams already using one of those systems, the practical question is which component improves their existing serving setup. Adopting an entire new stack is a larger decision than evaluating a routing or memory-management module. Our guide to AI infrastructure beyond the chips explains why the surrounding systems can determine how much useful work the hardware delivers.
Performance claims need a workload attached
NVIDIA reports inference improvements of up to seven times on Blackwell GPUs in benchmarks cited with the release. That is a vendor-reported upper result for particular configurations, not a promise that an arbitrary application will become seven times faster.
A useful deployment comparison would keep the model, hardware and traffic pattern consistent, then measure response delay and completed output under load. A service handling many repeated prefixes may benefit differently from one serving unrelated, short questions.
Dynamo 1.0 is available to developers worldwide. The announcement names cloud and application companies adopting NVIDIA’s inference platform, but those names do not establish identical configurations or results across their services.
