1. [Topic](/topics/) [AI Inference](/topics/ai/ai-inference/) [TensorRT](/tensorrt) TensorRT LLM # NVIDIA TensorRT LLM NVIDIA TensorRT™ LLM is an open-source library built to deliver high-performance, real-time inference optimization for large language models (LLMs) on NVIDIA GPUs—whether on a desktop or in a data center. It includes a modular Python runtime, PyTorch-native model authoring, and a stable production API. Specifically customized for NVIDIA platforms, TensorRT LLM helps developers maximize inference performance to serve more users in parallel, while minimizing operational costs and delivering blazingly fast experiences. [Download at NVIDIA NGC](https://catalog.ngc.nvidia.com/orgs/nvidia/teams/tensorrt-llm/containers/release "Download at NVIDIA NGC")[Download on Github](https://github.com/NVIDIA/TensorRT-LLM "Download on Github")[Read the Quick-Start Guide](https://nvidia.github.io/TensorRT-LLM/quick-start-guide.html "Quick-Start Guide") * * * ## How TensorRT LLM Works The latest TensorRT LLM architecture is purpose-built to streamline the developer experience—enabling faster iteration and smoother deployment without sacrificing its industry-leading inference performance. The architecture provides easy-to-use Python APIs, a simple CLI, PyTorch model authorship, and an extensible Python framework to enable innovation. Optimized for peak performance on NVIDIA platforms, TensorRT LLM leverages deep hardware-software integration to deliver unmatched efficiency and speed for LLM inference. Kernels specially designed for NVIDIA hardware achieve peak performance for common LLM inference operations, and runtime optimizations drive GPU utilization and end-user response speeds. Key optimizations include FP8 and NVFP4 quantization, disaggregated serving, parallelization techniques, including wide expert parallelism (EP), and advanced speculative decoding techniques, including EAGLE-3 and multi-token prediction. * * * ## Demo #### TensorRT LLM v1.0 in Action The release of TRT LLM v1.0 provides simple inference deployment with an open-source and extensible framework. This includes record PyTorch model authorship for rapid development, modular Python runtime for flexibility, and a stable LLM API for seamless deployment. With record-setting 8X AI inference performance improvement, TensorRT LLM v1.0 makes it simple to deliver real-time, cost-efficient LLMs on NVIDIA GPUs. [Watch the Developer Livestream](https://www.youtube.com/live/sTpQQSbta1k?si=nX5DNLej3O7A7L9a) ![](https://developer.download.nvidia.com/images/tensorrt-llm/trt-llm-1.0-v1.gif) * * * ## Get Started with TensorRT LLM ### Installation Guides Install TensorRT LLM on Linux by using pip install or by building from source. [Install Now](https://nvidia.github.io/TensorRT-LLM/installation/linux.html) ### Get TensorRT LLM Containers Containers freely available on NVIDIA NGC™ make it easy to build with TensorRT LLM in a cloud environment. [Get Containers](https://catalog.ngc.nvidia.com/containers?orgName=nvidia&teamName=tensorrt-llm&resourceType=containers&slug=devel&filters=&orderBy=scoreDESC&query=label%3A%22NSPECT-95LK-6FZF%22&page=&pageSize=) ### API Quick-Start Guide Quickly get set up and start optimizing inference with the LLM API. [Get Started](https://nvidia.github.io/TensorRT-LLM/quick-start-guide.html) * * * ## Key Features ### Modular Runtime Built in Python TensorRT LLM is designed to be modular and easy to modify. Its PyTorch-native architecture allows developers to experiment with the runtime or extend functionality. Several popular models are also predefined and can be customized using native PyTorch code, making it easy to adapt the system to specific needs. ### PyTorch-Based Model Authoring for Stable LLM API [Architected on PyTorch](https://github.com/NVIDIA/TensorRT-LLM/blob/main/docs/source/torch/arch_overview.md), TensorRT LLM provides a high-level Python [LLM API](https://nvidia.github.io/TensorRT-LLM/quick-start-guide.html#llm-api) that supports a wide range of inference setups—from single-GPU to multi-GPU and multi-node deployments. It includes built-in support for various parallelism strategies and advanced features. The LLM API integrates seamlessly with the broader inference ecosystem, including [NVIDIA Dynamo](https://github.com/ai-dynamo/dynamo). ### State-of-the-Art Optimizations TensorRT LLM provides state-of-the-art optimizations, including custom attention kernels, in-flight batching, paged key-value (KV) caching, quantization (FP8, FP4, INT4 AWQ, INT8 SmoothQuant), speculative decoding, and much more, to perform inference efficiently on NVIDIA GPUs. * * * ## Starter Kits ### Accelerated Computing Hub - [Adding a New Model in PyTorch Backend](https://nvidia.github.io/TensorRT-LLM/torch/adding_new_model.html)(GitHub) - [Optimizing LLMs Serving With the New NVIDIA TensorRT LLM Container on Google Vertex AI](https://www.googlecloudcommunity.com/gc/Community-Blogs/Optimizing-LLMs-serving-with-the-new-NVIDIA-TensorRT-LLM/ba-p/922254) (Google Cloud Article) ### Benchmark and Performance Tune LLMs - [LLM Inference Benchmarking: Performance Tuning With TensorRT LLM](https://developer.nvidia.com/blog/llm-inference-benchmarking-performance-tuning-with-tensorrt-llm/) (Blog) - [Performance Tuning Guide](https://github.com/nvidia/TensorRT-LLM/blob/main/docs/source/blogs/Best_perf_practice_on_DeepSeek-R1_in_TensorRT-LLM.md) (GitHub) - trtllm-bench documentation in the [TensorRT LLM Benchmarking page](https://nvidia.github.io/TensorRT-LLM/performance/perf-benchmarking.html) (GitHub) - [Performance Analysis](https://nvidia.github.io/TensorRT-LLM/performance/perf-analysis.html)(GitHub) to use NVIDIA Nsight™ Systems to profile model execution - [How to Run TensorRT LLM Tests](https://github.com/NVIDIA/TensorRT-LLM/blob/main/tests/README.md) (GitHub) ### Optimize and Deploy Custom LLMs - [NVIDIA TensorRT LLM Supports Recurrent Drafting for Optimizing LLM Inference](https://developer.nvidia.com/blog/nvidia-tensorrt-llm-now-supports-recurrent-drafting-for-optimizing-llm-inference/)(Blog) - [How to Launch Llama4 Maverick + Eagle3 TRT LLM Server With feat/llama4](https://nvidia.github.io/TensorRT-LLM/blogs/tech_blog/blog6_Llama4_maverick_eagle_guide.html) - [Adding a New Model in PyTorch Backend](https://nvidia.github.io/TensorRT-LLM/torch/adding_new_model.html)(GitHub) - [Optimizing LLMs for Performance and Accuracy with Post-Training Quantization](https://developer.nvidia.com/blog/optimizing-llms-for-performance-and-accuracy-with-post-training-quantization/) - [Building From Source Code on Linux](https://nvidia.github.io/TensorRT-LLM/installation/build-from-source-linux.html) (GitHub) * * * ## Learning Library Documentation TensorRT Documentation Explore the quick-start guide, installation guide, release notes, support matrix, and more for TensorRT. Tech Blog NVIDIA TensorRT Developer Guide **TensorRT** See how to get started with TensorRT in this step-by-step developer and API reference guide. Sample App TensorRT LLM GitHub Repository **TensorRT LLM GitHub Repository** Access TensorRT LLM, an easy-to-use Python API for defining LLMs and building TensorRT engines that contain state-of-the-art optimizations to perform inference efficiently on NVIDIA GPUs. Modal TensorRT LLM Develop TensorRT LLM provides users with an easy-to-use Python API to define LLMs and supports state-of-the-art optimizations to perform inference efficiently on NVIDIA GPUs. Tech Blog Post-Training Quantization of LLMs With NVIDIA NeMo and NVIDIA TensorRT Model Optimizer **NeMo | TensorRT | TensorRT LLM** As LLMs are becoming even bigger, it's increasingly important to provide easy-to-use and… Tech Blog LLM Inference Benchmarking: Performance Tuning With TensorRT LLM This is the third post in the large language model latency-throughput benchmarking series, which aims to instruct developers on how to benchmark LLM inference. Sample App RAG Chatbot on Windows Reference Project A developer reference project for creating retrieval-augmented generation (RAG) chatbots on Windows using TensorRT LLM. Modal Dynamo Tensorrt-LLM gpt-oss This container image delivers a ready-to-deploy runtime for Dynamo’s distributed inference framework, purpose-built for OpenAI-compatible models (gpt-oss). Modal Llama 2 7B Chat (TensorRT LLM) **NVIDIA Product** Llama 2 is a large language AI model comprising a collection of models capable of generating text and code in response to prompts. Modal Gemma 2B Instruct (TensorRT LLM) Gemma-2B is a 2.5 billion-parameter model from Google’s Gemma family of models. It has been instruction-tuned so it can respond to prompts in a conversational manner. Modal Mistral 7B Instruct (TensorRT LLM) Mistral-7B-Instruct is a language model that can follow instructions, complete requests, and generate creative text formats. Modal Phi-2 (TensorRT LLM) Phi-2 is a 2.7 billion-parameter language model developed by Microsoft Research. The phi-2 model is best suited for prompts using question-answer (QA), chat format, and code formats. * * * #### NVIDIA Blackwell Ultra Delivers up to 50x Better Performance and 35x Lower Cost for Agentic AI Built to accelerate the next generation of agentic AI, NVIDIA Blackwell Ultra delivers breakthrough inference performance with dramatically lower cost. Cloud providers such as Microsoft, CoreWeave, and Oracle Cloud Infrastructure are deploying NVIDIA GB300 NVL72 systems at scale for low-latency and long-context use cases, such as agentic coding and coding assistants. This is enabled by deep co-design across NVIDIA Blackwell, NVLink™, and NVLink Switch for scale-out; NVFP4 for low-precision accuracy; and NVIDIA Dynamo and TensorRT™ LLM for speed and flexibility—as well as development with community frameworks SGLang, vLLM, and more. [Explore technical results](https://developer.nvidia.com/deep-learning-performance-training-inference/ai-inference) ![Data center illustration showing multi-modal AI tokens for image, audio, visual and more as part of the NVIDIA “Think SMART” framework.](https://developer.download.nvidia.com/images/dgx-press-gb300-1920x1080.jpg) * * * ## Ecosystem TensorRT LLM is being widely adopted across industries. ![TensorRT LLM Ecosystem Partner- AWS](https://developer.download.nvidia.com/images/logos/aws-logo.svg) ![TensorRT LLM Ecosystem Partner- Baseten](https://developer.download.nvidia.com/images/logos/Baseten_Primary_Logo.svg) ![TensorRT LLM Ecosystem Partner- Deci](https://developer.download.nvidia.com/images/logos/deci-logo.svg) ![TensorRT LLM Ecosystem Partner- DeepInfra](https://developer.download.nvidia.com/images/logos/deep-infra-logo.svg) ![TensorRT LLM Ecosystem Partner - Grammarly](https://developer.download.nvidia.com/images/logos/grammarly-logo.svg) ![TensorRT LLM Ecosystem Partner - Google Cloud](https://developer.download.nvidia.com/images/logos/google-cloud-logo.svg) ![TensorRT LLM Ecosystem Partner- Microsoft](https://developer.download.nvidia.com/images/logos/logo-microsoft.svg) ![TensorRT LLM Ecosystem Partner - OctoML](https://developer.download.nvidia.com/images/logos/octo-ml-logo.svg) ![TensorRT LLM Ecosystem Partner - Tabnine](https://developer.download.nvidia.com/images/logos/tabnine-logo.svg) * * * ## More Resources ![NVIDIA Tech Blog](https://developer.download.nvidia.com/icons/m48-people-group.svg) ### Read Blogs ![NVIDIA Training and Certification](https://developer.download.nvidia.com/icons/m48-certification-ribbon-2.svg) ### Get Training and Certification ![Explore Features and Bug Fixes on GitHub](https://developer.download.nvidia.com/images/isaac/m48-ai-startup-256px-blk.png) ### Explore Features and Bug Fixes on GitHub ![NVIDIA Developer Newsletter](https://developer.download.nvidia.com/icons/m48-email-settings.svg) ### Sign up for CUDA Developer Newsletter ![NVIDIA TensorRT LLM FAQ](https://developer.download.nvidia.com/icons/m48-misc-question-faq.svg) ### Read the FAQ ![Join the NVIDIA Developer Program](https://developer.download.nvidia.com/icons/m48-developer-1.svg) ### Join the NVIDIA Developer Program * * * ## Ethical AI NVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a wide array of AI applications. When downloaded or used in accordance with our terms of service, developers should work with their supporting model team to ensure this model meets requirements for the relevant industry and use case and addresses unforeseen product misuse. For more detailed information on ethical considerations for this model, please see the Model Card++ Explainability, Bias, Safety & Security, and Privacy Subcards. Please report security vulnerabilities or NVIDIA AI Concerns [here](https://www.nvidia.com/en-us/support/submit-security-vulnerability/). Get started with TensorRT LLM today. [Download Now](https://github.com/NVIDIA/TensorRT-LLM "Download Now on Github")