Text Generation Inference

Text Generation Inference (TGI) is Hugging Face’s production server for serving HF models with batching, streaming, and tracing. Standard self-host option in HF-centric stacks. Serve open models behind HF/OpenAI-compatible APIs for agents. Large Language Model Text Generation Inference [!CAUTION] text-generation-inference is now in maintenance mode. Going forward, we will accept pull requests for minor bug fixes, documentation improvements and lightweight maintenance tasks. TGI has initiated the movement for optimized inference engines to rely on a transformers model architectures. This approach is now adopted by downstream inference engines, which we contribute to and recommend using going forward: vllm, SGLang, as well as local engines with inter-compatibility such as llama.cpp or MLX. A Rust, Python and gRPC server for text generation inference. Used in production.