Deploy this version​

docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
docker.litellm.ai/berriai/litellm:v1.80.15.rc.1

pip install litellm==1.80.15

Key Highlights​

• Manus API Support - New provider support for Manus API on /responses and GET /responses endpoints
• MiniMax Provider - Full support for MiniMax chat completions, TTS, and Anthropic native endpoint
• AWS Polly TTS - New TTS provider using AWS Polly API
• SSO Role Mapping - Configure role mappings for SSO providers directly in the UI
• Cost Estimator - New UI tool for estimating costs across multiple models and requests
• MCP Global Mode - Configure MCP servers globally with visibility controls
• Interactions API Bridge - Use all LiteLLM providers with the Interactions API
• RAG Query Endpoint - New RAG Search/Query endpoint for retrieval-augmented generation
• UI Usage - Endpoint Activity - Users can now see Endpoint Activity Metrics in the UI
• 50% Overhead Reduction - LiteLLM now sends 2.5× more requests to LLM providers

Performance - 50% Overhead Reduction​

LiteLLM now sends 2.5× more requests to LLM providers by replacing sequential if/elif chains with O(1) dictionary lookups for provider configuration resolution (92.7% faster). This optimization has a high impact because it runs inside the client decorator, which is invoked on every HTTP request made to the proxy server.

Before​

Note: Worse-looking provider metrics are a good sign here—they indicate requests spend less time inside LiteLLM.

============================================================
Fake LLM Provider Stats (When called by LiteLLM)
============================================================
Total Time:            0.56s
Requests/Second:       10746.68

Latency Statistics (seconds):
   Mean:               0.2039s
   Median (p50):       0.2310s
   Min:                0.0323s
   Max:                0.3928s
   Std Dev:            0.1166s
   p95:                0.3574s
   p99:                0.3748s

Status Codes:
   200: 6000

After​

============================================================
Fake LLM Provider Stats (When called by LiteLLM)
============================================================
Total Time:            1.42s
Requests/Second:       4224.49

Latency Statistics (seconds):
   Mean:               0.5300s
   Median (p50):       0.5871s
   Min:                0.0885s
   Max:                1.0482s
   Std Dev:            0.3065s
   p95:                0.9750s
   p99:                1.0444s

Status Codes:
   200: 6000

The benchmarks run LiteLLM locally with a lightweight LLM provider to eliminate network latency, isolating internal overhead and bottlenecks so we can focus on reducing pure LiteLLM overhead on a single instance.

UI Usage - Endpoint Activity​

Users can now see Endpoint Activity Metrics in the UI.

New Providers and Endpoints​

New Providers (11 new providers)​

New LLM API Endpoints (3 new endpoints)​

New Models / Updated Models​

New Model Support (100+ new models)​

Features​

• Gemini
  • Add image tokens in chat completion - PR #18327
  • Add usage object in image generation - PR #18328
  • Add thought signature support via tool call id - PR #18374
  • Add thought signature for non tool call requests - PR #18581
  • Preserve system instructions - PR #18585
  • Fix Gemini 3 images in tool response - PR #18190
  • Support snake_case for google_search tool parameters - PR #18451
  • Google GenAI adapter inline data support - PR #18477
  • Add deprecation_date for discontinued Google models - PR #18550
• Vertex AI
  • Add centralized get_vertex_base_url() helper for global location support - PR #18410
  • Convert image URLs to base64 for Vertex AI Anthropic - PR #18497
  • Separate Tool objects for each tool type per API spec - PR #18514
  • Add thought_signatures to VertexGeminiConfig - PR #18853
  • Add support for Vertex AI API keys - PR #18806
  • Add zai glm-4.7 model support - PR #18782
• Azure
  • Add Azure gpt-image-1.5 pricing to cost map - PR #18347
  • Add azure/gpt-5.2-chat model - PR #18361
  • Add support for image generation via Azure AD token - PR #18413
  • Add logprobs support for Azure OpenAI GPT-5.2 model - PR #18856
  • Add Azure BFL Flux 2 models for image generation and editing - PR #18764, PR #18766
• Bedrock
  • Add Bedrock Kimi K2 model support - PR #18797
  • Add support for model id in bedrock passthrough - PR #18800
  • Fix Nova model detection for Bedrock provider - PR #18250
  • Ensure toolUse.input is always a dict when converting from OpenAI format - PR #18414
• Databricks
  • Add enhanced authentication, security features, and custom user-agent support - PR #18349
• MiniMax
  • Add MiniMax chat completion support - PR #18380
  • Add Anthropic native endpoint support for MiniMax - PR #18377
  • Add support for MiniMax TTS - PR #18334
  • Add MiniMax provider support to UI dashboard - PR #18496
• Together AI
  • Add supports_response_schema to all supported Together AI models - PR #18368
• OpenRouter
  • Add OpenRouter embeddings API support - PR #18391
• Anthropic
  • Pass server_tool_use and tool_search_tool_result blocks - PR #18770
  • Add Anthropic cache control option to image tool call results - PR #18674
• Ollama
  • Add dimensions for ollama embedding - PR #18536
  • Extract pure base64 data from data URLs for Ollama - PR #18465
• Watsonx
  • Add Watsonx fields support - PR #18569
  • Fix Watsonx Audio Transcription - filter model field - PR #18810
• SAP
  • Add SAP creds for list in proxy UI - PR #18375
  • Pass through extra params from allowed_openai_params - PR #18432
  • Add client header for SAP AI Core Tracking - PR #18714
• Fireworks AI
  • Correct deepseek-v3p2 pricing - PR #18483
• ZAI
  • Add GLM-4.7 model with reasoning support - PR #18476
• Codestral
  • Correctly route codestral chat and FIM endpoints - PR #18467
• Azure AI
  • Fix authentication errors at messages API via azure_ai - PR #18500

New Provider Support​

• AWS Polly - Add AWS Polly API for TTS - PR #18326
• GigaChat - Add GigaChat provider support - PR #18564
• LlamaGate - Add LlamaGate as a new provider - PR #18673
• Abliteration AI - Add abliteration.ai provider - PR #18678
• Manus - Add Manus API support on /responses, GET /responses - PR #18804
• 5 AI Providers via openai_like - Add 5 AI providers using openai_like - PR #18362

Bug Fixes​

• Gemini
  • Properly catch context window exceeded errors - PR #18283
  • Remove prompt caching headers as support has been removed - PR #18579
  • Fix generate content request with audio file id - PR #18745
  • Fix google_genai streaming adapter provider handling - PR #18845
• Groq
  • Remove deprecated Groq models and update model registry - PR #18062
• Vertex AI
  • Handle unsupported region for Vertex AI count tokens endpoint - PR #18665
• General
  • Fix request body for image embedding request - PR #18336
  • Fix lost tool_calls when streaming has both text and tool_calls - PR #18316
  • Add all resolution for gpt-image-1.5 - PR #18586
  • Fix gpt-image-1 cost calculation using token-based pricing - PR #17906
  • Fix response_format leaking into extra_body - PR #18859
  • Align max_tokens with max_output_tokens for consistency - PR #18820

LLM API Endpoints​

Features​

• Responses API
  • Add new compact endpoint (v1/responses/compact) - PR #18697
  • Support more streaming callback hooks - PR #18513
  • Add mapping for reasoning effort to summary param - PR #18635
  • Add output_text property to ResponsesAPIResponse - PR #18491
  • Add annotations to completions responses API bridge - PR #18754
• Interactions API
  • Allow using all LiteLLM providers (interactions -> responses API bridge) - PR #18373
• RAG Search API
  • Add RAG Search/Query endpoint - PR #18376
• CountTokens API
  • Add Bedrock as a new provider for /v1/messages/count_tokens - PR #18858
• Generate Content
  • Add generate content in LLM route - PR #18405
• General
  • Enable async_post_call_failure_hook to transform error responses - PR #18348
  • Calculate total_tokens manually if missing and can be calculated - PR #18445
  • Add custom llm provider to get_llm_provider when sent via UI - PR #18638

Bugs​

• General
  • Handle empty error objects in response conversion - PR #18493
  • Preserve client error status codes in streaming mode - PR #18698
  • Return json error response instead of SSE format for initial streaming errors - PR #18757
  • Fix auth header for custom api base in generateContent request - PR #18637
  • Tool content should be string for Deepinfra - PR #18739
  • Fix incomplete usage in response object passed - PR #18799
  • Unify model names to provider-defined names - PR #18573

Management Endpoints / UI​

Features​

• SSO Configuration
  • Add SSO Role Mapping feature - PR #18090
  • Add SSO Settings Page - PR #18600
  • Allow adding role mappings for SSO - PR #18593
  • SSO Settings Page Add Role Mappings - PR #18677
  • SSO Settings Loading State + Deprecate Previous SSO Flow - PR #18617
• Virtual Keys
  • Allow deleting key expiry - PR #18278
  • Add optional query param "expand" to /key/list - PR #18502
  • Key Table Loading Skeleton - PR #18527
  • Allow column resizing on Keys Table - PR #18424
  • Virtual Keys Table Loading State Between Pages - PR #18619
  • Key and Team Router Setting - PR #18790
  • Allow router_settings on Keys and Teams - PR #18675
  • Use timedelta to calculate key expiry on generate - PR #18666
• Models + Endpoints
  • Add Model Clearer Flow For Team Admins - PR #18532
  • Model Page Loading State - PR #18574
  • Model Page Model Provider Select Performance - PR #18425
  • Model Page Sorting Sorts Entire Set - PR #18420
  • Refactor Model Hub Page - PR #18568
  • Add request provider form on UI - PR #18704
• Organizations & Teams
  • Allow Organization Admins to See Organization Tab - PR #18400
  • Resolve Organization Alias on Team Table - PR #18401
  • Resolve Team Alias in Organization Info View - PR #18404
  • Allow Organization Admins to View Their Organization Info - PR #18417
  • Allow editing team_member_budget_duration in /team/update - PR #18735
  • Reusable Duration Select + Team Update Member Budget Duration - PR #18736
• Usage & Spend
  • Add Error Code Filtering on Spend Logs - PR #18359
  • Add Error Code Filtering on UI - PR #18366
  • Usage Page User Max Budget fix - PR #18555
  • Add endpoint to Daily Activity Tables - PR #18729
  • Endpoint Activity in Usage - PR #18798
• Cost Estimator
  • Add Cost Estimator for AI Gateway - PR #18643
  • Add view for estimating costs across requests - PR #18645
  • Allow selecting many models for cost estimator - PR #18653
• CloudZero
  • Improve Create and Delete Path for CloudZero - PR #18263
  • Add CloudZero UI Docs - PR #18350
• Playground
  • Add MCP test support to completions on Playground - PR #18440
  • Add selectable MCP servers to the playground - PR #18578
  • Add custom proxy base URL support to Playground - PR #18661
• General UI
  • UI styling improvements and fixes - PR #18310
  • Add reusable "New" badge component for feature highlights - PR #18537
  • Hide New Badges - PR #18547
  • Change Budget page to Have Tabs - PR #18576
  • Clicking on Logo Directs to Correct URL - PR #18575
  • Add UI support for configuring meta URLs - PR #18580
  • Expire Previous UI Session Tokens on Login - PR #18557
  • Add license endpoint - PR #18311
  • Router Fields Endpoint + React Query for Router Fields - PR #18880

Bugs​

• UI Fixes
  • Fix Key Creation MCP Settings Submit Form Unintentionally - PR #18355
  • Fix UI Disappears in Development Environments - PR #18399
  • Fix Disable Admin UI Flag - PR #18397
  • Remove Model Analytics From Model Page - PR #18552
  • Useful Links Remove Modal on Adding Links - PR #18602
  • SSO Edit Modal Clear Role Mapping Values on Provider Change - PR #18680
  • UI Login Case Sensitivity fix - PR #18877
• API Fixes
  • Fix User Invite & Key Generation Email Notification Logic - PR #18524
  • Normalize Proxy Config Callback - PR #18775
  • Return empty data array instead of 500 when no models configured - PR #18556
  • Enforce org level max budget - PR #18813

AI Integrations​

New Integrations (4 new integrations)​

Logging​

• DataDog
  • Fix span kind fallback when parent_id missing - PR #18418
• Langfuse
  • Map Gemini cached_tokens to Langfuse cache_read_input_tokens - PR #18614
• Prometheus
  • Align prometheus metric names with DEFINED_PROMETHEUS_METRICS - PR #18463
  • Add Prometheus metrics for request queue time and guardrails - PR #17973
  • Add caching metrics for cache hits, misses, and tokens - PR #18755
  • Skip metrics for invalid API key requests - PR #18788
• Braintrust
  • Pass span_attributes in async logging and skip tags on non-root spans - PR #18409
• CloudZero
  • Add user email to CloudZero - PR #18584
• OpenTelemetry
  • Use already configured opentelemetry providers - PR #18279
  • Prevent LiteLLM from closing external OTEL spans - PR #18553
  • Allow configuring arize project name for OpenTelemetry service name - PR #18738
• LangSmith
  • Add support for LangSmith organization-scoped API keys with tenant ID - PR #18623
• Generic API Logger
  • Add log_format option to GenericAPILogger - PR #18587

Guardrails​

• Content Filter
  • Add content filter logs page - PR #18335
  • Log actual event type for guardrails - PR #18489
• Qualifire
  • Add Qualifire eval webhook - PR #18836
• Lasso Security
  • Add Lasso guardrail API docs - PR #18652
• Noma Security
  • Add MCP guardrail support for Noma - PR #18668
• Bedrock Guardrails
  • Remove redundant Bedrock guardrail block handling - PR #18634
• General
  • Generic guardrail API update - PR #18647
  • Prevent proxy startup failures from case-sensitive tool permission guardrail validation - PR #18662
  • Extend case normalization to ALL guardrail types - PR #18664
  • Fix MCP handling in unified guardrail - PR #18630
  • Fix embeddings calltype for guardrail precallhook - PR #18740

Spend Tracking, Budgets and Rate Limiting​

• Platform Fee / Margins - Add support for Platform Fee / Margins - PR #18427
• Negative Budget Validation - Add validation for negative budget - PR #18583
• Cost Calculation Fixes
  • Correct cost calculation when reasoning_tokens are without text_tokens - PR #18607
  • Fix background cost tracking tests - PR #18588
• Tag Routing - Support toggling tag matching between ANY and ALL - PR #18776

MCP Gateway​

• MCP Global Mode - Add MCP global mode - PR #18639
• MCP Server Visibility - Add configurable MCP server visibility - PR #18681
• MCP Registry - Add MCP registry - PR #18850
• MCP Stdio Header - Support MCP stdio header env overrides - PR #18324
• Parallel Tool Fetching - Parallelize tool fetching from multiple MCP servers - PR #18627
• Optimize MCP Server Listing - Separate health checks for optimized listing - PR #18530
• Auth Improvements
  • Require auth for MCP connection test endpoint - PR #18290
  • Fix MCP gateway OAuth2 auth issues and ClosedResourceError - PR #18281
• Bug Fixes
  • Fix MCP server health status reporting - PR #18443
  • Fix OpenAPI to MCP tool conversion - PR #18597
  • Remove exec() usage and handle invalid OpenAPI parameter names for security - PR #18480
  • Fix MCP error when using multiple servers simultaneously - PR #18855
• Migrate MCP Fetching Logic to React Query - PR #18352

Performance / Loadbalancing / Reliability improvements​

• 92.7% Faster Provider Config Lookup - LiteLLM now stresses LLM providers 2.5x more - PR #18867
• Lazy Loading Improvements
  • Consolidate lazy import handlers with registry pattern - PR #18389
  • Complete lazy loading migration for all 180+ LLM config classes - PR #18392
  • Lazy load additional components (types, callbacks, utilities) - PR #18396
  • Add lazy loading for get_llm_provider - PR #18591
  • Lazy-load heavy audio library and loggers - PR #18592
  • Lazy load 9 heavy imports in litellm/utils.py - PR #18595
  • Lazy load heavy imports to improve import time and memory usage - PR #18610
  • Implement lazy loading for provider configs, model info classes, streaming handlers - PR #18611
  • Lazy load 15 additional imports - PR #18613
  • Lazy load 15+ unused imports - PR #18616
  • Lazy load DatadogLLMObsInitParams - PR #18658
  • Migrate utils.py lazy imports to registry pattern - PR #18657
  • Lazy load get_llm_provider and remove_index_from_tool_calls - PR #18608
• Router Improvements
  • Validate routing_strategy at startup to fail fast with helpful error - PR #18624
  • Correct num_retries tracking in retry logic - PR #18712
  • Improve error messages and validation for wildcard routing with multiple credentials - PR #18629
• Memory Improvements
  • Add memory pattern detection test and fix bad memory patterns - PR #18589
  • Add unbounded data structure detection to memory test - PR #18590
  • Add memory leak detection tests with CI integration - PR #18881
• Database
  • Add idx on LOWER(user_email) for faster duplicate email checks - PR #18828
  • Proactive RDS IAM token refresh to prevent 15-min connection failed - PR #18795
  • Clarify database_connection_pool_limit applies per worker - PR #18780
  • Make base_connection_pool_limit default value the same - PR #18721
• Docker
  • Add libsndfile to database Docker image for audio processing - PR #18612
  • Add line_profiler support for performance analysis and fix Windows CRLF issues - PR #18773
• Helm
  • Add lifecycle support to Helm charts - PR #18517
• Authentication
  • Add Kubernetes ServiceAccount JWT authentication support - PR #18055
  • Use async anthropic client to prevent event loop blocking - PR #18435
• Logging Worker
  • Handle event loop changes in multiprocessing - PR #18423
• Security
  • Prevent expired key plaintext leak in error response - PR #18860
  • Mask extra header secrets in model info - PR #18822
  • Prevent duplicate User-Agent tags in request_tags - PR #18723
  • Properly use litellm api keys - PR #18832
• Misc
  • Remove double imports in main.py - PR #18406
  • Add LITELLM_DISABLE_LAZY_LOADING env var to fix VCR cassette creation issue - PR #18725
  • Add xiaomi_mimo to LlmProviders enum to fix router support - PR #18819
  • Allow installation with current grpcio on old Python - PR #18473
  • Add Custom CA certificates to boto3 clients - PR #18852
  • Fix bedrock_cache, metadata and max_model_budget - PR #18872
  • Fix LiteLLM SDK embedding headers missing field - PR #18844
  • Put automatic reasoning summary inclusion behind feat flag - PR #18688
  • turn_off_message_logging Does Not Redact Request Messages in proxy_server_request Field - PR #18897

Documentation Updates​

• Provider Documentation
  • Update MiniMax docs to be in proper format - PR #18403
  • Add docs for 5 AI providers - PR #18388
  • Fix gpt-5-mini reasoning_effort supported values - PR #18346
  • Fix PDF documentation inconsistency in Anthropic page - PR #18816
  • Update OpenRouter docs to include embedding support - PR #18874
  • Add LITELLM_REASONING_AUTO_SUMMARY in doc - PR #18705
• MCP Documentation
  • Agentcore MCP server docs - PR #18603
  • Mention MCP prompt/resources types in overview - PR #18669
  • Add Focus docs - PR #18837
• Guardrails Documentation
  • Qualifire docs hotfix - PR #18724
• Infrastructure Documentation
  • IAM Roles Anywhere docs - PR #18559
  • Fix formatting in proxy configs documentation - PR #18498
  • Fix GCS cache docs missing for proxy mode - PR #13328
  • Fix how to execute cloudzero sql - PR #18841
• General
  • LiteLLM adopters section - PR #18605
  • Remove redundant comments about setting litellm.callbacks - PR #18711
  • Update header to be markdown bold by removing space - PR #18846
  • Manus docs - new provider - PR #18817

New Contributors​

• @prasadkona made their first contribution in PR #18349
• @lucasrothman made their first contribution in PR #18283
• @aggeentik made their first contribution in PR #18317
• @mihidumh made their first contribution in PR #18361
• @Prazeina made their first contribution in PR #18498
• @systec-dk made their first contribution in PR #18500
• @xuan07t2 made their first contribution in PR #18514
• @RensDimmendaal made their first contribution in PR #18190
• @yurekami made their first contribution in PR #18483
• @agertz7 made their first contribution in PR #18556
• @yudelevi made their first contribution in PR #18550
• @smallp made their first contribution in PR #18536
• @kevinpauer made their first contribution in PR #18569
• @cansakiroglu made their first contribution in PR #18517
• @dee-walia20 made their first contribution in PR #18432
• @luxinfeng made their first contribution in PR #18477
• @cantalupo555 made their first contribution in PR #18476
• @andersk made their first contribution in PR #18473
• @majiayu000 made their first contribution in PR #18467
• @amangupta-20 made their first contribution in PR #18529
• @hamzaq453 made their first contribution in PR #18480
• @ktsaou made their first contribution in PR #18627
• @FlibbertyGibbitz made their first contribution in PR #18624
• @drorIvry made their first contribution in PR #18594
• @urainshah made their first contribution in PR #18524
• @mangabits made their first contribution in PR #18279
• @0717376 made their first contribution in PR #18564
• @nmgarza5 made their first contribution in PR #17330
• @wileykestner made their first contribution in PR #18445
• @minijeong-log made their first contribution in PR #14440
• @Isaac4real made their first contribution in PR #18710
• @marukaz made their first contribution in PR #18711
• @rohitravirane made their first contribution in PR #18712
• @lizzzcai made their first contribution in PR #18714
• @hkd987 made their first contribution in PR #18673
• @Mr-Pepe made their first contribution in PR #18674
• @gkarthi-signoz made their first contribution in PR #18726
• @Tianduo16 made their first contribution in PR #18723
• @wilsonjr made their first contribution in PR #18721
• @abliteration-ai made their first contribution in PR #18678
• @danialkhan02 made their first contribution in PR #18770
• @ihower made their first contribution in PR #18409
• @elkkhan made their first contribution in PR #18391
• @runixer made their first contribution in PR #18435
• @choby-shun made their first contribution in PR #18776
• @jutaz made their first contribution in PR #18853
• @sjmatta made their first contribution in PR #18250
• @andres-ortizl made their first contribution in PR #18856
• @gauthiermartin made their first contribution in PR #18844
• @mel2oo made their first contribution in PR #18845
• @DominikHallab made their first contribution in PR #18846
• @ji-chuan-che made their first contribution in PR #18540
• @raghav-stripe made their first contribution in PR #18858
• @akraines made their first contribution in PR #18629
• @otaviofbrito made their first contribution in PR #18665
• @chetanchoudhary-sumo made their first contribution in PR #18587
• @pascalwhoop made their first contribution in PR #13328
• @orgersh92 made their first contribution in PR #18652
• @DevajMody made their first contribution in PR #18497
• @matt-greathouse made their first contribution in PR #18247
• @emerzon made their first contribution in PR #18290
• @Eric84626 made their first contribution in PR #18281
• @LukasdeBoer made their first contribution in PR #18055
• @LingXuanYin made their first contribution in PR #18513
• @krisxia0506 made their first contribution in PR #18698
• @LouisShark made their first contribution in PR #18414

Full Changelog​

View complete changelog on GitHub

Deploy this version​

docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
docker.litellm.ai/berriai/litellm:v1.80.11-stable

pip install litellm==1.80.11

Key Highlights​

• Gemini 3 Flash Preview - Day 0 support for Google's Gemini 3 Flash Preview with reasoning capabilities
• Stability AI Image Generation - New provider for Stability AI image generation and editing
• LiteLLM Content Filter - Built-in guardrails for harmful content, bias, and PII detection with image support
• New Provider: Venice.ai - Support for Venice.ai API via providers.json
• Unified Skills API - Skills API works across Anthropic, Vertex, Azure, and Bedrock
• Azure Sentinel Logging - New logging integration for Azure Sentinel
• Guardrails Load Balancing - Load balance between multiple guardrail providers
• Email Budget Alerts - Send email notifications when budgets are reached
• Cloudzero Integration on UI - Setup your Cloudzero Integration Directly on the UI

Cloudzero Integration on UI​

Users can now configure their Cloudzero Integration directly on the UI.

Performance: 50% Reduction in Memory Usage and Import Latency for the LiteLLM SDK​

We've completely restructured litellm.__init__.py to defer heavy imports until they're actually needed, implementing lazy loading for 109 components.

This refactoring includes 41 provider config classes, 40 utility functions, cache implementations (Redis, DualCache, InMemoryCache), HTTP handlers, logging, types, and other heavy dependencies. Heavy libraries like tiktoken and boto3 are now loaded on-demand rather than eagerly at import time.

This makes LiteLLM especially beneficial for serverless functions, Lambda deployments, and containerized environments where cold start times and memory footprint matter.

New Providers and Endpoints​

New Providers (5 new providers)​

New LLM API Endpoints (2 new endpoints)​

New Models / Updated Models​

New Model Support (55+ new models)​

Features​

• Gemini
  • Add Gemini 3 Flash Preview day 0 support with reasoning - PR #18135
  • Support extra_headers in batch embeddings - PR #18004
  • Propagate token usage when generating images - PR #17987
  • Use JSON instead of form-data for image edit requests - PR #18012
  • Fix web search requests count - PR #17921
• Anthropic
  • Use dynamic max_tokens based on model - PR #17900
  • Fix claude-3-7-sonnet max_tokens to 64K default - PR #17979
  • Add OpenAI-compatible API with modify_params=True - PR #17106
• Vertex AI
  • Add Gemini 3 Flash Preview support - PR #18164
  • Add reasoning support for gemini-3-flash-preview - PR #18175
  • Fix image edit credential source - PR #18121
  • Pass credentials to PredictionServiceClient for custom endpoints - PR #17757
  • Fix multimodal embeddings for text + base64 image combinations - PR #18172
  • Add OCR support for DeepSeek model - PR #17971
• Azure AI
  • Add Azure Cohere 4 reranking models - PR #17961
  • Add Azure DeepSeek V3.2 versions - PR #18019
  • Return AzureAnthropicConfig for Claude models in get_provider_chat_config - PR #18086
• Fireworks AI
  • Add reasoning param support for Fireworks AI models - PR #17967
• Bedrock
  • Add Qwen 2 and Qwen 3 to get_bedrock_model_id - PR #18100
  • Remove ttl field when routing to bedrock - PR #18049
  • Add Bedrock Stability image edit models - PR #18254
• Perplexity
  • Use API-provided cost instead of manual calculation - PR #17887
• OpenAI
  • Add diarize model for audio transcription - PR #18117
  • Add gpt-image-1.5-2025-12-16 in model cost map - PR #18107
  • Fix cost calculation of gpt-image-1 model - PR #17966
• GitHub Copilot
  • Add github_copilot model info - PR #17858
• Custom LLM
  • Add image_edit and aimage_edit support - PR #17999

Bug Fixes​

• Gemini
  • Fix pricing for Gemini 3 Flash on Vertex AI - PR #18202
  • Add output_cost_per_image_token for gemini-2.5-flash-image models - PR #18156
  • Fix properties should be non-empty for OBJECT type - PR #18237
• Qwen
  • Add qwen3-embedding-8b input per token price - PR #18018
• General
  • Fix image URL handling - PR #18139
  • Support Signed URLs with Query Parameters in Image Processing - PR #17976
  • Add none to encoding_format instead of omitting it - PR #18042

LLM API Endpoints​

Features​

• Responses API
  • Add provider specific tools support - PR #17980
  • Add custom headers support - PR #18036
  • Fix tool calls transformation in completion bridge - PR #18226
  • Use list format with input_text for tool results - PR #18257
  • Add cost tracking in background mode - PR #18236
  • Fix Claude code responses API bridge errors - PR #18194
• Chat Completions API
  • Add support for agent skills - PR #18031
• Skills API
  • Unified Skills API works across Anthropic, Vertex, Azure, Bedrock - PR #18232
• Search API
  • Add new RAG Search API with rerankers - PR #18217
• Interactions API
  • Add Google Interactions API on SDK and AI Gateway - PR #18079, PR #18081
• Image Edit API
  • Add drop_params support and fix Vertex AI config - PR #18077
• General
  • Skip adding beta headers for Vertex AI as it is not supported - PR #18037
  • Fix managed files endpoint - PR #18046
  • Allow base_model for non-Azure providers in proxy - PR #18038

Bugs​

• General
  • Fix basemodel import in guardrail translation - PR #17977
  • Fix No module named 'fastapi' error - PR #18239

Management Endpoints / UI​

Features​

• Virtual Keys
  • Add master key rotation for credentials table - PR #17952
  • Fix tag management to preserve encrypted fields in litellm_params - PR #17484
  • Fix key delete and regenerate permissions - PR #18214
• Models + Endpoints
  • Add Models Conditional Rendering in UI - PR #18071
  • Add Health Check Model for Wildcard Model in UI - PR #18269
  • Auto Resolve Vector Store Embedding Model Config - PR #18167
• Vector Stores
  • Add Milvus Vector Store UI support - PR #18030
  • Persist Vector Store Settings in Team Update - PR #18274
• Logs & Spend
  • Add LiteLLM Overhead to Logs - PR #18033
  • Show LiteLLM Overhead in Logs UI - PR #18034
  • Resolve Team ID to Team Alias in Usage Page - PR #18275
  • Fix Usage Page Top Key View Button Visibility - PR #18203
• SSO & Health
  • Add SSO Readiness Health Check - PR #18078
  • Fix /health/test_connection to resolve env variables like /chat/completions - PR #17752
• CloudZero
  • Add CloudZero Cost Tracking UI - PR #18163
  • Add Delete CloudZero Settings Route and UI - PR #18168, PR #18170
• General
  • Update UI path handling for non-root Docker - PR #17989

Bugs​

• UI Fixes
  • Fix Login Page Failed To Parse JSON Error - PR #18159
  • Fix new user route user_id collision handling - PR #17559
  • Fix Callback Environment Variables Casing - PR #17912

AI Integrations​

Logging​

• Azure Sentinel
  • Add new Azure Sentinel Logger integration - PR #18146
• Prometheus
  • Add extraction of top level metadata for custom labels - PR #18087
• Langfuse
  • Fix not working log_failure_event - PR #18234
• Arize Phoenix
  • Fix nested spans - PR #18102
• General
  • Change extra_headers to additional_headers - PR #17950

Guardrails​

• LiteLLM Content Filter
  • Add built-in guardrails for harmful content, bias, etc. - PR #18029
  • Add support for running content filters on images - PR #18044
  • Add support for Brazil PII field - PR #18076
  • Add configurable guardrail options for content filtering - PR #18007
• Guardrails API
  • Support LLM tool call response checks on /chat/completions, /v1/responses, /v1/messages - PR #17619
  • Add guardrails load balancing - PR #18181
  • Fix guardrails for passthrough endpoint - PR #18109
  • Add headers to metadata for guardrails on pass-through endpoints - PR #17992
  • Various fixes for guardrail on OpenRouter models - PR #18085
• Lakera
  • Add monitor mode for Lakera - PR #18084
• Pillar Security
  • Add masking support and MCP call support - PR #17959
• Bedrock Guardrails
  • Add support for Bedrock image guardrails - PR #18115
  • Guardrails block action takes precedence over masking - PR #17968

Secret Managers​

• HashiCorp Vault
  • Add documentation for configurable Vault mount - PR #18082
  • Add per-team Vault configuration - PR #18150
• UI
  • Add secret manager settings controls to team management UI - PR #18149

Spend Tracking, Budgets and Rate Limiting​

• Email Budget Alerts - Send email notifications when budgets are reached - PR #17995

MCP Gateway​

• Auth Header Propagation - Add MCP auth header propagation - PR #17963
• Fix deepcopy error - Fix MCP tool call deepcopy error when processing requests - PR #18010
• Fix list tool - Fix MCP list_tools not working without database connection - PR #18161

Agent Gateway (A2A)​

• New Provider: Agent Gateway - Add pydantic ai agents support - PR #18013
• VertexAI Agent Engine - Add Vertex AI Agent Engine provider - PR #18014
• Fix model extraction - Fix get_model_from_request() to extract model ID from Vertex AI passthrough URLs - PR #18097

Performance / Loadbalancing / Reliability improvements​

• Lazy Imports - Use per-attribute lazy imports and extract shared constants - PR #17994
• Lazy Load HTTP Handlers - Lazy load http handlers - PR #17997
• Lazy Load Caches - Lazy load caches - PR #18001
• Lazy Load Types - Lazy load bedrock types, .types.utils, GuardrailItem - PR #18053, PR #18054, PR #18072
• Lazy Load Configs - Lazy load 41 configuration classes - PR #18267
• Lazy Load Client Decorators - Lazy load heavy client decorator imports - PR #18064
• Prisma Build Time - Download Prisma binaries at build time instead of runtime for security restricted environments - PR #17695
• Docker Alpine - Add libsndfile to Alpine image for ARM64 audio processing - PR #18092
• Security - Prevent LiteLLM API key leakage on /health endpoint failures - PR #18133

Documentation Updates​

• SAP Docs - Update SAP documentation - PR #17974
• Pydantic AI Agents - Add docs on using pydantic ai agents with LiteLLM A2A gateway - PR #18026
• Vertex AI Agent Engine - Add Vertex AI Agent Engine documentation - PR #18027
• Router Order - Add router order parameter documentation - PR #18045
• Secret Manager Settings - Improve secret manager settings documentation - PR #18235
• Gemini 3 Flash - Add version requirement in Gemini 3 Flash blog - PR #18227
• README - Expand Responses API section and update endpoints - PR #17354
• Amazon Nova - Add Amazon Nova to sidebar and supported models - PR #18220
• Benchmarks - Add infrastructure recommendations to benchmarks documentation - PR #18264
• Broken Links - Fix broken link corrections - PR #18104
• README Fixes - Various README improvements - PR #18206

Infrastructure / CI/CD​

• PR Templates - Add LiteLLM team PR template and CI/CD rules - PR #17983, PR #17985
• Issue Labeling - Improve issue labeling with component dropdown and more provider keywords - PR #17957
• PR Template Cleanup - Remove redundant fields from PR template - PR #17956
• Dependencies - Bump altcha-lib from 1.3.0 to 1.4.1 - PR #18017

New Contributors​

• @dongbin-lunark made their first contribution in PR #17757
• @qdrddr made their first contribution in PR #18004
• @donicrosby made their first contribution in PR #17962
• @NicolaivdSmagt made their first contribution in PR #17992
• @Reapor-Yurnero made their first contribution in PR #18085
• @jk-f5 made their first contribution in PR #18086
• @castrapel made their first contribution in PR #18077
• @dtikhonov made their first contribution in PR #17484
• @opleonnn made their first contribution in PR #18175
• @eurogig made their first contribution in PR #18084

Full Changelog​

View complete changelog on GitHub

Deploy this version​

docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
docker.litellm.ai/berriai/litellm:v1.80.10.rc.1

pip install litellm==1.80.10

Key Highlights​

• Agent (A2A) Gateway with Cost Tracking - Track agent costs per query, per token pricing, and view agent usage in the dashboard
• 2 New Agent Providers - LangGraph Agents and Azure AI Foundry Agents for agentic workflows
• New Provider: SAP Gen AI Hub - Full support for SAP Generative AI Hub with chat completions
• New Bedrock Writer Models - Add Palmyra-X4 and Palmyra-X5 models on Bedrock
• OpenAI GPT-5.2 Models - Full support for GPT-5.2, GPT-5.2-pro, and Azure GPT-5.2 models with reasoning support
• 227 New Fireworks AI Models - Comprehensive model coverage for Fireworks AI platform
• MCP Support on /chat/completions - Use MCP servers directly via chat completions endpoint
• Performance Improvements - Reduced memory leaks by 50%

Agent Gateway - 4 New Agent Providers​

This release adds support for agents from the following providers:

• LangGraph Agents - Deploy and manage LangGraph-based agents
• Azure AI Foundry Agents - Enterprise agent deployments on Azure
• Bedrock AgentCore - AWS Bedrock agent integration
• A2A Agents - Agent-to-Agent protocol support

AI Gateway admins can now add agents from any of these providers, and developers can invoke them through a unified interface using the A2A protocol.

For all agent requests running through the AI Gateway, LiteLLM automatically tracks request/response logs, cost, and token usage.

Agent (A2A) Usage UI​

Users can now filter usage statistics by agents, providing the same granular filtering capabilities available for teams, organizations, and customers.

Details:

• Filter usage analytics, spend logs, and activity metrics by agent ID
• View breakdowns on a per-agent basis
• Consistent filtering experience across all usage and analytics views

New Providers and Endpoints​

New Providers (5 new providers)​

New LLM API Endpoints (4 new endpoints)​

New Models / Updated Models​

New Model Support (270+ new models)​

Features​

• OpenAI
  • Add support for OpenAI GPT-5.2 models with reasoning_effort='xhigh' - PR #17836, PR #17875
  • Include 'user' param for responses API models - PR #17648
  • Use optimized async http client for text completions - PR #17831
• Azure
  • Add Azure GPT-5.2 models support - PR #17866
• Azure AI
  • Fix Azure AI Anthropic api-key header and passthrough cost calculation - PR #17656
  • Remove unsupported params from Azure AI Anthropic requests - PR #17822
• Anthropic
  • Prevent duplicate tool_result blocks with same tool - PR #17632
  • Handle partial JSON chunks in streaming responses - PR #17493
  • Preserve server_tool_use and web_search_tool_result in multi-turn conversations - PR #17746
  • Capture web_search_tool_result in streaming for multi-turn conversations - PR #17798
  • Add retrieve batches and retrieve file content support - PR #17700
• Bedrock
  • Add new Bedrock OSS models to model list - PR #17638
  • Add Bedrock Writer models (Palmyra-X4, Palmyra-X5) - PR #17685
  • Add EU Claude Opus 4.5 model - PR #17897
  • Add serviceTier support for Converse API - PR #17810
  • Fix header forwarding with custom API for Bedrock embeddings - PR #17872
• Gemini
  • Add support for computer use for Gemini - PR #17756
  • Handle context window errors - PR #17751
  • Add speechConfig to GenerationConfig for Gemini TTS - PR #17851
• Vertex AI
  • Add DeepSeek-V3.2 model support - PR #17770
  • Preserve systemInstructions for generate content request - PR #17803
• Mistral
  • Add Codestral 2508, Devstral 2512 models - PR #17801
• Cerebras
  • Add zai-glm-4.6 model support - PR #17683
  • Fix context window errors not recognized - PR #17587
• DeepSeek
  • Add native support for thinking and reasoning_effort params - PR #17712
• NVIDIA NIM Rerank
  • Add llama-3.2-nv-rerankqa-1b-v2 rerank model - PR #17670
• Fireworks AI
  • Add 227 new Fireworks AI models - PR #17692
• Dashscope
  • Fix default base_url error - PR #17584

Bug Fixes​

• Anthropic
  • Fix missing content in Anthropic to OpenAI conversion - PR #17693
  • Avoid error when we have just the tool_calls in input - PR #17753
• Azure
  • Fix error about encoding video id for Azure - PR #17708
• Azure AI
  • Fix LLM provider for azure_ai in model map - PR #17805
• Watsonx
  • Fix Watsonx Audio Transcription to only send supported params to API - PR #17840
• Router
  • Handle tools=None in completion requests - PR #17684
  • Add minimum request threshold for error rate cooldown - PR #17464

LLM API Endpoints​

Features​

• Responses API
  • Add usage details in responses usage object - PR #17641
  • Fix error for response API polling - PR #17654
  • Fix streaming tool_calls being dropped when text + tool_calls - PR #17652
  • Transform image content in tool results for Responses API - PR #17799
  • Fix responses api not applying tpm rate limits on api keys - PR #17707
• Containers API
  • Allow using LIST, Create Containers using custom-llm-provider - PR #17740
  • Add new container API file management + UI Interface - PR #17745
• Rerank API
  • Add support for forwarding client headers in /rerank endpoint - PR #17873
• Files API
  • Add support for expires_after param in Files endpoint - PR #17860
• Video API
  • Use litellm params for all videos APIs - PR #17732
  • Respect videos content db creds - PR #17771
• Embeddings API
  • Fix handling token array input decoding for embeddings - PR #17468
• Chat Completions API
  • Add v0 target storage support - store files in Azure AI storage and use with chat completions API - PR #17758
• generateContent API
  • Support model names with slashes on Gemini generateContent endpoints - PR #17743
• General
  • Use audio content for caching - PR #17651
  • Return 403 exception when calling GET responses API - PR #17629
  • Add nested field removal support to additional_drop_params - PR #17711
  • Async post_call_streaming_iterator_hook now properly iterates async generators - PR #17626

Bugs​

• General
  • Fix handle string content in is_cached_message - PR #17853

Management Endpoints / UI​

Features​

• UI Settings
  • Add Get and Update Backend Routes for UI Settings - PR #17689
  • UI Settings page implementation - PR #17697
  • Ensure Model Page honors UI Settings - PR #17804
  • Add All Proxy Models to Default User Settings - PR #17902
• Agent & Usage UI
  • Daily Agent Usage Backend - PR #17781
  • Agent Usage UI - PR #17797
  • Add agent cost tracking on UI - PR #17899
  • New Badge for Agent Usage - PR #17883
  • Usage Entity labels for filtering - PR #17896
  • Agent Usage Page minor fixes - PR #17901
  • Usage Page View Select component - PR #17854
  • Usage Page Components refactor - PR #17848
• Logs & Spend
  • Enhanced spend analytics in logs view - PR #17623
  • Add user info delete modal for user management - PR #17625
  • Show request and response details in logs view - PR #17928
• Virtual Keys
  • Fix x-litellm-key-spend header update - PR #17864
• Models & Endpoints
  • Model Hub Useful Links Rearrange - PR #17859
  • Create Team Model Dropdown honors Organization's Models - PR #17834
• SSO & Auth
  • Allow upserting user role when SSO provider role changes - PR #17754
  • Allow fetching role from generic SSO provider (Keycloak) - PR #17787
  • JWT Auth - allow selecting team_id from request header - PR #17884
  • Remove SSO Config Values from Config Table on SSO Update - PR #17668
• Teams
  • Attach team to org table - PR #17832
  • Expose the team alias when authenticating - PR #17725
• MCP Server Management
  • Add extra_headers and allowed_tools to UpdateMCPServerRequest - PR #17940
• Notifications
  • Show progress and pause on hover for Notifications - PR #17942
• General
  • Allow Root Path to Redirect when Docs not on Root Path - PR #16843
  • Show UI version number on top left near logo - PR #17891
  • Re-organize left navigation with correct categories and agents on root - PR #17890
  • UI Playground - allow custom model names in model selector dropdown - PR #17892

Bugs​

• UI Fixes
  • Fix links + old login page deprecation message - PR #17624
  • Filtering for Chat UI Endpoint Selector - PR #17567
  • Race Condition Handling in SCIM v2 - PR #17513
  • Make /litellm_model_cost_map public - PR #16795
  • Custom Callback on UI - PR #17522
  • Add User Writable Directory to Non Root Docker for Logo - PR #17180
  • Swap URL Input and Display Name inputs - PR #17682
  • Change deprecation banner to only show on /sso/key/generate - PR #17681
  • Change credential encryption to only affect db credentials - PR #17741
• Auth & Routes
  • Return 403 instead of 503 for unauthorized routes - PR #17723
  • AI Gateway Auth - allow using wildcard patterns for public routes - PR #17686

AI Integrations​

New Integrations (4 new integrations)​

Logging​

• Langfuse
  • Propagate Langfuse trace_id - PR #17669
  • Prefer standard trace id for Langfuse logging - PR #17791
  • Move query params to create_pass_through_route call in Langfuse passthrough - PR #17660
  • Add support for custom masking function - PR #17826
• Prometheus
  • Add 'exception_status' to prometheus logger - PR #17847
• OpenTelemetry
  • Add latency metrics (TTFT, TPOT, Total Generation Time) to OTEL payload - PR #17888
• General
  • Add polling via cache feature for async logging - PR #16862

Guardrails​

• HiddenLayer
  • Add HiddenLayer Guardrail Hooks - PR #17728
• Pillar Security
  • Add opt-in evidence results for Pillar Security guardrail during monitoring - PR #17812
• PANW Prisma AIRS
  • Add configurable fail-open, timeout, and app_user tracking - PR #17785
• Presidio
  • Add support for configurable confidence score thresholds and scope in Presidio PII masking - PR #17817
• LiteLLM Content Filter
  • Mask all regex pattern matches, not just first - PR #17727
• Regex Guardrails
  • Add enhanced regex pattern matching for guardrails - PR #17915
• Gray Swan Guardrail
  • Add passthrough mode for model response - PR #17102

Prompt Management​

• General
  • New API for integrating prompt management providers - PR #17829

Spend Tracking, Budgets and Rate Limiting​

• Service Tier Pricing - Extract service_tier from response/usage for OpenAI flex pricing - PR #17748
• Agent Cost Tracking - Track agent_id in SpendLogs - PR #17795
• Tag Activity - Deduplicate /tag/daily/activity metadata - PR #16764
• Rate Limiting - Dynamic Rate Limiter - allow specifying ttl for in memory cache - PR #17679

MCP Gateway​

• Chat Completions Integration - Add support for using MCPs on /chat/completions - PR #17747
• UI Session Permissions - Fix UI session MCP permissions across real teams - PR #17620
• OAuth Callback - Fix MCP OAuth callback routing and URL handling - PR #17789
• Tool Name Prefix - Fix MCP tool name prefix - PR #17908

Agent Gateway (A2A)​

• Cost Per Query - Add cost per query for agent invocations - PR #17774
• Token Counting - Add token counting non streaming + streaming - PR #17779
• Cost Per Token - Add cost per token pricing for A2A - PR #17780
• LangGraph Provider - Add LangGraph provider for Agent Gateway - PR #17783
• Bedrock & LangGraph Agents - Allow using Bedrock AgentCore, LangGraph agents with A2A Gateway - PR #17786
• Agent Management - Allow adding LangGraph, Bedrock Agent Core agents - PR #17802
• Azure Foundry Agents - Add Azure AI Foundry Agents support - PR #17845
• Azure Foundry UI - Allow adding Azure Foundry Agents on UI - PR #17909
• Azure Foundry Fixes - Ensure Azure Foundry agents work correctly - PR #17943

Performance / Loadbalancing / Reliability improvements​

• Memory Leak Fix - Cut memory leak in half - PR #17784
• Spend Logs Memory - Reduce memory accumulation of spend_logs - PR #17742
• Router Optimization - Replace time.perf_counter() with time.time() - PR #17881
• Filter Internal Params - Filter internal params in fallback code - PR #17941
• Gunicorn Suggestion - Suggest Gunicorn instead of uvicorn when using max_requests_before_restart - PR #17788
• Pydantic Warnings - Mitigate PydanticDeprecatedSince20 warnings - PR #17657
• Python 3.14 Support - Add Python 3.14 support via grpcio version constraints - PR #17666
• OpenAI Package - Bump openai package to 2.9.0 - PR #17818

Documentation Updates​

• Contributing - Update clone instructions to recommend forking first - PR #17637
• Getting Started - Improve Getting Started page and SDK documentation structure - PR #17614
• JSON Mode - Make it clearer how to get Pydantic model output - PR #17671
• drop_params - Update litellm docs for drop_params - PR #17658
• Environment Variables - Document missing environment variables and fix incorrect types - PR #17649
• SumoLogic - Add SumoLogic integration documentation - PR #17647
• SAP Gen AI - Add SAP Gen AI provider documentation - PR #17667
• Authentication - Add Note for Authentication - PR #17733
• Known Issues - Adding known issues to 1.80.5-stable docs - PR #17738
• Supported Endpoints - Fix Supported Endpoints page - PR #17710
• Token Count - Document token count endpoint - PR #17772
• Overview - Made litellm proxy and SDK difference cleaner in overview with a table - PR #17790
• Containers API - Add docs for containers files API + code interpreter on LiteLLM - PR #17749
• Target Storage - Add documentation for target storage - PR #17882
• Agent Usage - Agent Usage documentation - PR #17931, PR #17932, PR #17934
• Cursor Integration - Cursor Integration documentation - PR #17855, PR #17939
• A2A Cost Tracking - A2A cost tracking docs - PR #17913
• Azure Search - Update azure search docs - PR #17726
• Milvus Client - Fix milvus client docs - PR #17736
• Streaming Logging - Remove streaming logging doc - PR #17739
• Integration Docs - Update integration docs location - PR #17644
• Links - Updated docs links for mistral and anthropic - PR #17852
• Community - Add community doc link - PR #17734
• Pricing - Update pricing for global.anthropic.claude-haiku-4-5-20251001-v1:0 - PR #17703
• gpt-image-1-mini - Correct model type for gpt-image-1-mini - PR #17635

Infrastructure / Deployment​

• Docker - Use python instead of wget for healthcheck in docker-compose.yml - PR #17646
• Helm Chart - Add extraResources support for Helm chart deployments - PR #17627
• Helm Versioning - Add semver prerelease suffix to helm chart versions - PR #17678
• Database Schema - Add storage_backend and storage_url columns to schema.prisma for target storage feature - PR #17936

New Contributors​

• @xianzongxie-stripe made their first contribution in PR #16862
• @krisxia0506 made their first contribution in PR #17637
• @chetanchoudhary-sumo made their first contribution in PR #17630
• @kevinmarx made their first contribution in PR #17632
• @expruc made their first contribution in PR #17627
• @rcII made their first contribution in PR #17626
• @tamirkiviti13 made their first contribution in PR #16591
• @Eric84626 made their first contribution in PR #17629
• @vasilisazayka made their first contribution in PR #16053
• @juliettech13 made their first contribution in PR #17663
• @jason-nance made their first contribution in PR #17660
• @yisding made their first contribution in PR #17671
• @emilsvennesson made their first contribution in PR #17656
• @kumekay made their first contribution in PR #17646
• @chenzhaofei01 made their first contribution in PR #17584
• @shivamrawat1 made their first contribution in PR #17733
• @ephrimstanley made their first contribution in PR #17723
• @hwittenborn made their first contribution in PR #17743
• @peterkc made their first contribution in PR #17727
• @saisurya237 made their first contribution in PR #17725
• @Ashton-Sidhu made their first contribution in PR #17728
• @CyrusTC made their first contribution in PR #17810
• @jichmi made their first contribution in PR #17703
• @ryan-crabbe made their first contribution in PR #17852
• @nlineback made their first contribution in PR #17851
• @butnarurazvan made their first contribution in PR #17468
• @yoshi-p27 made their first contribution in PR #17915

Full Changelog​

View complete changelog on GitHub

Deploy this version​

docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
docker.litellm.ai/berriai/litellm:v1.80.8-stable

pip install litellm==1.80.8

Key Highlights​

• Agent Gateway (A2A) - Invoke agents through the AI Gateway with request/response logging and access controls
• Guardrails API v2 - Generic Guardrail API with streaming support, structured messages, and tool call checks
• Customer (End User) Usage UI - Track and visualize end-user spend directly in the dashboard
• vLLM Batch + Files API - Support for batch and files API with vLLM deployments
• Dynamic Rate Limiting on Teams - Enable dynamic rate limits and priority reservation on team-level
• Google Cloud Chirp3 HD - New text-to-speech provider with Chirp3 HD voices

Agent Gateway (A2A)​

This release introduces A2A Agent Gateway for LiteLLM, allowing you to invoke and manage A2A agents with the same controls you have for LLM APIs.

As a LiteLLM Gateway Admin, you can now do the following:

• Request/Response Logging - Every agent invocation is logged to the Logs page with full request and response tracking.
• Access Control - Control which Team/Key can access which agents.

As a developer, you can continue using the A2A SDK, all you need to do is point you A2AClient to the LiteLLM proxy URL and your API key.

Works with the A2A SDK:

from a2a.client import A2AClient

client = A2AClient(
    base_url="http://localhost:4000",  # Your LiteLLM proxy
    api_key="sk-1234"                   # LiteLLM API key
)

response = client.send_message(
    agent_id="my-agent",
    message="What's the status of my order?"
)

Get started with Agent Gateway here: Agent Gateway Documentation

Customer (End User) Usage UI​

Users can now filter usage statistics by customers, providing the same granular filtering capabilities available for teams and organizations.

Details:

• Filter usage analytics, spend logs, and activity metrics by customer ID
• View customer-level breakdowns alongside existing team and user-level filters
• Consistent filtering experience across all usage and analytics views

New Providers and Endpoints​

New Providers (5 new providers)​

New LLM API Endpoints (2 new endpoints)​

New Models / Updated Models​

New Model Support (33 new models)​

Features​

• OpenAI

  • Add gpt-5.1-codex-max model pricing and configuration - PR #17541
  • Add xhigh reasoning effort for gpt-5.1-codex-max - PR #17585
  • Add clear error message for empty LLM endpoint responses - PR #17445

• Azure OpenAI

  • Allow reasoning_effort='none' for Azure gpt-5.1 models - PR #17311

• Anthropic

  • Add claude-opus-4-5 alias to pricing data - PR #17313
  • Parse <budget:thinking> blocks for opus 4.5 - PR #17534
  • Update new Anthropic features as reviewed - PR #17142
  • Skip empty text blocks in Anthropic system messages - PR #17442

• Bedrock

  • Add Nova embedding support - PR #17253
  • Add support for Bedrock Qwen 2 imported model - PR #17461
  • Bedrock OpenAI model support - PR #17368
  • Add support for file content download for Bedrock batches - PR #17470
  • Make streaming chunk size configurable in Bedrock API - PR #17357
  • Add experimental latest-user filtering for Bedrock - PR #17282
  • Handle Cohere v4 embed response dictionary format - PR #17220
  • Remove not compatible beta header from Bedrock - PR #17301
  • Add model price and details for Global Opus 4.5 Bedrock endpoint - PR #17380

• Gemini (Google AI Studio + Vertex AI)

  • Add better handling in image generation for Gemini models - PR #17292
  • Fix reasoning_content showing duplicate content in streaming responses - PR #17266
  • Handle partial JSON chunks after first valid chunk - PR #17496
  • Fix Gemini 3 last chunk thinking block - PR #17403
  • Fix Gemini image_tokens treated as text tokens in cost calculation - PR #17554
  • Make sure that media resolution is only for Gemini 3 model - PR #17137

• Vertex AI

  • Add Google Cloud Chirp3 HD support on /speech - PR #17391
  • Add BGE Embeddings support - PR #17362
  • Handle global location for Vertex AI image generation endpoint - PR #17255
  • Add Google Private API Endpoint to Vertex AI fields - PR #17382

• Z.AI (Zhipu AI)

  • Add Z.AI as built-in provider - PR #17307

• GitHub Copilot

  • Add Embedding API support - PR #17278
  • Preserve encrypted_content in reasoning items for multi-turn conversations - PR #17130

• Databricks

  • Update Databricks model pricing and add new models - PR #17277

• OVHcloud

  • Add support of audio transcription for OVHcloud - PR #17305

• Mistral

  • Add Mistral Large 3 model support - PR #17547

• Moonshot

  • Fix missing Moonshot turbo models and fix incorrect pricing - PR #17432

• Together AI

  • Add context window exception mapping for Together AI - PR #17284

• WatsonX

  • Allow passing zen_api_key dynamically - PR #16655
  • Fix Watsonx Audio Transcription API - PR #17326
  • Fix audio transcriptions, don't force content type in request headers - PR #17546

• Fireworks AI

  • Add new model fireworks_ai/kimi-k2-instruct-0905 - PR #17328
  • Add fireworks/deepseek-v3p2 - PR #17395

• DeepSeek

  • Support Deepseek 3.2 with Reasoning - PR #17384

• Nova Lite 2

  • Add Nova Lite 2 reasoning support with reasoningConfig - PR #17371

• Ollama

  • Fix auth not working with ollama.com - PR #17191

• Groq

  • Fix supports_response_schema before using json_tool_call workaround - PR #17438

• vLLM

  • Fix empty response + vLLM streaming - PR #17516

• Azure AI

  • Migrate Anthropic provider to Azure AI - PR #17202
  • Fix GA path for Azure OpenAI realtime models - PR #17260

• Bedrock TwelveLabs

  • Add support for TwelveLabs Pegasus video understanding - PR #17193

Bug Fixes​

• Bedrock

  • Fix extra_headers in messages API bedrock invoke - PR #17271
  • Fix Bedrock models in model map - PR #17419
  • Make Bedrock converse messages respect modify_params as expected - PR #17427
  • Fix Anthropic beta headers for Bedrock imported Qwen models - PR #17467
  • Preserve usage from JSON response for OpenAI provider in Bedrock - PR #17589

• SambaNova

  • Fix acompletion throws error with SambaNova models - PR #17217

• General

  • Fix AttributeError when metadata is null in request body - PR #17306
  • Fix 500 error for malformed request - PR #17291
  • Respect custom LLM provider in header - PR #17290
  • Replace deprecated .dict() with .model_dump() in streaming_handler - PR #17359

LLM API Endpoints​

Features​

• Responses API

  • Add cost tracking for responses API - PR #17258
  • Map output_tokens_details of responses API to completion_tokens_details - PR #17458
  • Add image generation support for Responses API - PR #16586

• Batch API

  • Add vLLM batch+files API support - PR #15823
  • Fix optional parameter default value - PR #17434
  • Add status parameter as optional for FileObject - PR #17431

• Video Generation API

  • Add passthrough cost tracking for Veo - PR #17296

• OCR API

  • Add missing OCR and aOCR to CallTypes enum - PR #17435

• General

  • Support routing to only websearch supported deployments - PR #17500

Bugs​

• General
  • Fix streaming error validation - PR #17242
  • Add length validation for empty tool_calls in delta - PR #17523

Management Endpoints / UI​

Features​

• New Login Page

  • New Login Page UI - PR #17443
  • Refactor /login route - PR #17379
  • Add auto_redirect_to_sso to UI Config - PR #17399
  • Add Auto Redirect to SSO to New Login Page - PR #17451

• Customer (End User) Usage

  • Customer (end user) Usage feature - PR #17498
  • Customer Usage UI - PR #17506
  • Add Info Banner for Customer Usage - PR #17598

• Virtual Keys

  • Standardize API Key vs Virtual Key in UI - PR #17325
  • Add User Alias Column to Internal User Table - PR #17321
  • Delete Credential Enhancements - PR #17317

• Models + Endpoints

  • Show all credential values on Edit Credential Modal - PR #17397
  • Change Edit Team Models Shown to Match Create Team - PR #17394
  • Support Images in Compare UI - PR #17562

• Callbacks

  • Show all callbacks on UI - PR #16335
  • Credentials to use React Query - PR #17465

• Management Routes

  • Allow admin viewer to access global tag usage - PR #17501
  • Allow wildcard routes for nonproxy admin (SCIM) - PR #17178
  • Return 404 when a user is not found on /user/info - PR #16850

• OCI Configuration

  • Enable Oracle Cloud Infrastructure configuration via UI - PR #17159

Bugs​

• UI Fixes

  • Fix Request and Response Panel JSONViewer - PR #17233
  • Adding Button Loading States to Edit Settings - PR #17236
  • Fix Various Text, button state, and test changes - PR #17237
  • Fix Fallbacks Immediately Deleting before API resolves - PR #17238
  • Remove Feature Flags - PR #17240
  • Fix metadata tags and model name display in UI for Azure passthrough - PR #17258
  • Change labeling around Vertex Fields - PR #17383
  • Remove second scrollbar when sidebar is expanded + tooltip z index - PR #17436
  • Fix Select in Edit Membership Modal - PR #17524
  • Change useAuthorized Hook to redirect to new Login Page - PR #17553

• SSO

  • Fix the generic SSO provider - PR #17227
  • Clear SSO integration for all users - PR #17287
  • Fix SSO users not added to Entra synced team - PR #17331

• Auth / JWT

  • JWT Auth - Allow using regular OIDC flow with user info endpoints - PR #17324
  • Fix litellm user auth not passing issue - PR #17342
  • Add other routes in JWT auth - PR #17345
  • Fix new org team validate against org - PR #17333
  • Fix litellm_enterprise ensure imported routes exist - PR #17337
  • Use organization.members instead of deprecated organization field - PR #17557

• Organizations/Teams

  • Fix organization max budget not enforced - PR #17334
  • Fix budget update to allow null max_budget - PR #17545

AI Integrations (2 new integrations)​

Logging (1 new integration)​

New Integration​

• Weave
  • Basic Weave OTEL integration - PR #17439

Improvements & Fixes​

• DataDog

  • Fix Datadog callback regression when ddtrace is installed - PR #17393

• Arize Phoenix

  • Fix clean arize-phoenix traces - PR #16611

• MLflow

  • Fix MLflow streaming spans for Anthropic passthrough - PR #17288

• Langfuse

  • Fix Langfuse logger test mock setup - PR #17591

• General

  • Improve PII anonymization handling in logging callbacks - PR #17207

Guardrails (1 new integration)​

New Integration​

• Generic Guardrail API
  • Generic Guardrail API - allows guardrail providers to add INSTANT support for LiteLLM w/out PR to repo - PR #17175
  • Guardrails API V2 - user api key metadata, session id, specify input type (request/response), image support - PR #17338
  • Guardrails API - add streaming support - PR #17400
  • Guardrails API - support tool call checks on OpenAI /chat/completions, OpenAI /responses, Anthropic /v1/messages - PR #17459
  • Guardrails API - new structured_messages param - PR #17518
  • Correctly map a v1/messages call to the anthropic unified guardrail - PR #17424
  • Support during_call event type for unified guardrails - PR #17514

Improvements & Fixes​

• Noma Guardrail

  • Refactor Noma guardrail to use shared Responses transformation and include system instructions - PR #17315

• Presidio

  • Handle empty content and error dict responses in guardrails - PR #17489
  • Fix Presidio guardrail test TypeError and license base64 decoding error - PR #17538

• Tool Permissions

  • Add regex-based tool_name/tool_type matching for tool-permission - PR #17164
  • Add images for tool permission guardrail documentation - PR #17322

• AIM Guardrails

  • Fix AIM guardrail tests - PR #17499

• Bedrock Guardrails

  • Fix Bedrock Guardrail indent and import - PR #17378

• General Guardrails

  • Mask all matching keywords in content filter - PR #17521
  • Ensure guardrail metadata is preserved in request_data - PR #17593
  • Fix apply_guardrail method and improve test isolation - PR #17555

Secret Managers​

• CyberArk

  • Allow setting SSL verify to false - PR #17433

• General

  • Make email and secret manager operations independent in key management hooks - PR #17551

Spend Tracking, Budgets and Rate Limiting​

• Rate Limiting

  • Parallel Request Limiter with /messages - PR #17426
  • Allow using dynamic rate limit/priority reservation on teams - PR #17061
  • Dynamic Rate Limiter - Fix token count increases/decreases by 1 instead of actual count + Redis TTL - PR #17558

• Spend Logs

  • Deprecate spend/logs & add spend/logs/v2 - PR #17167
  • Optimize SpendLogs queries to use timestamp filtering for index usage - PR #17504

• Enforce User Param

  • Enforce support of enforce_user_param to OpenAI post endpoints - PR #17407

MCP Gateway​

• MCP Configuration

  • Remove URL format validation for MCP server endpoints - PR #17270
  • Add stack trace to MCP error message - PR #17269

• MCP Tool Results

  • Preserve tool metadata in CallToolResult - PR #17561

Agent Gateway (A2A)​

• Agent Invocation

  • Allow invoking agents through AI Gateway - PR #17440
  • Allow tracking request/response in "Logs" Page - PR #17449

• Agent Access Control

  • Enforce Allowed agents by key, team + add agent access groups on backend - PR #17502

• Agent Gateway UI

  • Allow testing agents on UI - PR #17455
  • Set allowed agents by key, team - PR #17511

Performance / Loadbalancing / Reliability improvements​

• Audio/Speech Performance

  • Fix /audio/speech performance by using shared_sessions - PR #16739

• Memory Optimization

  • Prevent memory leak in aiohttp connection pooling - PR #17388
  • Lazy-load utils to reduce memory + import time - PR #17171

• Database

  • Update default database connection number - PR #17353
  • Update default proxy_batch_write_at number - PR #17355
  • Add background health checks to db - PR #17528

• Proxy Caching

  • Fix proxy caching between requests in aiohttp transport - PR #17122

• Session Management

  • Fix session consistency, move Lasso API version away from source code - PR #17316
  • Conditionally pass enable_cleanup_closed to aiohttp TCPConnector - PR #17367

• Vector Store

  • Fix vector store configuration synchronization failure - PR #17525

Documentation Updates​

• Provider Documentation

  • Add Azure AI Foundry documentation for Claude models - PR #17104
  • Document responses and embedding API for GitHub Copilot - PR #17456
  • Add gpt-5.1-codex-max to OpenAI provider documentation - PR #17602
  • Update Instructions For Phoenix Integration - PR #17373

• Guides

  • Add guide on how to debug gateway error vs provider error - PR #17387
  • Agent Gateway documentation - PR #17454
  • A2A Permission management documentation - PR #17515
  • Update docs to link agent hub - PR #17462

• Projects

  • Add Google ADK and Harbor to projects - PR #17352
  • Add Microsoft Agent Lightning to projects - PR #17422

• Cleanup

  • Cleanup: Remove orphan docs pages and Docusaurus template files - PR #17356
  • Remove source .env from docs - PR #17466

Infrastructure / CI/CD​

• Helm Chart

  • Add ingress-only labels - PR #17348

• Docker

  • Add retry logic to apk package installation in Dockerfile.non_root - PR #17596
  • Chainguard fixes - PR #17406

• OpenAPI Schema

  • Refactor add_schema_to_components to move definitions to components/schemas - PR #17389

• Security

  • Fix security vulnerability: update mdast-util-to-hast to 13.2.1 - PR #17601
  • Bump jws from 3.2.2 to 3.2.3 - PR #17494

New Contributors​

• @weichiet made their first contribution in PR #17242
• @AndyForest made their first contribution in PR #17220
• @omkar806 made their first contribution in PR #17217
• @v0rtex20k made their first contribution in PR #17178
• @hxomer made their first contribution in PR #17207
• @orgersh92 made their first contribution in PR #17316
• @dannykopping made their first contribution in PR #17313
• @rioiart made their first contribution in PR #17333
• @codgician made their first contribution in PR #17278
• @epistoteles made their first contribution in PR #17277
• @kothamah made their first contribution in PR #17368
• @flozonn made their first contribution in PR #17371
• @richardmcsong made their first contribution in PR #17389
• @matt-greathouse made their first contribution in PR #17384
• @mossbanay made their first contribution in PR #17380
• @mhielpos-asapp made their first contribution in PR #17376
• @Joilence made their first contribution in PR #17367
• @deepaktammali made their first contribution in PR #17357
• @axiomofjoy made their first contribution in PR #16611
• @DevajMody made their first contribution in PR #17445
• @andrewtruong made their first contribution in PR #17439
• @AnasAbdelR made their first contribution in PR #17490
• @dominicfeliton made their first contribution in PR #17516
• @kristianmitk made their first contribution in PR #17504
• @rgshr made their first contribution in PR #17130
• @dominicfallows made their first contribution in PR #17489
• @irfansofyana made their first contribution in PR #17467
• @GusBricker made their first contribution in PR #17191
• @OlivverX made their first contribution in PR #17255
• @withsmilo made their first contribution in PR #17585

Full Changelog​

View complete changelog on GitHub

Deploy this version​

docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
docker.litellm.ai/berriai/litellm:v1.80.5-stable

pip install litellm==1.80.5

Key Highlights​

• Gemini 3 - Day-0 support for Gemini 3 models with thought signatures
• Prompt Management - Full prompt versioning support with UI for editing, testing, and version history
• MCP Hub - Publish and discover MCP servers within your organization
• Model Compare UI - Side-by-side model comparison interface for testing
• Batch API Spend Tracking - Granular spend tracking with custom metadata for batch and file creation requests
• AWS IAM Secret Manager - IAM role authentication support for AWS Secret Manager
• Logging Callback Controls - Admin-level controls to prevent callers from disabling logging callbacks in compliance environments
• Proxy CLI JWT Authentication - Enable developers to authenticate to LiteLLM AI Gateway using the Proxy CLI
• Batch API Routing - Route batch operations to different provider accounts using model-specific credentials from your config.yaml

Prompt Management​

This release introduces LiteLLM Prompt Studio - a comprehensive prompt management solution built directly into the LiteLLM UI. Create, test, and version your prompts without leaving your browser.

You can now do the following on LiteLLM Prompt Studio:

• Create & Test Prompts: Build prompts with developer messages (system instructions) and test them in real-time with an interactive chat interface
• Dynamic Variables: Use {{variable_name}} syntax to create reusable prompt templates with automatic variable detection
• Version Control: Automatic versioning for every prompt update with complete version history tracking and rollback capabilities
• Prompt Studio: Edit prompts in a dedicated studio environment with live testing and preview

API Integration:

Use your prompts in any application with simple API calls:

response = client.chat.completions.create(
    model="gpt-4",
    extra_body={
        "prompt_id": "your-prompt-id",
        "prompt_version": 2,  # Optional: specify version
        "prompt_variables": {"name": "value"}  # Optional: pass variables
    }
)

Get started here: LiteLLM Prompt Management Documentation

Performance – /realtime 182× Lower p99 Latency​

This update reduces /realtime latency by removing redundant encodings on the hot path, reusing shared SSL contexts, and caching formatting strings that were being regenerated twice per request despite rarely changing.

Results​

Test Setup​

Model Compare UI​

New interactive playground UI enables side-by-side comparison of multiple LLM models, making it easy to evaluate and compare model responses.

Features:

• Compare responses from multiple models in real-time
• Side-by-side view with synchronized scrolling
• Support for all LiteLLM-supported models
• Cost tracking per model
• Response time comparison
• Pre-configured prompts for quick and easy testing

Details:

• Parameterization: Configure API keys, endpoints, models, and model parameters, as well as interaction types (chat completions, embeddings, etc.)

• Model Comparison: Compare up to 3 different models simultaneously with side-by-side response views

• Comparison Metrics: View detailed comparison information including:

  • Time To First Token
  • Input / Output / Reasoning Tokens
  • Total Latency
  • Cost (if enabled in config)

• Safety Filters: Configure and test guardrails (safety filters) directly in the playground interface

Get Started with Model Compare

New Providers and Endpoints​

New Providers​

New Models / Updated Models​

New Model Support​

Features​

• Gemini (Google AI Studio + Vertex AI)

  • Add Day 0 gemini-3-pro-preview support - PR #16719
  • Add support for Gemini 3 Pro Image model - PR #16938
  • Add reasoning_content to streaming responses with tools enabled - PR #16854
  • Add includeThoughts=True for Gemini 3 reasoning_effort - PR #16838
  • Support thought signatures for Gemini 3 in responses API - PR #16872
  • Correct wrong system message handling for gemma - PR #16767
  • Gemini 3 Pro Image: capture image_tokens and support cost_per_output_image - PR #16912
  • Fix missing costs for gemini-2.5-flash-image - PR #16882
  • Gemini 3 thought signatures in tool call id - PR #16895

• Azure

  • Add azure gpt-5.1 models - PR #16817
  • Add Azure models 2025 11 to cost maps - PR #16762
  • Update Azure Pricing - PR #16371
  • Add SSML Support for Azure Text-to-Speech (AVA) - PR #16747

• OpenAI

  • Support GPT-5.1 reasoning.effort='none' in proxy - PR #16745
  • Add gpt-5.1-codex and gpt-5.1-codex-mini models to documentation - PR #16735
  • Inherit BaseVideoConfig to enable async content response for OpenAI video - PR #16708

• Anthropic

  • Add support for strict parameter in Anthropic tool schemas - PR #16725
  • Add image as url support to anthropic - PR #16868
  • Add thought signature support to v1/messages api - PR #16812
  • Anthropic - support Structured Outputs output_format for Claude 4.5 sonnet and Opus 4.1 - PR #16949

• Bedrock

  • Haiku 4.5 correct Bedrock configs - PR #16732
  • Ensure consistent chunk IDs in Bedrock streaming responses - PR #16596
  • Add Claude 4.5 to US Gov Cloud - PR #16957
  • Fix images being dropped from tool results for bedrock - PR #16492

• Vertex AI

  • Add Vertex AI Image Edit Support - PR #16828
  • Update veo 3 pricing and add prod models - PR #16781
  • Fix Video download for veo3 - PR #16875

• Snowflake

  • Snowflake provider support: added embeddings, PAT, account_id - PR #15727

• OCI

  • Add oci_endpoint_id Parameter for OCI Dedicated Endpoints - PR #16723

• XAI

  • Add support for Grok 4.1 Fast models - PR #16936

• Together AI

  • Add GLM 4.6 from together.ai - PR #16942

• Cerebras

  • Fix Cerebras GPT-OSS-120B model name - PR #16939

Bug Fixes​

• OpenAI

  • Fix for 16863 - openai conversion from responses to completions - PR #16864
  • Revert "Make all gpt-5 and reasoning models to responses by default" - PR #16849

• General

  • Get custom_llm_provider from query param - PR #16731
  • Fix optional param mapping - PR #16852
  • Add None check for litellm_params - PR #16754

LLM API Endpoints​

Features​

• Responses API

  • Add Responses API support for gpt-5.1-codex model - PR #16845
  • Add managed files support for responses API - PR #16733
  • Add extra_body support for response supported api params from chat completion - PR #16765

• Batch API

  • Support /delete for files + support /cancel for batches - PR #16387
  • Add config based routing support for batches and files - PR #16872
  • Populate spend_logs_metadata in batch and files endpoints - PR #16921

• Search APIs

  • Search APIs - error in firecrawl-search "Invalid request body" - PR #16943

• Vector Stores

  • Fix vector store create issue - PR #16804
  • Team vector-store permissions now respected for key access - PR #16639

• Audio Transcription

  • Fix audio transcription cost tracking - PR #16478
  • Add missing shared_sessions to audio/transcriptions - PR #16858

• Video Generation API

  • Fix videos tagging - PR #16770

Bugs​

• General
  • Responses API cost tracking with custom deployment names - PR #16778
  • Trim logged response strings in spend-logs - PR #16654

Management Endpoints / UI​

Features​

• Proxy CLI Auth

  • Allow using JWTs for signing in with Proxy CLI - PR #16756

• Virtual Keys

  • Fix Key Model Alias Not Working - PR #16896

• Models + Endpoints

  • Add additional model settings to chat models in test key - PR #16793
  • Deactivate delete button on model table for config models - PR #16787
  • Change Public Model Hub to use proxyBaseUrl - PR #16892
  • Add JSON Viewer to request/response panel - PR #16687
  • Standarize icon images - PR #16837

• Teams

  • Teams table empty state - PR #16738

• Fallbacks

  • Fallbacks icon button tooltips and delete with friction - PR #16737

• MCP Servers

  • Delete user and MCP Server Modal, MCP Table Tooltips - PR #16751

• Callbacks

  • Expose backend endpoint for callbacks settings - PR #16698
  • Edit add callbacks route to use data from backend - PR #16699

• Usage & Analytics

  • Allow partial matches for user ID in User Table - PR #16952

• General UI

  • Allow setting base_url in API reference docs - PR #16674
  • Change /public fields to honor server root path - PR #16930
  • Correct ui build - PR #16702
  • Enable automatic dark/light mode based on system preference - PR #16748

Bugs​

• UI Fixes

  • Fix flaky tests due to antd Notification Manager - PR #16740
  • Fix UI MCP Tool Test Regression - PR #16695
  • Fix edit logging settings not appearing - PR #16798
  • Add css to truncate long request ids in request viewer - PR #16665
  • Remove azure/ prefix in Placeholder for Azure in Add Model - PR #16597
  • Remove UI Session Token from user/info return - PR #16851
  • Remove console logs and errors from model tab - PR #16455
  • Change Bulk Invite User Roles to Match Backend - PR #16906
  • Mock Tremor's Tooltip to Fix Flaky UI Tests - PR #16786
  • Fix e2e ui playwright test - PR #16799
  • Fix Tests in CI/CD - PR #16972

• SSO

  • Ensure role from SSO provider is used when a user is inserted onto LiteLLM - PR #16794
  • Docs - SSO - Manage User Roles via Azure App Roles - PR #16796

• Auth

  • Ensure Team Tags works when using JWT Auth - PR #16797
  • Fix key never expires - PR #16692

• Swagger UI

  • Fixes Swagger UI resolver errors for chat completion endpoints caused by Pydantic v2 $defs not being properly exposed in the OpenAPI schema - PR #16784

AI Integrations​

Logging​

• Arize Phoenix

  • Fix arize phoenix logging - PR #16301
  • Arize Phoenix - root span logging - PR #16949

• Langfuse

  • Filter secret fields form Langfuse - PR #16842

• General

  • Exclude litellm_credential_name from Sensitive Data Masker (Updated) - PR #16958
  • Allow admins to disable, dynamic callback controls - PR #16750

Guardrails​

• IBM Guardrails

  • Fix IBM Guardrails optional params, add extra_headers field - PR #16771

• Noma Guardrail

  • Use LiteLLM key alias as fallback Noma applicationId in NomaGuardrail - PR #16832
  • Allow custom violation message for tool-permission guardrail - PR #16916

• Grayswan Guardrail

  • Grayswan guardrail passthrough on flagged - PR #16891

• General Guardrails

  • Fix prompt injection not working - PR #16701

Prompt Management​

• Prompt Management
  • Allow specifying just prompt_id in a request to a model - PR #16834
  • Add support for versioning prompts - PR #16836
  • Allow storing prompt version in DB - PR #16848
  • Add UI for editing the prompts - PR #16853
  • Allow testing prompts with Chat UI - PR #16898
  • Allow viewing version history - PR #16901
  • Allow specifying prompt version in code - PR #16929
  • UI, allow seeing model, prompt id for Prompt - PR #16932
  • Show "get code" section for prompt management + minor polish of showing version history - PR #16941

Secret Managers​

• AWS Secrets Manager
  • Adds IAM role assumption support for AWS Secret Manager - PR #16887

MCP Gateway​

• MCP Hub - Publish/discover MCP Servers within a company - PR #16857
• MCP Resources - MCP resources support - PR #16800
• MCP OAuth - Docs - mcp oauth flow details - PR #16742
• MCP Lifecycle - Drop MCPClient.connect and use run_with_session lifecycle - PR #16696
• MCP Server IDs - Add mcp server ids - PR #16904
• MCP URL Format - Fix mcp url format - PR #16940

Performance / Loadbalancing / Reliability improvements​

• Realtime Endpoint Performance - Fix bottlenecks degrading realtime endpoint performance - PR #16670
• SSL Context Caching - Cache SSL contexts to prevent excessive memory allocation - PR #16955
• Cache Optimization - Fix cache cooldown key generation - PR #16954
• Router Cache - Fix routing for requests with same cacheable prefix but different user messages - PR #16951
• Redis Event Loop - Fix redis event loop closed at first call - PR #16913
• Dependency Management - Upgrade pydantic to version 2.11.0 - PR #16909

Documentation Updates​

• Provider Documentation

  • Add missing details to benchmark comparison - PR #16690
  • Fix anthropic pass-through endpoint - PR #16883
  • Cleanup repo and improve AI docs - PR #16775

• API Documentation

  • Add docs related to openai metadata - PR #16872
  • Update docs with all supported endpoints and cost tracking - PR #16872

• General Documentation

  • Add mini-swe-agent to Projects built on LiteLLM - PR #16971

Infrastructure / CI/CD​

• UI Testing

  • Break e2e_ui_testing into build, unit, and e2e steps - PR #16783
  • Building UI for Testing - PR #16968
  • CI/CD Fixes - PR #16937

• Dependency Management

  • Bump js-yaml from 3.14.1 to 3.14.2 in /tests/proxy_admin_ui_tests/ui_unit_tests - PR #16755
  • Bump js-yaml from 3.14.1 to 3.14.2 - PR #16802

• Migration

  • Migration job labels - PR #16831

• Config

  • This yaml actually works - PR #16757

• Release Notes

  • Add perf improvements on embeddings to release notes - PR #16697
  • Docs - v1.80.0 - PR #16694

• Investigation

  • Investigate issue root cause - PR #16859

New Contributors​

• @mattmorgis made their first contribution in PR #16371
• @mmandic-coatue made their first contribution in PR #16732
• @Bradley-Butcher made their first contribution in PR #16725
• @BenjaminLevy made their first contribution in PR #16757
• @CatBraaain made their first contribution in PR #16767
• @tushar8408 made their first contribution in PR #16831
• @nbsp1221 made their first contribution in PR #16845
• @idola9 made their first contribution in PR #16832
• @nkukard made their first contribution in PR #16864
• @alhuang10 made their first contribution in PR #16852
• @sebslight made their first contribution in PR #16838
• @TsurumaruTsuyoshi made their first contribution in PR #16905
• @cyberjunk made their first contribution in PR #16492
• @colinlin-stripe made their first contribution in PR #16895
• @sureshdsk made their first contribution in PR #16883
• @eiliyaabedini made their first contribution in PR #16875
• @justin-tahara made their first contribution in PR #16957
• @wangsoft made their first contribution in PR #16913
• @dsduenas made their first contribution in PR #16891

Known Issues​

• /audit and /user/available_users routes return 404. Fixed in PR #17337

Full Changelog​

View complete changelog on GitHub

Deploy this version​

docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
docker.litellm.ai/berriai/litellm:v1.80.0-stable

pip install litellm==1.80.0

Key Highlights​

• 🆕 Agent Hub Support - Register and make agents public for your organization
• RunwayML Provider - Complete video generation, image generation, and text-to-speech support
• GPT-5.1 Family Support - Day-0 support for OpenAI's latest GPT-5.1 and GPT-5.1-Codex models
• Prometheus OSS - Prometheus metrics now available in open-source version
• Vector Store Files API - Complete OpenAI-compatible Vector Store Files API with full CRUD operations
• Embeddings Performance - O(1) lookup optimization for router embeddings with shared sessions

Agent Hub​

This release adds support for registering and making agents public for your organization. This is great for Proxy Admins who want a central place to make agents built in their organization, discoverable to their users.

Here's the flow:

1. Add agent to litellm.
2. Make it public.
3. Allow anyone to discover it on the public AI Hub page.

Get Started with Agent Hub

Performance – /embeddings 13× Lower p95 Latency​

This update significantly improves /embeddings latency by routing it through the same optimized pipeline as /chat/completions, benefiting from all previously applied networking optimizations.

Results​

Test Setup​

🆕 RunwayML​

Complete integration for RunwayML's Gen-4 family of models, supporting video generation, image generation, and text-to-speech.

Supported Endpoints:

• /v1/videos - Video generation (Gen-4 Turbo, Gen-4 Aleph, Gen-3A Turbo)
• /v1/images/generations - Image generation (Gen-4 Image, Gen-4 Image Turbo)
• /v1/audio/speech - Text-to-speech (ElevenLabs Multilingual v2)

Quick Start:

curl --location 'http://localhost:4000/v1/videos' \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer sk-1234' \
--data '{
    "model": "runwayml/gen4_turbo",
    "prompt": "A high quality demo video of litellm ai gateway",
    "input_reference": "https://example.com/image.jpg",
    "seconds": 5,
    "size": "1280x720"
}'

Get Started with RunwayML

Prometheus Metrics - Open Source​

Prometheus metrics are now available in the open-source version of LiteLLM, providing comprehensive observability for your AI Gateway without requiring an enterprise license.

Quick Start:

litellm_settings:
  success_callback: ["prometheus"]
  failure_callback: ["prometheus"]

Get Started with Prometheus

Vector Store Files API​

Complete OpenAI-compatible Vector Store Files API now stable, enabling full file lifecycle management within vector stores.

Supported Endpoints:

• POST /v1/vector_stores/{vector_store_id}/files - Create vector store file
• GET /v1/vector_stores/{vector_store_id}/files - List vector store files
• GET /v1/vector_stores/{vector_store_id}/files/{file_id} - Retrieve vector store file
• GET /v1/vector_stores/{vector_store_id}/files/{file_id}/content - Retrieve file content
• DELETE /v1/vector_stores/{vector_store_id}/files/{file_id} - Delete vector store file
• DELETE /v1/vector_stores/{vector_store_id} - Delete vector store

Quick Start:

curl --location 'http://localhost:4000/v1/vector_stores/vs_123/files' \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer sk-1234' \
--data '{
    "file_id": "file_abc"
}'

Get Started with Vector Stores

New Providers and Endpoints​

New Providers​

New LLM API Endpoints​

New Models / Updated Models​

New Model Support​

Video Generation Models​

Image Generation Models​

Audio Models​

Features​

• OpenAI

  • Add GPT-5.1 family support with reasoning capabilities - PR #16598
  • Add support for reasoning_effort='none' for GPT-5.1 - PR #16658
  • Add verbosity parameter support for GPT-5 family models - PR #16660
  • Fix forward OpenAI organization for image generation - PR #16607

• Gemini (Google AI Studio + Vertex AI)

  • Add support for reasoning_effort='none' for Gemini models - PR #16548
  • Add all Gemini image models support in image generation - PR #16526
  • Add Gemini image edit support - PR #16430
  • Fix preserve non-ASCII characters in function call arguments - PR #16550
  • Fix Gemini conversation format issue with MCP auto-execution - PR #16592

• Bedrock

  • Add support for filtering knowledge base queries - PR #16543
  • Ensure correct aws_region is used when provided dynamically for embeddings - PR #16547
  • Add support for custom KMS encryption keys in Bedrock Batch operations - PR #16662
  • Add bearer token authentication support for AgentCore - PR #16556
  • Fix AgentCore SSE stream iterator to async for proper streaming support - PR #16293

• Anthropic

  • Add context management param support - PR #16528
  • Fix preserve $defs for Anthropic tools input schema - PR #16648
  • Fix support Anthropic tool_use and tool_result in token counter - PR #16351

• Vertex AI

  • Add Vertex Kimi-K2-Thinking support - PR #16671
  • Add vertex_credentials support to litellm.rerank() - PR #16479

• Mistral

  • Fix Magistral streaming to emit reasoning chunks - PR #16434

• Moonshot (Kimi)

  • Add Kimi K2 thinking model support - PR #16445

• SambaNova

  • Fix SambaNova API rejecting requests when message content is passed as a list format - PR #16612

• VLLM

  • Fix use vllm passthrough config for hosted vllm provider instead of raising error - PR #16537
  • Add headers to VLLM Passthrough requests with success event logging - PR #16532

• Azure

  • Fix improve Azure auth parameter handling for None values - PR #14436

• Groq

  • Fix parse failed chunks for Groq - PR #16595

• Voyage

  • Add Voyage 3.5 and 3.5-lite embeddings pricing and doc update - PR #16641

• Fal.ai

  • Add fal-ai/flux/schnell support - PR #16580
  • Add all Imagen4 variants of fal ai in model map - PR #16579

Bug Fixes​

• General
  • Fix sanitize null token usage in OpenAI-compatible responses - PR #16493
  • Fix apply provided timeout value to ClientTimeout.total - PR #16395
  • Fix raising wrong 429 error on wrong exception - PR #16482
  • Add new models, delete repeat models, update pricing - PR #16491
  • Update model logging format for custom LLM provider - PR #16485

LLM API Endpoints​

New Endpoints​

• GET /providers
  • Add GET list of providers endpoint - PR #16432

Features​

• Video Generation API

  • Allow internal users to access video generation routes - PR #16472

• Vector Stores API

  • Vector store files stable release with complete CRUD operations - PR #16643
    • POST /v1/vector_stores/{vector_store_id}/files - Create vector store file
    • GET /v1/vector_stores/{vector_store_id}/files - List vector store files
    • GET /v1/vector_stores/{vector_store_id}/files/{file_id} - Retrieve vector store file
    • GET /v1/vector_stores/{vector_store_id}/files/{file_id}/content - Retrieve file content
    • DELETE /v1/vector_stores/{vector_store_id}/files/{file_id} - Delete vector store file
    • DELETE /v1/vector_stores/{vector_store_id} - Delete vector store
  • Ensure users can access search_results for both stream + non-stream response - PR #16459

Bugs​

• Video Generation API

  • Fix use GET for /v1/videos/{video_id}/content - PR #16672

• General

  • Fix remove generic exception handling - PR #16599

Management Endpoints / UI​

Features​

• Proxy CLI Auth

  • Fix remove strict master_key check in add_deployment - PR #16453

• Virtual Keys

  • UI - Add Tags To Edit Key Flow - PR #16500
  • UI - Test Key Page show models based on selected endpoint - PR #16452
  • UI - Expose user_alias in view and update path - PR #16669

• Models + Endpoints

  • UI - Add LiteLLM Params to Edit Model - PR #16496
  • UI - Add Model use backend data - PR #16664
  • UI - Remove Description Field from LLM Credentials - PR #16608
  • UI - Add RunwayML on Admin UI supported models/providers - PR #16606
  • Infra - Migrate Add Model Fields to Backend - PR #16620
  • Add API Endpoint for creating model access group - PR #16663

• Teams

  • UI - Invite User Searchable Team Select - PR #16454
  • Fix use user budget instead of key budget when creating new team - PR #16074

• Budgets

  • UI - Move Budgets out of Experimental - PR #16544

• Guardrails

  • UI - Config Guardrails should not be deletable from table - PR #16540
  • Fix remove enterprise restriction from guardrails list endpoint - PR #15333

• Callbacks

  • UI - New Callbacks table - PR #16512
  • Fix delete callbacks failing - PR #16473

• Usage & Analytics

  • UI - Improve Usage Indicator - PR #16504
  • UI - Model Info Page Health Check - PR #16416
  • Infra - Show Deprecation Warning for Model Analytics Tab - PR #16417
  • Fix Litellm tags usage add request_id - PR #16111

• Health Check

  • Add Langfuse OTEL and SQS to Health Check - PR #16514

• General UI

  • UI - Normalize table action columns appearance - PR #16657
  • UI - Button Styles and Sizing in Settings Pages - PR #16600
  • UI - SSO Modal Cosmetic Changes - PR #16554
  • Fix UI logos loading with SERVER_ROOT_PATH - PR #16618
  • Fix remove misleading 'Custom' option mention from OpenAI endpoint tooltips - PR #16622

• SSO

  • Ensure role from SSO provider is used when a user is inserted onto LiteLLM - PR #16794

Bugs​

• Management Endpoints
  • Fix inconsistent error responses in customer management endpoints - PR #16450
  • Fix correct date range filtering in /spend/logs endpoint - PR #16443
  • Fix /spend/logs/ui Access Control - PR #16446
  • Add pagination for /spend/logs/session/ui endpoint - PR #16603
  • Fix LiteLLM Usage shows key_hash - PR #16471
  • Fix app_roles missing from jwt payload - PR #16448

Logging / Guardrail / Prompt Management Integrations​

New Integration​

• 🆕 Zscaler AI Guard
  • Add Zscaler AI Guard hook for security policy enforcement - PR #15691

Logging​

• Langfuse

  • Fix handle null usage values to prevent validation errors - PR #16396

• CloudZero

  • Fix updated spend would not be sent to CloudZero - PR #16201

Guardrails​

• IBM Detector
  • Ensure detector-id is passed as header to IBM detector server - PR #16649

Prompt Management​

• Custom Prompt Management
  • Add SDK focused examples for custom prompt management - PR #16441

Spend Tracking, Budgets and Rate Limiting​

• End User Budgets
  • Allow pointing max_end_user budget to an id, so the default ID applies to all end users - PR #16456

MCP Gateway​

• Configuration
  • Add dynamic OAuth2 metadata discovery for MCP servers - PR #16676
  • Fix allow tool call even when server name prefix is missing - PR #16425
  • Fix exclude unauthorized MCP servers from allowed server list - PR #16551
  • Fix unable to delete MCP server from permission settings - PR #16407
  • Fix avoid crashing when MCP server record lacks credentials - PR #16601

Agents​

• Agent Registration (A2A Spec)
  • Support agent registration + discovery following Agent-to-Agent specification - PR #16615

Performance / Loadbalancing / Reliability improvements​

• Embeddings Performance

  • Use router's O(1) lookup and shared sessions for embeddings - PR #16344

• Router Reliability

  • Support default fallbacks for unknown models - PR #16419

• Callback Management

  • Add atexit handlers to flush callbacks for async completions - PR #16487

General Proxy Improvements​

• Configuration Management
  • Fix update model_cost_map_url to use environment variable - PR #16429

Documentation Updates​

• Provider Documentation

  • Fix streaming example in README - PR #16461
  • Update broken Slack invite links to support page - PR #16546
  • Fix code block indentation for fallbacks page - PR #16542
  • Documentation code example corrections - PR #16502
  • Document reasoning_effort summary field options - PR #16549

• API Documentation

  • Add docs on APIs for model access management - PR #16673
  • Add docs for showing how to auto reload new pricing data - PR #16675
  • LiteLLM Quick start - show how model resolution works - PR #16602
  • Add docs for tracking callback failure - PR #16474

• General Documentation

  • Fix container api link in release page - PR #16440
  • Add softgen to projects that are using litellm - PR #16423

New Contributors​

• @artplan1 made their first contribution in PR #16423
• @JehandadK made their first contribution in PR #16472
• @vmiscenko made their first contribution in PR #16453
• @mcowger made their first contribution in PR #16429
• @yellowsubmarine372 made their first contribution in PR #16395
• @Hebruwu made their first contribution in PR #16201
• @jwang-gif made their first contribution in PR #15691
• @AnthonyMonaco made their first contribution in PR #16502
• @andrewm4894 made their first contribution in PR #16487
• @f14-bertolotti made their first contribution in PR #16485
• @busla made their first contribution in PR #16293
• @MightyGoldenOctopus made their first contribution in PR #16537
• @ultmaster made their first contribution in PR #14436
• @bchrobot made their first contribution in PR #16542
• @sep-grindr made their first contribution in PR #16622
• @pnookala-godaddy made their first contribution in PR #16607
• @dtunikov made their first contribution in PR #16592
• @lukapecnik made their first contribution in PR #16648
• @jyeros made their first contribution in PR #16618

Full Changelog​

View complete changelog on GitHub

Deploy this version​

docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
docker.litellm.ai/berriai/litellm:v1.79.3-stable

pip install litellm==1.79.3.rc.1

Key Highlights​

• LiteLLM Custom Guardrail - Built-in guardrail with UI configuration support
• Performance Improvements - /responses API 19× Lower Median Latency
• Veo3 Video Generation (Vertex AI + Google AI Studio) - Use OpenAI Video API to generate videos with Vertex AI and Google AI Studio Veo3 models

Built-in Guardrails on AI Gateway​

This release introduces built-in guardrails for LiteLLM AI Gateway, allowing you to enforce protections without depending on an external guardrail API.

• Blocking Keywords - Block known sensitive keywords like "litellm", "python", etc.
• Pattern Detection - Block known sensitive patterns like emails, Social Security Numbers, API keys, etc.
• Custom Regex Patterns - Define custom regex patterns for your specific use case.

Get started with the built-in guardrails on AI Gateway here.

Performance – /responses 19× Lower Median Latency​

This update significantly improves /responses latency by integrating our internal network management for connection handling, eliminating per-request setup overhead.

Results​

Test Setup​

New Models / Updated Models​

New Model Support​

OCR Models​

Search Models​

Features​

• Azure

  • Add Azure GPT-5-Pro Responses API support with reasoning capabilities - PR #16235
  • Add gpt-image-1-mini pricing for Azure with quality tiers (low/medium/high) - PR #16182
  • Add support for returning Azure Content Policy error information when exceptions from Azure OpenAI occur - PR #16231
  • Fix Azure GPT-5 incorrectly routed to O-series config (temperature parameter unsupported) - PR #16246
  • Fix Azure doesn't accept extra body param - PR #16116
  • Fix Azure DALL-E-3 health check content policy violation by using safe default prompt - PR #16329

• Bedrock

  • Fix empty assistant message handling in AWS Bedrock Converse API to prevent 400 Bad Request errors - PR #15850
  • Fix: Filter AWS authentication params from Bedrock InvokeModel request body - PR #16315
  • Fix Bedrock proxy adding name to file content, breaks when cache_control in use - PR #16275
  • Fix global.anthropic.claude-haiku-4-5-20251001-v1:0 supports_reasoning flag and update pricing - PR #16263

• Gemini (Google AI Studio + Vertex AI)

  • Add gemini live audio model cost in model map - PR #16183
  • Fix translation problem with Gemini parallel tool calls - PR #16194
  • Fix: Send Gemini API key via x-goog-api-key header with custom api_base - PR #16085
  • Fix image_config.aspect_ratio not working for gemini-2.5-flash-image - PR #15999
  • Fix Gemini minimal reasoning env overrides disabling thoughts - PR #16347
  • Fix cache_read_input_token_cost for gemini-2.5-flash - PR #16354

• Anthropic

  • Fix Anthropic token counting for VertexAI - PR #16171
  • Fix anthropic-adapter: properly translate Anthropic image format to OpenAI - PR #16202
  • Enable automated prompt caching message format for Claude on Databricks - PR #16200
  • Add support for Anthropic Memory Tool - PR #16115
  • Propagate cache creation/read token costs for model info to fix Anthropic long context cost calculations - PR #16376

• Vertex AI

  • Add Vertex MiniMAX m2 model support - PR #16373
  • Correctly map 429 Resource Exhausted to RateLimitError - PR #16363
  • Add vertex_credentials support to litellm.rerank() for Vertex AI - PR #16266

• Databricks

  • Fix databricks streaming - PR #16368

• Deepgram

  • Return the diarized transcript when it's required in the request - PR #16133

• Fireworks

  • Update Fireworks audio endpoints to new api.fireworks.ai domains - PR #16346

• Cohere

  • Add cohere embed-v4.0 model support - PR #16358

• Watsonx

  • Support reasoning_effort for watsonx chat models - PR #16261

• OpenAI

  • Remove automatic summary from reasoning_effort transformation - PR #16210

• XAI

  • Remove Grok 4 Models Reasoning Effort Parameter - PR #16265

• Hosted VLLM

  • Fix HostedVLLMRerankConfig will not be used - PR #16352

New Provider Support​

• Bedrock Agentcore
  • Add Bedrock Agentcore as a provider on LiteLLM Python SDK and LiteLLM AI Gateway - PR #16252

LLM API Endpoints​

Features​

• OCR API

  • Add VertexAI OCR provider support + cost tracking - PR #16216
  • Add Azure AI Doc Intelligence OCR support - PR #16219

• Search API

  • Add firecrawl search API support with tiered pricing - PR #16257
  • Add searxng search API provider - PR #16259

• Responses API

  • Support responses API streaming in langfuse otel - PR #16153
  • Pass extra_body parameters to provider in Responses API requests - PR #16320

• Container API

  • Add E2E Container API Support - PR #16136
  • Update container documentation to be similar to others - PR #16327

• Video Generation API

  • Add Vertex and Gemini Videos API with Cost Tracking + UI support - PR #16323
  • Add custom_llm_provider support for video endpoints (non-generation) - PR #16121

• Audio API

  • Add gpt-4o-transcribe cost tracking - PR #16412

• Vector Stores

  • Milvus - search vector store support + support multi-part form data on passthrough - PR #16035
  • Azure AI Vector Stores - support "virtual" indexes + create vector store on passthrough API - PR #16160
  • Milvus - Passthrough API support - adds create + read vector store support via passthrough API's - PR #16170

• Embeddings API

  • Use valid CallTypes enum value in embeddings endpoint - PR #16328

• Rerank API

  • Generalize tiered pricing in generic cost calculator - PR #16150

Bugs​

• General
  • Fix index field not populated in streaming mode with n>1 and tool calls - PR #15962
  • Pass aws_region_name in litellm_params - PR #16321
  • Add retry-after header support for errors 502, 503, 504 - PR #16288

Management Endpoints / UI​

Features​

• Virtual Keys

  • UI - Delete Team Member with friction - PR #16167
  • UI - Litellm test key audio support - PR #16251
  • UI - Test Key Page Revert Model To Single Select - PR #16390

• Models + Endpoints

  • UI - Add Model Existing Credentials Improvement - PR #16166
  • UI - Add Azure AD Token field and Azure API Key optional - PR #16331
  • UI - Fixed Label for vLLM in Model Create Flow - PR #16285
  • UI - Include Model Access Group Models on Team Models Table - PR #16298
  • Fix /model_group/info Returning Entire Model List for SSO Users - PR #16296
  • Litellm non root docker Model Hub Table fix - PR #16282

• Guardrails

  • UI - Fix regression where Guardrail Entity Could not be selected and entity was not displayed - PR #16165
  • UI - Guardrail Info Page Show PII Config - PR #16164
  • Change guardrail_information to list type - PR #16127
  • UI - LiteLLM Guardrail - ensure you can see UI Friendly name for PII Patterns - PR #16382
  • UI - Guardrails - LiteLLM Content Filter, Allow Viewing/Editing Content Filter Settings - PR #16383
  • UI - Guardrails - allow updating guardrails through UI. Ensure litellm_params actually get updated in memory - PR #16384

• SSO Settings

  • Support dot notation on ui sso - PR #16135
  • UI - Prevent trailing slash in sso proxy base url input - PR #16244
  • UI - SSO Proxy Base URL input validation and remove normalizing / - PR #16332
  • UI - Surface SSO Create errors on create flow - PR #16369

• Usage & Analytics

  • UI - Tag Usage Top Model Table View and Label Fix - PR #16249
  • UI - Litellm usage date picker - PR #16264

• Cache Settings

  • UI - Cache Settings Redis Add Semantic Cache Settings - PR #16398

Bugs​

• General
  • UI - Remove encoding_format in request for embedding models - PR #16367
  • UI - Revert Changes for Test Key Multiple Model Select - PR #16372
  • UI - Various Small Issues - PR #16406

AI Integrations​

Logging​

• Langfuse

  • Fix langfuse input tokens logic for cached tokens - PR #16203

• Opik

  • Fix the bug with not incorrect attachment to existing trace & refactor - PR #15529

• S3

  • S3 logger, add support for ssl_verify when using minio logger - PR #16211
  • Strip base64 in s3 - PR #16157
  • Add allowing Key based prefix to s3 path - PR #16237
  • Add Prometheus metric to track callback logging failures in S3 - PR #16209

• OpenTelemetry

  • OTEL - Log Cost Breakdown on OTEL Logger - PR #16334

• DataDog

  • Add DD Agent Host support for datadog callback - PR #16379

Guardrails​

• Noma

  • Revert Noma Apply Guardrail implementation - PR #16214
  • Litellm noma guardrail support images - PR #16199

• PANW Prisma AIRS

  • PANW prisma airs guardrail deduplication and enhanced session tracking - PR #16273

• LiteLLM Custom Guardrail

  • Add LiteLLM Gateway built in guardrail - PR #16338
  • UI - Allow configuring LiteLLM Custom Guardrail - PR #16339
  • Bug Fix: Content Filter Guard - PR #16414

Secret Managers​

• CyberArk

  • Add CyberArk Secrets Manager Integration - PR #16278
  • Cyber Ark - Add Key Rotations support - PR #16289

• HashiCorp Vault

  • Add configurable mount name and path prefix for HashiCorp Vault - PR #16253
  • Secret Manager - Hashicorp, add auth via approle - PR #16374

• AWS Secrets Manager

  • Add tags and descriptions support to aws secrets manager - PR #16224

• Custom Secret Manager

  • Add Custom Secret Manager - Allow users to define and write a custom secret manager - PR #16297

• General

  • Email Notifications - Ensure Users get Key Rotated Email - PR #16292
  • Fix verify ssl on sts boto3 - PR #16313