> For the complete documentation index, see [llms.txt](https://docs.digitalapi.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.digitalapi.ai/api-gateway/plugin-reference/plugins-ai.md). # AI plugins Plugins in this category sit between consumers and large-language-model providers. They handle content moderation, prompt and response logging, token-budget enforcement, model routing across providers, and AI-specific rate limiting. Attach them at route scope on the routes that proxy to your LLM provider. The catalogue ships 8 AI plugins. {% hint style="info" %} **In this category:** [`ai-prompt-decorator`](#ai-prompt-decorator), [`ai-prompt-guard`](#ai-prompt-guard), [`ai-prompt-template`](#ai-prompt-template), [`ai-proxy`](#ai-proxy), [`ai-proxy-multi`](#ai-proxy-multi), [`ai-rag`](#ai-rag), [`ai-rate-limiting`](#ai-rate-limiting), [`ai-request-rewrite`](#ai-request-rewrite) {% endhint %} ### `ai-prompt-decorator` The `ai-prompt-decorator` plugin simplifies access to LLM providers, such as OpenAI and Anthropic, and their models by appending or prepending prompts into the request. {% hint style="info" %} **Priority** `1070` · **Category** AI {% endhint %}
Configuration and details | **Field** | **Required** | **Type** | **Description** | | ----------------- | --------------- | -------- | --------------------------------------------------- | | `prepend` | Conditionally\* | Array | An array of prompt objects to be prepended | | `prepend.role` | Yes | String | Role of the message (`system`, `user`, `assistant`) | | `prepend.content` | Yes | String | Content of the message. Minimum length: 1 | | `append` | Conditionally\* | Array | An array of prompt objects to be appended | | `append.role` | Yes | String | Role of the message (`system`, `user`, `assistant`) | | `append.content` | Yes | String | Content of the message. Minimum length: 1 | **Sample configuration** Prepend a system prompt in front of every incoming request: ```json { "ai-prompt-decorator": { "prepend": [ { "role": "system", "content": "You are a helpful assistant. Answer concisely." } ] } } ``` Append a message after the user prompt to steer the response: ```json { "ai-prompt-decorator": { "append": [ { "role": "user", "content": "Reply in English and cite your sources." } ] } } ```
### `ai-prompt-guard` The `ai-prompt-guard` plugin safeguards your AI endpoints by inspecting and validating incoming prompt messages. It checks the content of requests against user-defined allowed and denied patterns to ensure that only approved inputs are processed. Based on its configuration, the plugin can either examine just the latest message or the entire conversation history, and it can be set to check prompts from all roles or only from end users. {% hint style="info" %} **Priority** `1072` · **Category** AI {% endhint %}
Configuration and details When both **allow** and **deny** patterns are configured, the plugin first ensures that at least one allowed pattern is matched. If none match, the request is rejected with a *"Request doesn't match allow patterns"* error. If an allowed pattern is found, it then checks for any occurrences of denied patterns, rejecting the request with a *"Request contains prohibited content"* error if any are detected. | **Field** | **Required** | **Type** | **Description** | | --------------------------------- | ------------ | -------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | match\_all\_roles | No | boolean | If set to `true`, the plugin will check prompt messages from all roles. Otherwise, it only validates when its role is `"user"`. Default is `false`. | | match\_all\_conversation\_history | No | boolean | When enabled, all messages in the conversation history are concatenated and checked. If `false`, only the content of the last message is examined. Default is `false`. | | allow\_patterns | No | array | A list of regex patterns. When provided, the prompt must match **at least one** pattern to be considered valid. | | deny\_patterns | No | array | A list of regex patterns. If any of these patterns match the prompt content, the request is rejected. | **Sample configuration** Reject prompts that match a denied pattern: ```json { "ai-prompt-guard": { "deny_patterns": [ "block.*keyword", "(?i)ignore previous instructions" ] } } ``` Allow only prompts that match one of the approved patterns: ```json { "ai-prompt-guard": { "allow_patterns": [ "^support ticket:", "^order status:" ] } } ``` Check every role across the full conversation history against both lists: ```json { "ai-prompt-guard": { "match_all_roles": true, "match_all_conversation_history": true, "allow_patterns": [ "^support ticket:" ], "deny_patterns": [ "(?i)ignore previous instructions" ] } } ```
### `ai-prompt-template` The `ai-prompt-template` Plugin simplifies access to LLM providers, such as OpenAI and Anthropic, and their models. It pre-configures prompt templates that only accept user inputs in designated template variables, in a "fill in the blank" fashion. {% hint style="info" %} **Priority** `1071` · **Category** AI · **Required** `templates` {% endhint %}
Configuration and details | **Field** | **Required** | **Type** | **Description** | | ------------------------------------- | ------------ | -------- | ------------------------------------------------------------------------------------------------------------------------------------------ | | `templates` | Yes | Array | An array of template objects. | | `templates.name` | Yes | String | Name of the template. When requesting the route, the request should include the template name that corresponds to the configured template. | | `templates.template` | Yes | Object | Template specification. | | `templates.template.model` | Yes | String | Name of the AI Model, such as `gpt-4` or `gpt-3.5`. See your LLM provider API documentation for more available models. | | `templates.template.messages` | Yes | Array | Template message specification. | | `templates.template.messages.role` | Yes | String | Role of the message, such as `system`, `user`, or `assistant`. | | `templates.template.messages.content` | Yes | String | Content of the message (prompt). | **Sample configuration** A single named template that fixes the model and messages, leaving `{{ }}` variables for the caller to fill: ```json { "ai-prompt-template": { "templates": [ { "name": "product-description", "template": { "model": "gpt-4", "messages": [ { "role": "system", "content": "You write concise product descriptions." }, { "role": "user", "content": "Describe {{product_name}} in {{word_count}} words." } ] } } ] } } ```
### `ai-proxy` The `ai-proxy` Plugin simplifies access to LLM and embedding models by transforming Plugin configurations into the designated request format. It supports the integration with OpenAI, DeepSeek, Azure, AIMLAPI, Anthropic, OpenRouter, Gemini, Vertex AI, and other OpenAI-compatible APIs. {% hint style="info" %} **Priority** `1040` · **Category** AI · **Required** `provider`, `auth` {% endhint %}
Configuration and details In addition, the Plugin also supports logging LLM request information in the access log, such as token usage, model, time to the first response, and more. | Name | Type | Required | Default | Valid values | Description | | ------------------------------- | ------- | -------- | ------- | ------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | provider | string | True | | \[openai, deepseek, azure-openai, aimlapi, anthropic, openrouter, gemini, vertex-ai, openai-compatible] | LLM service provider. When set to `openai`, the Plugin will proxy the request to `https://api.openai.com/chat/completions`. When set to `deepseek`, the Plugin will proxy the request to `https://api.deepseek.com/chat/completions`. When set to `aimlapi`, the Plugin uses the OpenAI-compatible driver and proxies the request to `https://api.aimlapi.com/v1/chat/completions` by default. When set to `anthropic`, the Plugin will proxy the request to `https://api.anthropic.com/v1/chat/completions` by default. When set to `openrouter`, the Plugin uses the OpenAI-compatible driver and proxies the request to `https://openrouter.ai/api/v1/chat/completions` by default. When set to `gemini`, the Plugin uses the OpenAI-compatible driver and proxies the request to `https://generativelanguage.googleapis.com/v1beta/openai/chat/completions` by default. When set to `vertex-ai`, the Plugin will proxy the request to `https://aiplatform.googleapis.com` by default and requires `provider_conf` or `override`. When set to `openai-compatible`, the Plugin will proxy the request to the custom endpoint configured in `override`. | | provider\_conf | object | False | | | Configuration for the specific provider. Required when `provider` is set to `vertex-ai` and `override` is not configured. | | provider\_conf.project\_id | string | True | | | Google Cloud Project ID. | | provider\_conf.region | string | True | | | Google Cloud Region. | | auth | object | True | | | Authentication configurations. | | auth.header | object | False | | | Authentication headers. At least one of `header` or `query` must be configured. | | auth.query | object | False | | | Authentication query parameters. At least one of `header` or `query` must be configured. | | auth.gcp | object | False | | | Configuration for Google Cloud Platform (GCP) authentication. | | auth.gcp.service\_account\_json | string | False | | | Content of the GCP service account JSON file. This can also be configured by setting the `GCP_SERVICE_ACCOUNT` environment variable. | | auth.gcp.max\_ttl | integer | False | | minimum = 1 | Maximum TTL (in seconds) for caching the GCP access token. | | auth.gcp.expire\_early\_secs | integer | False | 60 | minimum = 0 | Seconds to expire the access token before its actual expiration time to avoid edge cases. | | options | object | False | | | Model configurations. In addition to `model`, you can configure additional parameters and they will be forwarded to the upstream LLM service in the request body. For instance, if you are working with OpenAI, you can configure additional parameters such as `temperature`, `top_p`, and `stream`. See your LLM provider's API documentation for more available options. | | options.model | string | False | | | Name of the LLM model, such as `gpt-4` or `gpt-3.5`. Refer to the LLM provider's API documentation for available models. | | override | object | False | | | Override setting. | | override.endpoint | string | False | | | Custom LLM provider endpoint, required when `provider` is `openai-compatible`. | | logging | object | False | | | Logging configurations. | | logging.summaries | boolean | False | false | | If true, logs request LLM model, duration, request, and response tokens. | | logging.payloads | boolean | False | false | | If true, logs request and response payload. | | timeout | integer | False | 30000 | ≥ 1 | Request timeout in milliseconds when requesting the LLM service. | | keepalive | boolean | False | true | | If true, keeps the connection alive when requesting the LLM service. | | keepalive\_timeout | integer | False | 60000 | ≥ 1000 | Keepalive timeout in milliseconds when connecting to the LLM service. | | keepalive\_pool | integer | False | 30 | | Keepalive pool size for the LLM service connection. | | ssl\_verify | boolean | False | true | | If true, verifies the LLM service's certificate. | **Sample configuration** Proxy to OpenAI, passing the API key as a bearer token header: ```json { "ai-proxy": { "provider": "openai", "auth": { "header": { "Authorization": "Bearer my-openai-api-key" } }, "options": { "model": "gpt-4" } } } ``` Proxy to Anthropic, passing the API key in the provider's own header: ```json { "ai-proxy": { "provider": "anthropic", "auth": { "header": { "x-api-key": "my-anthropic-api-key" } }, "options": { "model": "claude-3-5-sonnet" } } } ``` Authenticate with a query parameter instead of a header: ```json { "ai-proxy": { "provider": "openai", "auth": { "query": { "api_key": "my-openai-api-key" } } } } ``` Fuller config with tuning, timeout, and request logging: ```json { "ai-proxy": { "provider": "openai", "auth": { "header": { "Authorization": "Bearer my-openai-api-key" } }, "options": { "model": "gpt-4", "temperature": 0.5 }, "logging": { "summaries": true, "payloads": false }, "timeout": 30000, "ssl_verify": true } } ```
### `ai-proxy-multi` The `ai-proxy-multi` Plugin simplifies access to LLM and embedding models by transforming Plugin configurations into the designated request format for OpenAI, DeepSeek, Azure, AIMLAPI, Anthropic, OpenRouter, Gemini, Vertex AI, and other OpenAI-compatible APIs. It extends the capabilities of `ai-proxy` with load balancing, retries, fallbacks, and health checks. {% hint style="info" %} **Priority** `1041` · **Category** AI · **Required** `instances` {% endhint %}
Configuration and details In addition, the Plugin also supports logging LLM request information in the access log, such as token usage, model, time to the first response, and more. | Name | Type | Required | Default | Valid Values | Description | | ----------------------------------------- | --------------- | -------- | ---------- | ---------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | fallback\_strategy | string or array | False | |

string: "instance\_health\_and\_rate\_limiting", "http\_429", "http\_5xx"
array: \["rate\_limiting", "http\_429", "http\_5xx"]

| Fallback strategy. When set, the Plugin will check whether the specified instance’s token has been exhausted when a request is forwarded. If so, forward the request to the next instance regardless of the instance priority. When not set, the Plugin will not forward the request to low priority instances when token of the high priority instance is exhausted. | | balancer | object | False | | | Load balancing configurations. | | balancer.algorithm | string | False | roundrobin | \[roundrobin, chash] | Load balancing algorithm. When set to `roundrobin`, weighted round robin algorithm is used. When set to `chash`, consistent hashing algorithm is used. | | balancer.hash\_on | string | False | | \[vars, headers, cookie, consumer, vars\_combinations] | Used when `type` is `chash`. Support hashing on gateway variables, headers, cookie, consumer, or a combination of gateway variables. | | balancer.key | string | False | | | Used when `type` is `chash`. When `hash_on` is set to `header` or `cookie`, `key` is required. When `hash_on` is set to `consumer`, `key` is not required as the consumer name will be used as the key automatically. | | instances | array\[object] | True | | | LLM instance configurations. | | instances.name | string | True | | | Name of the LLM service instance. | | instances.provider | string | True | | \[openai, deepseek, azure-openai, aimlapi, anthropic, openrouter, gemini, vertex-ai, openai-compatible] | LLM service provider. When set to `openai`, the Plugin will proxy the request to `api.openai.com`. When set to `deepseek`, the Plugin will proxy the request to `api.deepseek.com`. When set to `aimlapi`, the Plugin uses the OpenAI-compatible driver and proxies the request to `api.aimlapi.com` by default. When set to `anthropic`, the Plugin will proxy the request to `api.anthropic.com` by default. When set to `openrouter`, the Plugin uses the OpenAI-compatible driver and proxies the request to `openrouter.ai` by default. When set to `gemini`, the Plugin uses the OpenAI-compatible driver and proxies the request to `generativelanguage.googleapis.com` by default. When set to `vertex-ai`, the Plugin will proxy the request to `aiplatform.googleapis.com` by default and requires `provider_conf` or `override`. When set to `openai-compatible`, the Plugin will proxy the request to the custom endpoint configured in `override`. | | instances.provider\_conf | object | False | | | Configuration for the specific provider. Required when `provider` is set to `vertex-ai` and `override` is not configured. | | instances.provider\_conf.project\_id | string | True | | | Google Cloud Project ID. | | instances.provider\_conf.region | string | True | | | Google Cloud Region. | | instances.priority | integer | False | 0 | | Priority of the LLM instance in load balancing. `priority` takes precedence over `weight`. | | instances.weight | string | True | 0 | greater or equal to 0 | Weight of the LLM instance in load balancing. | | instances.auth | object | True | | | Authentication configurations. | | instances.auth.header | object | False | | | Authentication headers. At least one of the `header` and `query` should be configured. | | instances.auth.query | object | False | | | Authentication query parameters. At least one of the `header` and `query` should be configured. | | instances.auth.gcp | object | False | | | Configuration for Google Cloud Platform (GCP) authentication. | | instances.auth.gcp.service\_account\_json | string | False | | | Content of the GCP service account JSON file. This can also be configured by setting the `GCP_SERVICE_ACCOUNT` environment variable. | | instances.auth.gcp.max\_ttl | integer | False | | minimum = 1 | Maximum TTL (in seconds) for caching the GCP access token. | | instances.auth.gcp.expire\_early\_secs | integer | False | 60 | minimum = 0 | Seconds to expire the access token before its actual expiration time to avoid edge cases. | | instances.options | object | False | | | Model configurations. In addition to `model`, you can configure additional parameters and they will be forwarded to the upstream LLM service in the request body. For instance, if you are working with OpenAI, DeepSeek, or AIMLAPI, you can configure additional parameters such as `max_tokens`, `temperature`, `top_p`, and `stream`. See your LLM provider's API documentation for more available options. | | instances.options.model | string | False | | | Name of the LLM model, such as `gpt-4` or `gpt-3.5`. See your LLM provider's API documentation for more available models. | | logging | object | False | | | Logging configurations. | | logging.summaries | boolean | False | false | | If true, log request LLM model, duration, request, and response tokens. | | logging.payloads | boolean | False | false | | If true, log request and response payload. | | logging.override | object | False | | | Override setting. | | logging.override.endpoint | string | False | | | LLM provider endpoint to replace the default endpoint with. If not configured, the Plugin uses the default OpenAI endpoint `https://api.openai.com/v1/chat/completions`. | | checks | object | False | | | Health check configurations. Note that at the moment, OpenAI, DeepSeek, and AIMLAPI do not provide an official health check endpoint. Other LLM services that you can configure under `openai-compatible` provider may have available health check endpoints. | | checks.active | object | True | | | Active health check configurations. | | checks.active.type | string | False | http | \[http, https, tcp] | Type of health check connection. | | checks.active.timeout | number | False | 1 | | Health check timeout in seconds. | | checks.active.concurrency | integer | False | 10 | | Number of upstream nodes to be checked at the same time. | | checks.active.host | string | False | | | HTTP host. | | checks.active.port | integer | False | | between 1 and 65535 inclusive | HTTP port. | | checks.active.http\_path | string | False | / | | Path for HTTP probing requests. | | checks.active.https\_verify\_certificate | boolean | False | true | | If true, verify the node's TLS certificate. | | timeout | integer | False | 30000 | greater than or equal to 1 | Request timeout in milliseconds when requesting the LLM service. | | keepalive | boolean | False | true | | If true, keep the connection alive when requesting the LLM service. | | keepalive\_timeout | integer | False | 60000 | greater than or equal to 1000 | Request timeout in milliseconds when requesting the LLM service. | | keepalive\_pool | integer | False | 30 | | Keepalive pool size for when connecting with the LLM service. | | ssl\_verify | boolean | False | true | | If true, verify the LLM service's certificate. | **Sample configuration** Two instances behind a round-robin balancer, one OpenAI and one Anthropic: ```json { "ai-proxy-multi": { "balancer": { "algorithm": "roundrobin" }, "instances": [ { "name": "openai-primary", "provider": "openai", "weight": 1, "auth": { "header": { "Authorization": "Bearer my-openai-api-key" } }, "options": { "model": "gpt-4" } }, { "name": "anthropic-secondary", "provider": "anthropic", "weight": 1, "auth": { "header": { "x-api-key": "my-anthropic-api-key" } }, "options": { "model": "claude-3-5-sonnet" } } ] } } ``` Prioritise one instance and fall back to the next when its token budget is exhausted: ```json { "ai-proxy-multi": { "fallback_strategy": "instance_health_and_rate_limiting", "instances": [ { "name": "openai-primary", "provider": "openai", "priority": 10, "weight": 1, "auth": { "header": { "Authorization": "Bearer my-openai-api-key" } } }, { "name": "openai-backup", "provider": "openai", "priority": 0, "weight": 1, "auth": { "header": { "Authorization": "Bearer my-backup-openai-api-key" } } } ] } } ``` Consistent-hashing balancer with active health checks and request logging: ```json { "ai-proxy-multi": { "balancer": { "algorithm": "chash", "hash_on": "headers", "key": "X-Consumer-Id" }, "checks": { "active": { "type": "http", "http_path": "/health", "timeout": 1, "https_verify_certificate": true } }, "logging": { "summaries": true }, "instances": [ { "name": "openai-a", "provider": "openai", "weight": 2, "auth": { "header": { "Authorization": "Bearer my-openai-api-key" } } }, { "name": "openai-b", "provider": "openai", "weight": 1, "auth": { "header": { "Authorization": "Bearer my-second-openai-api-key" } } } ] } } ```
### `ai-rag` The `ai-rag` Plugin provides Retrieval-Augmented Generation (RAG) capabilities with LLMs. It facilitates the efficient retrieval of relevant documents or information from external data sources, which are used to enhance the LLM responses, thereby improving the accuracy and contextual relevance of the generated outputs. {% hint style="info" %} **Priority** `1060` · **Category** AI · **Required** `embeddings_provider`, `vector_search_provider` {% endhint %}
Configuration and details The Plugin supports using [Azure OpenAI](https://azure.microsoft.com/en-us/products/ai-services/openai-service) and [Azure AI Search](https://azure.microsoft.com/en-us/products/ai-services/ai-search) services for generating embeddings and performing vector search. ***As of now only*** [***Azure OpenAI***](https://azure.microsoft.com/en-us/products/ai-services/openai-service) ***and*** [***Azure AI Search***](https://azure.microsoft.com/en-us/products/ai-services/ai-search) ***services are supported for generating embeddings and performing vector search respectively. PRs for introducing support for other service providers are welcomed.*** | Name | Required | Type | Description | | --------------------------------------------------- | -------- | ------ | ----------------------------------------------------------------------------------------------------------------------------------------- | | embeddings\_provider | True | object | Configurations of the embedding models provider. | | embeddings\_provider.azure\_openai | True | object | Configurations of [Azure OpenAI](https://azure.microsoft.com/en-us/products/ai-services/openai-service) as the embedding models provider. | | embeddings\_provider.azure\_openai.endpoint | True | string | Azure OpenAI embedding model endpoint. | | embeddings\_provider.azure\_openai.api\_key | True | string | Azure OpenAI API key. | | vector\_search\_provider | True | object | Configuration for the vector search provider. | | vector\_search\_provider.azure\_ai\_search | True | object | Configuration for Azure AI Search. | | vector\_search\_provider.azure\_ai\_search.endpoint | True | string | Azure AI Search endpoint. | | vector\_search\_provider.azure\_ai\_search.api\_key | True | string | Azure AI Search API key. | **Sample configuration** Azure OpenAI for embeddings and Azure AI Search for vector retrieval: ```json { "ai-rag": { "embeddings_provider": { "azure_openai": { "endpoint": "https://my-resource.openai.azure.com/openai/deployments/text-embedding-ada-002/embeddings?api-version=2024-02-01", "api_key": "my-azure-openai-api-key" } }, "vector_search_provider": { "azure_ai_search": { "endpoint": "https://my-search.search.windows.net/indexes/docs/docs/search?api-version=2024-07-01", "api_key": "my-azure-ai-search-api-key" } } } } ```
### `ai-rate-limiting` The `ai-rate-limiting` Plugin enforces token-based rate limiting for requests sent to LLM services. It helps manage API usage by controlling the number of tokens consumed within a specified time frame, ensuring fair resource allocation and preventing excessive load on the service. It is often used with `ai-proxy` or `ai-proxy-multi` plugin. {% hint style="info" %} **Priority** `1030` · **Category** AI {% endhint %}
Configuration and details | Name | Type | Required | Default | Valid values | Description | | -------------------------- | -------------- | -------- | ------------- | ---------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------- | | limit | integer | False | | >0 | The maximum number of tokens allowed within a given time interval. At least one of `limit` and `instances.limit` should be configured. | | time\_window | integer | False | | >0 | The time interval corresponding to the rate limiting `limit` in seconds. At least one of `time_window` and `instances.time_window` should be configured. | | show\_limit\_quota\_header | boolean | False | true | | If true, includes `X-AI-RateLimit-Limit-*`, `X-AI-RateLimit-Remaining-*`, and `X-AI-RateLimit-Reset-*` headers in the response, where `*` is the instance name. | | limit\_strategy | string | False | total\_tokens | \[total\_tokens, prompt\_tokens, completion\_tokens] | Type of token to apply rate limiting. `total_tokens` is the sum of `prompt_tokens` and `completion_tokens`. | | instances | array\[object] | False | | | LLM instance rate limiting configurations. | | instances.name | string | True | | | Name of the LLM service instance. | | instances.limit | integer | True | | >0 | The maximum number of tokens allowed within a given time interval for an instance. | | instances.time\_window | integer | True | | >0 | The time interval corresponding to the rate limiting `limit` in seconds for an instance. | | rejected\_code | integer | False | 503 | \[200, 599] | The HTTP status code returned when a request exceeding the quota is rejected. | | rejected\_msg | string | False | | | The response body returned when a request exceeding the quota is rejected. | **Sample configuration** A single token limit shared across all traffic on the route: ```json { "ai-rate-limiting": { "limit": 100000, "time_window": 3600 } } ``` Per-instance limits, each with its own token budget and window: ```json { "ai-rate-limiting": { "instances": [ { "name": "openai-primary", "limit": 100000, "time_window": 3600 }, { "name": "anthropic-secondary", "limit": 50000, "time_window": 3600 } ] } } ``` Limit on prompt tokens only, with a custom rejection code and message: ```json { "ai-rate-limiting": { "limit": 20000, "time_window": 60, "limit_strategy": "prompt_tokens", "rejected_code": 429, "rejected_msg": "Token quota exceeded, please retry later." } } ```
### `ai-request-rewrite` The `ai-request-rewrite` plugin intercepts client requests before they are forwarded to the upstream service. It sends a predefined prompt, along with the original request body, to a specified LLM service. The LLM processes the input and returns a modified request body, which is then used for the upstream request. This allows dynamic transformation of API requests based on AI-generated content. {% hint style="info" %} **Priority** `1073` · **Category** AI · **Required** `prompt`, `provider`, `auth` {% endhint %}
Configuration and details | **Field** | **Required** | **Type** | **Description** | | ------------------------------- | ------------ | -------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | prompt | Yes | String | The prompt send to LLM service. | | provider | Yes | String | Name of the LLM service. Available options: openai, deekseek, azure-openai, aimlapi, anthropic, openrouter, gemini, vertex-ai, and openai-compatible. When `aimlapi` is selected, the plugin uses the OpenAI-compatible driver with a default endpoint of `https://api.aimlapi.com/v1/chat/completions`. | | provider\_conf | No | Object | Configuration for the specific provider. Required when `provider` is set to `vertex-ai` and `override` is not configured. | | provider\_conf.project\_id | Yes | String | Google Cloud Project ID. | | provider\_conf.region | Yes | String | Google Cloud Region. | | auth | Yes | Object | Authentication configuration | | auth.header | No | Object | Authentication headers. Key must match pattern `^[a-zA-Z0-9._-]+$`. | | auth.query | No | Object | Authentication query parameters. Key must match pattern `^[a-zA-Z0-9._-]+$`. | | auth.gcp | No | Object | Configuration for Google Cloud Platform (GCP) authentication. | | auth.gcp.service\_account\_json | No | String | Content of the GCP service account JSON file. This can also be configured by setting the `GCP_SERVICE_ACCOUNT` environment variable. | | auth.gcp.max\_ttl | No | Integer | Maximum TTL (in seconds) for caching the GCP access token. Minimum: 1. | | auth.gcp.expire\_early\_secs | No | Integer | Seconds to expire the access token before its actual expiration time to avoid edge cases. Minimum: 0. Default: 60. | | options | No | Object | Key/value settings for the model | | options.model | No | String | Model to execute. Examples: "gpt-3.5-turbo" for openai, "deepseek-chat" for deekseek, or "qwen-turbo" for openai-compatible or aimlapi services | | override.endpoint | No | String | Override the default endpoint when using OpenAI-compatible services (e.g., self-hosted models or third-party LLM services). When the provider is 'openai-compatible', the endpoint field is required. | | timeout | No | Integer | Total timeout in milliseconds for requests to LLM service, including connect, send, and read timeouts. Range: 1 - 60000. Default: 30000 | | keepalive | No | Boolean | Enable keepalive for requests to LLM service. Default: true | | keepalive\_timeout | No | Integer | Keepalive timeout in milliseconds for requests to LLM service. Minimum: 1000. Default: 60000 | | keepalive\_pool | No | Integer | Keepalive pool size for requests to LLM service. Minimum: 1. Default: 30 | | ssl\_verify | No | Boolean | SSL verification for requests to LLM service. Default: true | **Sample configuration** Ask OpenAI to redact personal data from the request body before it reaches the upstream: ```json { "ai-request-rewrite": { "prompt": "Remove all personally identifiable information from the request body and return only the modified JSON.", "provider": "openai", "auth": { "header": { "Authorization": "Bearer my-openai-api-key" } }, "options": { "model": "gpt-3.5-turbo" } } } ``` Fuller config using Anthropic with a query-parameter key, timeout, and TLS verification: ```json { "ai-request-rewrite": { "prompt": "Rewrite the request body to use formal English.", "provider": "anthropic", "auth": { "query": { "api_key": "my-anthropic-api-key" } }, "options": { "model": "claude-3-5-sonnet" }, "timeout": 30000, "ssl_verify": true } } ```
--- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation by asking a question. Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters: ``` GET https://docs.digitalapi.ai/api-gateway/plugin-reference/plugins-ai.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.