> For the complete documentation index, see [llms.txt](https://docs.digitalapi.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.digitalapi.ai/api-gateway/plugin-reference/plugins-observability.md).
# Observability plugins
Plugins in this category emit signal about every request: access logs, structured metrics, traces, audit events. Attach them once per environment (typically global scope) and route the output to your existing log aggregator, metrics backend, or tracing system. The catalogue ships 20 Observability plugins.
{% hint style="info" %}
**In this category:** [`clickhouse-logger`](#clickhouse-logger), [`datadog`](#datadog), [`elasticsearch-logger`](#elasticsearch-logger), [`file-logger`](#file-logger), [`google-cloud-logging`](#google-cloud-logging), [`http-logger`](#http-logger), [`kafka-logger`](#kafka-logger), [`lago`](#lago), [`loggly`](#loggly), [`loki-logger`](#loki-logger), [`prometheus`](#prometheus), [`rocketmq-logger`](#rocketmq-logger), [`skywalking-logger`](#skywalking-logger), [`sls-logger`](#sls-logger), [`splunk-hec-logging`](#splunk-hec-logging), [`syslog`](#syslog), [`tcp-logger`](#tcp-logger), [`tencent-cloud-cls`](#tencent-cloud-cls), [`udp-logger`](#udp-logger), [`zipkin`](#zipkin)
{% endhint %}
### `clickhouse-logger`
The `clickhouse-logger` Plugin is used to push logs to [ClickHouse](https://clickhouse.com/) database.
{% hint style="info" %}
**Priority** `398` · **Category** Observability
{% endhint %}
Configuration and details
| Name | Type | Required | Default | Valid values | Description |
| ------------------------- | ---------- | -------- | ------------------- | -------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| endpoint\_addr | Deprecated | True | | | Use `endpoint_addrs` instead. ClickHouse endpoints. |
| endpoint\_addrs | array | True | | | ClickHouse endpoints. |
| database | string | True | | | Name of the database to store the logs. |
| logtable | string | True | | | Table name to store the logs. |
| user | string | True | | | ClickHouse username. |
| password | string | True | | | ClickHouse password. |
| timeout | integer | False | 3 | \[1,...] | Time to keep the connection alive for after sending a request. |
| name | string | False | "clickhouse logger" | | Unique identifier for the logger. If you use Prometheus to monitor AILIX metrics, the name is exported in `ailix_batch_process_entries`. |
| ssl\_verify | boolean | False | true | \[true,false] | When set to `true`, verifies SSL. |
| log\_format | object | False | | | Log format declared as key-value pairs in JSON. Values support strings and nested objects (up to five levels deep; deeper fields are truncated). Within strings, AILIX or [NGINX](http://nginx.org/en/docs/varindex.html) variables can be referenced by prefixing with `$`. |
| include\_req\_body | boolean | False | false | \[false, true] | When set to `true` includes the request body in the log. If the request body is too big to be kept in the memory, it can't be logged due to Nginx's limitations. |
| include\_req\_body\_expr | array | False | | | Filter for when the `include_req_body` attribute is set to `true`. Request body is only logged when the expression set here evaluates to `true`. See lua-resty-expr for more. |
| include\_resp\_body | boolean | False | false | \[false, true] | When set to `true` includes the response body in the log. |
| include\_resp\_body\_expr | array | False | | | Filter for when the `include_resp_body` attribute is set to `true`. Response body is only logged when the expression set here evaluates to `true`. See lua-resty-expr for more. |
### `datadog`
The `datadog` monitoring Plugin is for seamless integration of AILIX with [Datadog](https://www.datadoghq.com/), one of the most used monitoring and observability platform for cloud applications.
{% hint style="info" %}
**Priority** `495` · **Category** Observability
{% endhint %}
Configuration and details
When enabled, the Plugin supports multiple metric capture types for request and response cycles.
This Plugin, pushes its custom metrics to the [DogStatsD](https://docs.datadoghq.com/developers/dogstatsd/?tab=hostagent) server over UDP protocol and comes bundled with [Datadog agent](https://docs.datadoghq.com/agent/).
DogStatsD implements the StatsD protocol which collects the custom metrics for the AILIX agent, aggregates them into a single data point, and sends it to the configured Datadog server.
This Plugin provides the ability to push metrics as a batch to the external Datadog agent, reusing the same datagram socket. It might take some time to receive the log data. It will be automatically sent after the timer function in the batch processor expires.
| Name | Type | Required | Default | Valid values | Description |
| --------------- | ------- | -------- | ------- | ------------- | ---------------------------------------------------------------------------------------------------------------- |
| prefer\_name | boolean | False | true | \[true,false] | When set to `false`, uses Route/Service ID instead of name (default) with metric tags. |
| include\_path | boolean | False | false | \[true,false] | When set to `true`, includes the path pattern in metric tags. |
| include\_method | boolean | False | false | \[true,false] | When set to `true`, includes the HTTP method in metric tags. |
| constant\_tags | array | False | \[] | | Static tags to embed into all metrics generated by this route. Useful for grouping metrics over certain signals. |
### `elasticsearch-logger`
The `elasticsearch-logger` Plugin pushes request and response logs in batches to [Elasticsearch](https://www.elastic.co) and supports the customization of log formats. When enabled, the Plugin will serialize the request context information to [Elasticsearch Bulk format](https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-bulk.html#docs-bulk) and add them to the queue, before they are pushed to Elasticsearch. See batch processor for more details.
{% hint style="info" %}
**Priority** `413` · **Category** Observability
{% endhint %}
Configuration and details
| Name | Type | Required | Default | Description |
| ------------------------- | -------------- | -------- | ------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| endpoint\_addrs | array\[string] | True | | Elasticsearch API endpoint addresses. If multiple endpoints are configured, they will be written randomly. |
| field | object | True | | Elasticsearch `field` configuration. |
| field.index | string | True | | Elasticsearch [\_index field](https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-index-field.html#mapping-index-field). |
| log\_format | object | False | | Custom log format as key-value pairs in JSON. Values support strings and nested objects (up to five levels deep; deeper fields are truncated). Within strings, AILIX or [NGINX variables](http://nginx.org/en/docs/varindex.html) can be referenced by prefixing with `$`. |
| auth | array | False | | Elasticsearch [authentication](https://www.elastic.co/guide/en/elasticsearch/reference/current/setting-up-authentication.html) configuration. |
| auth.username | string | True | | Elasticsearch [authentication](https://www.elastic.co/guide/en/elasticsearch/reference/current/setting-up-authentication.html) username. |
| auth.password | string | True | | Elasticsearch [authentication](https://www.elastic.co/guide/en/elasticsearch/reference/current/setting-up-authentication.html) password. |
| ssl\_verify | boolean | False | true | If true, perform SSL verification. |
| timeout | integer | False | 10 | Elasticsearch send data timeout in seconds. |
| include\_req\_body | boolean | False | false | If true, include the request body in the log. Note that if the request body is too big to be kept in the memory, it can not be logged due to NGINX's limitations. |
| include\_req\_body\_expr | array\[array] | False | | An array of one or more conditions in the form of lua-resty-expr. Used when the `include_req_body` is true. Request body would only be logged when the expressions configured here evaluate to true. |
| include\_resp\_body | boolean | False | false | If true, include the response body in the log. |
| include\_resp\_body\_expr | array\[array] | False | | An array of one or more conditions in the form of lua-resty-expr. Used when the `include_resp_body` is true. Response body would only be logged when the expressions configured here evaluate to true. |
### `file-logger`
The `file-logger` Plugin is used to push log streams to a specific location.
{% hint style="info" %}
**Priority** `399` · **Category** Observability
{% endhint %}
Configuration and details
:::tip
* `file-logger` plugin can count request and response data for individual routes locally, which is useful for debugging.
* `file-logger` plugin can get AILIX variables and [NGINX variables](http://nginx.org/en/docs/varindex.html), while `access.log` can only use NGINX variables.
* `file-logger` plugin support hot-loaded so that we can change its configuration at any time with immediate effect.
* `file-logger` plugin saves every data in JSON format.
* The user can modify the functions executed by the `file-logger` during the `log phase` to collect the information they want.
:::
| Name | Type | Required | Description |
| ------------------------- | ------------- | -------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| path | string | True | Log file path. |
| log\_format | object | False | Log format declared as key-value pairs in JSON. Values support strings and nested objects (up to five levels deep; deeper fields are truncated). Within strings, AILIX or [NGINX](http://nginx.org/en/docs/varindex.html) variables can be referenced by prefixing with `$`. |
| include\_req\_body | boolean | False | When set to `true` includes the request body in the log. If the request body is too big to be kept in the memory, it can't be logged due to Nginx's limitations. |
| include\_req\_body\_expr | array | False | Filter for when the `include_req_body` attribute is set to `true`. Request body is only logged when the expression set here evaluates to `true`. See lua-resty-expr for more. |
| include\_resp\_body | boolean | False | When set to `true` includes the response body in the log file. |
| include\_resp\_body\_expr | array | False | When the `include_resp_body` attribute is set to `true`, use this to filter based on lua-resty-expr. If present, only logs the response into file if the expression evaluates to `true`. |
| match | array\[array] | False | Logs will be recorded when the rule matching is successful if the option is set. See lua-resty-expr for a list of available expressions. |
### `google-cloud-logging`
The `google-cloud-logging` Plugin is used to send AILIX access logs to [Google Cloud Logging Service](https://cloud.google.com/logging/).
{% hint style="info" %}
**Priority** `407` · **Category** Observability
{% endhint %}
Configuration and details
This plugin also allows to push logs as a batch to your Google Cloud Logging Service. It might take some time to receive the log data. It will be automatically sent after the timer function in the batch processor expires.
| Name | Required | Default | Description |
| -------------------------- | ---------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| auth\_config | True | | Either `auth_config` or `auth_file` must be provided. |
| auth\_config.client\_email | True | | Email address of the Google Cloud service account. |
| auth\_config.private\_key | True | | Private key of the Google Cloud service account. |
| auth\_config.project\_id | True | | Project ID in the Google Cloud service account. |
| auth\_config.token\_uri | True | | Token URI of the Google Cloud service account. |
| auth\_config.entries\_uri | False | | Google Cloud Logging Service API. |
| auth\_config.scope | False | \["", "", "", ""] | Access scopes of the Google Cloud service account. See [OAuth 2.0 Scopes for Google APIs](https://developers.google.com/identity/protocols/oauth2/scopes#logging). |
| auth\_config.scopes | Deprecated | \["", "", "", ""] | Access scopes of the Google Cloud service account. Use `auth_config.scope` instead. |
| auth\_file | True | | Path to the Google Cloud service account authentication JSON file. Either `auth_config` or `auth_file` must be provided. |
| ssl\_verify | False | true | When set to `true`, enables SSL verification as mentioned in [OpenResty docs](https://github.com/openresty/lua-nginx-module#tcpsocksslhandshake). |
| resource | False | {"type": "global"} | Google monitor resource. See [MonitoredResource](https://cloud.google.com/logging/docs/reference/v2/rest/v2/MonitoredResource) for more details. |
| log\_id | False | ailix | Google Cloud logging ID. See [LogEntry](https://cloud.google.com/logging/docs/reference/v2/rest/v2/LogEntry) for details. |
| log\_format | False | | Log format declared as key-value pairs in JSON. Values support strings and nested objects (up to five levels deep; deeper fields are truncated). Within strings, AILIX or [NGINX](http://nginx.org/en/docs/varindex.html) variables can be referenced by prefixing with `$`. |
### `http-logger`
The `http-logger` Plugin is used to push log data requests to HTTP/HTTPS servers.
{% hint style="info" %}
**Priority** `410` · **Category** Observability · **Required** `uri`
{% endhint %}
Configuration and details
This will allow the ability to send log data requests as JSON objects to monitoring tools and other HTTP servers.
| Name | Type | Required | Default | Valid values | Description |
| ------------------------- | ------- | -------- | ------- | ---------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| uri | string | True | | | URI of the HTTP/HTTPS server. |
| auth\_header | string | False | | | Authorization headers if required. |
| timeout | integer | False | 3 | \[1,...] | Time to keep the connection alive for after sending a request. |
| log\_format | object | False | | | Log format declared as key-value pairs in JSON. Values support strings and nested objects (up to five levels deep; deeper fields are truncated). Within strings, AILIX or [NGINX](http://nginx.org/en/docs/varindex.html) variables can be referenced by prefixing with `$`. |
| include\_req\_body | boolean | False | false | \[false, true] | When set to `true` includes the request body in the log. If the request body is too big to be kept in the memory, it can't be logged due to Nginx's limitations. |
| include\_req\_body\_expr | array | False | | | Filter for when the `include_req_body` attribute is set to `true`. Request body is only logged when the expression set here evaluates to `true`. See lua-resty-expr for more. |
| include\_resp\_body | boolean | False | false | \[false, true] | When set to `true` includes the response body in the log. |
| include\_resp\_body\_expr | array | False | | | When the `include_resp_body` attribute is set to `true`, use this to filter based on lua-resty-expr. If present, only logs the response if the expression evaluates to `true`. |
| concat\_method | string | False | "json" | \["json", "new\_line"] | Sets how to concatenate logs. When set to `json`, uses `json.encode` for all pending logs and when set to `new_line`, also uses `json.encode` but uses the newline (`\n`) to concatenate lines. |
| ssl\_verify | boolean | False | false | \[false, true] | When set to `true` verifies the SSL certificate. |
### `kafka-logger`
The `kafka-logger` Plugin is used to push logs as JSON objects to Apache Kafka clusters. It works as a Kafka client driver for the ngx\_lua Nginx module.
{% hint style="info" %}
**Priority** `403` · **Category** Observability
{% endhint %}
Configuration and details
It might take some time to receive the log data. It will be automatically sent after the timer function in the batch processor expires.
| Name | Type | Required | Default | Valid values | Description |
| ------------------------------ | ------- | -------- | -------------- | -------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| broker\_list | object | True | | | Deprecated, use `brokers` instead. List of Kafka brokers. (nodes). |
| brokers | array | True | | | List of Kafka brokers (nodes). |
| brokers.host | string | True | | | The host of Kafka broker, e.g, `192.168.1.1`. |
| brokers.port | integer | True | | \[0, 65535] | The port of Kafka broker |
| brokers.sasl\_config | object | False | | | The sasl config of Kafka broker |
| brokers.sasl\_config.mechanism | string | False | "PLAIN" | \["PLAIN", "SCRAM-SHA-256", "SCRAM-SHA-512"] | The mechaism of sasl config |
| brokers.sasl\_config.user | string | True | | | The user of sasl\_config. If sasl\_config exists, it's required. |
| brokers.sasl\_config.password | string | True | | | The password of sasl\_config. If sasl\_config exists, it's required. |
| kafka\_topic | string | True | | | Target topic to push the logs for organisation. |
| producer\_type | string | False | async | \["async", "sync"] | Message sending mode of the producer. |
| required\_acks | integer | False | 1 | \[1, -1] | Number of acknowledgements the leader needs to receive for the producer to consider the request complete. This controls the durability of the sent records. The attribute follows the same configuration as the Kafka `acks` attribute. `required_acks` cannot be 0. See Apache Kafka documentation for more. |
| key | string | False | | | Key used for allocating partitions for messages. |
| timeout | integer | False | 3 | \[1,...] | Timeout for the upstream to send data. |
| name | string | False | "kafka logger" | | Unique identifier for the batch processor. If you use Prometheus to monitor AILIX metrics, the name is exported in `ailix_batch_process_entries`. |
| meta\_format | enum | False | "default" | \["default"，"origin"] | Format to collect the request information. Setting to `default` collects the information in JSON format and `origin` collects the information with the original HTTP request. See [examples](#meta_format-example) below. |
| log\_format | object | False | | | Log format declared as key-value pairs in JSON. Values support strings and nested objects (up to five levels deep; deeper fields are truncated). Within strings, AILIX or [NGINX](http://nginx.org/en/docs/varindex.html) variables can be referenced by prefixing with `$`. |
| include\_req\_body | boolean | False | false | \[false, true] | When set to `true` includes the request body in the log. If the request body is too big to be kept in the memory, it can't be logged due to Nginx's limitations. |
| include\_req\_body\_expr | array | False | | | Filter for when the `include_req_body` attribute is set to `true`. Request body is only logged when the expression set here evaluates to `true`. See lua-resty-expr for more. |
| max\_req\_body\_bytes | integer | False | 524288 | >=1 | Maximum request body allowed in bytes. Request bodies falling within this limit will be pushed to Kafka. If the size exceeds the configured value, the body will be truncated before being pushed to Kafka. |
| include\_resp\_body | boolean | False | false | \[false, true] | When set to `true` includes the response body in the log. |
| include\_resp\_body\_expr | array | False | | | Filter for when the `include_resp_body` attribute is set to `true`. Response body is only logged when the expression set here evaluates to `true`. See lua-resty-expr for more. |
| max\_resp\_body\_bytes | integer | False | 524288 | >=1 | Maximum response body allowed in bytes. Response bodies falling within this limit will be pushed to Kafka. If the size exceeds the configured value, the body will be truncated before being pushed to Kafka. |
| cluster\_name | integer | False | 1 | \[0,...] | Name of the cluster. Used when there are two or more Kafka clusters. Only works if the `producer_type` attribute is set to `async`. |
| producer\_batch\_num | integer | optional | 200 | \[1,...] | `batch_num` parameter in [lua-resty-kafka](https://github.com/doujiang24/lua-resty-kafka). The merge message and batch is send to the server. Unit is message count. |
| producer\_batch\_size | integer | optional | 1048576 | \[0,...] | `batch_size` parameter in [lua-resty-kafka](https://github.com/doujiang24/lua-resty-kafka) in bytes. |
| producer\_max\_buffering | integer | optional | 50000 | \[1,...] | `max_buffering` parameter in [lua-resty-kafka](https://github.com/doujiang24/lua-resty-kafka) representing maximum buffer size. Unit is message count. |
| producer\_time\_linger | integer | optional | 1 | \[1,...] | `flush_time` parameter in [lua-resty-kafka](https://github.com/doujiang24/lua-resty-kafka) in seconds. |
| meta\_refresh\_interval | integer | optional | 30 | \[1,...] | `refresh_interval` parameter in [lua-resty-kafka](https://github.com/doujiang24/lua-resty-kafka) specifies the time to auto refresh the metadata, in seconds. |
### `lago`
The `lago` plugin pushes requests and responses to [Lago Self-hosted](https://github.com/getlago/lago) and [Lago Cloud](https://getlago.com) via the Lago REST API. the plugin allows you to use it with a variety of AILIX built-in features, such as the AILIX consumer and the request-id plugin.
{% hint style="info" %}
**Priority** `415` · **Category** Observability · **Required** `endpoint_addrs`, `token`, `event_transaction_id`, `event_subscription_id`, `event_code`
{% endhint %}
Configuration and details
This allows for API monetization or let AILIX to be an AI gateway for AI tokens billing scenarios.
:::note disclaimer
Lago owns its trademarks and controls its commercial products and open source projects.
The project uses the `AGPL-3.0` license instead of the `Apache-2.0` license that is the same as AILIX. As a user, you will need to evaluate for yourself whether it is applicable to your business to use the project in a compliant way or to obtain another type of license from Lago. AILIX community does not endorse it.
The plugin does not contain any proprietary code or SDKs from Lago, it is contributed by contributors to AILIX and licensed under the `Apache-2.0` license, which is in line with any other part of AILIX and you don't need to worry about its compliance.
:::
When enabled, the plugin will collect information from the request context (e.g. event code, transaction ID, associated subscription ID) as configured and serialize them into [Event JSON objects](https://getlago.com/docs/api-reference/events/event-object) as required by Lago. They will be added to the buffer and sent to Lago in batches of up to 100. This batch size is a [requirement](https://getlago.com/docs/api-reference/events/batch) from Lago. If you want to modify it, see batch processor for more details.
| Name | Type | Required | Default | Valid values | Description |
| ----------------------- | -------------- | -------- | -------------------- | ------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| endpoint\_addrs | array\[string] | True | | | Lago API address, such as `http://127.0.0.1:3000`. It supports both self-hosted Lago and Lago Cloud. If multiple endpoints are configured, the log will be pushed to a randomly selected endpoint from the list. |
| endpoint\_uri | string | False | /api/v1/events/batch | | Lago API endpoint for [batch usage events](https://docs.getlago.com/api-reference/events/batch). |
| token | string | True | | | Lago API key created in the Lago dashboard. |
| event\_transaction\_id | string | True | | | Event's transaction ID, used to identify and de-duplicate the event. It supports string templates containing AILIX and NGINX variables, such as `req_${request_id}`, which allows you to use values returned by upstream services or the `request-id` plugin. |
| event\_subscription\_id | string | True | | | Event's subscription ID, which is automatically generated or configured when you assign the plan to the customer on Lago. This is used to associate API consumption to a customer subscription and supports string templates containing AILIX and NGINX variables, such as `cus_${consumer_name}`, which allows you to use values returned by upstream services or AILIX consumer. |
| event\_code | string | True | | | Lago billable metric's code for associating an event to a specified billable item. |
| event\_properties | object | False | | | Event's properties, used to attach information to an event. This allows you to send certain information on an event to Lago, such as the HTTP status to exclude failed requests from billing, or the AI token consumption in the response body for accurate billing. The keys are fixed strings, while the values can be string templates containing AILIX and NGINX variables, such as `${status}`. |
| ssl\_verify | boolean | False | true | | If true, verify Lago's SSL certificates. |
| timeout | integer | False | 3000 | \[1, 60000] | Timeout for the Lago service HTTP call in milliseconds. |
| keepalive | boolean | False | true | | If true, keep the connection alive for multiple requests. |
| keepalive\_timeout | integer | False | 60000 | >=1000 | Keepalive timeout in milliseconds. |
| keepalive\_pool | integer | False | 5 | >=1 | Maximum number of connections in the connection pool. |
### `loggly`
The `loggly` Plugin is used to forward logs to [SolarWinds Loggly](https://www.solarwinds.com/loggly) for analysis and storage.
{% hint style="info" %}
**Priority** `411` · **Category** Observability · **Required** `customer_token`
{% endhint %}
Configuration and details
When the Plugin is enabled, AILIX will serialize the request context information to [Loggly Syslog](https://documentation.solarwinds.com/en/success_center/loggly/content/admin/streaming-syslog-without-using-files.htm?cshid=loggly_streaming-syslog-without-using-files) data format which is Syslog events with [RFC5424](https://datatracker.ietf.org/doc/html/rfc5424) compliant headers.
When the maximum batch size is exceeded, the data in the queue is pushed to Loggly enterprise syslog endpoint. See batch processor for more details.
| Name | Type | Required | Default | Description |
| ------------------------- | ------------- | -------- | -------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| customer\_token | string | True | | Unique identifier used when sending logs to Loggly to ensure that they are sent to the right organisation account. |
| severity | string (enum) | False | INFO | Syslog log event severity level. Choose between: `DEBUG`, `INFO`, `NOTICE`, `WARNING`, `ERR`, `CRIT`, `ALERT`, and `EMEGR`. |
| severity\_map | object | False | nil | A way to map upstream HTTP response codes to Syslog severity. Key-value pairs where keys are the HTTP response codes and the values are the Syslog severity levels. For example `{"410": "CRIT"}`. |
| tags | array | False | | Metadata to be included with any event log to aid in segmentation and filtering. |
| log\_format | object | False | {"host": "$host", "@timestamp": "$time\_iso8601", "client\_ip": "$remote\_addr"} | Log format declared as key-value pairs in JSON. Values support strings and nested objects (up to five levels deep; deeper fields are truncated). Within strings, AILIX or [NGINX](http://nginx.org/en/docs/varindex.html) variables can be referenced by prefixing with `$`. |
| include\_req\_body | boolean | False | false | When set to `true` includes the request body in the log. If the request body is too big to be kept in the memory, it can't be logged due to Nginx's limitations. |
| include\_req\_body\_expr | array | False | | Filter for when the `include_req_body` attribute is set to `true`. Request body is only logged when the expression set here evaluates to `true`. See lua-resty-expr for more. |
| include\_resp\_body | boolean | False | false | When set to `true` includes the response body in the log. |
| include\_resp\_body\_expr | array | False | | When the `include_resp_body` attribute is set to `true`, use this to filter based on lua-resty-expr. If present, only logs the response if the expression evaluates to `true`. |
### `loki-logger`
The `loki-logger` Plugin pushes request and response logs in batches to [Grafana Loki](https://grafana.com/oss/loki/), via the [Loki HTTP API](https://grafana.com/docs/loki/latest/reference/loki-http-api/#loki-http-api) `/loki/api/v1/push`. The Plugin also supports the customization of log formats.
{% hint style="info" %}
**Priority** `414` · **Category** Observability · **Required** `endpoint_addrs`
{% endhint %}
Configuration and details
When enabled, the Plugin will serialize the request context information to [JSON objects](https://grafana.com/docs/loki/latest/api/#push-log-entries-to-loki) and add them to the queue, before they are pushed to Loki. See batch processor for more details.
| Name | Type | Required | Default | Valid values | Description |
| ------------------------- | -------------- | -------- | ----------------- | ------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| endpoint\_addrs | array\[string] | True | | | Loki API base URLs, such as `http://127.0.0.1:3100`. If multiple endpoints are configured, the log will be pushed to a randomly determined endpoint from the list. |
| endpoint\_uri | string | False | /loki/api/v1/push | | URI path to the Loki ingest endpoint. |
| tenant\_id | string | False | fake | | Loki tenant ID. According to Loki's [multi-tenancy documentation](https://grafana.com/docs/loki/latest/operations/multi-tenancy/#multi-tenancy), the default value is set to `fake` under single-tenancy. |
| headers | object | False | | | Key-value pairs of request headers (settings for `X-Scope-OrgID` and `Content-Type` will be ignored). |
| log\_labels | object | False | {job = "ailix"} | | Loki log label. Support [NGINX variables](https://nginx.org/en/docs/varindex.html) and constant strings in values. Variables should be prefixed with a `$` sign. For example, the label can be `{"origin" = "ailix"}` or `{"origin" = "$remote_addr"}`. |
| ssl\_verify | boolean | False | true | | If true, verify Loki's SSL certificates. |
| timeout | integer | False | 3000 | \[1, 60000] | Timeout for the Loki service HTTP call in milliseconds. |
| keepalive | boolean | False | true | | If true, keep the connection alive for multiple requests. |
| keepalive\_timeout | integer | False | 60000 | >=1000 | Keepalive timeout in milliseconds. |
| keepalive\_pool | integer | False | 5 | >=1 | Maximum number of connections in the connection pool. |
| log\_format | object | False | | | Custom log format as key-value pairs in JSON. Values support strings and nested objects (up to five levels deep; deeper fields are truncated). Within strings, AILIX variables and [NGINX variables](http://nginx.org/en/docs/varindex.html) can be referenced by prefixing with `$`. |
| name | string | False | loki-logger | | Unique identifier of the Plugin for the batch processor. If you use Prometheus to monitor AILIX metrics, the name is exported in `ailix_batch_process_entries`. |
| include\_req\_body | boolean | False | false | | If true, include the request body in the log. Note that if the request body is too big to be kept in the memory, it can not be logged due to NGINX's limitations. |
| include\_req\_body\_expr | array\[array] | False | | | An array of one or more conditions in the form of lua-resty-expr. Used when the `include_req_body` is true. Request body would only be logged when the expressions configured here evaluate to true. |
| include\_resp\_body | boolean | False | false | | If true, include the response body in the log. |
| include\_resp\_body\_expr | array\[array] | False | | | An array of one or more conditions in the form of lua-resty-expr. Used when the `include_resp_body` is true. Response body would only be logged when the expressions configured here evaluate to true. |
### `prometheus`
The `prometheus` Plugin provides the capability to integrate AILIX with [Prometheus](https://prometheus.io).
{% hint style="info" %}
**Priority** `500` · **Category** Observability
{% endhint %}
Configuration and details
After enabling the Plugin, AILIX will start collecting relevant metrics, such as API requests and latencies, and exporting them in a [text-based exposition format](https://prometheus.io/docs/instrumenting/exposition_formats/#exposition-formats) to Prometheus. You can then create event monitoring and alerting in Prometheus to monitor the health of your API gateway and APIs.
### `rocketmq-logger`
The `rocketmq-logger` Plugin provides the ability to push logs as JSON objects to your RocketMQ clusters.
{% hint style="info" %}
**Priority** `402` · **Category** Observability · **Required** `nameserver_list`, `topic`
{% endhint %}
Configuration and details
It might take some time to receive the log data. It will be automatically sent after the timer function in the batch processor expires.
| Name | Type | Required | Default | Valid values | Description |
| ------------------------- | ------- | -------- | ----------------- | --------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| nameserver\_list | object | True | | | List of RocketMQ nameservers. |
| topic | string | True | | | Target topic to push the data to. |
| key | string | False | | | Key of the messages. |
| tag | string | False | | | Tag of the messages. |
| log\_format | object | False | | | Log format declared as key-value pairs in JSON. Values support strings and nested objects (up to five levels deep; deeper fields are truncated). Within strings, AILIX or [NGINX](http://nginx.org/en/docs/varindex.html) variables can be referenced by prefixing with `$`. |
| timeout | integer | False | 3 | \[1,...] | Timeout for the upstream to send data. |
| use\_tls | boolean | False | false | | When set to `true`, uses TLS. |
| access\_key | string | False | "" | | Access key for ACL. Setting to an empty string will disable the ACL. |
| secret\_key | string | False | "" | | secret key for ACL. |
| name | string | False | "rocketmq logger" | | Unique identifier for the batch processor. If you use Prometheus to monitor AILIX metrics, the name is exported in `ailix_batch_process_entries`. processor. |
| meta\_format | enum | False | "default" | \["default"，"origin"] | Format to collect the request information. Setting to `default` collects the information in JSON format and `origin` collects the information with the original HTTP request. See [examples](#meta_format-example) below. |
| include\_req\_body | boolean | False | false | \[false, true] | When set to `true` includes the request body in the log. If the request body is too big to be kept in the memory, it can't be logged due to Nginx's limitations. |
| include\_req\_body\_expr | array | False | | | Filter for when the `include_req_body` attribute is set to `true`. Request body is only logged when the expression set here evaluates to `true`. See lua-resty-expr for more. |
| include\_resp\_body | boolean | False | false | \[false, true] | When set to `true` includes the response body in the log. |
| include\_resp\_body\_expr | array | False | | | Filter for when the `include_resp_body` attribute is set to `true`. Response body is only logged when the expression set here evaluates to `true`. See lua-resty-expr for more. |
### `skywalking-logger`
The `skywalking-logger` Plugin pushes request and response logs as JSON objects to SkyWalking OAP server in batches and supports the customization of log formats.
{% hint style="info" %}
**Priority** `408` · **Category** Observability · **Required** `endpoint_addr`
{% endhint %}
Configuration and details
If there is an existing tracing context, it sets up the trace-log correlation automatically and relies on SkyWalking Cross Process Propagation Headers Protocol.
| Name | Type | Required | Default | Valid values | Description |
| ------------------------- | ------------- | -------- | --------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------ |
| endpoint\_addr | string | True | | | URI of the SkyWalking OAP server. |
| service\_name | string | False | "AILIX" | | Service name for the SkyWalking reporter. |
| service\_instance\_name | string | False | "AILIX Instance Name" | | Service instance name for the SkyWalking reporter. Set it to `$hostname` to directly get the local hostname. |
| log\_format | object | False | | Custom log format as key-value pairs in JSON. Values support strings and nested objects (up to five levels deep; deeper fields are truncated). Within strings, AILIX or [NGINX variables](http://nginx.org/en/docs/varindex.html) can be referenced by prefixing with `$`. | |
| timeout | integer | False | 3 | \[1,...] | Time to keep the connection alive for after sending a request. |
| name | string | False | "skywalking logger" | | Unique identifier to identify the logger. If you use Prometheus to monitor AILIX metrics, the name is exported in `ailix_batch_process_entries`. |
| include\_req\_body | boolean | False | false | If true, include the request body in the log. Note that if the request body is too big to be kept in the memory, it can not be logged due to NGINX's limitations. | |
| include\_req\_body\_expr | array\[array] | False | | An array of one or more conditions in the form of lua-resty-expr. Used when the `include_req_body` is true. Request body would only be logged when the expressions configured here evaluate to true. | |
| include\_resp\_body | boolean | False | false | If true, include the response body in the log. | |
| include\_resp\_body\_expr | array\[array] | False | | An array of one or more conditions in the form of lua-resty-expr. Used when the `include_resp_body` is true. Response body would only be logged when the expressions configured here evaluate to true. | |
### `sls-logger`
The `sls-logger` Plugin is used to push logs to [Alibaba Cloud log Service](https://www.alibabacloud.com/help/en/log-service/latest/use-the-syslog-protocol-to-upload-logs) using [RF5424](https://tools.ietf.org/html/rfc5424).
{% hint style="info" %}
**Priority** `406` · **Category** Observability · **Required** `host`, `port`, `project`, `logstore`, `access_key_id`, `access_key_secret`
{% endhint %}
Configuration and details
It might take some time to receive the log data. It will be automatically sent after the timer function in the batch processor expires.
| Name | Required | Description |
| ------------------------- | -------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| host | True | IP address or the hostname of the TCP server. See [Alibaba Cloud log service documentation](https://www.alibabacloud.com/help/en/log-service/latest/endpoints) for details. Use IP address instead of domain. |
| port | True | Target upstream port. Defaults to `10009`. |
| timeout | False | Timeout for the upstream to send data. |
| log\_format | False | Log format declared as key-value pairs in JSON. Values support strings and nested objects (up to five levels deep; deeper fields are truncated). Within strings, AILIX or [NGINX](http://nginx.org/en/docs/varindex.html) variables can be referenced by prefixing with `$`. |
| project | True | Project name in Alibaba Cloud log service. Create SLS before using this Plugin. |
| logstore | True | logstore name in Ali Cloud log service. Create SLS before using this Plugin. |
| access\_key\_id | True | AccessKey ID in Alibaba Cloud. See [Authorization](https://www.alibabacloud.com/help/en/log-service/latest/create-a-ram-user-and-authorize-the-ram-user-to-access-log-service) for more details. |
| access\_key\_secret | True | AccessKey Secret in Alibaba Cloud. See [Authorization](https://www.alibabacloud.com/help/en/log-service/latest/create-a-ram-user-and-authorize-the-ram-user-to-access-log-service) for more details. |
| include\_req\_body | True | When set to `true`, includes the request body in the log. |
| include\_req\_body\_expr | No | Filter for when the `include_req_body` attribute is set to `true`. Request body is only logged when the expression set here evaluates to `true`. See lua-resty-expr for more. |
| include\_resp\_body | No | When set to `true` includes the response body in the log. |
| include\_resp\_body\_expr | No | Filter for when the `include_resp_body` attribute is set to `true`. Response body is only logged when the expression set here evaluates to `true`. See lua-resty-expr for more. |
| name | False | Unique identifier for the batch processor. If you use Prometheus to monitor AILIX metrics, the name is exported in `ailix_batch_process_entries`. |
### `splunk-hec-logging`
The `splunk-hec-logging` Plugin is used to forward logs to [Splunk HTTP Event Collector (HEC)](https://docs.splunk.com/Documentation/Splunk/8.2.6/Data/UsetheHTTPEventCollector) for analysis and storage.
{% hint style="info" %}
**Priority** `409` · **Category** Observability · **Required** `endpoint`
{% endhint %}
Configuration and details
When the Plugin is enabled, AILIX will serialize the request context information to [Splunk Event Data format](https://docs.splunk.com/Documentation/Splunk/latest/Data/FormateventsforHTTPEventCollector#Event_metadata) and submit it to the batch queue. When the maximum batch size is exceeded, the data in the queue is pushed to Splunk HEC. See batch processor for more details.
| Name | Required | Default | Description |
| --------------------------- | -------- | ------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| endpoint | True | | Splunk HEC endpoint configurations. |
| endpoint.uri | True | | Splunk HEC event collector API endpoint. |
| endpoint.token | True | | Splunk HEC authentication token. |
| endpoint.channel | False | | Splunk HEC send data channel identifier. Read more: [About HTTP Event Collector Indexer Acknowledgment](https://docs.splunk.com/Documentation/Splunk/8.2.3/Data/AboutHECIDXAck). |
| endpoint.timeout | False | 10 | Splunk HEC send data timeout in seconds. |
| endpoint.keepalive\_timeout | False | 60000 | Keepalive timeout in milliseconds. |
| ssl\_verify | False | true | When set to `true` enables SSL verification as per [OpenResty docs](https://github.com/openresty/lua-nginx-module#tcpsocksslhandshake). |
| log\_format | False | | Log format declared as key-value pairs in JSON. Values support strings and nested objects (up to five levels deep; deeper fields are truncated). Within strings, AILIX or [NGINX](http://nginx.org/en/docs/varindex.html) variables can be referenced by prefixing with `$`. |
### `syslog`
The `syslog` Plugin is used to push logs to a Syslog server.
{% hint style="info" %}
**Priority** `401` · **Category** Observability · **Required** `host`, `port`
{% endhint %}
Configuration and details
Logs can be set as JSON objects.
| Name | Type | Required | Default | Valid values | Description |
| ------------------------- | ------- | -------- | ------------ | -------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| host | string | True | | | IP address or the hostname of the Syslog server. |
| port | integer | True | | | Target port of the Syslog server. |
| name | string | False | "sys logger" | | Identifier for the server. If you use Prometheus to monitor AILIX metrics, the name is exported in `ailix_batch_process_entries`. |
| timeout | integer | False | 3000 | \[1, ...] | Timeout in ms for the upstream to send data. |
| tls | boolean | False | false | | When set to `true` performs TLS verification. |
| flush\_limit | integer | False | 4096 | \[1, ...] | Maximum size of the buffer (KB) and the current message before it is flushed and written to the server. |
| drop\_limit | integer | False | 1048576 | | Maximum size of the buffer (KB) and the current message before the current message is dropped because of the size limit. |
| sock\_type | string | False | "tcp" | \["tcp", "udp] | Transport layer protocol to use. |
| pool\_size | integer | False | 5 | \[5, ...] | Keep-alive pool size used by `sock:keepalive`. |
| log\_format | object | False | | | Log format declared as key-value pairs in JSON. Values support strings and nested objects (up to five levels deep; deeper fields are truncated). Within strings, AILIX or [NGINX](http://nginx.org/en/docs/varindex.html) variables can be referenced by prefixing with `$`. |
| include\_req\_body | boolean | False | false | \[false, true] | When set to `true` includes the request body in the log. |
| include\_req\_body\_expr | array | False | | | Filter for when the `include_req_body` attribute is set to `true`. Request body is only logged when the expression set here evaluates to `true`. See lua-resty-expr for more. |
| include\_resp\_body | boolean | False | false | \[false, true] | When set to `true` includes the response body in the log. |
| include\_resp\_body\_expr | array | False | | | When the `include_resp_body` attribute is set to `true`, use this to filter based on lua-resty-expr. If present, only logs the response if the expression evaluates to `true`. |
### `tcp-logger`
The `tcp-logger` Plugin can be used to push log data requests to TCP servers.
{% hint style="info" %}
**Priority** `405` · **Category** Observability · **Required** `host`, `port`
{% endhint %}
Configuration and details
This provides the ability to send log data requests as JSON objects to monitoring tools and other TCP servers.
This plugin also allows to push logs as a batch to your external TCP server. It might take some time to receive the log data. It will be automatically sent after the timer function in the batch processor expires.
| Name | Type | Required | Default | Valid values | Description |
| ------------------------- | ------- | -------- | ------- | -------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| host | string | True | | | IP address or the hostname of the TCP server. |
| port | integer | True | | \[0,...] | Target upstream port. |
| timeout | integer | False | 1000 | \[1,...] | Timeout for the upstream to send data. |
| log\_format | object | False | | | Log format declared as key-value pairs in JSON. Values support strings and nested objects (up to five levels deep; deeper fields are truncated). Within strings, AILIX or [NGINX](http://nginx.org/en/docs/varindex.html) variables can be referenced by prefixing with `$`. |
| tls | boolean | False | false | | When set to `true` performs SSL verification. |
| tls\_options | string | False | | | TLS options. |
| include\_req\_body | boolean | False | false | \[false, true] | When set to `true` includes the request body in the log. |
| include\_req\_body\_expr | array | No | | | Filter for when the `include_req_body` attribute is set to `true`. Request body is only logged when the expression set here evaluates to `true`. See lua-resty-expr for more. |
| include\_resp\_body | boolean | No | false | \[false, true] | When set to `true` includes the response body in the log. |
| include\_resp\_body\_expr | array | No | | | Filter for when the `include_resp_body` attribute is set to `true`. Response body is only logged when the expression set here evaluates to `true`. See lua-resty-expr for more. |
### `tencent-cloud-cls`
The `tencent-cloud-cls` Plugin uses [TencentCloud CLS](https://cloud.tencent.com/document/product/614) API to forward AILIX logs to your topic.
{% hint style="info" %}
**Priority** `397` · **Category** Observability · **Required** `cls_host`, `cls_topic`, `secret_id`, `secret_key`
{% endhint %}
Configuration and details
| Name | Type | Required | Default | Valid values | Description |
| ------------------------- | ------- | -------- | ------- | -------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| cls\_host | string | Yes | | | CLS API host，please refer [Uploading Structured Logs](https://www.tencentcloud.com/document/api/614/16873). |
| cls\_topic | string | Yes | | | topic id of CLS. |
| secret\_id | string | Yes | | | SecretId of your API key. |
| secret\_key | string | Yes | | | SecretKey of your API key. |
| sample\_ratio | number | No | 1 | \[0.00001, 1] | How often to sample the requests. Setting to `1` will sample all requests. |
| include\_req\_body | boolean | No | false | \[false, true] | When set to `true` includes the request body in the log. If the request body is too big to be kept in the memory, it can't be logged due to NGINX's limitations. |
| include\_req\_body\_expr | array | No | | | Filter for when the `include_req_body` attribute is set to `true`. Request body is only logged when the expression set here evaluates to `true`. See lua-resty-expr for more. |
| include\_resp\_body | boolean | No | false | \[false, true] | When set to `true` includes the response body in the log. |
| include\_resp\_body\_expr | array | No | | | Filter for when the `include_resp_body` attribute is set to `true`. Response body is only logged when the expression set here evaluates to `true`. See lua-resty-expr for more. |
| global\_tag | object | No | | | kv pairs in JSON，send with each log. |
| log\_format | object | No | | | Log format declared as key-value pairs in JSON. Values support strings and nested objects (up to five levels deep; deeper fields are truncated). Within strings, AILIX or [NGINX](http://nginx.org/en/docs/varindex.html) variables can be referenced by prefixing with `$`. |
### `udp-logger`
The `udp-logger` Plugin can be used to push log data requests to UDP servers.
{% hint style="info" %}
**Priority** `400` · **Category** Observability · **Required** `host`, `port`
{% endhint %}
Configuration and details
This provides the ability to send log data requests as JSON objects to monitoring tools and other UDP servers.
This plugin also allows to push logs as a batch to your external UDP server. It might take some time to receive the log data. It will be automatically sent after the timer function in the batch processor expires.
| Name | Type | Required | Default | Valid values | Description |
| ------------------------- | ------- | -------- | ------------ | -------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| host | string | True | | | IP address or the hostname of the UDP server. |
| port | integer | True | | \[0,...] | Target upstream port. |
| timeout | integer | False | 3 | \[1,...] | Timeout for the upstream to send data. |
| log\_format | object | False | | | Log format declared as key-value pairs in JSON. Values support strings and nested objects (up to five levels deep; deeper fields are truncated). Within strings, AILIX or [NGINX](http://nginx.org/en/docs/varindex.html) variables can be referenced by prefixing with `$`. |
| name | string | False | "udp logger" | | Unique identifier for the batch processor. If you use Prometheus to monitor AILIX metrics, the name is exported in `ailix_batch_process_entries`. processor. |
| include\_req\_body | boolean | False | false | \[false, true] | When set to `true` includes the request body in the log. |
| include\_req\_body\_expr | array | No | | | Filter for when the `include_req_body` attribute is set to `true`. Request body is only logged when the expression set here evaluates to `true`. See lua-resty-expr for more. |
| include\_resp\_body | boolean | No | false | \[false, true] | When set to `true` includes the response body in the log. |
| include\_resp\_body\_expr | array | No | | | Filter for when the `include_resp_body` attribute is set to `true`. Response body is only logged when the expression set here evaluates to `true`. See lua-resty-expr for more. |
### `zipkin`
[Zipkin](https://github.com/openzipkin/zipkin) is an open-source distributed tracing system. The `zipkin` Plugin instruments AILIX and sends traces to Zipkin based on the [Zipkin API specification](https://zipkin.io/pages/instrumenting.html).
{% hint style="info" %}
**Priority** `12011` · **Category** Observability · **Required** `endpoint`, `sample_ratio`
{% endhint %}
Configuration and details
The Plugin can also send traces to other compatible collectors, such as [Jaeger](https://www.jaegertracing.io/docs/1.51/getting-started/#migrating-from-zipkin) and Apache SkyWalking, both of which support Zipkin [v1](https://zipkin.io/zipkin-api/zipkin-api.yaml) and [v2](https://zipkin.io/zipkin-api/zipkin2-api.yaml) APIs.
| Name | Type | Required | Default | Valid values | Description |
| ------------- | ------- | -------- | --------------------------- | ------------- | ------------------------------------------------------------------------------------------------ |
| endpoint | string | True | | | Zipkin span endpoint to POST to, such as `http://127.0.0.1:9411/api/v2/spans`. |
| sample\_ratio | number | True | | \[0.00001, 1] | Frequency to sample requests. Setting to `1` means sampling every request. |
| service\_name | string | False | "AILIX" | | Service name for the Zipkin reporter to be displayed in Zipkin. |
| server\_addr | string | False | the value of `$server_addr` | IPv4 address | IPv4 address for the Zipkin reporter. For example, you can set this to your external IP address. |
| span\_version | integer | False | 2 | \[1, 2] | Version of the span type. |
---
# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.
## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.
Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:
```
GET https://docs.digitalapi.ai/api-gateway/plugin-reference/plugins-observability.md?ask=&goal=
```
`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.
Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.