> For the complete documentation index, see [llms.txt](https://docs.digitalapi.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.digitalapi.ai/api-gateway/observability/investigating-with-request-traces.md).

# Investigating with request traces

A **request trace** is the end-to-end record of one request: gateway ingress through plugin chain, through upstream call, through response. The trace breaks down timing per plugin and per upstream so you can answer the question *where did the 4 seconds go?* When the observability surfaces show errors spiking, request traces tell you why.

### Opening the request traces page

#### Open request traces

**Before you start**

* A symptom and a time window. *All 5xx in the last 15 minutes* is investigable; *everything is slow* is not.

To open the page:

1. From the left sidebar, choose **Observability** → **Request Traces** (`/observability/traces`).
2. The filter bar defaults to **All Services** and **All Status**, and the trace list shows the most recent requests at the top.

![Figure 9-1. The Request Traces page.](/files/LxfGCwduKKPViM26vn5Z)

**Verify**

1. The URL is `<your-helix-portal-domain>/observability/traces`.
2. The trace list populates with the most recent requests across the gateway.

{% hint style="success" %}
**Result:** A recent request stream, ready to filter.
{% endhint %}

Related tasks:

* [Filter traces by service or status](#filter-traces-by-service-or-status): narrow the list.

### Filtering the trace list

#### Filter traces by service or status

**Before you start**

* The service or status class you want. Common starting points are all `5xx` in the last 15 minutes, or the specific service that a custom widget flagged.

To filter:

1. Click **All Services** and select one.
2. Click **All Status** and select a class (`2xx`, `4xx`, or `5xx`) or a specific status code.
3. Bound the time window.
4. Click any row to open the trace detail with plugin-by-plugin timing.

**Verify**

1. The filter buttons reflect the active selection.
2. The list shrinks to rows matching every active filter.
3. The detail view renders per-plugin and per-upstream timing.

{% hint style="success" %}
**Result:** An isolated trace set, with full per-phase timing for each request.
{% endhint %}

{% hint style="success" %}
**Tip:** When a trace's total time looks bad, read the per-phase split before drawing conclusions. *4 ms gateway plus 3,996 ms upstream* tells a different story than *4 s gateway with a fast upstream*: the first sends you to the backend team, the second sends you to AILIX support.
{% endhint %}

{% hint style="danger" %}
**Caution:** Trace retention is bounded by your AILIX deployment's storage configuration; older traces age out. For long-term forensics, export the trace at investigation time rather than assuming it remains available a week later.
{% endhint %}

Related tasks:

* [Building custom reports](/api-gateway/observability/building-custom-reports.md#building-custom-reports): save filter combinations for recurring investigations.
* [Building custom widgets](/api-gateway/observability/building-custom-widgets.md#building-custom-widgets): see the aggregate metric this trace contributed to.


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://docs.digitalapi.ai/api-gateway/observability/investigating-with-request-traces.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.