> For the complete documentation index, see [llms.txt](https://docs.digitalapi.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.digitalapi.ai/api-gateway/observability/investigating-with-request-traces.md).

# Investigating with request traces

A **request trace** is the end-to-end record of one request: gateway ingress through the plugin chain, through the upstream call, through the response. The trace breaks down timing per plugin and per upstream so you can answer the question *where did the 4 seconds go?* When the observability surfaces show errors spiking, request traces tell you why.

Tracing in Helix is **session-based and scoped to a single API**. Instead of a permanently-on global stream, you start a **tracing session** on the API you are investigating: the gateway captures matching requests for a bounded window, then you browse the captured sessions and drill into per-phase timing.

### Opening traces for an API

#### Open the Traces tab

**Before you start**

* The API you want to investigate, deployed to an environment that receives traffic.
* A symptom and a time window. *All 5xx in the last 15 minutes* is investigable; *everything is slow* is not.

To open traces:

1. From the left sidebar, choose **API Gateway** → **APIs** and open the API.
2. Select the **Observability** tab, then **Traces**.

![Figure 9-1. The Traces tab for an API, before any session has run.](https://3578638071-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FoZvCoRmyRavMxaTP3MiP%2Fuploads%2Fgit-blob-ae6e2d40f621048a2ac21f6c87b4342a69e7eb96%2Fapi-observability-traces.png?alt=media)

**Verify**

1. The URL is `<your-helix-portal-domain>/api-gateway/apis/<uuid>?tab=observability`.
2. The page shows your existing tracing sessions, or an empty state inviting you to start one.

{% hint style="success" %}
**Result:** The Traces tab for the API, ready to start a capture.
{% endhint %}

Related tasks:

* [Start a tracing session](#start-a-tracing-session): capture live requests for a bounded window.

### Capturing requests

#### Start a tracing session

**Before you start**

* The environment whose traffic you want to capture (for example `test` or `production`).
* A sense of how long to capture. Short windows on high-volume APIs fill quickly; long windows suit intermittent issues.

To start a session:

1. On the Traces tab, click **Start trace session**.
2. Choose the **Environment** to capture from.
3. Pick a **capture window**. Presets are **Quick check** (1 min), **Standard debug** (5 min), **Deep dive** (15 min), and **Long capture** (1 hour), or set a **Custom** value in seconds (60 to 3600).
4. Click **Start tracing**. The session runs until the timer ends or you stop it.

![Figure 9-2. The Start tracing session dialog: environment and capture window.](https://3578638071-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FoZvCoRmyRavMxaTP3MiP%2Fuploads%2Fgit-blob-dbec66e9382712a9ce1e6bd598c647aaa534d7cc%2Ftraces-start-session.png?alt=media)

The session captures method, URI, status, latency, headers, and plugin phases for live requests on the API.

**Verify**

1. A running session appears on the Traces tab with its remaining time.
2. As matching requests arrive, the session's captured-request count increases.

{% hint style="success" %}
**Result:** A live tracing session collecting matching requests for the chosen window.
{% endhint %}

{% hint style="success" %}
**Tip:** On a busy API, prefer a **capture filter** over a long window. Filtering to the one route or status you care about keeps the session small and readable instead of burying the interesting request in thousands of healthy ones.
{% endhint %}

#### Capture extra context (advanced)

Two optional panels in the Start tracing session dialog refine what each trace records.

**Variables to capture.** By default the gateway captures a standard set of variables. Expand **Advanced: variables to capture** to record more: choose **All resolved variables** for the broadest capture, pick specific variables from the list, or add a plugin context field (for example `$helix` or `$consumer_name`). Use the variable name without the leading `$`.

![Figure 9-3. Advanced: variables to capture.](https://3578638071-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FoZvCoRmyRavMxaTP3MiP%2Fuploads%2Fgit-blob-f3f2f1af2583dbd8200c53d86aea9bbd816f1335%2Ftraces-capture-variables.png?alt=media)

**Filter which requests to capture.** Expand **Advanced: filter which requests to capture** to record only the requests that match a condition (for example a specific route, method, or status), instead of everything hitting the API.

![Figure 9-4. Advanced: filter which requests to capture.](https://3578638071-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FoZvCoRmyRavMxaTP3MiP%2Fuploads%2Fgit-blob-7d03b8ca49298decf828583c63262af80619d6ac%2Ftraces-capture-filter.png?alt=media)

**Verify**

1. The captured traces include the extra variables you selected.
2. With a filter set, only matching requests appear in the session.

{% hint style="success" %}
**Result:** Traces that carry exactly the context you need, without the noise of unrelated traffic.
{% endhint %}

### Reading a trace

#### Browse a session and read per-phase timing

To read the results:

1. On the Traces tab, open a captured session.
2. The session lists its captured requests with method, URI, status, and total latency.
3. Click any request to open its trace detail with plugin-by-plugin and upstream timing.

**Verify**

1. The request list shows the traffic captured during the session window.
2. The detail view renders per-plugin and per-upstream timing for a single request.

{% hint style="success" %}
**Result:** Full per-phase timing for each captured request, so you can see where the time went.
{% endhint %}

{% hint style="success" %}
**Tip:** When a trace's total time looks bad, read the per-phase split before drawing conclusions. *4 ms gateway plus 3,996 ms upstream* tells a different story than *4 s gateway with a fast upstream*: the first sends you to the backend team, the second sends you to Helix support.
{% endhint %}

{% hint style="info" %}
**Note:** You can also upload a saved trace file to analyse a capture offline, without starting a live session.
{% endhint %}

{% hint style="danger" %}
**Caution:** Tracing sessions and their captured requests are retained per your Helix deployment's storage configuration; older sessions age out. For long-term forensics, export a trace at investigation time rather than assuming it remains available a week later.
{% endhint %}

#### What the trace detail shows

Beyond per-phase timing, a trace makes the request's outcome explicit:

* **Outcome.** A trace that reached your backend shows a normal outcome. When a plugin answered the request itself (for example an authentication `401` or a rate-limit `429`), the trace is marked **Upstream not called**: the plugin chain short-circuited and your backend was never contacted. This reads as a warning, not an error.
* **Fail reason.** When a plugin fails open or fails closed, the trace names the plugin and its **fail reason**, so you can see exactly why a request was blocked, or allowed through despite an error.
* **Upstream host.** The **Upstream** tab shows the **Upstream host** the request was proxied to, alongside the upstream timing.
* **Captured variables.** Any Helix context values you captured appear as individual fields in the trace snapshot, and the request's analytics entry is attached to the trace.
* **Only the plugins that acted.** Phases where a plugin did no real work are marked **passive** and hidden by default, so each plugin appears once, in the phase where it did its work, rather than in every phase of the chain. When a trace contains passive executions, a **Show passive executions** toggle appears in the Request chain section; turn it on to reveal them, each tagged with a reason: **Helper** (declared gateway plumbing), **Disabled by config** (a setting made it a no-op here), or **Other phase** (it is configured for a different phase). When a request or response body is not captured, the trace records why (for example the body was empty, or request buffering was off).

![Figure 9-5. A trace detail: the per-phase Request chain and the Inspector.](https://3578638071-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FoZvCoRmyRavMxaTP3MiP%2Fuploads%2Fgit-blob-1dfc195a9fc05cd6dc54a15a2237858b834cb431%2Ftrace-request-chain.png?alt=media)

In Figure 9-5 the **Request chain** panel groups each plugin under the phase where it ran. A plugin that altered the request or response carries a **changes** badge (`3 changes`, `2 changes`, `1 change`); the `mocking` plugin answered the request itself, so its row is marked **Upstream not called**. The **Inspector** on the right shows the captured request and response for the selected trace: headers, query, body, and the raw JSON.

The session list is ordered newest first, and an API's name in the session and trace views is a link that opens that API in a new tab.

Related tasks:

* [Building custom reports](/api-gateway/observability/building-custom-reports.md#building-custom-reports): save filter combinations for recurring investigations.
* [Building custom widgets](/api-gateway/observability/building-custom-widgets.md#building-custom-widgets): see the aggregate metric this trace contributed to.


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://docs.digitalapi.ai/api-gateway/observability/investigating-with-request-traces.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.