> For the complete documentation index, see [llms.txt](https://docs.digitalapi.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.digitalapi.ai/api-gateway/reference/troubleshooting.md). # Troubleshooting This chapter pairs the failure modes you are most likely to hit in the first month with their causes and one- or two-step fixes. The symptom-cause-fix table is for quick lookup; the detailed entries below it expand the same set of issues with the reproduction notes and the navigation path you need to resolve them. Coverage spans the surfaces introduced earlier in the guide: services (`Add Service` permission, missing routes, no traffic reaching the backend), upstreams (the Add Upstream wizard, health-check failures, egress reachability), plugins (execution-order surprises), Custom Widgets (empty widgets), and Setup (SAML configurations with zero mappings, invitation emails that do not arrive, role assignment caching). If a symptom is not in this chapter, jump to the matching chapter for the surface you are working on. The *Verify* sections in each task subsection are deliberately written to catch the failure modes that do not warrant their own troubleshooting entry. For anything still unresolved, the contact-support and account-manager channels listed under Additional resources are the next stops. ### Symptom-cause-fix table | Symptom | Likely cause | What to do | | ---------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `/welcome` renders the marketing page | Session expired or token cache cleared. | Sign in again. If a fresh sign-in still lands on the marketing page, the SPA's auth check is failing: verify the identity provider if you use SAML, or check that the email is provisioned in the organisation. | | `Add Service` button is disabled | The signed-in user lacks `api.create`. | Have an admin grant a role with `api.create`; refresh after the role is assigned. | | `Add Upstream` wizard rejects **Next** on the Overview step | A required field is empty. **Name** and **Environment** are both required on Overview before Next is enabled. | Fill **Name** and pick an environment from **Select an API environment**; **Next** becomes enabled. | | Upstream health check fails for a backend that responds to direct curl | AILIX dials the host:port from its own egress; the backend may be reachable from your network but not from AILIX's. | Confirm the backend is internet-reachable (or that the AILIX deployment is in-network). Allow-list AILIX's egress IPs at the backend's edge if needed. | | A custom widget shows ‘No data’ for a service that is taking traffic | The widget's filters scope to a service, environment, or time window that has no matching traffic. | Loosen the filters on the widget (clear environment, broaden the time window) and re-save. | | SAML configuration saved but users still see the email-and-password form | The configuration's mapping count is 0; saved but inactive. | Open the configuration, complete **Mappings** to bind an IdP attribute to an AILIX user, and save. | | Invitation email never arrives | SMTP filtered as spam, or the AILIX outbound configuration misroutes. | Check the user's spam folder. Re-invite if not delivered; if repeated re-invitation also fails, escalate to AILIX support. | | New role does not appear in assignment dropdowns | Client-side caching after creation. | Hard-refresh (⌘-Shift-R or Ctrl-Shift-R) the **Users** or **Groups** page. | | Revision create returns 400 with `required property 'plugins' not found` | Route objects in `routeSpec` are missing the `plugins` field. | Add `"plugins": {}` to every route object, even routes with no route-level plugins. | | Revision create returns 400 with `$.script: must be at least 10 characters long` | A route object includes `"script": ""` (empty string). | Remove the `script` field entirely. Only include it with a real Lua script (minimum 10 characters). Same applies to `service_id`: omit it, do not set to empty string. | | Deploy returns 400 with `failed to check the configuration of plugin X` | Plugin config passes service-spec validation but fails AILIX runtime validation. | Fetch the plugin's schema first: `GET /plugin-schemas/{pluginName}`. Check required fields — AI plugins especially have non-obvious requirements (`ai-prompt-template` needs `templates[]` array, `ai-proxy` needs `auth`). | | Deploy returns 400 with `Upstream not configured for environment` | Upstream binding was set via `PATCH .../upstream-bindings` (replace) instead of `.../upstream-bindings/add`. The replace endpoint silently clears bindings if the payload format is unexpected. | Always use `PATCH .../upstream-bindings/add`. Verify bindings stuck: check `upstreamBindings` array in the revision response. | | Deploy returns 409 with route conflicts | Two APIs define the same URI + method combination in the same environment. | Prefix each API's routes with a unique namespace (e.g. `/open-banking/api/...`). Use `proxy-rewrite` plugin to strip the prefix before forwarding. | | Import-openapi returns 400 with `desc: must be at most 256 characters long` | The OAS `info.description` or an operation description exceeds 256 characters. | Truncate all descriptions to under 200 characters. Single line, no markdown. | | API calls return 401 | Firebase JWT token expired (lifetime: 1 hour). | Refresh the Helix Portal page in the browser to trigger token refresh. Copy the new `Authorization: Bearer` header from DevTools → Network. | | `GET /plugin-schemas` returns only AI plugins | Built-in AILIX plugins (key-auth, cors, limit-count, etc.) are available but do not appear in the schema list. | Built-in plugins work when configured — their validation happens at the AILIX runtime level during deploy, not via the schema endpoint. | | `loki-logger` plugin fails validation with `property "endpoint_addrs" is required` | Config uses `endpoint_addr` (singular) instead of `endpoint_addrs` (plural, array). | Use `"endpoint_addrs": ["http://host:3100"]` not `"endpoint_addr": "..."`. | | `proxy-cache` fails with `cache_key variable $request_method unsupported` | `$request_method` is not a valid cache key variable. | Use only `$uri`, `$host`, and supported Nginx variables in `cache_key`. | ### Detailed entries #### A service is registered but no traffic reaches the backend A service alone forwards nothing. Confirm in order: 1. The service has at least one route. 2. The route targets an upstream. 3. The service is deployed to an environment. 4. Consumer traffic is hitting the environment's gateway hostname (not the portal hostname). 5. The route's match rules (path, host, method) are not stricter than expected. Any one of those five missing breaks the path. #### Cannot delete a developer because their apps are still active Developer records are referenced by apps. Either delete each app (or change its developer assignment) before deleting the developer, or use the cascade-delete flow if your role includes the relevant `force-delete` permission. The audit log captures both paths. #### Service detail page shows ‘No environments’ after a deployment finished The service detail page caches its environment list briefly. Wait thirty seconds and refresh. If the deployed environment still does not appear, open `/api-gateway/environments`, find the environment, and confirm the service is listed under **Deployed Services**. If the environment list shows the service but the service detail does not, raise it with AILIX support: the two views are reconciled in the background and a persistent disagreement is a bug. #### Plugin order produces unexpected behaviour Plugins at the same scope run in a documented default order. Each plugin in the catalogue ships with a default execution order (a small integer); lower numbers run first. The default order is set by the plugin's author and matches the typical chain: authentication before rate-limiting, transformation before observability, and so on. Some plugins expose an **Order** field on the attachment form, which overrides the default for that specific attachment. Plugins without an Order field run at the default order and cannot be re-ranked without editing the plugin itself. The most common surprise is auth running after rate-limit at route scope: a misconfigured Order field rate-limits anonymous traffic before authenticating it, which fills the rate-limit bucket with junk and starves real consumers. Verify the order on the route's **Plugins** tab; if a plugin exposes an Order field, edit it. If not, the default order applies. #### Upstream wizard's Targets step accepts no nodes The Targets step expects at least one backend node before **Next** is enabled. If the node row is filled in but **Next** stays disabled, check the **Service discovery** toggle: when enabled, manual node entry is hidden and the upstream expects discovery to populate nodes at run time. For static-node upstreams, leave service discovery off. #### API call returns 200 on service-spec update but deploy fails This is the most common trap when managing services via the API. The control plane validates structural correctness (JSON schema, field types, string lengths) when you `PUT` a service spec or route spec, but plugin-level validation only runs when the AILIX runtime processes the deploy. This means: 1. Service-spec update: 200 OK (even with broken plugin config) 2. Route-spec update: 200 OK 3. Deploy: 400 `"failed to check the configuration of plugin X err: property Y is required"` Before deploying, validate your plugin config against the schema endpoint: `GET /api/orgs/{orgId}/plugin-schemas/{pluginName}`. AI plugins are especially prone to this because their schemas differ from upstream APISIX documentation. Known differences from APISIX open-source documentation: * `ai-prompt-template`: requires `templates` array where each entry needs `name` (string) and `template` (object with `model` and `messages[]`). * `ai-proxy`: requires `auth` field (not optional as in OSS docs). * `loki-logger`: uses `endpoint_addrs` (plural, array) not `endpoint_addr` (singular, string). * `proxy-cache`: does not support `$request_method` in `cache_key`. {% hint style="warning" %} **Pre-deploy checklist:** Before calling the deploy endpoint, verify: (1) all routes have `plugins` (even if `{}`), (2) no route includes empty-string `service_id` or `script`, (3) plugin configs match the schema from `GET /plugin-schemas/{name}`, (4) upstream bindings are set via `/add` not the base endpoint. {% endhint %} --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter: ``` GET https://docs.digitalapi.ai/api-gateway/reference/troubleshooting.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.