Monitoring the REST Gateway
The REST Gateway can count what an operator needs to watch it: the HTTP requests it finished, how long they took, the ones callers abandoned, the requests in flight, its conversations with the key server, the requests its rate and capacity control refused, whether its build is admitted, failed token releases and live sessions. It exposes these as Prometheus metrics on a separate management listener that is off by default and binds loopback only.
Every release carries a recipe for scraping them from your own Prometheus, safely, under
integrations/monitoring/ in the Console bundle: the metrics contract, the scrape job, seven
alert rules with their tests, the queries, an nginx scrape proxy and a runbook.
What is measured, and what is not
Measured, by the Gateway, about the Gateway:
- every HTTP response it finished, by route pattern (
/api/v1/principals/:id, never an id), method and status class, and how long it took; - every request whose caller left before the answer (an abort), counted once and never also as a completion;
- every conversation it held with the key server, whether it ended in an error, and its wall time;
- the requests its rate and capacity control refused, by stage;
- whether a release manifest is configured, whether the build is admitted, and whether the full surface is up;
- token releases attempted and failed, live sessions and the session ceiling.
Not measured:
- The key server's own work. The key-server timings are the Gateway's wall time for each conversation, including the wait for a pooled connection, not the key server's processing time. A slow conversation can be a full pool, not a slow key server.
- The Console. It exposes no metrics.
- The host and the database. Use the standard host and PostgreSQL exporters.
- Tracing. There are no spans.
- Who did what. No label, sample or help text carries a path with its ids, a query string, a
principal, a service, a session, an object, a T-tag, an address, a bearer, a proof or error text.
Every label value comes from a closed set, so one Gateway's metrics are bounded at 5000
series; the Gateway logs its own maximum at boot
(
[boot] metrics: 125 (route, method) pairs, at most 2550 series). The audit trail answers who did what.
Turning on the management listener
The listener is turned on by one variable, TE_GATEWAY_METRICS_ADDR=<host>:<port>. The host must
be 127.0.0.1, ::1 (written [::1]:<port>) or localhost. Any other host refuses the
Gateway's start, naming the variable. Unset, nothing is counted and no second port is opened.
Both tier provisioners write it, validating the value exactly as the Gateway does. On Linux:
sudo tricryption-tier-provision --metrics-addr 127.0.0.1:9464
On Windows, from an elevated prompt:
powershell.exe -NoProfile -ExecutionPolicy Bypass -File 'C:\Program Files\PiSoft\Tricryption\libexec\console-gateway\tricryption-tier-provision.ps1' -MetricsAddr 127.0.0.1:9464
The variable is written into the Gateway's environment file
(/etc/pisoft/console-gateway/gateway.env on Linux,
%ProgramData%\PiSoft\Tricryption\console-gateway\gateway.env on Windows). The rules are the same
on both:
- The value must be
<host>:<port>with a loopback host and a port from 1 to 65535. - It may not be combined with
--identity-out(-IdentityOut): a Gateway that runs elsewhere publishes its own listener. - A later run without
--metrics-addrremoves the line, which turns the listener off.
The listener answers GET /metrics (and HEAD) with the Prometheus text format, any other
path 404 and any other method 405. It collects one scrape at a time; a second concurrent
scrape is answered 503 and counted. It never opens a session, never reaches the key server and
is not itself counted. The public listener has no metrics path, and the REST API gains
none.
The production shape
your Prometheus --HTTPS + monitoring client certificate--> nginx (scrape proxy) --HTTP--> management listener
(anywhere) on the Gateway host 127.0.0.1:9464
- The Gateway listens for metrics on loopback only. It holds no new TLS listener.
- nginx runs on the Gateway host, configured from the recipe's
scrape-proxy/nginx.conf.template. It terminates TLS, demands a client certificate issued by the monitoring CA (that CA only, never a system bundle), admits exactly one certificate subject, forwardsGET /metricsunchanged to the loopback listener, and answers every other path404. It runs as the systemd service the recipe ships (scrape-proxy/tricryption-scrape-proxy.unit, installed astricryption-scrape-proxy.serviceand enabled), so it starts again with the machine: an nginx started by hand is gone after a reboot, and Prometheus then finds the target down. - The monitoring identity is a dedicated client certificate (
extendedKeyUsage = clientAuth) from a CA used for nothing else, held only by Prometheus. It authorizes reading metrics and nothing else. The Gateway never sees it. - Your Prometheus gets the
tricryption-gatewayjob from the recipe'sprometheus/prometheus.yml:https, the monitoring certificate, one target per Gateway host (the proxy's address, for examplegateway.example.test:9465). Itsinstanceanddeploymentlabels are set by your job; the Gateway never labels itself.
What the proxy refuses (every row but the last measured against the shipped template; the last is the template's own rule):
| The client presents | The proxy answers |
|---|---|
| no client certificate | 400, after the handshake; nothing is forwarded |
| a certificate from any other CA, even one carrying the admitted subject | 400; nothing is forwarded |
| a monitoring-CA certificate with another subject | 403; nothing is forwarded |
the monitoring identity, any path but /metrics | 404 |
the monitoring identity, /metrics, any method but GET | 405 |
The commands that issue the certificates, render and start the proxy and add the job are in the runbook's Enable section.
The metric families
| Family | Type | Labels | What it counts |
|---|---|---|---|
te_gateway_http_requests_total | counter | route method status_class | HTTP responses the Gateway finished |
te_gateway_http_request_duration_seconds | histogram | route method | from the request's arrival to its finish |
te_gateway_http_aborted_total | counter | route method | requests whose connection closed before the response finished |
te_gateway_http_in_flight | gauge | requests open now | |
te_gateway_ks_conversations_total | counter | outcome context | conversations with the key server |
te_gateway_ks_conversation_duration_seconds | histogram | outcome | each conversation's wall time, pool wait included |
te_gateway_overload_refusals_total | counter | stage | requests the rate and capacity control refused |
te_gateway_build_admission_configured | gauge | 1 when a release manifest is configured | |
te_gateway_build_admitted | gauge | 1 when the build is admitted | |
te_gateway_booted | gauge | 1 once the full surface is up | |
te_gateway_token_release_attempts_total | counter | token releases attempted (after boot) | |
te_gateway_token_release_failures_total | counter | token releases that failed (after boot) | |
te_gateway_sessions_live | gauge | live sessions (after boot) | |
te_gateway_sessions_max | gauge | the session ceiling (after boot) | |
te_gateway_metrics_series | gauge | the number of series in this exposition | |
te_gateway_metrics_scrape_failures_total | counter | reason | collections refused or failed |
te_gateway_process_start_time_seconds | gauge | when the process started |
The label values:
| Label | Values |
|---|---|
route | a registered route pattern; /oidc/ for the login page; unmatched for everything no route matched (404s, 405s, early refusals, the 503 BUILD_NOT_ADMITTED answers before boot) |
method | GET HEAD POST PUT PATCH DELETE OPTIONS other |
status_class | 1xx 2xx 3xx 4xx 5xx |
outcome | ok error |
context | request (made for a caller's request) background (made outside any request) |
stage | capacity source identity concurrency sessions |
reason | busy too_large error |
Both histograms use the buckets 0.005 0.01 0.025 0.05 0.1 0.25 0.5 1 2.5 5 10 +Inf seconds.
The recipe's METRICS.md and metrics-contract.json are the full contract, and queries.md
holds the queries an operator types, each with the trap it avoids.
Reading the metrics
- Counters reset when the Gateway restarts. Read every
_totalwithrate()orincrease(), never raw. A change inte_gateway_process_start_time_secondsmarks a restart. - A restart loses what the old process counted after its last scrape.
increase()across a restart can fall short by up to one scrape interval of traffic. Measured on one host: 297 of 344 requests over a 95-second window spanning a restart, at a 15-second interval. - Absent is not zero. A route-labelled series exists only once it has been observed, so a
route that never failed has no 5xx series and a Gateway that never saw an abort has no abort
series. A ratio over them needs an
or ... * 0arm, as the recipe's queries show, or it reads "no data" where the answer is 0. The four after-boot families are absent untilte_gateway_bootedis 1. outcome="error"includes the key server's refusals. A403 ACL_DENIEDis a conversation that ended in an error. A key server that is down shows as errors together with 5xx answers.- Overload refusals answered
503are also 5xx. Read the two together. - Aggregate histogram buckets before
histogram_quantile, keepingleandinstance. A mean of per-instance p95s is not a p95. - Never sum two scrapers of one Gateway. Two Prometheus servers see the same counters.
- Readiness is build admission, not health.
te_gateway_bootedandte_gateway_build_admittedsay nothing about the key server, its database or the audit pipeline.
The alerts
The rule file prometheus/rules/tricryption-gateway.rules.yml defines seven alerts. Every
threshold is an example to tune, not a service-level objective. Each alert names its section of
the runbook, and the rule tests (promtool test rules) give each one a case that fires and one
that does not.
| Alert | Severity | Fires when |
|---|---|---|
TricryptionGatewayTargetDown | critical | Prometheus has not scraped a Gateway for 2 minutes |
TricryptionGatewayHigh5xx | warning | more than 5% of a Gateway's responses are 5xx for 5 minutes, above 0.1 requests a second |
TricryptionGatewayOverloadRefusals | warning | the rate and capacity control refused a request at some stage in the last 5 minutes |
TricryptionGatewayHighLatencyP95 | warning | the p95 request latency is above 2.5 seconds for 10 minutes, above 0.1 requests a second |
TricryptionGatewayTokenReleaseFailures | warning | a token release failed in the last 15 minutes |
TricryptionGatewayBuildNotAdmitted | critical | a release manifest is configured and the build has not been admitted for 5 minutes |
TricryptionGatewayMetricsCollectionFailing | info | the Gateway refused or failed a metrics collection in the last 10 minutes |
What to check first when each one fires, and what not to conclude, is in the runbook that ships with the release:
- Linux:
/opt/pisoft/tricryption/console-gateway/integrations/monitoring/runbook.md - Windows:
C:\Program Files\PiSoft\Tricryption\console-gateway\integrations\monitoring\runbook.md
The runbook also holds the Enable, Verify a scrape, Rotate the monitoring identity and Disable and roll back procedures.
Turning it off
Re-run the tier provisioner without --metrics-addr (-MetricsAddr), or remove
TE_GATEWAY_METRICS_ADDR from the Gateway's environment file, and restart the Gateway. No
listener is opened and nothing is counted; the business service is unaffected. Then stop the
scrape proxy and remove the tricryption-gateway job and the rule file from Prometheus.
Prometheus keeps the old series until its retention expires them.
What it costs
The cost was measured on one host, a Gateway in front of a local key server, with a closed-loop load of 8 workers sending encrypt requests (a 20-second warm-up excluded, 60 seconds recorded, three repeats per configuration, the rate limits lifted so that the limiter was not what was measured):
| Configuration | Throughput | p95 latency | Gateway CPU per 1,000 requests |
|---|---|---|---|
listener off (TE_GATEWAY_METRICS_ADDR unset) | 152.75 requests/s | 56.89 ms | 1.878 s |
| listener on, scraped every 15 seconds through the proxy | −0.0% | +0.2% | +0.3% |
| listener on, scraped every second through the proxy | −0.1% | +0.4% | +1.6% |
Both measured configurations are inside a 5% budget for p95 latency and for CPU. That load was bound by the key server (a median of 52 ms in every configuration), so the CPU column is the sharper measure of what counting costs. On your own traffic, compare the latency your load balancer or your clients see, and the Gateway process's CPU, before and after you turn the listener on.