Metrics

This page describes the metrics exposed by the Connect platform and the signals we recommend building dashboards and alerts around. For instructions on configuring telemetry export, see Monitoring.

There are four sources of metrics in the Connect platform:

• The Connect API, which exports metrics over OTLP to the OpenTelemetry endpoint you configure.
• The SPIRE server that runs alongside Connect in the control plane, which serves a Prometheus endpoint for scraping.
• The SPIRE server in each trust zone, which also serves a Prometheus endpoint for scraping.
• Credex, which exports metrics over OTLP to the OpenTelemetry endpoint you configure.

These metrics cover the Connect platform specifically. We recommend monitoring CPU, memory, pod restarts and database health with your existing Kubernetes and database monitoring.

Prerequisites

The Connect API metrics described on this page require Connect v0.69.0 or later.

To collect them, enable Connect telemetry.

To collect SPIRE server metrics, enable the SPIRE Prometheus endpoint on the control plane SPIRE server and on each trust zone SPIRE server.

To collect Credex metrics, enable Credex telemetry. The Credex metrics described on this page require Credex v0.21.0 or later.

Connect API metrics

Connect records a histogram for every API call, following the OpenTelemetry semantic conventions for RPC metrics. The metric is rpc.server.call.duration, measured in seconds, and carries the following attributes:

Attribute	Description
rpc.method	The fully qualified method, for example proto.connect.trust_zone_service.v1alpha1.TrustZoneService/ListTrustZones.
rpc.response.status_code	OK, or a gRPC status code such as PERMISSION_DENIED.
error.type	Present on failed calls only.

The API reference lists the services and methods that can appear in rpc.method.

Depending on your metrics backend, metric and attribute names may be translated on ingestion. For example, Prometheus replaces dots with underscores and appends the unit, so the metric is stored as rpc_server_call_duration_seconds.

When building panels from this metric, exclude two kinds of call:

• Health checks. Kubernetes probes call grpc.health.v1.Health/Check, which dominates the traffic.
• Long-lived streams. WorkloadService/PublishWorkloads and WorkloadService/PublishWorkloadEvents record the whole lifetime of the stream as their duration, so they always fall into the +Inf bucket of the histogram.

SPIRE server metrics

The SPIRE server metrics and signals on this page apply to both the control plane SPIRE server and the SPIRE server in each trust zone. Trust zone SPIRE servers issue the identities used by your workloads, so we recommend monitoring each of them alongside the control plane SPIRE server.

SPIRE servers expose metrics with the spire_server_ prefix. Many of its metrics are call counters, which carry a status label holding the gRPC status code of the call. A status other than OK indicates a failed call.

Some metrics carry a trust_domain_id label, but most do not. To tell SPIRE servers apart in dashboards and alerts, group by the labels your scrape configuration attaches to each target, such as the job, cluster or namespace.

The full list of SPIRE server metrics is in the upstream SPIRE telemetry reference.

Credex metrics

Credex records metrics for both inbound and outbound requests, following the OpenTelemetry semantic conventions for HTTP metrics and RPC metrics.

Inbound requests

Credex records a histogram for every request to its HTTP API. The metric is http.server.request.duration, measured in seconds, and carries the following attributes:

Attribute	Description
http.route	The matched route, for example /token. Requests that match no route do not carry this attribute.
http.request.method	The HTTP method, for example POST.
http.response.status_code	The HTTP status code of the response, for example 403.

Data points also carry standard attributes such as url.scheme, server.address and network.protocol.version. Credex also records request and response sizes as http.server.request.body.size and http.server.response.body.size.

The /token endpoint returns errors as defined by OAuth 2.0. A 403 response means the exchange was denied by an exchange policy or an external hook, a 400 or 401 response means the request or client authentication was invalid, and a 5xx response indicates a failure within Credex.

When building panels from this metric, exclude the /health route, which is called by Kubernetes probes. In Prometheus, the metric is stored as http_server_request_duration_seconds.

Outbound requests

Credex records http.client.request.duration, in seconds, for HTTP calls to downstream OAuth authorization servers in OAuth Bridge exchanges, to external hooks and to the JWKS endpoints of token issuers. The server.address attribute identifies the service that was called and http.response.status_code holds the status code it returned. http.response.status_code will not be present for calls that fail before a response is received, for example because of a connection error or timeout.

Credex records rpc.client.call.duration, in seconds, for gRPC calls to the SPIRE server’s custom JWT signer when using the spire signing method, and to the SPIRE agent’s Delegated Identity API for OIDC to SPIFFE exchanges. As with the Connect API metric, rpc.method holds the fully qualified method, for example cofide.api.server.jwtsigner.v1.JWTSigner/SignJWT, and rpc.response.status_code holds OK or a gRPC status code such as UNAVAILABLE.

Recommended signals

X.509 CA expiry

The spire_server_manager_x509_ca_rotate_expiration gauge holds the expiry time of the active X.509 CA as a Unix timestamp, labelled by trust_domain_id. Subtract the current time from this value to get the time remaining until the CA expires.

SPIRE prepares a new CA when the active CA reaches half of its lifetime, and activates it when one sixth of the lifetime remains. If the time remaining drops below one sixth of the CA lifetime, rotation has not happened as expected and should be investigated before the CA expires.

Signing key operations

The spire_server_server_key_manager_generate_key and spire_server_server_key_manager_get_public_keys call counters record operations on the SPIRE server’s signing keys. Calls with a status other than OK indicate a failure in the key manager. For example, the permissions granted to the SPIRE server for an external key management service may have been changed.

Node attestation failures

The spire_server_rpc_agent_v1_agent_attest_agent call counter records node attestation attempts by SPIRE agents. Watch for calls with a status other than OK. A workload cannot obtain an identity until its node has successfully attested.

SVID issuance volume

The rates of spire_server_server_ca_sign_x509_svid and spire_server_server_ca_sign_jwt_svid give the number of X.509 and JWT SVIDs being issued. These are useful as a baseline, and sudden spikes or drops are worth investigating.

Federated trust bundle refreshes

The spire_server_bundle_manager_fetch_federated_bundle call counter records fetches of trust bundles from federated trust domains, labelled by trust_domain_id. Watch for calls with a status other than OK. If the bundle of a federated trust domain cannot be fetched, key rotation in that trust domain will not be detected and federation with it will eventually break.

Connect API authentication and authorization failures

Filter rpc.server.call.duration to calls where rpc.response.status_code is UNAUTHENTICATED or PERMISSION_DENIED, grouped by rpc.method. This shows callers that are failing to authenticate or are attempting operations that their role bindings do not allow.

Connect API request rate

The rate of rpc.server.call.duration, grouped by rpc.method, gives a baseline of Connect API traffic.

Credex token exchange failures

Filter http.server.request.duration to the /token route, grouped by http.response.status_code. A 5xx response indicates a failure within Credex and should be investigated. An increase in 403 responses means more exchanges are being denied, which can indicate a misconfigured exchange policy or a workload attempting an exchange it is not permitted to make.

Credex token exchange latency

Track the 95th and 99th percentiles of http.server.request.duration for the /token route. The duration of a token exchange includes any calls to external hooks, downstream OAuth authorization servers and the JWT signer, so compare it with the outbound request metrics to find the source of increased latency.

Credex dependency failures

Filter http.client.request.duration to calls with a 5xx status code or no status code, grouped by server.address, and rpc.client.call.duration to calls where rpc.response.status_code is not OK. Token exchanges that depend on a failing service will fail. In particular, failures calling the SPIRE server’s custom JWT signer prevent Credex from issuing any tokens when using the spire signing method.

Credex request rate

The rate of http.server.request.duration for the /token route gives a baseline of token exchange traffic.