Monitoring

The Cofide Connect platform provides metrics, traces and other telemetry insights for workload identity components. These are intended to augment, rather than replace, existing monitoring solutions for Kubernetes and cloud-native systems. This page describes how to configure telemetry export. For the metrics that are available and the signals we recommend monitoring, see Metrics.

Connect

OpenTelemetry

The Cofide Connect API is instrumented using OpenTelemetry (OTel) and can export telemetry using OTLP over gRPC to a compatible endpoint, including an OpenTelemetry Collector. Connect supports OTLP/gRPC receiver endpoints only.

To enable telemetry in Connect, add the following Helm chart values:

connect:
  telemetry:
    enabled: true
    collectorEndpoint: https://<your-collector-endpoint>:4317
    auth:
      # One of the auth methods described below

Authentication

Connect supports three production-ready authentication methods for the OTLP/gRPC receiver endpoint: OIDC via a Kubernetes Service Account, OIDC via JWT-SVID and SPIFFE mTLS.

An insecure mode which uses unauthenticated plain text connections is also supported for testing and development purposes only - this should never be used in production.

Kubernetes Service Account

When the target OTel collector is running in the same Kubernetes cluster as Connect, the Kubernetes Service Account authentication method is the most straightforward. Connect authenticates to the collector using its own Projected Service Account Token (PSAT) as an OIDC bearer token. The collector uses the standard OTel OIDC auth plugin to validate the token.

The Connect Kubernetes cluster’s API server must have OIDC discovery functionality enabled to allow the OTel collector to verify Connect’s PSAT signature. The Kubernetes API server’s --service-account-issuer can be used to set the issuer URL for service account tokens. Alternatively, for cloud-provider-managed Kubernetes services, a public discovery endpoint is often made available by default - check the documentation for your Kubernetes service provider.

The following Helm values provide an example of the required OTel collector configuration to receive telemetry data from Connect:

config:
  extensions:
    oidc:
      providers:
        - issuer_url: https://kubernetes.default.svc.cluster.local
          issuer_ca_path: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
          audience: otel-collector
  receivers:
    otlp:
      protocols:
        grpc:
          auth:
            authenticator: oidc
          tls:
            # TLS certificates must be trusted by Connect.
            cert_file: /path/to/tls.crt
            key_file: /path/to/tls.key
  exporters:
    # Use this to verify the collector is working
    # Replace with your desired export targets
    debug: {}
  service:
    extensions: [oidc]
    pipelines:
      traces: { receivers: [otlp], exporters: [debug] }
      metrics: { receivers: [otlp], exporters: [debug] }

This configuration assumes the Kubernetes API server’s --service-account-issuer argument is set to https://kubernetes.default.svc.cluster.local (i.e. the local in-cluster API address). The issuer_ca_path allows the OTel collector to trust the Kubernetes API server’s in-cluster TLS certificate. For public OIDC discovery endpoints, the issuer URL should be changed and the issuer CA path can be omitted (assuming the OIDC discovery endpoint uses a publicly-trusted certificate). The grpc.tls config section should be used to add appropriate TLS certificates to the OTel collector’s gRPC endpoint.

Use the following Connect Helm values to enable this authentication method:

connect:
  telemetry:
    enabled: true
    collectorEndpoint: https://<your-collector-endpoint>:4317
    auth:
      k8sPSAT:
        enabled: true
        # Audience must match OTel collector config
        audience: otel-collector
        # Trust root for the OTel collector's gRPC endpoint
        tls:
          caFile: /path/to/tls.crt # Empty uses system roots.

To protect against replay attacks using Connect’s PSAT, the OTel endpoint must be secured with TLS. The caFile config key can be used to provide the collector’s CA certificate to Connect. This certificate should match the one provided in the grpc.tls section of the OTel configuration values above.

SPIFFE JWT-SVID

As an alternative to Kubernetes PSAT authentication, Connect also supports using a JWT-SVID as an OIDC bearer token. The OTel collector validates the signed token via the SPIRE OIDC discovery provider, providing SPIFFE-native security for Connect to OTel collector communication. This mode is useful when the OTel endpoint is outside the Connect cluster or when Kubernetes Service Account authentication is undesirable.

The OIDC discovery provider must be deployed to the Connect cluster - it can be enabled using the following SPIRE Helm chart values:

spiffe-oidc-discovery-provider:
  enabled: true

By default, the SPIRE OIDC discovery provider endpoints are secured using a SPIRE-issued TLS certificate which is not trusted by the OTel collector. The tls configuration section in the OIDC discovery provider’s Helm values provide various options for obtaining a TLS certificate which is trusted by the collector (e.g. a publicly-trusted certificate obtained via cert-manager).

The following Helm values provide an example of the required OTel collector configuration to receive telemetry data from Connect:

config:
  extensions:
    oidc:
      providers:
        - issuer_url: <JWT-SVID issuer configured in SPIRE Helm values>
          audience: otel-collector
  receivers:
    otlp:
      protocols:
        grpc:
          auth:
            authenticator: oidc
          tls:
            # TLS certificates must be trusted by Connect.
            cert_file: /path/to/tls.crt
            key_file: /path/to/tls.key
  exporters:
    # Use this to verify the collector is working
    # Replace with your desired export targets
    debug: {}
  service:
    extensions: [oidc]
    pipelines:
      traces: { receivers: [otlp], exporters: [debug] }
      metrics: { receivers: [otlp], exporters: [debug] }

The above configuration should be combined with any other OTel collector configuration required for your environment.

On the Connect side, set the following Helm values to enable the SPIFFE JWT-SVID authentication method:

connect:
  telemetry:
    enabled: true
    collectorEndpoint: https://<your-collector-endpoint>:4317
    auth:
      spiffeJwt:
        enabled: true
        # Audience must match OTel collector config
        audience: otel-collector
        # Trust root for the OTel collector's gRPC endpoint
        tls:
          caFile: /path/to/tls.crt # Empty uses system roots.

As with the Kubernetes Service Account authentication method, the OTLP/gRPC receiver endpoint must be secured with TLS to protect against replay attacks using Connect’s JWT-SVID. The tls.caFile key can be used to ensure Connect trusts the OTel collector’s certificate authority.

SPIFFE Mutual TLS

SPIFFE mTLS is the most secure authentication method which enables zero-trust telemetry.

To enable SPIFFE mTLS authentication between Connect and the target OTel receiver endpoint, add the following to the connect.telemetry section of the Connect Helm values:

connect:
  telemetry:
    enabled: true
    collectorEndpoint: https://<your-collector-endpoint>:4317
    auth:
      spiffeMtls:
        enabled: true
        collectorSpiffeId: "spiffe://example.org/ns/otel-collector/sa/otel-collector"

The standard OTel Collector does not support native SPIFFE mTLS, so a SPIFFE helper sidecar must be used to deliver X.509 certificates to the collector. This requires:

• SPIFFE CSI driver to be installed on the cluster (enabled by default in the SPIRE Helm chart)
• SPIFFE helper to be granted permission to obtain an X.509 SVID on behalf of the OTel Collector

To grant the SPIFFE helper this permission, create the following ClusterSPIFFEID resource on the Connect Kubernetes cluster:

apiVersion: spire.spiffe.io/v1alpha1
kind: ClusterSPIFFEID
metadata:
  name: otel-collector
spec:
  className: spire-mgmt-spire
  spiffeIDTemplate: "spiffe://{{ .TrustDomain }}/ns/{{ .PodMeta.Namespace }}/sa/{{ .PodSpec.ServiceAccountName }}"
  namespaceSelector:
    matchLabels: { kubernetes.io/metadata.name: otel-collector }
  podSelector:
    matchLabels: { app.kubernetes.io/name: opentelemetry-collector }

A ClusterSPIFFEID is required because the SPIRE server backing Connect cannot (by definition) use Connect for attestation policies, so the SPIRE controller manager is instead used to handle workload registration.

To configure the OTel collector’s spiffe-helper sidecar, create an otel-collector namespace and the following ConfigMap:

apiVersion: v1
kind: ConfigMap
metadata:
  name: spiffe-helper
  namespace: otel-collector
data:
  helper.conf: |
    agent_address = "/spiffe-workload-api/spire-agent.sock"
    cert_dir = "/certs"
    svid_file_name = "tls.crt"
    svid_key_file_name = "tls.key"
    svid_bundle_file_name = "ca.pem"
    daemon_mode = true

Once the above ClusterSPIFFEID and ConfigMap are in place, the following minimal Helm values can be used with the OTel Collector Helm chart to deploy a SPIFFE mTLS-enabled collector:

mode: deployment
image:
  repository: otel/opentelemetry-collector-k8s
# Run as non-root; fsGroup 0 lets the collector read the group-root cert files.
podSecurityContext:
  fsGroup: 0
securityContext:
  runAsUser: 65532
  runAsGroup: 0
# spiffe-helper fetches + rotates the collector's SVID into the shared mem volume.
extraContainers:
  - name: spiffe-helper
    image: ghcr.io/spiffe/spiffe-helper:0.10.1
    securityContext:
      runAsUser: 65532
      runAsGroup: 65532
    args: ["-config", "config/helper.conf"]
    volumeMounts:
      - { name: spiffe-helper-config, mountPath: /config, readOnly: true }
      - {
          name: spiffe-workload-api,
          mountPath: /spiffe-workload-api,
          readOnly: true,
        }
      - { name: certs, mountPath: /certs }
extraVolumes:
  - name: spiffe-helper-config
    configMap:
      name: spiffe-helper
  - name: spiffe-workload-api
    csi:
      driver: csi.spiffe.io
      readOnly: true
  - name: certs
    emptyDir:
      medium: Memory
extraVolumeMounts:
  - { name: certs, mountPath: /certs, readOnly: true }
config:
  receivers:
    otlp:
      protocols:
        grpc:
          endpoint: 0.0.0.0:4317
          tls:
            cert_file: /certs/tls.crt
            key_file: /certs/tls.key
            client_ca_file: /certs/ca.pem
            client_ca_file_reload: true
            reload_interval: 1m
  exporters:
    # Use this to verify the collector is working
    # Replace with your desired export targets
    debug: {}
  service:
    pipelines:
      traces: { receivers: [otlp], exporters: [debug] }
      metrics: { receivers: [otlp], exporters: [debug] }

To deploy the OTel Collector, use

helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts
helm install otel-collector open-telemetry/opentelemetry-collector \
  -n otel-collector \
  --version 0.133.0 \
  --values values.yaml

Note

The configuration above assumes that the Helm release name and namespace are both “otel-collector” so that the SPIFFE ID matches the value Connect expects during the mTLS handshake. Adjust the namespace and service account references in each of the above resources as appropriate for your target OTel collector.

Once the OTel Collector is running and configured correctly, the debug exporter should begin printing log lines of the form:

info    Traces    {"resource": {"service.instance.id": "6c7e67dc-1b5e-4836-9e80-75eb606c5189", "service.name": "otelcol-k8s", "service.version": "0.134.0"}, "otelcol.component.id": "debug", "otelcol.component.kind": "exporter", "otelcol.signal": "traces", "resource spans": 1, "spans": 2}

in the OTel Collector pod logs.

Finally, replace the debug exporter with dedicated configuration to export Connect’s telemetry into your chosen OTel-compatible backend.

SPIRE

This section applies to both the SPIRE server that backs the Connect control plane and the SPIRE server in each trust zone, which are deployed from the same Helm chart.

SPIRE exposes metrics in Prometheus format when both recommendations.enabled: true and recommendations.prometheus.enabled: true are set under the global.spire section in SPIRE Helm configuration. This is the default behaviour for Helm values generated using the cofidectl trust-zone helm values command.

To disable metrics on the SPIRE server and all Kubernetes-based SPIRE agents, set recommendations.prometheus.enabled: false or, to disable metrics on individual components, use:

spire-server:
  telemetry:
    prometheus:
      enabled: false
spire-agent:
  telemetry:
    prometheus:
      enabled: false

An example Grafana dashboard for monitoring SPIRE metrics is provided by the upstream SPIRE project.

OpenTelemetry

For environments which standardise on OpenTelemetry, the same scrape config can be provided to the OTel Collector Helm Chart under the config.receivers.prometheus configuration section.

Prometheus

If your existing monitoring infrastructure uses the kube-prometheus operator, you can enable automatic scraping of SPIRE metrics by setting telemetry.prometheus.podMonitor.enabled: true in both the spire-server and spire-agent blocks to automatically create PodMonitor resources.

For other Prometheus deployments, a basic scrape config for SPIRE server and agent metrics is shown below:

- job_name: spire-server
  kubernetes_sd_configs:
    - role: pod
      namespaces:
        names: [spire-server]
  relabel_configs:
    - source_labels: [__meta_kubernetes_pod_label_app_kubernetes_io_component]
      regex: server
      action: keep
    - source_labels: [__meta_kubernetes_pod_label_app_kubernetes_io_instance]
      regex: spire
      action: keep
    - source_labels: [__meta_kubernetes_pod_label_app_kubernetes_io_name]
      regex: server
      action: keep
    - source_labels: [__address__]
      regex: ([^:]+)(?::\d+)?
      target_label: __address__
      replacement: $1:9988
  metrics_path: /metrics
  scheme: http
- job_name: spire-agent
  kubernetes_sd_configs:
    - namespaces:
        names: [spire-system]
      role: pod
  metrics_path: /metrics
  relabel_configs:
    - action: keep
      regex: agent
      source_labels:
        - __meta_kubernetes_pod_label_app_kubernetes_io_name
    - action: keep
      regex: spire
      source_labels:
        - __meta_kubernetes_pod_label_app_kubernetes_io_instance
    - source_labels:
        - __meta_kubernetes_pod_name
      target_label: kubernetes_pod_name
    - regex: ([^:]+)(?::\d+)?
      replacement: $1:9988
      source_labels:
        - __address__
      target_label: __address__
  scheme: http

Credex

OpenTelemetry

Credex is instrumented to provide trace coverage for all major steps in a token exchange request, including a parent span covering the end-to-end HTTP request to the /token endpoint as well as child spans for requests to external services such as policy hooks, downstream OAuth authorisation servers and JWT signers.

Similar to the Connect API, Credex supports exporting metrics and traces using OTLP over gRPC to a compatible endpoint, including an OpenTelemetry Collector. To enable Credex telemetry, use the following Helm values:

credex:
  telemetry:
    enabled: true
    collectorEndpoint: https://<your-collector-endpoint>:4317
    auth:
      # One of the auth methods described below

Authentication

Credex supports the same three authentication methods for the OTLP/gRPC receiver endpoint as Connect.

Kubernetes Service Account

When the OTel collector is running in the same Kubernetes cluster as Credex, the Kubernetes PSAT authentication method allows Credex to securely send telemetry data to the collector with minimal additional configuration. To enable Kubernetes Projected Service Account auth, follow the OTel collector configuration steps in the corresponding Connect section then deploy Credex with the following Credex Helm values:

credex:
  telemetry:
    enabled: true
    collectorEndpoint: https://<your-collector-endpoint>:4317
    auth:
      k8sPSAT:
        enabled: true
        # Audience must match OTel collector config
        audience: otel-collector
        # Trust root for the OTel collector's gRPC endpoint
        tls:
          caFile: /path/to/tls.crt # Empty uses system roots.

As with Connect, the OTel receiver endpoint must be protected by TLS to prevent replay attacks using the Credex PSAT. The caFile key can be used to provide Credex with a trusted OTel collector certificate.

SPIFFE JWT-SVID

To authenticate Credex to an OTel receiver endpoint using its JWT-SVID as an OIDC bearer token, follow the Collector configuration steps in the corresponding Connect section then deploy Credex with the following Helm values:

credex:
  telemetry:
    enabled: true
    collectorEndpoint: https://<your-collector-endpoint>:4317
    auth:
      spiffeJwt:
        enabled: true
        # Audience must match OTel collector config
        audience: otel-collector
        # Trust root for the OTel collector's gRPC endpoint
        tls:
          caFile: /path/to/tls.crt # Empty uses system roots.

As with Connect, the OTel receiver endpoint must be protected by TLS to prevent replay attacks using the Credex JWT-SVID. The caFile key can be used to provide Credex with a trusted OTel collector certificate.

SPIFFE Mutual TLS

Credex’s SPIFFE-native mTLS authentication method for an OTLP/gRPC receiver endpoint is similar to Connect’s, so the same prerequisites around the SPIFFE CSI driver and X.509 SVID issuance apply. However, since Credex runs in a workload cluster rather than the control plane, we use a Connect Attestation Policy (instead of a ClusterSPIFFEID resource) to register the workload cluster’s OTel Collector and give it a SPIFFE ID:

cofidectl attestation-policy add static \
    --name otel-collector-$unique_suffix \
    --parent-id-path cluster/$cluster/spire/agents \
    --selectors "k8s:ns:otel-collector" \
    --selectors "k8s:sa:otel-collector-opentelemetry-collector" \
    --spiffe-id-path "ns/otel-collector/sa/otel-collector-opentelemetry-collector" \
    --trust-zone $trust_zone

where $cluster and $trust_zone must be replaced by the names of the Connect Cluster and TrustZone resources respectively.

Once the attestation policy has been created, install the Collector using the same Helm configuration as the Connect section above.

To enable telemetry in Credex, set the following Credex Helm values:

credex:
  telemetry:
    enabled: true
    collectorEndpoint: https://<your-collector-endpoint>:4317
    auth:
      spiffeMtls:
        enabled: true
        collectorSpiffeId: "spiffe://$trust_domain/ns/otel-collector/sa/otel-collector-opentelemetry-collector"

replacing $trust_domain with the domain of the trust zone Credex runs within (see cofidectl trust-zone list).

As with the Connect case, the OTel Collector pod logs should include output similar to the following when it is collecting Credex telemetry:

info    Traces    {"resource": {"service.instance.id": "456d495d-ef02-4c9c-85a4-56f962537016", "service.name": "otelcol-k8s", "service.version": "0.134.0"}, "otelcol.component.id": "debug", "otelcol.component.kind": "exporter", "otelcol.signal": "traces", "resource spans": 2, "spans": 3}

After configuring a non-debug exporter to push telemetry to an OTel-compatible backend, traces will appear with under the ‘credex’ service name.