Monitoring

The Cofide Connect platform provides metrics, traces and other telemetry insights for workload identity components. These are intended to augment, rather than replace, existing monitoring solutions for Kubernetes and cloud-native systems.

Connect

Export to OTel Collector

The Cofide Connect API is instrumented using Open Telemetry and is able to push data to a dedicated local OTel Collector using SPIFFE mTLS authentication for zero-trust telemetry.

To enable telemetry in Connect, add the following Helm chart values:

connect:
  telemetryEnabled: true
  extraEnv:
  - name: OTEL_EXPORTER_OTLP_ENDPOINT
    value: https://otel-collector-opentelemetry-collector.otel-collector.svc.cluster.local:4317

OTel Collector does not support native SPIFFE mTLS, so a SPIFFE helper sidecar must be used to deliver X.509 certificates to the collector. This requires:

• SPIFFE CSI driver to be installed on the cluster (enabled by default in the SPIRE Helm chart)
• SPIFFE helper to be granted permission to obtain an X.509 SVID on behalf of the OTel Collector

To grant the SPIFFE helper this permission, create the following ClusterSPIFFEID resource on the Connect Kubernetes cluster:

apiVersion: spire.spiffe.io/v1alpha1
kind: ClusterSPIFFEID
metadata:
  name: otel-collector
spec:
  className: spire-mgmt-spire
  spiffeIDTemplate: "spiffe://{{ .TrustDomain }}/ns/{{ .PodMeta.Namespace }}/sa/{{ .PodSpec.ServiceAccountName }}"
  namespaceSelector:
    matchLabels: { kubernetes.io/metadata.name: otel-collector }
  podSelector:
    matchLabels: { app.kubernetes.io/name: opentelemetry-collector }

A ClusterSPIFFEID is used here because the SPIRE server backing Connect cannot (by definition) use Connect for attestation policies, so the upstream SPIRE controller manager is used to handle workload registration.

To configure the OTel collector’s spiffe-helper sidecar, create an otel-collector namespace and the following ConfigMap:

apiVersion: v1
kind: ConfigMap
metadata:
  name: spiffe-helper
  namespace: otel-collector
data:
  helper.conf: |
    agent_address = "/spiffe-workload-api/spire-agent.sock"
    cert_dir = "/certs"
    svid_file_name = "tls.crt"
    svid_key_file_name = "tls.key"
    svid_bundle_file_name = "ca.pem"
    daemon_mode = true

Once the above ClusterSPIFFEID and ConfigMap are in place, the following minimal Helm values can be used with the OTel Collector Helm chart to deploy a SPIFFE mTLS-enabled collector:

mode: deployment
image:
  repository: otel/opentelemetry-collector-k8s
# Run as non-root; fsGroup 0 lets the collector read the group-root cert files.
podSecurityContext:
  fsGroup: 0
securityContext:
  runAsUser: 65532
  runAsGroup: 0
# spiffe-helper fetches + rotates the collector's SVID into the shared mem volume.
extraContainers:
  - name: spiffe-helper
    image: ghcr.io/spiffe/spiffe-helper:0.10.1
    securityContext:
      runAsUser: 65532
      runAsGroup: 65532
    args: ["-config", "config/helper.conf"]
    volumeMounts:
      - { name: spiffe-helper-config, mountPath: /config, readOnly: true }
      - { name: spiffe-workload-api, mountPath: /spiffe-workload-api, readOnly: true }
      - { name: certs, mountPath: /certs }
extraVolumes:
  - name: spiffe-helper-config
    configMap:
      name: spiffe-helper
  - name: spiffe-workload-api
    csi:
      driver: csi.spiffe.io
      readOnly: true
  - name: certs
    emptyDir:
      medium: Memory
extraVolumeMounts:
  - { name: certs, mountPath: /certs, readOnly: true }
config:
  receivers:
    otlp:
      protocols:
        grpc:
          endpoint: 0.0.0.0:4317
          tls:
            cert_file: /certs/tls.crt
            key_file: /certs/tls.key
            client_ca_file: /certs/ca.pem
            client_ca_file_reload: true
            reload_interval: 1m
  exporters:
    # Use this to verify the collector is working
    # Replace with your desired export targets
    debug: {}
  service:
    pipelines:
      traces: { receivers: [otlp], exporters: [debug] }
      metrics: { receivers: [otlp], exporters: [debug] }

To deploy the OTel Collector, use

helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts
helm install otel-collector open-telemetry/opentelemetry-collector \
  -n otel-collector \
  --version 0.133.0 \
  --values values.yaml

Note

The release name and namespace must be “otel-collector” so that the SPIFFE ID matches the value Connect expects during the mTLS handshake.

Once the OTel Collector is running and configured correctly, the debug exporter should begin printing log lines of the form:

info    Traces    {"resource": {"service.instance.id": "6c7e67dc-1b5e-4836-9e80-75eb606c5189", "service.name": "otelcol-k8s", "service.version": "0.134.0"}, "otelcol.component.id": "debug", "otelcol.component.kind": "exporter", "otelcol.signal": "traces", "resource spans": 1, "spans": 2}

in the OTel Collector pod logs.

Finally, replace the debug exporter with dedicated configuration to export Connect’s telemetry into your chosen OTel-compatible backend.

SPIRE

SPIRE exposes metrics in Prometheus format when both recommendations.enabled: true and recommendations.prometheus.enabled: true are set under the global.spire section in SPIRE Helm configuration. This is the default behaviour for Helm values generated using the cofidectl trust-zone helm values command.

To disable metrics on the SPIRE server and all Kubernetes-based SPIRE agents, set recommendations.prometheus.enabled: false or, to disable metrics on individual components, use:

spire-server:
  telemetry:
    prometheus:
      enabled: false
spire-agent:
  telemetry:
    prometheus:
      enabled: false

An example Grafana dashboard for monitoring SPIRE metrics is provided by the upstream SPIRE project.

Export to OTel Collector

For environments which standardise on Open Telemetry, the same scrape config can be provided to the OTel Collector Helm Chart under the config.receivers.prometheus configuration section.

Export to Prometheus

If your existing monitoring infrastructure uses the kube-prometheus operator, you can enable automatic scraping of SPIRE metrics by setting telemetry.prometheus.podMonitor.enabled: true in both the spire-server and spire-agent blocks to automatically create PodMonitor resources.

For other Prometheus deployments, a basic scrape config for SPIRE server and agent metrics is shown below:

- job_name: spire-server
  kubernetes_sd_configs:
    - role: pod
      namespaces:
        names: [spire-server]
  relabel_configs:
    - source_labels: [__meta_kubernetes_pod_label_app_kubernetes_io_component]
      regex: server
      action: keep
    - source_labels: [__meta_kubernetes_pod_label_app_kubernetes_io_instance]
      regex: spire
      action: keep
    - source_labels: [__meta_kubernetes_pod_label_app_kubernetes_io_name]
      regex: server
      action: keep
    - source_labels: [__address__]
      regex: ([^:]+)(?::\d+)?
      target_label: __address__
      replacement: $1:9988
  metrics_path: /metrics
  scheme: http
- job_name: spire-agent
  kubernetes_sd_configs:
    - namespaces:
        names: [spire-system]
      role: pod
  metrics_path: /metrics
  relabel_configs:
    - action: keep
      regex: agent
      source_labels:
        - __meta_kubernetes_pod_label_app_kubernetes_io_name
    - action: keep
      regex: spire
      source_labels:
        - __meta_kubernetes_pod_label_app_kubernetes_io_instance
    - source_labels:
        - __meta_kubernetes_pod_name
      target_label: kubernetes_pod_name
    - regex: ([^:]+)(?::\d+)?
      replacement: $1:9988
      source_labels:
        - __address__
      target_label: __address__
  scheme: http

Credex

Credex is instrumented to provide trace coverage for all major steps in a token exchange request, including a parent span covering the end-to-end HTTP request to the /token endpoint as well as child spans for requests to external services such as policy hooks, downstream OAuth authorisation servers and JWT signers.

Export to OTel Collector

Similar to the Connect API, Credex supports pushing metrics and traces to an OTel Collector using SPIFFE-native mTLS authentication; as such, the same pre-requisites around the SPIFFE CSI driver and X.509 SVID issuance apply. However, since Credex runs in a workload cluster rather than the control plane, we use a Connect Attestation Policy (instead of a ClusterSPIFFEID resource) to register the workload cluster’s OTel Collector and give it a SPIFFE ID:

cofidectl attestation-policy add static \
    --name otel-collector-$unique_suffix \
    --parent-id-path cluster/$cluster/spire/agents \
    --selectors "k8s:ns:otel-collector" \
    --selectors "k8s:sa:otel-collector-opentelemetry-collector" \
    --spiffe-id-path "ns/otel-collector/sa/otel-collector-opentelemetry-collector"

cofidectl attestation-policy-binding add \
    --trust-zone $trust_zone \
    --attestation-policy otel-collector-$unique_suffix

where $cluster and $trust_zone must be replaced by the names of the Connect Cluster and TrustZone resources respectively.

Once the attestation policy and binding have been created, install the Collector using the same Helm configuration as the Connect section above.

To enable telemetry in Credex, set the following Credex Helm values:

credex:
  telemetry:
    enabled: true
    targetSPIFFEID: "spiffe://$trust_domain/ns/otel-collector/sa/otel-collector-opentelemetry-collector"
  extraEnv:
    - name: OTEL_EXPORTER_OTLP_ENDPOINT
      value: "https://otel-collector-opentelemetry-collector.otel-collector.svc.cluster.local:4317"

replacing $trust_domain with the domain of the trust zone Credex runs within (see cofidectl trust-zone list).

As with the Connect case, the OTel Collector pod logs should include output similar to the following when it is collecting Credex telemetry:

info    Traces    {"resource": {"service.instance.id": "456d495d-ef02-4c9c-85a4-56f962537016", "service.name": "otelcol-k8s", "service.version": "0.134.0"}, "otelcol.component.id": "debug", "otelcol.component.kind": "exporter", "otelcol.signal": "traces", "resource spans": 2, "spans": 3}

After configuring a non-debug exporter to push telemetry to an OTel-compatible backend, Credex traces will appear with a root span name of ‘credex-server’.