Kubernetes

One agent per node, as a DaemonSet: every container's logs, kubelet and node metrics, and an OTLP endpoint for your applications — named by pod annotations.

The agent runs on every node, control plane included. It reads the container log files the kubelet writes, asks the kubelet for node, pod and container metrics, and receives traces, metrics and logs your applications send with an OpenTelemetry SDK. Everything is tagged with the pod, namespace, node and workload it came from, and named after the pod's zharp.io annotations.

Install with Helm

1. 1
   Get an API key

   Create one under Integrations & API → Main API keys with the Ingest only scope. One key serves every node of every cluster.

2. 2
   Install the chart

   The chart is in the agent repository. environment becomes deployment.environment.name on everything whose pod has no zharp.io/env annotation; clusterName is sent as k8s.cluster.name.

   Helm

   git clone https://github.com/zharp-io/agent.git
   helm install zharp-agent ./agent/deploy/helm/zharp-agent \
     --namespace zharp-system --create-namespace \
     --set apiKey=YOUR_API_KEY \
     --set environment=production \
     --set clusterName=prod-eu

   To keep the key out of Helm values, create the Secret yourself and point the chart at it:

   Helm, with your own Secret

   kubectl create namespace zharp-system
   kubectl -n zharp-system create secret generic zharp-agent --from-literal=api-key=YOUR_API_KEY
   
   helm install zharp-agent ./agent/deploy/helm/zharp-agent -n zharp-system \
     --set existingSecret=zharp-agent

3. 3
   Check it runs on every node

   kubectl -n zharp-system get pods -l app.kubernetes.io/name=zharp-agent -o wide

   Logs show up in Logs within a minute, each under its service name. Metrics follow at the next collection (every 30 seconds).

Without Helm

deploy/kubernetes holds the same objects, rendered from the chart with its defaults (zharp-agent.yaml, regenerated by render.sh). The API key comes from a Secret you create first:

kubectl create namespace zharp-system
kubectl -n zharp-system create secret generic zharp-agent --from-literal=api-key=YOUR_API_KEY
kubectl apply -k "github.com/zharp-io/agent/deploy/kubernetes?ref=main"

What it collects

The agent's own pods are never read. Every record also carries service.name, deployment.environment.name, service.version and, for logs, zharp.source — see below.

Naming your workloads

Put annotations on the pod template. Without them, the service is the pod's app.kubernetes.io/name label, else the deployment, statefulset, daemonset, cronjob or job name, else the container name; the version falls back to app.kubernetes.io/version, and the source to a guess from the image (nginx, postgres, redis, eclipse-temurin, python …). An application that sets its own service.name through an OpenTelemetry SDK keeps it unless the pod is annotated.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: checkout
spec:
  template:
    metadata:
      annotations:
        zharp.io/service: checkout
        zharp.io/env: production
        zharp.io/version: "2.3.1"
        zharp.io/source: java
        # this pod's own rules, on top of the agent-wide ones
        zharp.io/logs-exclude: 'GET /actuator/health'
        zharp.io/logs-mask: '[{"pattern":"(card_holder=)\\w+","replace":"${1}***"}]'
    spec:
      containers:
        - name: app
          image: ghcr.io/acme/checkout:2.3.1

The annotations are read on the node and removed before sending. Log Configuration has the multi-line rules, the rules' order and the built-in masks.

Sending traces from your apps

Point the SDK at the agent's Service. Its internalTrafficPolicy: Local sends each pod's data to the agent on the same node, which tags it with that pod and its annotations — an app that doesn't set service.name is named like its logs.

env:
  - name: OTEL_EXPORTER_OTLP_ENDPOINT
    value: http://zharp-agent.zharp-system.svc:4318
  # or, with otlp.hostPort.enabled=true, the agent on the pod's own node:
  # - name: HOST_IP
  #   valueFrom: { fieldRef: { fieldPath: status.hostIP } }
  # - name: OTEL_EXPORTER_OTLP_ENDPOINT
  #   value: http://$(HOST_IP):4318

Chart values

environment: production
clusterName: prod-eu
logs:
  excludeNamespaces: [kube-system]
rules:
  exclude:
    - 'GET /healthz'
  mask:
    - pattern: '(password=)\S+'
      replace: '${1}***'
  disableDefaultMasks: [jwt]

What it needs

• Read-only RBAC: get, list and watch on pods, namespaces, nodes, endpoints, replicasets, deployments, statefulsets, daemonsets, jobs and cronjobs (to name workloads and read annotations), and get on nodes/stats, nodes/proxy and nodes/metrics for the kubelet.
• Host paths, read-only: /var/log/pods, /var/lib/docker/containers (where Docker-runtime nodes keep the files), and / as /hostfs for node metrics. One writable path, /var/lib/zharp-agent, remembers how far each file was read, so a restart neither repeats nor skips lines.
• User: the container runs as root, because the kubelet's log files are readable only by root; it drops every capability and has a read-only root filesystem.