Log Configuration

Name each container's logs, join stack traces into one record, and drop or mask what shouldn't leave the server — with labels, annotations or one agent-wide rule.

Everything here works the same for Docker containers, Kubernetes pods, PM2 apps, systemd services and log files: one set of rules in the agent, applied before a line is sent. Nothing has to be configured; the defaults name every container, join stack traces and mask card numbers and credentials.

Labels and annotations

Docker containers are configured with labels, Kubernetes pods with annotations on the pod template. A pod with several containers can set any annotation for one container as zharp.io/<container>.<key>, which wins over the pod-wide one.

Which name wins

• Docker: the label, then the container's ZHARP_SERVICE / ZHARP_ENV / ZHARP_VERSION environment variables, then the Compose service name, then the container name. Labels are read when a container starts, so recreate it to change them.
• Kubernetes: the container annotation, then the pod annotation, then a service.name the app sets itself through an OpenTelemetry SDK, then the app.kubernetes.io/name label (app.kubernetes.io/version for the version), then the deployment, statefulset, daemonset, cronjob or job name, then the container name. Pods without zharp.io/env get the chart's environment.
• Source: when none is set, well-known images give one — nginx/openresty → nginx, httpd → apache, postgres → postgresql, mysql/mariadb → mysql, redis/valkey → redis, mongo → mongodb, node → node, python → python, openjdk/eclipse-temurin/amazoncorretto/tomcat → java, golang → go (by the last part of the image name, so bitnami/postgresql counts).

services:
  checkout:
    image: ghcr.io/acme/checkout:1.4.2
    labels:
      zharp.service: checkout
      zharp.env: production
      zharp.version: "1.4.2"
      zharp.source: java
      zharp.logs.exclude: 'GET /actuator/health'
      zharp.logs.mask: '[{"pattern":"(card_holder=)\\w+","replace":"${1}***"}]'
  worker:
    image: ghcr.io/acme/worker:1.4.2
    environment:
      ZHARP_ENV: production        # read from the container when there is no label
      ZHARP_VERSION: "1.4.2"
  redis:
    image: redis:7                 # service "redis", source "redis" from the image
  debug-shell:
    image: busybox
    labels:
      zharp.logs: "false"          # not collected

A label or annotation that can't be used — a regex that doesn't compile, a mask that isn't JSON — is reported once in the agent's log and left out; the container's other settings still apply. For Kubernetes, see Kubernetes.

Multi-line records

A stack trace is one event, so the agent sends it as one record: the first line's time and level, the whole trace as the message. This is on by default (auto) for Docker containers, Kubernetes pods, PM2 apps, journal services and log files.

2026-10-19 12:00:01.123 ERROR [http-1] c.e.Checkout - charge failed
java.lang.IllegalStateException: Order 42 has no lines
	at com.example.shop.OrderService.total(OrderService.java:57)
Caused by: java.sql.SQLException: Connection is not available
	... 52 common frames omitted

2026-10-19 12:00:03,456 ERROR payments: charge 7 failed
Traceback (most recent call last):
  File "/app/app.py", line 9, in charge
    1 / 0
ZeroDivisionError: division by zero

The auto rules

• A record starts at a line that begins with a timestamp (ISO 8601, 2026/10/19, syslog, Apache, klog …), a level word (ERROR, [WARN], INFO:), a JSON object, or a crash header (panic:, Exception in thread).
• A line continues the record before it when it is indented (Java \tat …, Node at … and Python File … lines) or blank, or has a shape stack traces print unindented: Caused by:, ... 52 more, Traceback (most recent call last):, an exception line such as java.lang.IllegalStateException: … or ZeroDivisionError: …, Go's goroutine 1 [running]: and function frames.
• Anything else is a record of its own, so access logs and plain output stay one line per record.
• Bounds: a record ends after 1,000 lines or 256 KiB (the rest becomes the next record), and is sent once no line has followed for a second.

Set zharp.logs.multiline (or zharp.io/logs-multiline) to a regex to say exactly where records start — every line that doesn't match joins the one before — or to off for one record per line. Each stream (a container's stdout, its stderr, a file, a systemd unit) is joined on its own, so interleaved output doesn't mix.

Processing rules

Exclude drops a record whose text matches a regex. Mask rewrites every match before the record leaves the server. They run on the joined record, in this order: the container's or pod's own rules, then the agent-wide rules, then the built-in masks. Exclusion is checked before masking. Patterns are RE2 regular expressions (syntax).

Agent-wide rules

# drop health checks from every source
sudo zharp-collector add rule exclude --pattern 'GET /healthz?'

# rewrite secrets before they leave the server ($1 keeps the first group)
sudo zharp-collector add rule mask --pattern '(password=)\S+' --replace '$1***'
sudo zharp-collector add rule mask --pattern '\b\d{3}-\d{2}-\d{4}\b'      # → [REDACTED]

# the built-in masks
sudo zharp-collector add rule default-masks --off
sudo zharp-collector add rule default-masks --on

# or at install time
curl -fsSL https://api.zharp.io/api/v1/install/agent.sh | sh -s -- --token YOUR_TOKEN \
  --log-exclude 'GET /healthz?' --log-mask '(?i)ssn=\d+' --no-default-masks

add rule writes the zharp_logs processor in /etc/zharp-collector/config.yaml and restarts the agent; you can also edit it there. On Kubernetes they are the chart's rules.exclude and rules.mask values (Chart values).

processors:
  zharp_logs:
    default_masks: true
    disable_default_masks: [jwt]     # keep the others
    rules:
      - type: exclude
        pattern: 'GET /healthz?'
      - type: mask
        pattern: '(password=)\S+'
        replace: '$$1***'             # $ is written $$ in config.yaml

Built-in masks

On by default, everywhere, after your own rules:

Turn them all off with zharp-collector add rule default-masks --off, --no-default-masks at install, default_masks: false in config.yaml, or rules.defaultMasks=false in the chart. Turn off single ones with disable_default_masks (chart: rules.disableDefaultMasks).