How the Agent Works

The Zharp Agent is an OpenTelemetry Collector built for Zharp. It runs on your server, reads what is there, and pushes it to Zharp over HTTPS.

It only ever connects out. Nothing on the server has to accept connections from the internet, which is what lets Zharp monitor databases and services behind a firewall. To install it, see Agent Setup.

Where the data goes

Everything goes to the Zharp API, by default https://api.zharp.io/api/v1, with the workspace API key in the X-API-KEY header.

Which address it uses: the install command writes the address it was downloaded from into the agent's config, so an agent installed from your dashboard reports to the same Zharp. An agent with no address configured falls back to https://api.zharp.io/api/v1. For self-hosted Zharp, install with --endpoint https://your-zharp/api/v1, or set endpoint: under the zharp exporter in the config.

If Zharp can't be reached, the agent keeps running and sends again as soon as it can; what it collected during the gap isn't sent later. A host that hasn't sent a heartbeat for 2 minutes shows as idle, and after 15 as offline.

What stays on the server

Database passwords never leave it. They live in the agent's env file and are used to connect locally; Zharp only receives the metrics. The install command uses an Ingest only key made for the agent: it can send metrics and logs to your workspace but can't read or change anything through the API. Anyone holding it can still push data in your name, so keep it as private as the server. Revoking it under Integrations & API → Main API keys cuts off every host that uses it.

On Windows the same files are in C:\ProgramData\zharp-collector\, with env.ps1 in place of env.

Application traces are accepted on 127.0.0.1:4317 (gRPC) and :4318 (HTTP) — localhost only, since a trace can carry request URLs, query text and error messages.

What it finds by itself

At install the agent looks for nginx, Apache, IIS, system logs, Docker, Redis, PostgreSQL, MySQL and MongoDB, and for application logs: Docker containers, systemd services and PM2 apps. Whatever works with its defaults is collected straight away; a database that needs a monitoring user is listed on the host in the dashboard as Needs setup, with the steps to finish it.

Application logs. The agent runs as the unprivileged user zharp-collector and reads each source the way that needs the least access:

• Docker containers through the Docker API socket (the user is in the docker group, as for container metrics) — not from /var/lib/docker. Each record carries service.name (the zharp.service label, the container's ZHARP_SERVICE, the Compose service, else the container name), the environment, version and source from the same labels, the container's id, name and image, and the stream; a level is set only when the line states one (a JSON level/severity field, level=, or a leading ERROR/WARN/INFO/DEBUG). stderr isn't treated as an error. Containers labelled zharp.logs=false, and the agent's own container, are skipped. If Docker restarts, the agent reconnects by itself.
• systemd services through journalctl (the user is in systemd-journal): the service is the unit without .service (the syslog identifier when there is no unit), and the level comes from the priority — 0–2 fatal, 3 error, 4 warning, 5–6 info, 7 debug. Left out as operating system rather than application: systemd, systemd-*, init.scope, dbus, cron/crond/anacron, ssh/sshd, polkit, NetworkManager, snapd, udisks2, accounts-daemon, rsyslog, getty@, user@, session-*, sudo, su, login, kernel and audit records, the agent's own zharp-collector unit, and containers using Docker's journald log driver (already read through the API). Commands typed in a shell or run by hand (session scopes) are left out too. The list is in the filter step of the journald receiver in config.yaml if you want to change it.
• PM2 by tailing <home>/.pm2/logs of root and every user (and $PM2_HOME/logs). app-out.log, app-error.log and the cluster files app-out-3.log all report as app. The install grants read access with ACLs to those folders only — setfacl -m u:zharp-collector:x on each folder above that others can't enter, setfacl -R -m u:zharp-collector:rX on the logs folder and a default ACL so new files are readable too. Nothing is chmod-ed.

Journal and syslog. On most distributions rsyslog writes the journal's lines into /var/log/syslog or /var/log/messages, so reading both would send every line twice. While the journal is read, those two files aren't tailed; System logs keeps auth.log and kern.log — logins and the kernel, which the journal source leaves out — so nothing is collected twice.

Before a line leaves the server. Stack traces are joined into one record per stream, then every log record — whatever its source — passes through the zharp_logs processor: the container's or pod's own exclude and mask rules, the agent-wide rules, and the built-in masks for card numbers, bearer tokens, JWTs, AWS keys and private keys. See Log Configuration.

Each source starts at the end on first run: the history from before the install isn't sent. After that the agent remembers where it stopped, so a restart picks up without repeating or skipping lines. Install with --no-app-logs (or ZHARP_NO_APP_LOGS=1) to leave application logs out, and add any of them later with zharp-collector add docker-logs, add journal or add pm2.

For databases the agent reports connection usage against the server's limit, and whether it could connect at all. A private database monitor goes down when the agent can't connect, when connections pass 90% of the limit, when MySQL refuses connections at its limit, or when the database or the agent stops reporting — see Private Databases.

An agent can report on several databases of one engine. The first is added as usual; each further one gets a name with --name, and runs in a pipeline of its own whose points carry zharp.database (for example postgresql/orders). Zharp registers each name as a separate database of that host, so each can have its own private monitor. Hosts with one database per engine report exactly as before.

Commands

On Windows, run the same commands in an elevated PowerShell, without sudo.

Containers and Kubernetes

On Kubernetes the agent runs as a DaemonSet from its Helm chart: it reads /var/log/pods, asks the kubelet for node, pod and container metrics, tags everything with its pod and workload and names it from the zharp.io/* annotations — see Kubernetes.

The agent is also published as ghcr.io/zharp-io/agent, with ready-made manifests for Kubernetes, Docker Compose, Swarm and ECS in the agent repository. Its source and every release are at github.com/zharp-io/agent.

On ECS Fargate, Google Cloud Run and Azure Container Apps the same image runs as a sidecar in your task, service or app: started with ZHARP_MODE=sidecar, it builds its config from environment variables, receives OTLP, FireLens logs and StatsD on localhost, and adds the platform's attributes — see Serverless & Sidecars.