Custom Monitors

Watch anything that can produce JSON. Zharp polls your endpoint or receives what you send it, and your trigger rules decide when that means trouble.

How a custom monitor works

A Custom monitor gets its data one of two ways, chosen under SOURCE TYPE:

Either way the data — the parsed response body, or the body you posted — is run through the monitor's TRIGGER RULES. A rule describes trouble: when it matches, it fires. Each evaluation is a check, and it fails when:

• the rules fire — one of them, or all of them, depending on Trigger evaluation — and at least one fired rule is Down or Alert (an Info rule is only recorded);
• the Success condition isn't met (by default there is none beyond the rules);
• in pull mode, the request can't be completed — no connection, or no response within the timeout;
• in push mode with Expect data every set, nothing has arrived for that long.

Failed checks open an incident following the Confirmation period and Recovery period under Incident behavior, the same as every monitor — see Monitors.

Create a custom monitor

1. 1
   Choose the type

   On the Monitors page, click New, open the type picker and choose Custom monitor, listed under Custom.

2. 2
   Pick the source type

   Under SOURCE TYPE, choose Pull (Poll) to have Zharp call an endpoint, or Push (API) to send data yourself. New monitors start on pull.

3. 3
   Set up the source

   For pull, build the request under REQUEST BUILDER (see Pull mode). For push, copy the URL shown under Send data to this monitor via API (see Push mode).

4. 4
   Add trigger rules

   Click Add trigger rule, choose Builder or Script, fill it in and click Save Rule. A monitor needs at least one rule.

5. 5
   Tune and create

   Under Trigger & execution, set how rules combine and the execution limits (see the settings table). For pull, set Check frequency under Incident behavior. Pick who is alerted and click Create monitor.

Pull mode

Under REQUEST BUILDER, pick the method — GET, POST, PUT, PATCH, DELETE or HEAD — and enter the endpoint, such as https://api.example.com/health (https:// is added if you leave it out). The tabs underneath add Headers, query Params and a request Body. The body is sent exactly as typed, so add a Content-Type header if your endpoint needs one; an API key or Authorization header goes under Headers.

The endpoint must be reachable from the internet; private and internal addresses are refused. The request runs at the monitor's Check frequency, set under Incident behavior, from each region you select.

Your rules then see:

• the response body as data — parsed when it is JSON, otherwise the raw text (and a builder rule then finds no fields);
• the response status code and headers, to script rules.

Push mode

With Push (API) chosen, the form shows the monitor's own URL under Send data to this monitor via API. Copy it with the button beside it — the token at the end is the only credential, so keep it secret. POST a JSON object to it whenever you have data:

curl -X POST https://api.zharp.io/api/v1/push/YOUR_PUSH_TOKEN \
  -H "Content-Type: application/json" \
  -d '{"queue_depth": 1500, "workers": 4, "status": "degraded"}'

Rules run on the payload straight away, and the reply says what they decided — status is UP or DOWN, and firedTriggers lists each rule that fired:

{
  "statusCode": 200,
  "message": "Payload received and evaluated",
  "data": {
    "status": "DOWN",
    "firedTriggers": [
      { "name": "Queue overflow", "severity": "down", "reason": "queue_depth (1500) greater than 1000" }
    ]
  }
}

An unknown token gets 404 with Invalid push token. While the monitor is paused, payloads are ignored: the reply is 200 with Monitor is paused; payload ignored and status PAUSED, and nothing is evaluated or recorded. The most recent payload is kept and shown on the monitor page. Check frequency does nothing for a push monitor.

Alert when data stops

A push monitor is quiet between payloads, and by default silence is not a failure. To be alerted when your sender stops, set Expect data every under Trigger & execution to 15 minutes, 1 hour, 6 hours or 24 hours. If that long passes after the last payload with nothing new, the monitor goes down with No data received for 3720s, expected every 3600s. — or, if nothing has ever arrived, No data received from this monitor yet. The next payload is evaluated as usual.

Trigger rules

Each rule has a RULE NAME and is either a Builder condition or a Script. Give every rule a different name: a rule's timing and its line in the incident are tracked by name.

Builder rules

A builder rule reads one field and compares it. Under CONDITION, enter the field's path after When, pick an operator after is, and enter the value to compare with.

• Write the path as it appears in the JSON, with dots between levels: queue_depth, stats.cpu.usage, or items.0.status for the first element of an array. A leading payload. or data. is allowed and ignored, so payload.queue_depth reads queue_depth.
• A path that isn't in the payload reads as missing.

The four number comparisons never fire when either side isn't a number, so a payload with a missing or non-numeric field doesn't fire them.

DURATION sets how long the condition must hold before the rule fires — For a number of seconds, minutes or hours, consecutive. It is timed from the first evaluation where the condition was true; an evaluation where it is false starts the clock again. A new rule starts at 5 minutes; set 0 to fire on the first match. Until then the rule is pending, not fired, and the check passes. In push mode the clock only moves when payloads arrive. Only builder rules have a duration.

SEVERITY is Alert, Down (the default) or Info. See Severity and trigger evaluation.

Script rules

A script rule is JavaScript under TRIGGER SCRIPT. It must export a default function, which may be async. The function gets the data and returns whether to fire:

export default async function trigger({ data, headers, status, meta }) {
  if (data.queue_depth > 1000) {
    return { fire: true, severity: 'down', message: `Queue at ${data.queue_depth}` }
  }
  return { fire: false }
}

A script rule has no duration: it fires on the first evaluation that returns fire: true. To wait before alerting, use the Confirmation period under Incident behavior.

Click Run above the editor to test the script. It runs against the example JSON in the SAMPLE PAYLOAD box that opens; edit the JSON and click Run again. The result reads Fires (with the severity, if returned), Does not fire, or the error, followed by anything the script logged with console.log. A test run gets the sample as data, status 200 and empty headers, and has 5 seconds.

What a script can use

Scripts run in an isolated JavaScript sandbox — not Node.js. Available: the JavaScript language itself (JSON, Math, Date, Promise, regular expressions…), console, setTimeout and the other timers, URL, URLSearchParams, atob, btoa, and fetch with Headers and Response. There is no require, import, process, file system or environment variables.

Script examples

// Fire when more than 5% of requests failed.
export default function trigger({ data }) {
  const rate = data.errors / Math.max(data.requests, 1)
  if (rate > 0.05) {
    return { fire: true, severity: 'alert', message: `Error rate ${(rate * 100).toFixed(1)}%` }
  }
  return { fire: false }
}

// Pull mode: judge the HTTP response itself.
export default function trigger({ data, status, headers }) {
  if (status >= 500) {
    return { fire: true, message: `Endpoint returned ${status}` }
  }
  if (headers['x-maintenance'] === 'on') {
    return { fire: true, severity: 'info', message: 'Maintenance flag is set' }
  }
  return { fire: false }
}

// Fire when any job in the list has failed.
export default function trigger({ data }) {
  const failed = (data.jobs || []).filter((job) => job.state === 'failed')
  if (failed.length > 0) {
    return { fire: true, message: `${failed.length} failed: ${failed.map((j) => j.name).join(', ')}` }
  }
  return { fire: false }
}

// Cross-check another service before firing.
export default async function trigger({ data }) {
  if (data.queue_depth <= 1000) return { fire: false }

  const res = await fetch('https://status.example.com/api/workers')
  const workers = await res.json()

  if (workers.online < 2) {
    return { fire: true, severity: 'down', message: `Queue at ${data.queue_depth} with ${workers.online} workers` }
  }
  return { fire: false }
}

Severity and trigger evaluation

Trigger evaluation decides how rules combine. With Any rule matches (default), one fired rule fails the check. With All rules match, the check fails only when every rule has fired in the same evaluation; a pending rule, or a script that errored, keeps it passing.

Severity decides what a fired rule does:

With All rules match, every rule must fire and at least one of them must be Down or Alert. When rules of different severities fire together, the incident takes the most severe one.

A failed check's message lists the reason for every Down or Alert rule that fired, separated by ; — for a builder rule the field, the value it had, the operator and the threshold, such as queue_depth (1500) greater than 1000. A passing check reads No triggers fired.

Trigger & execution settings

When it goes down

The monitor page

• The source — PUSH with the start of the push URL, or PULL with the endpoint and method — and the status: HEALTHY, the number of rules FIRED, FAILING (failed with no rule fired) or WAITING (no data yet).
• A chart: Ingested events / min for push, Fetch + eval latency for pull.
• A strip with STATUS, TRIGGERS PASSING, how many fired, MATCH MODE (ANY or ALL) and the success condition.
• Triggers: each builder rule with its condition, the value RECEIVED in the latest data, and the severity it FIRES AS; script rules are shown as code. A matched Info rule is marked with an info icon rather than as failing. Script rules that failed to run on the latest evaluation are listed below with their errors.
• Recent evaluations: one row per evaluation, marked PASS, INFO (only Info rules fired), FIRED or FAILED, with the rule that fired and the value observed. Expand a row for the values the rules saw.
• Beside it, Ingress for push (events per minute, last received) or Data source for pull (endpoint, method, auth, interval — the Check frequency — and timeout), and Webhook when a webhook integration alerts for this monitor.

Troubleshooting

A builder rule never fires

• Check the path against the payload, level by level: stats.cpu for { "stats": { "cpu": 90 } }. The RECEIVED column on the monitor page shows — when the path finds nothing.
• Number comparisons need numbers on both sides. A value like "85%" isn't one.
• The rule's DURATION may not have passed yet: a new rule waits 5 minutes.
• An Info rule fires without taking the monitor down — look for INFO on the evaluation.
• In pull mode, make sure the endpoint returns JSON; a text body has no fields.

A script rule never fires

Look for a script error under Triggers on the monitor page, and test it with Run and a payload like the real one. It must return { fire: true } — fire: 1 or a thrown error doesn't count. A rule returning severity: 'info' fires without taking the monitor down.

Every push fails

Read the message on a failed evaluation: it names the rules that fired. With Exit code 0, a payload whose exit_code isn't 0 fails. Also send the body as JSON with Content-Type: application/json.

The pull monitor stays up although the endpoint returns errors

With No triggers fired, an error status passes unless a rule catches it. Set Success condition to HTTP 2xx response.