Escalation & On-call
On-call schedules say who is on duty. Escalation policies say who gets alerted about an incident, in what order, how, and what happens if nobody answers.
Both live under Team Members, on the On-Call Schedules and Escalation Policies tabs.
Who gets paged by default
When an incident opens on a monitor, Zharp works it out in this order:
- If the monitor has an escalation policy, the policy runs.
- Otherwise, whoever is on call on your primary on-call schedule is paged.
- If there is no schedule, or nobody is on call right now, nobody is paged: the monitor's “is DOWN” alert to its owner and subscribers is all that goes out. With Escalate to the team chosen for when nobody acknowledges, the team is paged once after 5 minutes — when there is a schedule.
- A page goes out once. It repeats only when the escalation policy, or Repeat the notification, says so.
Incidents reported by hand with Whoever is on call (default), and every report sent through the public report link, follow the same route. See Incidents.
On-call schedules
- 1Create the schedule
On Team Members → On-Call Schedules, click Create Schedule and give it a Name.
- 2Add responders
Under Responders, search colleagues and add them. They take turns in the order you add them; drag them to change the order.
- 3Set the rotation
Change shift every sets how long each turn lasts: a number of days, weeks or months. Starting at sets when the first responder's turn begins; nobody is on call on this schedule before that moment. Shifts then change at that same time of day — every 2 weeks from Monday 09:00 hands over every other Monday at 09:00. A monthly rotation hands over on the same day of the month, or the month's last day when it is shorter.
The Starting at picker shows a month calendar — use its arrows to move between months — with today ringed, in the schedule's timezone. Pick a day, set the hour and minute, and click Apply changes.
- 4Pick the schedule's timezone
The timezone is in the Starting at picker, next to the globe, and starts as your browser's. The start time, every shift change and the active hours are read in this timezone, whoever is looking at the schedule and wherever Zharp's servers are. Handovers stay at the same local time across daylight saving changes.
- 5Choose when the schedule is active
Under Rotation method, Every day keeps someone on call around the clock. Specific days and times lets you add rows of Days, Starts at and Ends at, in the schedule's timezone; outside those hours nobody is on call on this schedule. A row whose end is before its start runs overnight into the next day.
- 6Save
Preview shows who will be on call, worked out the same way paging is. Month shows who covers most of each day of the month, counting a change set with Apply changes from; Week shows the week hour by hour (see The week view below). The arrows page through months in month view and weeks in week view. Click Create schedule. To change it later, open the schedule and click Configure.
Someone you want on the rotation isn't in the workspace yet? Invite colleagues at the top of the page opens the same invite dialog as Team → Members (if your role can invite members). Once they accept, they can be added as responders.
Scheduling a change
When you edit a schedule, Apply changes from (optional) lets the new rotation — responders, Change shift every and Starting at — take over at a set date and time, in the schedule's timezone. Until then the current rotation keeps running, and the calendar shows both: the current one up to the handover, the new one after it. Leave it empty to apply the new rotation straight away. The name, timezone and active hours always change immediately. Opening the schedule again before the handover shows the scheduled rotation, so you can adjust or replace it.
Who is on call now
The On-Call Schedules list shows each schedule's ON-CALL NOW. A schedule's own page says <name> is on-call, or Nobody is currently on-call. Both are worked out exactly the way paging is — overrides, active hours, the rotation in effect and the schedule's timezone — and the calendar's month view shows who is on shift at midday on each day, in the schedule's timezone; its week view shows every shift and handover.
The week view
The calendar on a schedule's page, and the Preview while you create or configure one, have a Week / Month toggle. Week shows the seven days from Monday to Sunday, one column per day, with every shift that day: who is on call and from when to when, such as 09:00 – 17:01 or 00:00 – 24:00. Times are in the schedule's timezone, which the view names. When a shift carries over from the day before it starts with ←, and when it runs into the next day it ends with →; hover a shift to see when it starts and ends in full. Dashed Nobody on call blocks mark the gaps — outside the active hours, or before the rotation starts or takes over with Apply changes from.
Shifts are worked out exactly the way paging is: overrides first (on the schedule's page, and the saved ones while configuring), shown as dashed blocks marked Override; then, inside the active hours, the rotation in effect. An active-hours row ends at the end of its Ends at minute, so a row ending at 17:00 is on call until 17:01. While you configure a schedule with Apply changes from set, the week shows the current rotation up to the handover and the new one after it.
In week view the arrows move a week at a time; in month view, a month at a time. Switching between them keeps the same date in view. On a schedule's page, click a day in either view to create an override for it.
Exporting a schedule
On a saved schedule, Export schedule downloads its shifts as a calendar file (.ics) for the Next 30 days, Next 90 days or Next 12 months, to import into Google Calendar, Outlook or Apple Calendar. Each shift is an event named On call: <name>. Shifts are worked out the way paging works them out — rotation, active hours, overrides and handovers set with Apply changes from — and times are written in UTC, so your calendar shows them in your own timezone. The file is a snapshot: export again after changing the schedule or adding overrides.
Overrides
To swap a shift, open the schedule and click Create override, or click a day on the calendar to cover that whole day. Pick the On-call responder, set Start on-call at and End on-call at as dd/mm/yyyy hh:mm, and click Override. The times are in the schedule's timezone, which the dialog names. For that period the person you picked is on call instead of the rotation — even outside the schedule's active hours.
Escalation policies
A policy is a list of steps that run in order. Create one on Team Members → Escalation Policies with Create Policy: give it an Escalation policy name, add steps with the buttons below the steps, drag steps to reorder them, and click Create escalation policy. A policy needs at least one step.
The Escalation Policies list marks a policy Active when at least one monitor escalates through it, and Inactive when none does yet; hover the label for the number of monitors. An inactive policy still runs when you pick it under Report to or with the Escalate button.
| Step | What it does |
|---|---|
| Escalate & alert | Alerts everyone you list under Who should we alert?, all at once. |
| Instructions & todo list | Posts your instructions as a comment on the incident's timeline. |
| Time-based rule | Checks the current day and time, and can stop the policy or change its severity. |
| Metadata rule | Checks a metadata field of the monitor or incident, and can stop the policy or change its severity. |
| Resolve or remove | Resolves or deletes the incident, and ends the policy. |
Waiting between steps
Each step starts with a wait: Wait for a number of minutes, counted from when the policy started — not from the step before. Steps set to 0, 5 and 10 minutes run at 0, 5 and 10 minutes; that is why each new step is pre-filled a little later than the last. A step whose time has already passed when the step before it finishes runs straight away.
So a step with a lower wait than an earlier step can't run any sooner than that step — it runs right after it. The editor warns you on such a step ("This step waits 5 min, less than the 10 min of an earlier step…"); raise its wait to space the steps out.
Use the step's … menu and Wait until... to wait for a time of day in a timezone instead, for example to hold non-urgent alerts until 09:00 in Europe/London. The steps after a Wait until step count their minutes from when it ran: if it is set to 5 and the next step to 10, the next step runs 5 minutes after it.
Escalate & alert
Under Who should we alert? add any mix of:
- Teammates — a person, or Entire team.
- Groups — everyone in the group who is an active member of the workspace.
- On-call schedules — whoever is on call on that schedule. If nobody is, the entire team is alerted instead and the timeline says No responder on shift — the same as naming that schedule under Report to or with the Escalate button. If the same step also alerts Entire team, the team is alerted once.
- Integrations — a connection such as Slack, Discord, Microsoft Teams, Google Chat, Mattermost, Telegram, PagerDuty, Pushover, Pushbullet, Splunk, Zapier or a webhook. The connection is posted to even if its own event filter wouldn't include the incident.
How urgent is this alert? picks the step's severity, which decides how people are reached. With no severity, each person is reached the way their own alert settings say.
Severities
Create one from a step with Set up custom severity. Give it a Severity name and choose, under When there's a new incident, how people are reached: E-mail, Push notification and/or Critical alert. A critical alert is a push notification that can break through Do Not Disturb on phones that allow it, and through quiet hours. Call and SMS are shown but not available yet: SMS and phone call alerts aren't available in your country yet.
Instructions & todo list
Write the steps your responder should follow; markdown works. Lines like - [ ] Check the database dashboard become checkboxes anyone can tick on the incident page. You can use placeholders such as {monitor_name}, {monitor_url}, {incident_title}, {root_cause} or any key from the monitor's Metadata; they're filled in when the comment is posted.
Tick Follow up every and set a number of hours unless all checkboxes are marked as checked to remind the people this incident has already been escalated to, by e-mail and push, until every box is ticked or the incident is resolved.
Time-based and metadata rules
A Time-based rule matches on the days you select, between From and To in the chosen Timezone. A window can run past midnight; the same time in both fields means the whole day.
A Metadata rule matches when If the metadata field has one of the comma-separated values in matches any of the following values; case doesn't matter. Leave the values empty to match any value. The field can be a key from the monitor's Metadata section, or monitor_name, monitor_url, incident_title or root_cause.
When a rule matches, it does what you chose; when it doesn't, the policy carries on with the next step.
| If it matches | What happens |
|---|---|
| Do not escalate | The policy stops here. |
| Continue to the next step | The policy carries on. |
| Escalate as a severity | The rest of this run alerts with that severity. |
If no one acknowledges
After the last step, the setting at the bottom of the policy decides what happens: don't repeat, repeat 1 time, repeat 2 times, repeat 3 times or repeat until acknowledged. A repeat runs the whole policy again from the first step, at least 5 minutes after the last step. repeat until acknowledged stops after 24 repeats.
Exporting and importing a policy
Export policy, at the top of the policy editor, downloads the policy as a JSON file — what is in the editor at that moment, so on a new policy it works before you save. The file has the name, description, the If no one acknowledges setting and every step with all its fields: Wait for minutes or Wait until a time with its timezone, who each step alerts, its severity, instructions and follow-ups, rule days, times, timezones and outcomes, and resolve or remove steps. Teammates, groups, on-call schedules, connections and severities are written with both their id and their name (and a teammate's e-mail), so the file stays readable and can move between workspaces.
To use a file, create a policy and click Import policy next to Export policy. The file fills the editor, replacing anything already in it (Zharp asks first); nothing is saved until you click Create escalation policy. Each teammate, group, schedule, connection and severity in the file is matched to this workspace:
- by its id, when that id exists in this workspace — the same workspace, or the same person in another one;
- otherwise by name — teammates by e-mail first, then name; connections only to one of the same integration; case doesn't matter, and a name that fits more than one doesn't match;
- otherwise it is left out, and a warning above the form lists everything that couldn't be matched, by step.
A step left alerting nobody can't be saved until you pick someone. A step whose severity wasn't found has none picked, and a rule set to Escalate as an unknown severity is set to Continue to the next step — both are in the warning. A file that isn't a Zharp policy export, or has a missing or wrong field, isn't imported: the editor lists each problem, by step and field.
What stops escalation
Before every step, Zharp checks the incident. Once someone clicks Acknowledge, or the incident is resolved, no further steps or repeats run. Every step that runs is recorded on the incident's timeline, including rules that matched or didn't.
A maintenance window stops it too. When one starts on the incident's monitor, the escalation already running stops — no further steps, repeats, checklist reminders or on-duty re-pages — and the timeline says Escalation stopped. The same check runs before each step, so a window that began between two steps is never missed. Incidents reported by hand are the exception: someone chose to page about those, so they keep escalating.
When a monitor fails again soon after recovering and Zharp reopens its incident, the incident starts unacknowledged again and its response starts over: the escalation policy runs from the first step, or the on-call responder is paged. Steps still waiting from the earlier run don't run. Reopening an incident by hand with Reopen also makes it unacknowledged, but pages nobody by itself — use Escalate for that.
Using a policy
On a monitor
In the monitor's settings, under How should we escalate incidents?, choose:
- Notify the on-duty member — pages whoever is on call on the primary schedule. Choose how under How should we notify the on-duty member? (E-mail, Push notification, Slack alert). Under If the on-duty member doesn't acknowledge the incident: Do nothing; Escalate to the team pages everyone in the workspace if nobody acknowledges within 5 minutes; Repeat the notification pages the on-duty member again every 5 minutes until someone acknowledges, up to 6 times in all.
- Escalation policy — pick a policy, or + Create new policy to write one in place.
On a single incident
Choose a policy under Report to when you report an incident, or add it with the Escalate button on an open incident. The policy starts from its first step.
Personal notification preferences
Each person controls how they're reached. In the dashboard, Alerts & Reporting in your account menu turns Email Notifications and Push Notifications on or off. When a policy step has no severity, these decide how that person is paged.
In the Zharp mobile app, the Notifications settings add:
- Maintenance windows — a push when a maintenance window starts on a monitor you get alerts for.
- Quiet hours — a nightly Schedule, in your phone's timezone. During quiet hours, pages reach you by e-mail only. Critical alerts always notify (Bypass quiet hours when a monitor goes down) lets outage pages through anyway. A critical alert always gets through.