Incidents

An incident is one problem, from the moment it starts until it is resolved. Zharp opens most of them for you when a monitor fails; you can also report one by hand.

Every incident has a page under Incidents with its cause, who has been alerted, and a timeline of everything that happened. The list has three tabs: All incidents, Ongoing and Resolved. Ongoing includes incidents someone is already investigating.

Automatic incidents

A monitor is checked from every region it is set up for. When more than half of those regions report a failure, the monitor goes down and Zharp opens an incident named <monitor> is DOWN. The incident page shows the cause in plain words under Cause, and the timeline lists each region that failed and what it got back.

When the incident opens, two things happen at once:

• The response starts. If the monitor has an escalation policy, the policy runs. Otherwise Zharp pages whoever is on call on your primary on-call schedule, once. With no schedule, or nobody on call, the alert is all that goes out. See Escalation & On-call.
• The monitor's alerts go out. The workspace owner and the monitor's subscribers get an e-mail, people who allow it get a push notification, and the monitor's connections are posted to. See Alert Channels.

No incidents are opened for a monitor while it is inside a maintenance window — log monitors included. If a window starts while one of its monitors already has an open incident, that incident's escalation stops: no more steps, repeats or on-duty re-pages run, and the timeline says Escalation stopped. Incidents you reported by hand keep escalating.

Resolved on recovery

When the monitor passes its checks again, the open incident is resolved automatically and the timeline says Resolved automatically. You don't have to close it yourself.

Flapping monitors reopen the same incident

If a monitor fails again within 30 minutes of recovering, and for the same cause, Zharp reopens the incident it just resolved instead of opening a new one. The timeline marks each return as Recurrence #2, Recurrence #3 and so on. The reopened incident is unacknowledged again, and the response starts over as for a new outage: the monitor's escalation policy from its first step, or the on-call responder.

Degraded monitors

When a monitor is still up overall but failing in some of its regions, no incident opens and nobody is alerted: a monitor goes down — and alerts — when more than half of its regions fail. The monitor page shows Failing in … under its status, with the regions, until every region passes in the same round. If the failure spreads to most regions, the monitor goes down and an outage incident opens as usual.

Reporting an incident by hand

Use this for problems no monitor catches, like a broken sign-up flow a customer told you about.

1. 1
   Open the report form

   On Incidents, click Report an incident.

2. 2
   Describe what's going on

   Fill in Brief description (this becomes the incident's title) and In-depth description. Under Attach files or screenshots you can add PNG, JPG or PDF files up to 10MB each.

3. 3
   Choose who to tell

   Pick an option under Report to, then choose how to reach people under Immediately notify via: E-mail, Push notification or Critical alert. Pick at least one.

4. 4
   Create it

   Click Create new incident. You're taken to the new incident's page.

Whichever you pick, a report on a monitor is also posted to the monitor's own connections that get Down — once each, even if you also picked one of them under Report to.

Critical alert is a push notification that can break through Do Not Disturb on phones that allow it, and through quiet hours.

Public report links

People without a Zharp account can report an incident through your workspace's report link, https://www.zharp.io/report/<token>. Find it at the top of the report form under Share this URL with your team and click Copy URL. Share it on your team's Slack or wiki.

The public form asks for Brief description, In-depth description and Your e-mail, and which of E-mail, Push notification and Critical alert to use. A public reporter can't choose who to notify: the report always goes to whoever is on call on your primary schedule, or the entire team if nobody is.

A public report isn't credited to anyone in your workspace. The e-mail the reporter gave is kept as the reporter: it's what Created by shows on the incident page and what the incident list shows as the reporter, and the timeline's opening event names it.

Responding to an incident

Acknowledge

Click Acknowledge to tell everyone you're on it. The incident moves to Investigating, your name appears under Acknowledged by, and escalation stops: no further escalation policy steps run, and the on-duty member isn't paged again. For a monitor's incident, connections that chose to hear about acknowledgements are told.

Resolve and reopen

Once an incident is acknowledged, the button becomes Resolve. A resolved incident can be brought back with Reopen for 24 hours after it was resolved; after that, report a new incident instead. Reopening sets it back to Ongoing and unacknowledged.

Escalate

Click Escalate on an open incident to pull more people in right away. Under Who to notify, search and add any mix of:

• Teammates — a person, or Entire team.
• Groups — everyone in the group.
• Escalation policies — the policy starts from its first step.
• On-call schedules — whoever is on call; the entire team if nobody is.
• Integrations — a connection such as a Slack channel or a webhook.

Under How to notify choose Email, Push notification and/or Critical alert, then click Escalate Incident. Each escalation is recorded on the timeline with who did it and who was reached.

Timeline, comments and attachments

The Timeline records everything that happens to the incident: the check that opened it, who was alerted and how, acknowledgements, escalations, recurrences and the resolution.

To add an update, write in the comment box (markdown works) and click Post. Use the paperclip to attach a file to the comment. Files attached to a report are posted to the timeline as a comment too.

When an escalation policy posts instructions with checkboxes, they appear on the timeline as a checklist anyone on the team can tick. See Escalation & On-call.

Severities

Incidents don't carry a severity of their own. A severity belongs to an escalation policy step and decides how the people in that step are reached — for example e-mail and push, or a critical alert. Rules in a policy can switch the rest of the run to a different severity. Severities are set up from the escalation policy editor; see Escalation & On-call.