link_offMonitors

Broken Links

Crawl your site on a schedule and find every link and image that leads to an error page, a dead server or nowhere at all.

How the crawl works

A Broken links monitor starts at the URL you give it, reads the page, and collects every link (<a href>) and image (<img src>) on it. It requests each one, reads the pages among them that are on your site, collects their links and images in turn, and so on, level by level, until it reaches Max crawl depth.

Depth. The start page is level 0. With Max crawl depth at 3 (the default), pages up to two clicks from the start are read for links, and the links found on them — three clicks away — are checked but not read.
Your site. A page counts as yours when its hostname is exactly the start URL's. www.example.com and example.com are different hosts, and so is every subdomain.
Other sites. Links to other hosts are always checked — one request each — but their pages aren't read, unless Follow external links is Yes. Then external pages are crawled too, to the same depth.
Once per URL. Each address is requested once per crawl, however many pages link to it. The part after # is dropped first, so /pricing#faq and /pricing are the same link.
Not followed. mailto:, tel:, javascript:, data: and links that are only a #fragment. Stylesheets, scripts, srcset images and links added by JavaScript after the page loads aren't seen: the crawler reads the HTML as the server sends it.

Pages that will be read for links are fetched with GET; everything else — images, external links, and links on the last level — with HEAD, which asks only for the status. Up to five requests are in flight at once, and no more than Request rate limit start each second. Each request gets Page timeout to answer and follows up to 5 redirects, and identifies itself with the User agent you chose. The crawler doesn't read robots.txt or honour rel="nofollow".

What counts as broken

A status of 400 or above after redirects — 404, 410, 500, 503 and so on.
No answer within Page timeout — shown as timeout.
No connection at all: the name doesn't resolve, the connection is refused, the certificate isn't valid, or there are more than 5 redirects.
A link to a private or loopback address, which is never requested.
An address that isn't a valid URL.

One broken link is enough: the check fails when the crawl finds any, and passes when it finds none. Each selected region runs the whole crawl, and the results are combined as described in Core Concepts.

Limits on a crawl

A crawl stops when there is nothing left to check, when it has checked Max pages to crawl addresses, or when the monitor's Request timeout runs out — 30 seconds unless changed. That timeout is the budget for the whole crawl, not for each request. A crawl that runs out of time reports what it found so far, adds how many addresses were left, and fails only if it found something broken.

lightbulb

Large sites

For a site with many pages, set Request timeout under Request parameters to 60 seconds, lower Max crawl depth, or point separate monitors at different sections. If the check message often ends with Stopped after timeout, the crawl isn't reaching the whole site.

Create a broken links monitor

1
Choose the type
On the Monitors page, click New, open the type picker and choose Broken links monitor. It is listed under Website & availability.
2
Enter the start URL
Fill in Website URL with the page the crawl starts from, such as https://example.com or https://example.com/docs. Without https://, it is added for you. Only pages on this URL's host are crawled.
3
Set the crawler
Open Advanced settings. Under Crawler settings, choose how fast and how far the crawl goes, and list anything to skip in Ignore URL patterns. Under Request parameters, Request timeout sets how long the whole crawl may take. If your site needs them, add a header under Request headers and credentials under HTTP authentication.
4
Create it
Pick who is alerted under How should we escalate incidents? and click Create monitor. Check frequency defaults to 3 minutes; links rarely break that fast, so 30 minutes — the longest — is usually enough. Every region you leave selected crawls your site on each check.

Crawler settings

Setting	Default	What it does
Request rate limit	5 req/sec	1, 2, 5 or 10. At most this many requests start each second, across the whole crawl. Lower it if the crawl strains your server.
Max crawl depth	3 levels	1, 2, 3 or 5. How many clicks from the start URL the crawl reaches (see Depth above).
Max pages to crawl	500 pages	100, 250, 500 or 1,000. The crawl stops after checking this many addresses — every link and image counts, not just pages.
User agent	Zharp Bot	How the crawler introduces itself: Zharp Bot (`Zharp-Broken-Link-Monitor/1.0`), Chrome (a desktop Chrome browser) or Googlebot (Google's crawler). Pick one your bot protection lets through.
Page timeout	30 seconds	10, 20, 30 or 60 seconds. How long each request may take before the link counts as a timeout. It never runs past the end of the crawl's Request timeout.
Follow external links	No	Links to other hosts are always checked. No: their pages aren't read. Yes: other hosts' pages are crawled too, which can make a crawl much larger.
Ignore URL patterns	Empty	Comma-separated. An address matching any pattern is neither checked nor crawled. `` stands for any characters, and a pattern can match anywhere in the address: `/admin/` skips everything under `/admin/`, `.pdf` skips PDF files. A pattern without `` matches as plain text, so `/admin/` works too.
Request timeout	30 seconds	10, 20, 30, 45 or 60 seconds. How long the whole crawl may run (under Request parameters).
Request headers	None	A header sent with every request to the start URL's host — for example a token your firewall expects. A `User-Agent` header here replaces the one chosen above.
HTTP authentication	None	Basic authentication credentials, sent with every request to the start URL's host, for sites behind a login prompt.

Headers and credentials go only to your own host. Links to other sites are requested without them, so they never leave your site.

info

Not used by the crawl

The crawler chooses its own method — GET or HEAD — and follows redirects itself, so a broken links monitor has no method, body, redirect or proxy settings.

When it goes down

What happened	Message on the check
The crawl found broken links	`Checked 214 links, found 3 broken.`
It also ran out of time	`Checked 380 links, found 1 broken. (Stopped after timeout: 52 left in queue)`
The start URL isn't a valid URL	`Invalid Start URL: https://example com`

If the start page itself can't be loaded, it is reported as a broken link with no Found on page. The monitor stays down, with its incident open, until a crawl finds nothing broken — fix or remove the links and the next check closes it.

The monitor page

The bar at the top: the start URL, the depth, EXTERNAL PAGES CRAWLED when external pages are crawled, the number of links checked, and ALL OK or how many are broken.
A chart of Links scanned per crawl, with the latest Links scanned, Broken and Pages crawled (pages actually read for links).
Broken links: every broken link from the latest crawl, with its Status (the HTTP code, timeout or error), the Broken URL, whether it is a Link or an Image, the page it was Found on, and when it was last seen. Expand a row for the anchor or alt text, the method used, redirects and final URL, how many times and on how many pages it appears, when it was first seen broken, and when it last worked. Up to 50 broken links are kept per crawl.
Scan settings: start URL, crawl depth, max pages, whether external pages are crawled, schedule and the last scan.

Troubleshooting

A link works in my browser but is reported broken

Some servers answer HEAD with 405 or 404 while GET works. Check the Method in the expanded row; if it is HEAD, the server is the cause. Add the address to Ignore URL patterns if you can't change it.
Bot protection and rate limits answer automated requests with 403 or 429. Allow the crawler, choose another User agent, lower Request rate limit, or send a header your protection accepts under Request headers.
A page slower than Page timeout shows as timeout, even if it loads eventually. Raise Page timeout for slow pages.

Pages I expected aren't crawled

Pages on www. or another subdomain are a different host from the start URL. Start from the host your pages live on.
Links added by JavaScript aren't in the HTML the crawler reads.
The crawl stopped early — at Max pages to crawl or Request timeout. Compare Links scanned with the size of your site.

The crawl is too heavy for my server

Lower Request rate limit — at 1 request per second the crawl never sends more than one new request a second. Each region crawls on each check, so choose fewer regions or a longer Check frequency too, a lower Max crawl depth or Max pages to crawl, and skip large sections with Ignore URL patterns. A slower rate means a longer crawl, so raise Request timeout to match.

language

Websites & Keywords

Check that the pages themselves are up.

monitor_heart

Monitor Types

Settings every monitor shares.

crisis_alert

Incidents

How incidents open and resolve.

notifications_active

Alert Channels

Where alerts are sent.

Mail Servers

Security Scans