Monitor settings

Per-site settings: what gets crawled, how often, and who hears about it.

Crawl scope

Start URL — where discovery begins, usually the homepage.

Include and exclude patterns — path patterns deciding what is in. Excluding is the main lever on both noise and cost, and the usual candidates are:

Paginated archives (/page/*)
Tag and author archives
Search result pages
Anything behind a login
Staging or preview paths that answer publicly

Sitemap — point at yours if it is not at the default location. Without one, pages nothing links to are invisible to the crawler.

Page limit — a ceiling, so a runaway URL space (faceted filters generating infinite combinations) cannot consume a crawl.

Frequency

How quickly you want to know. Daily suits most sites; a documentation site under active edit benefits from it, a brochure site changing quarterly does not need it.

Frequency multiplies pages, and pages are what Monitor is billed on. Crawling everything hourly is usually paying for precision nobody acts on.

Rendering

Whether to execute JavaScript before checking. Necessary for a site that renders content client-side, unnecessary and slower for one that does not.

If your pages are server-rendered, leaving this off makes crawls faster and cheaper. If you are unsure, turn it on once and compare a page’s rendered output against its source — the difference is the answer.

Politeness

Request rate and concurrency. The defaults are conservative deliberately: a crawler that hammers a small host looks like an attack and gets itself blocked.

Raise it only if your host is comfortable, and lower it if you see the crawl being rate-limited in crawl activity.

Alerts

Who is notified, through which channel, and at what severity. Set per site — see alerts and incidents.

Pausing

Pause a site during a migration or a redesign. History is kept, so when you resume the comparison picks up from where it was rather than treating everything as new.

Pausing during a planned change is much better than muting the resulting alerts and forgetting.