Pages and crawl activity

The Pages tab lists every page Monitor tracks on a site, and opening a page shows everything Monitor recorded about it. The Crawl activity tab answers a different question: which of your pages search engine bots actually fetched, based on access logs you paste in. This page covers both.

Before you start: Monitor must be enabled for your workspace. For Crawl activity, you also need access to your server or CDN access logs.

The Pages tab

The Pages tab lets you filter, sort and open any page. It has two views:

Rows: the list of matching pages.
Shape: charts of the matching pages by Status, Indexable, Health, Importance and Depth from the home page.

Columns

The list starts with Page, Status, Health, Importance and Checked. Select Columns to show or hide the others. Select a column heading to sort by it (every column except Indexable sorts).

Column	What it shows
Page	The page’s title and URL
Status	The HTTP status from the last check
Indexable	Whether search engines may index the page
Health	The page’s health score, out of 1,000
Importance	0 to 10, estimated from click depth, internal links and traffic. More important pages are checked more often
Depth	How many clicks the page is from the home page
Inbound links	How many internal links point at the page
Checked	When Monitor last checked the page
Column	What it shows
Page	The page’s title and URL
Status	The HTTP status from the last check
Indexable	Whether search engines may index the page
Health	The page’s health score, out of 1,000
Importance	0 to 10, estimated from click depth, internal links, traffic and how often the content changes. More important pages are checked more often
Depth	How many clicks the page is from the home page
Inbound links	How many internal links point at the page
Checked	When Monitor last checked the page

Filters

Above the list, you can filter by text in the URL, by Status (for example Broken (4xx/5xx) or Not checked yet) and by Indexable, and set a minimum Importance and a maximum Health. Select Clear to remove the filters.

To see the pages affected by one check, open that check from the Issues tab. The Pages tab then shows only the affected pages; select Clear issue filter to go back to all pages.

Segments

A segment is a named group of pages, such as your blog. Segments appear in the rail beside the list, under All pages. Every new site starts with one segment, Important pages. To create one, select Add segment, give it a Name, and add criteria on URL, Indexable, Status, Health, Importance, Depth or Discovered.

You can use a segment to stop reporting a finding across a group of pages, to set different thresholds, and to scope an alert rule. See Issues and severity and Monitor site settings.

Views

Under Views, you can save the current set of columns with a name. A view is visible only to you unless you select Share with the team, which lets everyone with access to the site use and edit it.

Page detail

Select a page to open it. The header shows its Importance and Page health, and links to act on it:

Open page opens the live page.
Edit in Workover appears when the page was published from a Workover document.
Edit in WordPress appears when Monitor can link to the post in the WordPress editor.

Beside the page, a list of its checks shows what is Open and Everything else, with the state of each check. See Issues and severity.

The page has these tabs:

Tab	What it shows
Current version	A screenshot, then indexability and content details, the segments the page is in, search and traffic figures, Website Specification results and Core Web Vitals
DOM / SRC	The page as served (SRC) compared with the page after JavaScript runs in a browser (DOM): title, canonical, meta robots, H1, word count and internal links. Needs Render pages with a browser
Tracked changes	Every recorded change to a watched attribute on this page, with before and after
History	Stored checks of this page. Pick two to compare them
Source	The stored source of the page

On Current version, indexability details include Indexable, HTTPS, Status, Depth, Canonical, Meta robots, X-Robots-Tag, Redirected to, Inbound internal links and Found via. Content details include Title, Meta description, H1, Words, Images, Structured data and Open Graph title.

Structured data lists which schema types the page declares. Monitor does not validate the markup against those types.

Search & traffic appears when Monitor has click or session figures for the page, which needs Search Console or Google Analytics connected under Settings > Reporting.

A new snapshot is stored the first time a page is checked, and again whenever something Monitor watches changes. Changes to body copy, images and scripts are not tracked. See Changes and diffs.

Crawl activity

Crawl activity shows what search engine bots did on your site, from your own access logs. Every hit is verified by reverse DNS, so bots that only claim to be Googlebot are not counted. Monitor recognizes Googlebot, Bingbot, YandexBot and Applebot.

Until you add logs, this tab is empty and Monitor’s view of crawl frequency is an estimate.

Add access logs

Export lines from your web server or CDN access log in combined log format. A recent tail or a full day works.
On the site’s Crawl activity tab, paste the lines under Add log lines.
Select Ingest.

Monitor reports how many lines it read, how many claimed to be crawlers, how many it verified, and how many pages they matched. One paste can be at most about 5 MB, so add a large log in several pastes.

What you see

Section	What it shows
Verified hits	Crawler requests that passed reverse DNS verification
Pages crawled	Tracked pages that at least one verified crawler fetched
Pages never crawled	Indexable tracked pages absent from every log you have added. The count stops at 50
Important pages nothing has fetched	Those pages, most important first, with their importance
Most crawled	The pages crawlers fetch most, which shows where your crawl budget goes

Crawl activity does not show Monitor’s own checking.