# Pages and crawl activity

The **Pages** tab lists every page Monitor tracks on a site, and opening a page shows everything Monitor recorded about it. The **Crawl activity** tab answers a different question: which of your pages search engine bots actually fetched, based on access logs you paste in. This page covers both.

> **Before you start:** Monitor must be enabled for your workspace. For **Crawl activity**, you also need access to your server or CDN access logs.

## The Pages tab

The Pages tab lets you filter, sort and open any page. It has two views:

- **Rows**: the list of matching pages.
- **Shape**: charts of the matching pages by **Status**, **Indexable**, **Health**, **Importance** and **Depth from the home page**.

### Columns

The list starts with **Page**, **Status**, **Health**, **Importance** and **Checked**. Select **Columns** to show or hide the others. Select a column heading to sort by it (every column except **Indexable** sorts).

| Column | What it shows |
| --- | --- |
| **Page** | The page’s title and URL |
| **Status** | The HTTP status from the last check |
| **Indexable** | Whether search engines may index the page |
| **Health** | The page’s health score, out of 1,000 |
| **Importance** | 0 to 10, estimated from click depth, internal links and traffic. More important pages are checked more often |
| **Depth** | How many clicks the page is from the home page |
| **Inbound links** | How many internal links point at the page |
| **Checked** | When Monitor last checked the page |
| Column | What it shows |
| **Page** | The page’s title and URL |
| **Status** | The HTTP status from the last check |
| **Indexable** | Whether search engines may index the page |
| **Health** | The page’s health score, out of 1,000 |
| **Importance** | 0 to 10, estimated from click depth, internal links, traffic and how often the content changes. More important pages are checked more often |
| **Depth** | How many clicks the page is from the home page |
| **Inbound links** | How many internal links point at the page |
| **Checked** | When Monitor last checked the page |

### Filters

Above the list, you can filter by text in the **URL**, by **Status** (for example **Broken (4xx/5xx)** or **Not checked yet**) and by **Indexable**, and set a minimum **Importance** and a maximum **Health**. Select **Clear** to remove the filters.

To see the pages affected by one check, open that check from the **Issues** tab. The Pages tab then shows only the affected pages; select **Clear issue filter** to go back to all pages.

### Segments

A segment is a named group of pages, such as your blog. Segments appear in the rail beside the list, under **All pages**. Every new site starts with one segment, **Important pages**. To create one, select **Add segment**, give it a **Name**, and add criteria on **URL**, **Indexable**, **Status**, **Health**, **Importance**, **Depth** or **Discovered**.

You can use a segment to stop reporting a finding across a group of pages, to set different thresholds, and to scope an alert rule. See [Issues and severity](/docs/issues-and-severity/) and [Monitor site settings](/docs/monitor-settings/).

### Views

Under **Views**, you can save the current set of columns with a name. A view is visible only to you unless you select **Share with the team**, which lets everyone with access to the site use and edit it.

## Page detail

Select a page to open it. The header shows its **Importance** and **Page health**, and links to act on it:

- **Open page** opens the live page.
- **Edit in Workover** appears when the page was published from a Workover document.
- **Edit in WordPress** appears when Monitor can link to the post in the WordPress editor.

Beside the page, a list of its checks shows what is **Open** and **Everything else**, with the state of each check. See [Issues and severity](/docs/issues-and-severity/).

The page has these tabs:

| Tab | What it shows |
| --- | --- |
| **Current version** | A screenshot, then indexability and content details, the segments the page is in, search and traffic figures, Website Specification results and Core Web Vitals |
| **DOM / SRC** | The page as served (SRC) compared with the page after JavaScript runs in a browser (DOM): title, canonical, meta robots, H1, word count and internal links. Needs **Render pages with a browser** |
| **Tracked changes** | Every recorded change to a watched attribute on this page, with before and after |
| **History** | Stored checks of this page. Pick two to compare them |
| **Source** | The stored source of the page |

On **Current version**, indexability details include **Indexable**, **HTTPS**, **Status**, **Depth**, **Canonical**, **Meta robots**, **X-Robots-Tag**, **Redirected to**, **Inbound internal links** and **Found via**. Content details include **Title**, **Meta description**, **H1**, **Words**, **Images**, **Structured data** and **Open Graph title**.

**Structured data** lists which schema types the page declares. Monitor does not validate the markup against those types.

**Search & traffic** appears when Monitor has click or session figures for the page, which needs Search Console or Google Analytics connected under **Settings** > **Reporting**.

A new snapshot is stored the first time a page is checked, and again whenever something Monitor watches changes. Changes to body copy, images and scripts are not tracked. See [Changes and diffs](/docs/changes-and-diffs/).

## Crawl activity

Crawl activity shows what search engine bots did on your site, from your own access logs. Every hit is verified by reverse DNS, so bots that only claim to be Googlebot are not counted. Monitor recognizes Googlebot, Bingbot, YandexBot and Applebot.

Until you add logs, this tab is empty and Monitor’s view of crawl frequency is an estimate.

### Add access logs

1. Export lines from your web server or CDN access log in combined log format. A recent tail or a full day works.
2. On the site’s **Crawl activity** tab, paste the lines under **Add log lines**.
3. Select **Ingest**.

Monitor reports how many lines it read, how many claimed to be crawlers, how many it verified, and how many pages they matched. One paste can be at most about 5 MB, so add a large log in several pastes.

### What you see

| Section | What it shows |
| --- | --- |
| **Verified hits** | Crawler requests that passed reverse DNS verification |
| **Pages crawled** | Tracked pages that at least one verified crawler fetched |
| **Pages never crawled** | Indexable tracked pages absent from every log you have added. The count stops at 50 |
| **Important pages nothing has fetched** | Those pages, most important first, with their importance |
| **Most crawled** | The pages crawlers fetch most, which shows where your crawl budget goes |

Crawl activity does not show Monitor’s own checking.

## Related

- [Issues and severity](/docs/issues-and-severity/)
- [Changes and diffs](/docs/changes-and-diffs/)
- [Monitor site settings](/docs/monitor-settings/)
- [What Monitor does](/docs/monitor-overview/)