Lessons  ·  Lesson 01

Crawl budget

Nothing else matters until a crawler can reach the page, render it, and decide to keep it. This is the cheapest lesson on the site, because almost everything in it is free to fix.

If a word here is unfamiliar it is defined in the SEO glossary. Two public penalties and one problem we found on our own network are pulled apart in the case studies, and the longer pieces live in the library.

Part 1

What a crawler actually requests

A crawler asks your server for a file, the same way a browser does, and it gets whatever your server hands back. It does not see your content management system, your preview, or the version you are looking at while logged in. If the page needs a session or a login to show its text, the crawler receives an empty shell and stores that.

Open the page in a private window with JavaScript switched off. What survives is roughly what gets collected first.
Part 2

Why rendering changes the answer

Modern search engines do run JavaScript, but on their own schedule and with their own budget. A page whose main text appears only after a script has fetched it will usually be collected twice: once as the shell, later as the full thing. On a large site the second pass can lag badly, and pages that change often suffer most.

If the words you want to rank for exist only in the rendered version, treat indexing delays as expected rather than mysterious.
Part 3

The four ways a page blocks itself

A noindex tag in the head or in an x-robots-tag header. A path disallowed in robots.txt, which stops the fetch before anything is read. A canonical pointing at a different address, which hands the credit elsewhere. And a status code in the four hundreds or five hundreds. Any one of them is enough on its own, and all four are invisible to a person reading the page.

Run the free check on the page. Those four are the first four things it reports.
Part 4

Reading the index report without panic

Coverage reports list reasons, not verdicts. Excluded by noindex means somebody did it, deliberately or not. Discovered but not currently indexed usually means the page is known and judged not worth fetching yet, which is a quality and internal linking signal rather than a bug. Duplicate without user selected canonical means the search engine picked a different page as the master.

Sort by reason, fix the reason that appears most, and re-check in a fortnight rather than daily.
Next step

Want a free consultation on your own page? Leave your details.

Run the free check first, then tell us what it found. We come back with what we would fix first and why, at no cost.

Run the free check