How search engines work: crawling, indexing and ranking explained

Search engines do three separate jobs: they crawl to find your pages, index to store and understand them, and rank to decide which to show for a given search. Almost every SEO problem is a failure at one specific stage, and diagnosing which one is the difference between fixing something in an afternoon and guessing for six months.

That distinction is the whole point of this article. Most SEO advice skips it and jumps straight to tactics, which is why people end up rewriting title tags on pages Google has never been able to reach.

Stage one: crawling

Crawling is discovery. Search engines run automated programs – Googlebot, Bingbot, and increasingly GPTBot and PerplexityBot – that follow links from page to page, collecting what they find.

A crawler reaches your site in one of three ways: it follows a link from a site it already knows, it reads your XML sitemap, or you ask it directly through Search Console.

That first route matters more than people expect. If nobody links to your site and you have not submitted it anywhere, a search engine has no reason to know you exist. This is why brand-new sites feel invisible: not because the content is poor, but because nothing has pointed a crawler at it.

Three things commonly block crawling:

robots.txt disallow rules. One line can make an entire site invisible. Check yours at yoursite.com/robots.txt

Pages with no links pointing to them. If a page is not linked from anywhere and is not in your sitemap, it may never be found.

Content that only appears after JavaScript runs. Google handles this reasonably well now. Several other crawlers, including some AI crawlers, do not.

How to check: in Google Search Console, use URL Inspection on any page. It will tell you plainly whether Google has crawled it and when.

Stage two: indexing

Crawling finds the page. Indexing is the decision about whether to keep it, and what it is about.

This is where a lot of people are surprised. Being crawled does not mean being indexed. Google routinely crawls a page, evaluates it, and decides not to store it. Search Console reports this as “Crawled – currently not indexed”, and it usually means one of three things: the page is thin, it duplicates something else, or the site as a whole has not yet earned enough trust for Google to bother.

During indexing, a search engine works out what the page is about. It reads your headings, your body content, your structured data, and the words other sites use when linking to you. Structured data matters here more than most people realise: it is the difference between a machine inferring what you are and being told.

How to check: search

site:yoursite.com

in Google. That shows roughly what is indexed. For detail, use the Pages report in Search Console – it groups every URL by status and gives you the reason for exclusions.

Stage three: ranking

Only now does the ordering question arise. When someone searches, the engine pulls relevant pages from its index and sorts them.

Google uses many signals, and nobody outside Google knows the full weighting. But the broad categories are well established:

Relevance – does the page actually answer this query, including queries phrased differently from your wording?

Quality and authority – is this a credible source? Links from other sites remain the strongest external signal.

Experience – does the page load quickly, work on a phone, and avoid interrupting the reader?

Context – the searcher’s location, device and history all shift results.

The uncomfortable truth for new sites is that the second category takes time. You can control relevance and experience immediately. Authority is earned by other people, and that cannot be rushed or bought safely.

Why the three stages matter for diagnosis

Here is the practical value of separating them. When a page is not performing, the question is not “how do I improve my SEO?” – it is “which stage is failing?”

Symptom	Failing stage	Where to look
Page not in site: results at all	Crawling	robots.txt, internal links, sitemap
Crawled but not indexed	Indexing	Thin or duplicate content, site trust
Indexed but ranked low	Ranking	Relevance, backlinks, competition
Ranked but no clicks	Presentation	Title tag and meta description

Each row has a completely different fix. Rewriting your title tags will do nothing for a page blocked in robots.txt. Building links will not help a page Google cannot crawl.

What changed with AI answers

The three stages still apply, but a fourth consideration now sits alongside them: citation.

When someone asks an AI assistant a question, it does not return ten links. It composes an answer and cites a handful of sources. Getting cited depends on things that overlap with ranking but are not identical to it:

Content that answers a question outright, in the opening lines, rather than building up to it

Structured data that states clearly what you are and what you do

A named, traceable author rather than anonymous content

Being mentioned on sites other than your own

Notably, some sites that rank poorly get cited frequently, because they state facts plainly and are marked up well. And some sites that rank well never get quoted, because their answers are buried in the sixth paragraph beneath a personal anecdote.

The order to fix things in

If you take one thing from this: work in the order the stages happen.

Make sure your pages can be crawled

Make sure they are worth indexing

Then, and only then, work on ranking

And write so a machine can quote you

Optimising content on a page search engines cannot reach is the most common wasted effort in this field. It is also entirely avoidable, because Search Console will tell you exactly which stage you are stuck at, for free, in about thirty seconds.