A search engine works in three steps: it crawls the web to discover pages, indexes those pages to store and understand them, then ranks the relevant ones whenever someone searches. Get any one of those stages wrong and your page never shows up, no matter how good it is. This guide walks through each step the way Google actually does it, and where you can intervene.
What a search engine actually does
At its core, a search engine is a giant, constantly updated library plus a very fast librarian. It reads billions of pages ahead of time, files them by topic, and then — in the fraction of a second after you hit enter — pulls the handful it thinks best answers your specific question.
The important mental shift for anyone new to SEO for beginners is this: Google does not search the live web when you type a query. It searches its index, a pre-built copy of the web. Everything you do to rank happens before the search, at the crawl and index stage.
Three jobs, in order:
| Stage | What happens | What can go wrong |
|---|---|---|
| Crawling | Bots discover and fetch your URLs | Page is blocked, orphaned, or too slow to fetch |
| Indexing | Content is parsed, rendered and stored | Thin content, noindex tag, duplicate flagged |
| Ranking | Indexed pages are scored per query | Weak relevance, poor links, bad page experience |
Each stage is a gate. Pass crawling but fail indexing, and you are invisible. Pass indexing but fail ranking, and you are on page 8.
Crawling: how bots discover your pages
Crawling is the discovery phase. Google uses an automated program called Googlebot — a crawler, sometimes called a spider or bot — to browse the web the way a person clicking links would, only far faster and at massive scale.
Googlebot starts from a list of URLs it already knows and finds new pages three main ways:
- Following links from pages it has already crawled, which is why internal linking matters so much
- Reading your XML sitemap, a file that lists the URLs you want found
- Discovering references across the web, such as a new backlink pointing at your page
Each time it visits, Googlebot fetches the HTML, then queues any new links it finds. Modern crawling also renders the page — it runs the JavaScript and loads the layout like a real browser — so it can see content that only appears after scripts execute. If your key content only appears after a heavy client-side render, crawling can be slower and less reliable.
Crawl budget and what limits it
Google does not crawl every page equally. Each site gets a rough crawl budget — how many pages Googlebot will fetch in a given window — shaped by your site's authority, how fast your server responds, and how often your content changes. A slow server or thousands of low-value URLs waste that budget on pages that do not deserve it.
You control crawling with a robots.txt file at the root of your domain. It tells bots which paths they may or may not fetch. Be careful: blocking a page in robots.txt stops it being crawled, but it does not reliably stop it being indexed if other sites link to it. To keep a page out of results, use a noindex meta tag instead — which only works if the page is crawlable.
Indexing: how pages get stored and understood
Once a page is crawled, Google tries to understand and store it. This is indexing. The engine parses your text, images, videos and structured data, works out what the page is about, and files it in the index against the topics and terms it covers.
During indexing Google decides things like:
- The page's primary topic and the keywords it can answer
- Whether it is a duplicate of another page, and if so which version is canonical
- Signals of quality and trust that feed ranking later
- Whether structured data (schema markup) makes it eligible for rich results
Not every crawled page gets indexed. Google openly says it may skip pages that are low quality, duplicative, or blocked by a directive. If you have ever seen "Crawled - currently not indexed" in Google Search Console, that is Google telling you it looked and chose not to store the page — usually a content-quality or duplication signal, not a bug.
The practical takeaway: indexing is a quality filter, not an automatic step. Thin pages, near-duplicate product variants, and boilerplate templates are exactly what gets left out.
Ranking: how Google orders the results
Ranking is the part everyone obsesses over, and it is the most misunderstood. When you search, Google scans its index for pages relevant to your query and orders them using hundreds of signals combined into a score. There is no single "ranking number" you can optimise.
The big signal families are:
- Relevance — does the page actually match the words and intent of the query? Google reads meaning, not just exact keywords, so it understands that "how google works" and "how does google search function" want the same answer.
- Content quality and helpfulness — depth, originality, and evidence that a real expert wrote it. This maps to Google's E-E-A-T guidance.
- Links — pages linked to by many trustworthy sites are treated as more authoritative. This is the off-page side of SEO.
- Page experience — speed, mobile-friendliness, and stability, measured partly through Core Web Vitals.
- Context — your location, language, device, and search history all reshape results.
For a full breakdown of the individual inputs, see our guide to SEO ranking factors. The key point is that ranking is query-specific and relative: you are not scored in isolation, you are scored against every other page competing for the same search.
Rankings also move constantly. Watching them by hand across dozens of keywords is a losing game, which is where a tool like DeployFlare's rank tracker earns its place — it records daily positions so you can tie changes to what you actually shipped.
The role of algorithms and machine learning
The "hundreds of signals" are combined by ranking systems, not a single formula. Over the years Google has layered machine-learning systems on top of classic signals. RankBrain helped interpret never-before-seen queries. Later systems like BERT and MUM improved understanding of natural language, so Google grasps prepositions, nuance and context rather than matching strings.
What this means for you is simple and slightly opinionated: writing for the reader beats writing for a keyword. Machine-learning ranking rewards content that satisfies the intent behind a search, and it is very good at spotting pages stuffed with keywords but empty of substance. That is also why black-hat tactics that game old signals tend to collapse after each core update.
How AI Overviews and answer engines fit in
AI Overviews — the generated summaries at the top of some results — and answer engines like ChatGPT or Perplexity feel like a new kind of search. Under the hood, they mostly are not. Google's AI Overviews still rely on the same crawled and indexed pages; a generative model summarises trusted indexed sources and cites them.
So the fundamentals do not change:
- Your page still has to be crawlable and indexed to be quoted
- It still has to be relevant and trustworthy to be chosen as a source
- Clear structure, direct answers and schema markup make you easier to extract
The newer discipline of optimising to be cited by AI answers is often called GEO (generative engine optimisation), but it sits on the same crawl-index-rank foundation. Tracking whether AI engines mention your brand is becoming its own metric — DeployFlare covers this alongside traditional rank tracking.
How to help search engines crawl and index your site
You are not a passive spectator. Here is a practical checklist to make each stage easier for Google.
| Do | Don't |
|---|---|
| Submit an XML sitemap in Search Console | Bury pages with zero internal links (orphans) |
| Use descriptive internal links between related pages | Block important pages in robots.txt by accident |
| Keep server response and page load fast | Publish thin, near-duplicate pages at scale |
| Set correct canonical tags on similar pages | Rely on JavaScript to render critical content |
| Fix errors in the Page Indexing report | Use noindex and then wonder why you don't rank |
A few specifics worth calling out:
- Submit and maintain a sitemap. It is the fastest way to tell Google which URLs matter, especially on a new site with few backlinks.
- Link internally with intent. A page three clicks from your homepage with no internal links pointing at it may never be crawled. Strong internal linking spreads crawl priority and authority.
- Watch Google Search Console weekly. The Page Indexing and Crawl Stats reports tell you exactly which pages are excluded and why. This is the single best free diagnostic you have.
- Earn quality links. Links remain a core discovery and ranking signal. A handful of relevant, trustworthy links does more than a hundred spammy ones.
If you want the full step order from scratch, our SEO checklist turns all of this into a repeatable process, and types of SEO explains how technical, on-page and off-page work split up.
Once you understand that Google crawls, indexes, then ranks — and that each is a separate gate — SEO stops feeling like guesswork. Most "we don't rank" problems trace back to failing one specific stage, and now you know where to look.