Crawling and indexing: how Google finds your pages

Crawling and indexing are two separate gates, and most ranking problems fail at the second one

If a page "isn't ranking," check first whether it's even in Google's index. Crawling and indexing are two distinct steps: crawling is Googlebot fetching your page, indexing is Google deciding to store that page and let it compete in results. A page can be crawled and never indexed. When that happens, no amount of keyword tweaking or link building helps, because the page isn't in the running at all.

Here's the shift most guides miss. The dominant reason pages don't get indexed is no longer technical (broken robots rules, noindex tags, server errors) but quality. The Search Console status "Crawled - currently not indexed" usually means Google fetched the page, evaluated it, and chose to leave it out. The fix is content depth and internal links, not resubmitting your sitemap for the tenth time.

This is a spoke of our technical SEO pillar. For the full picture on how search engines process a site, start there, then come back here for the crawl-and-index mechanics.

The pipeline: discover, crawl, render, index, serve

Google Search runs pages through five stages, and each is a gate a page can fail at. Google's own in-depth guide to how Search works lays out the same sequence.

1. Discover — Google learns a URL exists. It follows links from pages it already knows, reads your XML sitemap best practices, and picks up URLs from other signals. A page with no internal links pointing to it and no sitemap entry can stay undiscovered indefinitely.
2. Crawl — Googlebot requests the URL and downloads the HTML. Your robots.txt rules decide whether Googlebot is even allowed to fetch it. Googlebot is the crawler doing this fetch, and a blocked URL never gets past this gate.
3. Render — Google runs the page's JavaScript to see the final DOM, the way a browser would. If your content only appears after client-side JS execution, this stage matters a lot. See JavaScript SEO for the failure modes.
4. Index — Google analyzes the rendered content, works out what the page is about, checks canonical tags to pick the representative URL, and decides whether the page is worth storing. This is where quality judgments happen.
5. Serve — Indexed pages compete for rankings when someone searches.

Ranking, the thing everyone obsesses over, is the last gate. If a page fails at gate 1, 2, or 4, it never reaches gate 5.

How Google finds new pages

Discovery has three main channels, in rough order of reliability:

• Internal links. Googlebot crawls by following links. A page linked from your navigation or from a high-traffic article gets found fast. An orphan page with no internal links may never be found. This is the single most underrated indexing lever: adding three or four internal links from established pages does more than any resubmit.
• XML sitemaps. Your sitemap is a list of URLs you're telling Google you care about. It helps discovery, especially for new or deep pages, but a sitemap entry is a hint, not a command. Being in the sitemap does not guarantee indexing.
• External links. Backlinks from other sites also surface your URLs to Googlebot.

For a single new page you want crawled sooner, use the URL Inspection tool in Search Console and click Request Indexing. Google's guidance on asking it to recrawl is explicit that this is for individual URLs, not bulk submission, that there's a quota, and that "requesting a recrawl multiple times for the same URL won't get it crawled any faster." It speeds up crawling, not the quality decision.

Map each Search Console status to the gate it fails at

The Page indexing report in Search Console tells you exactly where a URL is stuck. Read the status literally, because the wording maps to a specific gate.

The distinction between the top two is the one people get wrong. "Discovered" means the URL is queued but not yet fetched, frequently because Google rescheduled the crawl to avoid overloading your site. "Crawled - currently not indexed" means Google already fetched it and made a judgment call. Google's documentation says there's no need to resubmit a "Crawled - currently not indexed" URL, which is the tell: the ball is in your court to make the page better, not Google's to try again.

Fixing "Crawled - currently not indexed"

This is the status that trips up most sites, and resubmitting does nothing. Work through these in order:

• Content depth. Thin, near-duplicate, or templated pages get dropped. Programmatic pages that differ by only a city name or a single variable are the classic culprit. Each indexed page needs a reason to exist.
• Internal links. Pages Google considers low-priority get deprioritized for indexing. Linking to the page from relevant, already-indexed content signals that it matters. A technical SEO audit finds orphan and weakly-linked pages fast.
• Uniqueness against your own site. If ten pages say roughly the same thing, Google keeps one. Consolidate or differentiate.
• Overall site quality. Google evaluates pages partly in the context of the whole site. A site full of thin pages makes it harder for any single page to earn indexing.

Structured data won't force indexing, but clean structured data markup helps Google understand a page and can improve how it appears once indexed. Fix the content and links first.

Crawl budget: stop worrying unless you're big

Crawl budget is the amount of crawling Google is willing to do on your site, set by two things: crawl capacity (how much your server can handle without slowing down) and crawl demand (how much Google wants your content). It's real, but it's almost certainly not your problem.

Google's guide to managing crawl budget states plainly that it's an advanced topic aimed at large sites (1 million-plus pages with content changing about weekly), medium-or-larger sites with very rapidly changing content (10,000-plus pages updated daily), or sites with a large share of "Discovered - currently not indexed" URLs. Google adds that these numbers are "a rough estimate," not exact thresholds. If you run a few hundred or a few thousand pages, crawl budget is not why pages aren't indexed. Chasing it is a distraction from the content-and-links work that moves the needle.

If you are large enough to care, the levers are: cut duplicate and low-value URLs (faceted navigation and infinite parameter combinations are the usual waste), keep your server fast because slow responses lower crawl capacity, and make sure your most important pages are the easiest to reach. Faster pages get crawled more, so tightening Core Web Vitals has a direct crawl benefit at scale.

How to check and speed things up

• Audit index coverage. Open the Page indexing report in Search Console and read the status breakdown. It tells you what share of your pages are indexed and why the rest aren't.
• Inspect individual URLs. Use URL Inspection to see the exact status of any page and request indexing for genuinely new or updated ones.
• Fix discovery gaps at scale. A site audit surfaces orphan pages, broken internal links, forgotten noindex tags, and canonical conflicts across your whole site, which is faster than checking URLs one at a time.
• Watch it over time. Indexing isn't permanent. Pages get dropped when quality slips or duplicates pile up. Connecting your Search Console data to monitoring lets you catch a drop in indexed pages before it costs you traffic. DeployFlare does this from ₹499/month, billed in INR with UPI and GST, and the same index tracking applies whether you rank in Delhi, Dubai, or Denver.

The mental model to keep: crawling is a plumbing problem, indexing is a quality problem, and ranking only happens after both. Most "not ranking" cases are really "not indexed," and most "not indexed" cases today are content and internal-link problems wearing a technical costume.