How crawling works
Googlebot and Bingbot find URLs through links from other sites, your own internal links and XML sitemaps. They fetch each page and render it, so content loaded by JavaScript can count. Crawling has limits. If your server slows down or returns errors, Google crawls less, and new pages take longer to appear.
robots.txt controls crawling, not indexing. Google says a blocked URL can still appear in results, just without a description. To keep a page out of search, use a noindex tag or a password.
How indexing works
After crawling, the search engine reads the main content, headings, links, images, structured data and metadata. It groups duplicates, such as HTTP and HTTPS versions of a page, and picks one canonical URL to show. Redirects, canonical tags and consistent internal links point it to the right version.
Google may crawl a thin or duplicate page and still leave it out of the index. The Page indexing report in Search Console shows which pages made it and why others did not.
How ranking works
Ranking happens at search time. The engine reads the query, including intent and location, then orders the indexed pages. Relevance comes first, then quality, authority and usability. For local searches, the distance to the searcher counts too.
How to help all three
- Link core service pages from the menu and related pages.
- Block only low-value areas like internal search results.
- Give each page one topic, a clear title and unique copy.
- Earn links and reviews, and update pages when facts change.
Read more on what robots.txt does and why pages go unindexed. Pages stuck outside Google? Our technical SEO work fixes crawl paths and index signals first.