· ~14 min
SEO configuration essentials: get indexed first
Discovery, crawl, index, and appearance—robots, sitemaps, canonicals, titles, pagination, and common Search Console reasons. For any public site.
Many owners treat “not indexed” as one problem. Split it: did the crawler discover the URL, can it fetch parseable HTML, will the engine put it in the index, and do the title and snippet read like a human wrote them? Bad configuration usually fails in the first three steps. Keywords and long-tail only matter after the page is actually indexed.
Follow this order. Rules follow what Google Search Central, Bing Webmaster Guidelines, and Baidu’s webmaster college state clearly. GEO Cool scores whether AI can cite you; this article is about whether search engines can index you first. Same technical base, different goal.
1. Four separate steps
- Discovery: internal links, sitemaps, and webmaster submissions so crawlers know the URL exists.
- Crawl: robots allows it; the server returns body HTML—not a login wall or an empty shell that appears only after a click.
- Index: the engine decides to keep the page. Crawled does not mean indexed. noindex, duplicates, soft 404s, and thin quality stop here.
- Appearance: titles, snippets, and rich results. No index means no appearance; indexed still does not mean rank #1 for every query.
Priority for any site: get important URLs crawled and indexed before chasing rankings or AI citation. Unindexed pages are unstable sources for AI too.
2. robots.txt controls crawl, not “remove from search”
robots.txt tells automated clients which URLs they may fetch. Google is explicit: it mainly prevents crawlers from overwhelming the site; it is not the way to keep a page out of results. To stay out of search, use noindex or login protection. A URL blocked by robots can still appear from other links—often as a URL-only result without a snippet.
If the same page is both disallowed in robots and marked noindex, the crawler cannot read noindex, so the crawl block cancels the indexing directive. Do not put noindex inside robots.txt—Google does not support that. Googlebot also ignores Crawl-delay; throttle with Search Console crawl-rate settings or remove junk URLs from the sitemap.
- Disallow: / for Googlebot, Bingbot, or Baiduspider is a site-wide outage.
- Keep Baiduspider allowed for Chinese audiences; keep Bingbot for Bing-powered results.
- Bing: robots governs crawl; use NOINDEX to leave Bing search, Copilot, and grounding.
- End every robots file with a real Sitemap: line on the chosen canonical host.
Baidu’s college once suggested “do not create robots.txt if you want everything indexed.” Keep a permissive robots anyway: it declares Sitemap and prevents someone from later writing a full Disallow by mistake.
3. Sitemaps are hints, not commands
Google treats sitemap submission as a hint: download is not guaranteed, nor is crawling every URL. Cap one file at 50,000 URLs or 50MB uncompressed; split larger sets with a sitemap index. Use UTF-8 and absolute URLs. Google ignores changefreq and priority. lastmod helps only when it stays accurate and matches real page changes—bumping a copyright year does not count.
List only URLs you want indexed: HTTP 200, self-canonical, no noindex. Skip parameter spam, print views, admin, already-301’d URLs, and site-search results. If the CMS already emits one sitemap, do not hang a second conflicting list.
On Baidu: periodic sitemap fetches do not refresh content when the URL stays the same. New forum replies are not “fixed” by changing sitemap cadence. For faster discovery of new URLs, use Baidu’s push API; it is faster than sitemaps. An ETag on the sitemap file makes them recheck the file itself more often.
4. Canonicals and redirects
Pick one host, e.g. https://www.example.com/. 301 everything else to it: http→https, apex→www if you chose www; do not treat trailing slash and default documents as separate indexable URLs.
| Setting | Takeaway |
|---|---|
| canonical | A hint, not a command; on-page rel=canonical beats the sitemap; the canonical page must self-point |
| 301 / 308 | Permanent; the target can become canonical—use for domain moves and host consolidation |
| 302 / 307 | Temporary; results often keep the old URL—use for short maintenance |
| Avoid | Using robots or temporary removal as canonicalization; canonical to 404 or every page to the homepage |
With hreflang, canonical should point to the same-language page. Do not use noindex to pick a canonical—that removes the page from search. Use canonical for duplicates. Redirect old URLs one-to-one to new ones; do not dump everything onto the new homepage.
5. noindex: hide what should stay out
Put noindex in meta robots or the X-Robots-Tag header. The page must still be crawlable or the rule is never seen and the URL may still appear. Effect waits for the next crawl. Urgent takedown tools are temporary hides, not canonicalization.
Use noindex for: site search, filter parameter pages, cart, checkout, admin, one-off reports, previews, thank-you pages, and thin archives with no query value. Do not noindex article bodies, product detail, or important category lists. Under mobile-first indexing, mobile and desktop robots meta must match; a mobile-only noindex drops the page from the mobile-first index.
nosnippet removes text and video previews across web search, images, Discover, and AI Mode. It does not block indexing. If you still want snippets cited, do not put nosnippet on body pages.
6. Titles, long-tail, and people-first content
Result titles are not locked to your tag. Google may pick from title, visible headline, H1, og:title, prominent text, body, anchors, and WebSite schema. Changes wait for recrawl. If robots blocks the page, Google may fall back to off-site anchors.
- title and H1 say the same thing: unique object first, site name after.
- One title per page. Category, article, and product pages must not share the homepage title.
- Subheads follow reader questions: what it is, how it differs, how to do it, what to watch for.
- There is no official “N characters to rank.” Long titles only get truncated.
- Long-tail is not a special tag or an indexing switch: one concrete fact per URL, answer in the first paragraphs, clear anchors from category pages.
- Cloning one body and swapping one word across dozens of URLs is doorway spam, not long-tail.
Google’s essentials: technically crawlable, not spam, and useful to people. Meeting them still does not guarantee crawl, index, or serve. Three questions for each page: who wrote it, what is the evidence, why does it exist. There is no preferred word count. Changing dates without real edits does not help.
7. Pagination, filters, and JavaScript
Each paginated page needs its own URL and a self-canonical. Do not canonicalize page 2 to page 1. Link pages with normal href anchors. Do not put page numbers only after #—Google ignores fragments. Google no longer uses rel=next / rel=prev. “Load more” and infinite scroll: crawlers do not click buttons; items must still be reachable via pagination links or the sitemap.
Same list with only sort or filter changes: noindex them or crawl them less. Keep only the clean URL in the sitemap.
If main content needs JavaScript: put first-screen body in the initial HTML; use <a href>; SPA routing with the History API, not #/products alone; do not put noindex in the original HTML hoping a script will remove it—Google may skip rendering. Do not Disallow CSS/JS the body depends on.
8. Common “not indexed” reasons
| Report reason | Check first |
|---|---|
| Blocked by robots.txt | Allow crawl; a block also hides noindex |
| Excluded by noindex | Did a body page get noindex by mistake? |
| Page with redirect | Index should be the redirect target |
| Soft 404 | Real content should not look empty on 200; gone pages use 404/410 |
| Alternate page with proper canonical | Expected; confirm the canonical is the one you want |
| Discovered – currently not indexed | Link from important pages; more “Request indexing” clicks do not raise quota |
| Crawled – currently not indexed | Duplicates, soft 404, or no unique value; resubmitting cannot force indexing |
Third-party “instant index” tools only call webmaster APIs; they cannot order Google to index. The Indexing API is only for JobPosting or BroadcastEvent inside VideoObject pages, default 200 publishes/day—do not sweep ordinary business, content, or commerce sites with it.
9. By site type
- Blog / CMS: match the admin site URL to the canonical host; one SEO plugin for title, canonical, sitemap, and schema.
- Forums: robots at the site root; keep search and empty archives out of the sitemap; new replies are not refreshed by sitemap cadence alone.
- Behind a CDN: force HTTPS and www choice to match canonical; crawlers must get the same 200 as browsers on body URLs.
- Domain moves: 301 old URLs one-to-one; submit the new sitemap; keep redirects for months.
10. Acceptance and anti-patterns
After changing robots, sitemaps, redirects, or sitewide title templates, verify: canonical-host robots returns 200 without a full Disallow; five sitemap URLs are 200, self-canonical, not noindex; http and apex 301 to the canonical host; a body page’s source shows title and main paragraphs; the host is verified and the sitemap succeeds in Search Console or Baidu.
- Do not stuff keywords into titles, footers, or hidden text.
- Do not show long text to crawlers and a redirect to people.
- Do not buy links, build doorways, or clone one body across dozens of domains.
- Do not run two SEO plugins outputting schema.
- Do not block search crawlers for AI, or citation crawlers for SEO.
How this pairs with GEO
Search indexing is the base for stable AI citation. Indexed but hard to cite → fix answer-first copy, JSON-LD, and citation bots per the GEO guides. Cite-ready but Search Console shows blocked → return to robots, CDN, and noindex here. GEO Cool can score the same public URL for citation readiness; indexing truth still comes from each engine’s webmaster reports.