WebFora.io

Glossary

What is robots.txt

Also: robots file, crawl rules, noindex

robots.txt is a small text file at the root of your site that tells search engine crawlers which parts they may or may not visit. A wrong line can hide your whole site from Google.

robots.txt lives at yourdomain.com/robots.txt and contains rules for crawlers: «Googlebot, don't visit /admin», «here is the Sitemap (sitemap.xml)A sitemap is a machine-readable list of all your site's pages (sitemap.xml) that helps Google find and index them faster. Generated automatically by most platforms; submit it in Search Console.→». It's the «staff only» sign on certain doors. It does not password-protect anything — it's a polite request that search engines honour — and it is also a common cause of invisible sites: a leftover Disallow: / from the development stage blocks everything. To keep a single page out of search results, a noindex Meta tags (title & description)Meta tags are the page title and description that Google shows in search results and browsers show in the tab. Good ones get more clicks; each page should have its own.→ is the right tool.

  • Check yours: open /robots.txt in the browser — Disallow: / under User-agent: * means Google is locked out.
  • AI crawlers (GPTBot, ClaudeBot…) read the same file; allowing them helps your business appear in AI answers.
  • Platforms manage it for you; you rarely edit it by hand.

Related terms

Order · €39Audit