Glossary
What is robots.txt
Also: robots file, crawl rules, noindex
robots.txt is a small text file at the root of your site that tells search engine crawlers which parts they may or may not visit. A wrong line can hide your whole site from Google.
robots.txt lives at yourdomain.com/robots.txt and contains rules for crawlers: «Googlebot, don't visit /admin», «here is the Sitemap (sitemap.xml)A sitemap is a machine-readable list of all your site's pages (sitemap.xml) that helps Google find and index them faster. Generated automatically by most platforms; submit it in Search Console.→». It's the «staff only» sign on certain doors. It does not password-protect anything — it's a polite request that search engines honour — and it is also a common cause of invisible sites: a leftover Disallow: / from the development stage blocks everything. To keep a single page out of search results, a noindex Meta tags (title & description)Meta tags are the page title and description that Google shows in search results and browsers show in the tab. Good ones get more clicks; each page should have its own.→ is the right tool.
- Check yours: open
/robots.txtin the browser —Disallow: /underUser-agent: *means Google is locked out. - AI crawlers (GPTBot, ClaudeBot…) read the same file; allowing them helps your business appear in AI answers.
- Platforms manage it for you; you rarely edit it by hand.