Generate a robots.txt file for your website. Control which pages search engines can crawl and index.
Want to jump straight in? Use our free Robots.txt Generator tool directly.
Go to the Robots.txt Generator on UtilsNow. It helps create properly formatted robots.txt files.
Choose which search engine bots to configure rules for (Googlebot, Bingbot, or all bots with *).
Specify which paths to allow (e.g., /) and disallow (e.g., /admin/, /private/) for each bot.
Include your XML sitemap URL so search engines can discover all your pages efficiently.
Copy or download the generated robots.txt file and place it in the root directory of your website.
Use our free Robots.txt Generator tool — no signup, no uploads, works directly in your browser.
Open Robots.txt GeneratorThe robots.txt file must be placed in the root directory of your domain. For example: https://example.com/robots.txt.
robots.txt prevents crawling but not indexing. Google may still index a page (without content) if other sites link to it. Use noindex meta tag to prevent indexing.
No, Google recommends allowing access to CSS and JS files so it can render your pages properly for indexing.