🤖 Robots.txt & .htaccess Generator
Visually build robots.txt and .htaccess files with common presets, live preview, basic syntax checks, and one-click download.
Last updated: November 8, 2026 · By the BoltQuickTools team
Write crawler rules and Apache config without memorising the syntax
Site owners editing robots.txt or .htaccess by hand can take down search traffic or a whole site with one wrong line. This page assembles both files from form controls, so you start from correct directives and can read what will be deployed. It only writes text: nothing here tests your server.
Building a robots.txt
- Start from a preset: Allow All, Block All, Block AI Crawlers, Standard (block admin, api, private) or WordPress Default. Applying one replaces everything you have entered.
- Adjust each block under User-Agent Rules: pick a bot or Custom..., use + Add Rule for more Allow and Disallow paths, and fill in Crawl-delay if you want one.
- Add URLs with + Add Sitemap. Check the Validation line, then use Copy or Download as robots.txt.
The Standard preset with a sitemap URL and a Crawl-delay of 10 produces:
User-agent: *
Disallow: /admin/
Disallow: /api/
Disallow: /private/
Disallow: /tmp/
Allow: /
Crawl-delay: 10
Sitemap: https://example.com/sitemap.xml
Building an .htaccess
Switch to the .htaccess tab and turn on what you need. Sections appear in the preview in a fixed order. With Force HTTPS redirect and Force www on, you get:
# Force HTTPS
RewriteEngine On
RewriteCond %{HTTPS} off
RewriteRule ^(.*)$ https://%{HTTP_HOST}%{REQUEST_URI} [L,R=301]
# Force www
RewriteEngine On
RewriteCond %{HTTP_HOST} !^www\. [NC]
RewriteRule ^(.*)$ https://www.%{HTTP_HOST}%{REQUEST_URI} [L,R=301]
Back up the existing file before uploading. A syntax error here returns a 500 for the whole directory, and only the robots.txt side has a validator. The .htaccess fields are not checked, so a cache period of banana goes straight into the output.
Serious traps in the .htaccess output
- Do not combine Block IP Addresses with Password Protection. Apache treats separate top-level authorization rules as "any of these", so the generated
<RequireAll>block lets every unblocked visitor in without a password. To use both, moveRequire valid-userinside the<RequireAll>block by hand. - IP blocking needs Apache 2.4.
Require not ipdoes not exist in 2.2. - Force www and Force non-www always redirect to https. Only enable them on a site with a working certificate. Enabling one turns the other off.
- Redirect rules match by prefix.
Redirect 301 /old-page /new-pagealso sends/old-page/xto/new-page/x. Paths containing spaces are written unquoted and will break the line. - Hotlink protection matches the start of the referrer, so
example.com.other.testalso passes. Empty referrers are allowed on purpose. - This is Apache syntax. Nginx does not read
.htaccessat all.
Things robots.txt cannot do
- It is a request, not a lock. It keeps polite crawlers out, and it publishes your
/admin/path to anyone who reads it. Use authentication for private areas. - Blocking a URL does not remove it from search results. A page needs to be crawlable with a noindex tag for that, which the meta tag generator can write.
- Google ignores
Crawl-delayand the Host Directive. Other crawlers vary. A delay below 1 is left out of the file. - Block AI Crawlers names GPTBot, CCBot, Google-Extended, anthropic-ai and ClaudeBot, and nothing else. Add others through Custom... if you want them covered.
- WordPress Default is more aggressive than WordPress's own defaults. Blocking
/wp-includes/and plugins can hide CSS and scripts that search engines need to render pages. It also inserts anexample.comsitemap that you must replace, and the validator will not flag it.
Disallow, validation and password files
Does an empty Disallow block everything?
No, it is the opposite. Disallow: with no path allows all URLs for that bot, and a single / blocks them all.
What does the validator actually check?
Empty user-agent names, an Allow with no path, paths that do not start with / or *, and sitemap URLs without http or https. It does not check that a path exists or that rules make sense.
Where do the password file and permissions come in?
Path to .htpasswd file must be an absolute server path, and that file has to be created separately. The chmod calculator helps with its permissions, and HTTP status codes explains the 301, 302 and 403 responses these rules return.