🤖 Robots.txt & .htaccess Generator

Visually build robots.txt and .htaccess files with common presets, live preview, basic syntax checks, and one-click download.

Last updated: November 8, 2026 · By the BoltQuickTools team

Presets
User-Agent Rules
Sitemaps
Host Directive
Validation
No issues found.
Preview
HTTPS & WWW
Custom Redirects
Custom Error Pages
CORS Headers
Browser Caching
Compression
Hotlink Protection
Block IP Addresses
Password Protection
Preview

Write crawler rules and Apache config without memorising the syntax

Site owners editing robots.txt or .htaccess by hand can take down search traffic or a whole site with one wrong line. This page assembles both files from form controls, so you start from correct directives and can read what will be deployed. It only writes text: nothing here tests your server.

Building a robots.txt

  1. Start from a preset: Allow All, Block All, Block AI Crawlers, Standard (block admin, api, private) or WordPress Default. Applying one replaces everything you have entered.
  2. Adjust each block under User-Agent Rules: pick a bot or Custom..., use + Add Rule for more Allow and Disallow paths, and fill in Crawl-delay if you want one.
  3. Add URLs with + Add Sitemap. Check the Validation line, then use Copy or Download as robots.txt.

The Standard preset with a sitemap URL and a Crawl-delay of 10 produces:

User-agent: *
Disallow: /admin/
Disallow: /api/
Disallow: /private/
Disallow: /tmp/
Allow: /
Crawl-delay: 10

Sitemap: https://example.com/sitemap.xml

Building an .htaccess

Switch to the .htaccess tab and turn on what you need. Sections appear in the preview in a fixed order. With Force HTTPS redirect and Force www on, you get:

# Force HTTPS
RewriteEngine On
RewriteCond %{HTTPS} off
RewriteRule ^(.*)$ https://%{HTTP_HOST}%{REQUEST_URI} [L,R=301]

# Force www
RewriteEngine On
RewriteCond %{HTTP_HOST} !^www\. [NC]
RewriteRule ^(.*)$ https://www.%{HTTP_HOST}%{REQUEST_URI} [L,R=301]

Back up the existing file before uploading. A syntax error here returns a 500 for the whole directory, and only the robots.txt side has a validator. The .htaccess fields are not checked, so a cache period of banana goes straight into the output.

Serious traps in the .htaccess output

Things robots.txt cannot do

Disallow, validation and password files

Does an empty Disallow block everything?

No, it is the opposite. Disallow: with no path allows all URLs for that bot, and a single / blocks them all.

What does the validator actually check?

Empty user-agent names, an Allow with no path, paths that do not start with / or *, and sitemap URLs without http or https. It does not check that a path exists or that rules make sense.

Where do the password file and permissions come in?

Path to .htpasswd file must be an absolute server path, and that file has to be created separately. The chmod calculator helps with its permissions, and HTTP status codes explains the 301, 302 and 403 responses these rules return.