Robots.txt generator
Build a robots.txt file with sitemap reference and optional rules for AI training crawlers and AI search bots.
Control which crawlers can access your site, including the AI training crawlers many businesses now want to exclude while still allowing AI search engines that send traffic.
Notes
- Blocking AI training crawlers does not affect Google search rankings.
Fill in the fields and press Calculate.
Other free calculators
Hosting
Hosting plan calculator
Answer four questions and find out whether shared, private shared or VPS hosting fits your website, with the monthly price.
Web design
Website cost calculator
Estimate what your website project will cost based on pages, content, languages and functionality.
Hosting
Downtime cost calculator
Work out what an hour of website downtime costs your business, and what different uptime guarantees mean in real hours per year.
Want us to do this for you?
We host, build and market business websites across the Gulf and worldwide. Send the brief and we will reply within one working day.