Free Tool
Free Robots.txt Checker
Check whether robots.txt is reachable and whether the site exposes crawl guidance and sitemap references.
What this tool checks
Run one URL scan and get a complete report.
- robots.txt status
- robots.txt sample
- Sitemap reference hint
- Indexability warnings
How it works
The scanner fetches the public page, follows a limited redirect chain, collects headers/resources/HTML, applies self-owned detection rules, checks SEO/security/accessibility/performance signals, and returns evidence-backed fixes.
Private IPs, localhost, file URLs, and internal hostnames are blocked to reduce SSRF and abuse risk.
Related tools