Check robots.txt

web.robots
Add to your agent
Site or robots.txt URL
robots.txt content, instead of a URL
Crawler name, like "Googlebot" or "GPTBot"
Paths or URLs to check
Log in to run it
Results appear here

Provider

Opentools$0.0010

Use it from your agent

View schema →
Once Opentools is connected, just ask your agent:
“Is GPTBot allowed to crawl our /docs pages?”“What does nytimes.com's robots.txt block?”
Connect Claude, Cursor or another MCP client →

Often used with

Page to Markdownweb.markdownAny web page as clean Markdown.HTML to Markdownhtml.to_markdownConvert HTML you already have.Page metadataweb.metadataTitle, Open Graph, JSON-LD, icons.Page linksweb.linksEvery link on a page.