Crawl a site

web.crawl
Add to your agent
Where to start
Only crawl paths matching one of these globs, e.g. ["/docs/**"]. * matches within a segment, ** across.
Skip paths matching these globs, e.g. ["/blog/tag/**", "**/*.pdf"]
Log in to run it
Results appear here

Provider

OpenTools$0.0040

Use it from your agent

View schema →
Once OpenTools is connected, just ask your agent:
“Crawl our docs site and give me every page as Markdown.”“Crawl /blog on this site, two levels deep, and skip tag pages.”
Connect Claude, Cursor or another MCP client →

Often used with

Page to Markdownweb.markdownAny web page as clean Markdown.HTML to Markdownhtml.to_markdownConvert HTML you already have.Page metadataweb.metadataTitle, Open Graph, JSON-LD, icons.Page linksweb.linksEvery link on a page.