Your website, as your agent's knowledge
Give it a URL and it reads your pages into the knowledge base your agent answers from, no copying, no maintaining a separate FAQ.
The Website Crawler turns your existing site into agent knowledge. Point it at a URL, and the pages are crawled, chunked and embedded into the same knowledge base your agent retrieves from when it answers. When your site changes, re-crawl and the answers follow. It needs no account and no credentials, just the address of a site you own.
What your agent can do
Crawl a site into knowledge
Give it a starting URL and it walks the pages, turning them into knowledge your agent can quote from in a conversation.
Re-crawl when things change
Run it again after a site update and the refreshed pages replace what was there. Your agent stops answering from last month's copy.
- 03
Sits alongside your uploads
Crawled pages live in the same knowledge base as your uploaded documents, so the agent draws on both without you choosing between them.
How to set it up
- 1
Add the Website Crawler
No account to link and nothing to authorise. It only needs a URL.
- 2
Give it your address
Enter the site you want read. Use a site you own or have permission to crawl.
- 3
Crawl, then check
Start the crawl and watch the pages land in Knowledge, where you can see exactly what your agent can now draw on.
Frequently asked
Does it keep my site up to date automatically?
No. You choose when to re-crawl, so the knowledge only changes when you decide it should.
Can I crawl a site I don't own?
Please don't. Point it at sites you own or have permission to crawl. You're responsible for what you feed your agent.
Where do the crawled pages end up?
In your space's knowledge base, next to your uploaded files, where you can review and remove them.
Related pages
Give your AI agent Website Crawler
Start free, connect your tools, and put an AI agent to work, cancel anytime.