Webpages are one of the most powerful ways to enrich your Datastore.
By connecting a webpage, your agent can directly reference and learn from the content of online resources like documentation, blogs, FAQs, or changelogs — ensuring answers stay accurate and updated.This guide walks through how to add a webpage as a data source, configure Auto-Sync schedules, and monitor updates with Sync History.
Select the Datastore where you want to add external knowledge.
Inside, go to the Data Sources tab. Here you’ll see all linked sources that feed information into your Datastore.Click Add new data source to start adding content.
Data Sources: Begin adding a new source
2
Choose Webpage as the data source type
The system supports multiple types of sources (Files, Text, GitHub, REST API, etc.).Select Webpage, paste the target URL, and click Create.
This will crawl the page content and add it into the Datastore.
Pick Webpage and provide the URL
3
Confirm the Webpage is added successfully
Once processed, the webpage source will appear in the list of sources.Each entry shows:
Source Type (Webpage, File, etc.)
Title of the page
Inclusion Status (e.g., Included in knowledge corpus)
Processing State (FINISHED, PENDING, etc.)
At this point, your agent can already use this content.
Webpage data source created and active
4
Enable Auto-Sync to keep data fresh
Webpages often change — new docs, updated FAQs, or product details.
To avoid outdated answers, toggle Auto-Resync on the source card.This ensures the system automatically re-crawls the page at set intervals.
Toggle Auto-Sync on the webpage source
5
Choose a sync frequency
When Auto-Sync is enabled, you can define how frequently the page should be re-processed.Options include:
Every hour / 3h / 6h / 12h / 24h
Specific weekdays (e.g., every Monday, every Friday)
Pick a schedule that matches how often the page changes. For example, news sites may need hourly syncs, while product docs may only need daily updates.
Set Auto-Resync frequency
6
Use Sync Now for instant updates
If urgent changes happen on the webpage and you don’t want to wait for the next scheduled sync, click Sync Now.This immediately reprocesses the page and refreshes the Datastore with the latest version.
Manual Sync Now option
7
Monitor Sync History
To track all re-crawls, click History on the source card.
Here you can see:
Number of syncs performed
Success rate (were syncs successful or failed)
Content changes (if any new text was detected)
Average processing time
This is helpful for auditing and ensuring data freshness.
