Adding Webpage Data Sources

​

Step-by-Step Guide

1

Open a Datastore and navigate to Data Sources

Select the Datastore where you want to add external knowledge.
Inside, go to the Data Sources tab. Here you’ll see all linked sources that feed information into your Datastore.Click Add new data source to start adding content.

Data Sources: Begin adding a new source

2

Choose Webpage as the data source type

The system supports multiple types of sources (Files, Text, GitHub, REST API, etc.).Select Webpage, paste the target URL, and click Create.
This will crawl the page content and add it into the Datastore.

Pick Webpage and provide the URL

3

Confirm the Webpage is added successfully

Once processed, the webpage source will appear in the list of sources.Each entry shows:

• Source Type (Webpage, File, etc.)
• Title of the page
• Inclusion Status (e.g., Included in knowledge corpus)
• Processing State (FINISHED, PENDING, etc.)

At this point, your bot can already use this content.

Webpage data source created and active

4

Enable Auto-Sync to keep data fresh

Webpages often change — new docs, updated FAQs, or product details.
To avoid outdated answers, toggle Auto-Resync on the source card.This ensures the system automatically re-crawls the page at set intervals.

Toggle Auto-Sync on the webpage source

5

Choose a sync frequency

When Auto-Sync is enabled, you can define how frequently the page should be re-processed.Options include:

• Every hour / 3h / 6h / 12h / 24h
• Specific weekdays (e.g., every Monday, every Friday)

Pick a schedule that matches how often the page changes. For example, news sites may need hourly syncs, while product docs may only need daily updates.

Set Auto-Resync frequency

6

Use Sync Now for instant updates

If urgent changes happen on the webpage and you don’t want to wait for the next scheduled sync, click Sync Now.This immediately reprocesses the page and refreshes the Datastore with the latest version.

Manual Sync Now option

7

Monitor Sync History

To track all re-crawls, click History on the source card.
Here you can see:

• Number of syncs performed
• Success rate (were syncs successful or failed)
• Content changes (if any new text was detected)
• Average processing time

This is helpful for auditing and ensuring data freshness.

Check sync history for detailed logs

​

✅ Key Takeaways

• Webpages are a simple way to continuously feed dynamic, public content into your Datastore.
• Use Auto-Sync for hands-free updates, and Sync Now for urgent refreshes.
• Track updates with Sync History to ensure your bot always has the most accurate knowledge.