Website Crawling

Website crawling is the easiest way to train your chatbot. BubblaV scans your pages, extracts content, and makes it searchable so your bot can answer questions accurately.

​

Adding Your Main Website

When you create a new website in your dashboard, you’ll be prompted to enter your main website URL. The crawling process starts automatically after you add the website. To add a main website later:

---

​

How Crawling Works

When you add a website or sub-website, BubblaV automatically starts crawling:

1. Checks for llms-full.txt - looks for AI-optimized content first
2. Falls back to sitemap if llms-full.txt not available
3. Visits your URL and extracts all text content
4. Follows links to discover other pages on your domain
5. Processes content into searchable chunks with embeddings
6. Updates status for each page (Crawled, Pending, Failed)

---

​

AI-Optimized Content (llms-full.txt)

BubblaV automatically detects and uses the llms.txt standard when available. This is an emerging standard for providing AI-ready documentation.

​

What is llms-full.txt?

Many modern documentation sites now provide a /llms-full.txt file - a single file containing all their documentation in a structured format. Major companies like Anthropic, Vercel, and Stripe use this format.

​

How BubblaV Uses It

When you add a website, BubblaV checks for /llms-full.txt first:

1. If llms-full.txt exists: Content is extracted directly from this file, no page-by-page crawling needed
2. If llms-full.txt doesn’t exist: Falls back to standard sitemap-based crawling

​

Benefits

​

Incremental Updates

For sites with llms-full.txt:

• We store a fingerprint of the file
• During incremental crawls, we check if the file changed
• Only re-process if content is updated
• This is more efficient than checking each page

​

Checking if Your Site Supports It

Visit https://yoursite.com/llms-full.txt in your browser. If it loads, your site supports this feature. Popular documentation platforms that support llms-full.txt:

• Mintlify
• GitBook
• Docusaurus (with plugin)
• ReadMe

---

​

Adding More Content Sources

​

Sub-websites

What is a sub-website? A sub-website is an additional website or domain that you want to include in your knowledge base alongside your main website. This allows you to train your chatbot on content from multiple related sites. To add a sub-website:

1. Go to Knowledge → Websites
2. Click Add Website
3. Enter the sub-website URL (e.g., https://blog.example.com)
4. Click Add - crawling starts automatically

Use cases:

• Blog on a subdomain (e.g., https://blog.example.com)
• Help center on a different domain
• Regional or language-specific sites
• Multiple related websites you want to include in one knowledge base

​

Individual Pages

Add specific URLs that aren’t linked from your main site:

1. Go to Knowledge → Pages
2. Click Add Page
3. Paste the full URL
4. Click Add - the page will be crawled automatically

Use cases:

• Landing pages
• PDF documents hosted online
• Specific product pages

​

Sitemap Import

Import all URLs from your sitemap at once:

1. Go to Knowledge → Sitemaps
2. Click Add Sitemap
3. Enter your sitemap URL (e.g., https://example.com/sitemap.xml)
4. Click Import

All URLs in the sitemap will be automatically queued for crawling.

---

​

Managing Crawled Pages

​

Enable/Disable Pages

Toggle pages on/off to control what the bot knows:

• Enabled: Bot can use this content to answer questions
• Disabled: Content is stored but not used

​

Delete Pages

Permanently remove pages from your knowledge base:

1. Find the page in the list
2. Click the delete icon
3. Confirm deletion

---

​

Automatic Incremental Crawling

BubblaV automatically performs incremental crawls to keep your knowledge base up to date. The system detects changes on your website and only crawls new or updated pages, making the process efficient and fast. How it works:

• The system monitors your websites for changes
• New pages are automatically discovered and crawled
• Updated pages are re-indexed when changes are detected
• No manual action is required

Sync frequency by plan: Incremental crawls run automatically in the background. You don’t need to manually trigger re-crawls.

---

​

Plan Page Limits

When you hit your limit, new pages won’t be crawled and you won’t be able to upload files. Upgrade your plan for more capacity.

---

​

Best Practices

---

​

Troubleshooting

---

​

Next Steps