About Corebee-Bot

Corebee-Bot/1.0 (+https://corebee.ai/bot)

What we do

When a Corebee customer pastes their own website URL into our agent-build wizard, our backend fetches a small number of pages from that site to auto-detect basic brand information for the customer's support agent:

• The homepage HTML.
• Up to four common public paths: /contact, /about, /sitemap.xml, and a small number of well-known help-centre URLs.
• Same-origin <link rel="stylesheet"> files (capped at 5 sheets / 200 KB) so the colour and font extractors can read CSS that lives outside the homepage HTML.

We extract: a logo, primary and secondary brand colours, a font, a public role-prefixed contact email (e.g. support@, contact@ — never personal staff addresses), office hours from the page footer, and a short voice sample. We do not save raw HTML, do not follow off-site links, and do not crawl recursively beyond the URLs above.

When we crawl

• Only on demand. Corebee-Bot fires only when one of our customers types a URL into the agent-build wizard. We do not run a continuous crawl.
• Single visit, bounded. A single wizard scan issues at most ~10 requests to your site, all within an 8-second wall-clock budget, with a 5 MB per-response cap and a 3-redirect hop cap.
• Rate limited per Corebee customer. Each authenticated Corebee customer can run at most 10 scans per hour.

How to opt out

Corebee-Bot honours the Robots Exclusion Protocol (RFC 9309). To block all access from Corebee-Bot, add the following to your /robots.txt:

User-Agent: Corebee-Bot
Disallow: /

We refresh our cached copy of robots.txt once an hour. Once your rule is in place, the next wizard scan from any Corebee customer that targets your site will return a "Site disallows our crawler" error and no page content will be fetched.

You can also disallow individual paths the same way — for example, to allow homepage detection but block the contact page:

User-Agent: Corebee-Bot
Disallow: /contact

What we do not do

• We do not crawl a site that has not been entered into the wizard by one of our customers. We do not run a discovery crawl across the public web.
• We do not extract personal staff emails. The email scraper is limited to a closed allowlist of role-prefixed locals (support@, help@, contact@, and a few others — never jane.smith@ or ceo@).
• We do not train AI models on the content we fetch. The scrape feeds a one-time auto-detect for the Corebee customer who entered the URL.
• We do not crawl through authentication or paywalls. Only publicly reachable HTTPS URLs are fetched.

Identifying us

• User-Agent: Corebee-Bot/1.0 (+https://corebee.ai/bot)
• Outbound origin: Vercel serverless functions in Mumbai, India (bom1). The exact IPs are managed by Vercel and rotate; a reverse-DNS check against our User-Agent string is the most reliable way to identify legitimate Corebee-Bot traffic.

Reporting abuse

If you believe Corebee-Bot is misbehaving — fetching too frequently, ignoring your robots.txt, following redirects you did not expect, or impersonating Corebee-Bot from an unrelated origin — please contact us:

• Security: jonathan@corebee.ai
• Privacy: jonathan@corebee.ai

See also our Privacy Policy, Data Processing Agreement, and Security page.