Clearbot

Clearbot is Clearnode's website crawler. Clearnode checks websites for accessibility, site health, and uptime. Our customers are the businesses and agencies that own or manage those websites: they add their sites, and Clearnode checks them on a schedule. If you have seen Clearnode in your server logs, this page explains how to recognize our requests, what they do, and how to control them.

How to recognize Clearbot

User agent. Clearnode requests use a standard Chrome user agent with this identifier appended:

Clearnode/1.0 (+https://clearnode.app/bot)

IP addresses for page and uptime checks. These requests typically come from addresses in Laravel Cloud's published IP address list. The addresses are shared with other applications, so an IP address alone does not identify Clearnode. If you allowlist those addresses, also match the Clearnode user agent or use the secret header described below. We link to the provider's current list rather than copying addresses that may change.

IP addresses for browser rendering. Rendering checks use a portable browser client, so their source addresses depend on the client and cannot be listed reliably here. For questions about identifying or allowing rendering traffic, email [email protected].

What Clearbot does

  • Page checks: Fetches page HTML to look for broken links, errors, and missing titles, headings, or descriptions.
  • Browser rendering: Loads a page's styles, fonts, and images in a browser and runs an accessibility audit. It blocks analytics, advertising, and video. If the site's owner has configured it to do so, it closes the cookie banner first.
  • Discovery: Reads robots.txt and sitemaps to find pages on a site.
  • Uptime checks: Sends one request to an address chosen by the site's owner, as often as once a minute depending on the customer's plan.

Most monitored pages are checked less often than uptime targets: key pages are typically checked daily, other monitored pages weekly, and all pages in a full scan about every six months. A site's owner can also request a rescan.

How Clearbot behaves

  • For page checks and rendering, it follows robots.txt rules for the Clearnode user agent, including Crawl-delay.
  • Across our customers, it sends one request at a time to a website, at least one second apart or longer if Crawl-delay requires it. It renders one page at a time.
  • It reads pages. It does not sign in or submit forms.

There are two customer-controlled exceptions. A customer who manages a site can choose to ignore its robots.txt rules, for example when a staging site blocks all crawlers. Uptime checks do not read robots.txt; each is a single request to an address the customer chose to monitor.

How to stop Clearbot

To stop page checks and rendering, add these rules to your site's robots.txt:

User-agent: Clearnode
Disallow: /

These rules do not stop uptime checks, and a customer who manages the site can choose to override robots.txt. To stop all Clearnode requests to your site, email [email protected] with your website's address and ask us to block it. We add blocked sites to a list that customer settings cannot override.

How to allow Clearbot

If you are a Clearnode customer and your firewall, CDN, or bot protection blocks checks, you can allow them in these ways:

  1. Secret header (most precise): In your site's Clearnode settings, set a value for the X-Clearnode-Key header. Clearnode sends it with requests to that site. Allow requests whose header exactly matches your private value.
  2. User agent: Match the Clearnode/1.0 (+https://clearnode.app/bot) identifier above. A user agent can be imitated, so do not treat it alone as proof of identity.
  3. IP addresses and user agent: For page and uptime checks, match the relevant addresses in Laravel Cloud's current IP list together with the user agent. Rendering-check addresses vary by client; contact us if you need help allowing them.

Contact

For questions, opt-outs, or abuse reports, email [email protected].