ClarvithBot
Clarvith Actors identify as ClarvithBot/1.0 (+https://clarvith.com/bot; public job-board lookup for search and reference; not for AI training).
This page describes what the crawler fetches and how it behaves.
What it fetches
- Public ATS HTTP APIs: Greenhouse, Lever, Ashby, and Workable job-board endpoints documented in the Actor README.
-
Company websites for discovery: homepage and, when allowed, careers paths such as
/careers,/jobs, and/company/careers. -
robots.txtfor each host before other paths, per RFC 9309.
robots.txt (RFC 9309)
- A 2xx robots.txt is obeyed (up to five redirects on that file).
- 4xx on robots.txt means the host is allowed.
- 5xx, timeout, or unreachable robots.txt blocks the host for that run.
- 429 responses pause the host (60 seconds) and reduce rate; repeated 429 may disable a host.
Rate limits (as implemented)
Per-host request pacing in the shared HTTP client:
- Lever and Workable API hosts: at most 1 request per second.
- Greenhouse and Ashby API hosts: at most 3 requests per second.
- Other hosts (company sites): default 1 request per second unless configured otherwise.
Concurrent company fetches respect these limits per host. Workable may be disabled for the remainder of a run after a second 429.
Opt out
To request blocking or discuss crawling, email [email protected] with the hostname and your role. Standard robots.txt rules remain the primary control mechanism.