DataInsideBot
Our crawler reads public pages on company websites to keep company records and business contact details accurate. Here is exactly how it behaves, and how to stop it.
How it behaves
- robots.txt comes first
- We read your robots.txt before anything else and follow it, including Crawl-delay. If robots.txt disallows a page, we never request it; if it disallows the whole site, or answers 401, 403 or 429, we stop and don't come back on that visit.
- One request at a time
- We send at most one request at a time to a site, at least one second apart (two by default, longer if your Crawl-delay asks).
- A few pages, rarely
- Each visit reads the home page and up to a dozen pages about the company: team, leadership, about, contact, press and careers pages. We come back about once a month, and skip pages that haven't changed.
- Public pages only
- No logins, forms, search boxes, paywalls or CAPTCHAs, and we never try to get around a block.
- Honest identity
- Every request says DataInsideBot in its user agent, with a link to this page and our email address.
What we keep
- About the company: its name, what it does, its offices, and its general contact details such as a main phone number or info@ address.
- About people in a business role who are named on the company's own pages: name, job title, and work contact details the page publishes for them.
- A compressed copy of each page read, so we can show where a fact came from and re-read it without visiting again.
We never keep personal email addresses, home addresses, photos or anything sensitive. Details about people are not shown or sold to anyone until we have approved that use for the person's country under its law.
Block it on your site
Add this to your robots.txt. We read it on every visit, so the change takes effect next time.
User-agent: DataInsideBot Disallow: /
To slow it down instead, add Crawl-delay: 10 (seconds) under the same line. To have pages we already read about your company removed, email hello@datainside.ai from an address at your domain.
Remove yourself
If your name or work contact details appear on DataInside, enter your work email in the Privacy Center: we email you a private link to see what we hold, the page each detail came from, and to correct it or have it removed. Once removed, you stay removed: we keep only a scrambled marker so your details don't come back from another page.