NuclearDirectoryBot
NuclearDirectoryBot is the crawler that keeps Nuclear Directory up to date. It reads public pages, such as regulator filings, company press releases and news reports, and looks for facts about the US nuclear industry.
How to recognize it
Every request sends this user agent:
NuclearDirectoryBot/0.1 (+https://nuclear-directory.org/bot)
How it behaves
- It checks every page against the site's robots.txt, including pages it's redirected to. It reads a site's robots.txt once per visit, not once per page.
- It waits 3 seconds between requests to the same site.
- It fetches each page at most once a day and asks only for changes (ETag and Last-Modified), so an unchanged page costs a short reply.
- When a site blocks it, it stops. It doesn't retry, work around the block or pretend to be a browser.
How to block it
Add this to your robots.txt. It stops on its next visit.
User-agent: NuclearDirectoryBot Disallow: /
Questions about a visit
Email hello@nuclear-directory.org with the time and the page, or the reference number from your firewall's block page.