WebsiteSEOChecker-Audit crawler

Website SEO Checker · Last updated: September 29, 2026

In short. Our crawler visits a site only when a person asks for a website audit of it. It reads at most 25 pages per check, two at a time with a pause, follows robots.txt and never submits forms or logs in.

How to recognise it

User agent:

Mozilla/5.0 (compatible; WebsiteSEOChecker-Audit/1.0; +https://www.seolytics.app/website-seo-checker/bot)

Requests come from Supabase Edge Functions servers, so there is no fixed IP list.

How it behaves

Starts from the address a person entered and follows links on the same site, then URLs from the sitemap.
Up to 25 pages per check, 2 requests at a time, at least 0.3 s between requests. Crawl-delay is respected up to 5 s.
Skips login, cart, checkout, admin, search and feed URLs, files and URLs with sort or filter parameters.
Reads only the HTML. It does not run JavaScript, load images or send cookies.
Stops when the site answers with a bot check or rate limit and reports that it was blocked.

How to block it

Add a group to robots.txt. The crawler follows the rules for websiteseochecker, or for * if there is none:

User-agent: websiteseochecker Disallow: /

How to allow it through bot protection

If you want to audit your own site and it sits behind Cloudflare, Vercel or a similar service, add a rule that skips the bot check for user agents containing WebsiteSEOChecker-Audit. Or use the Chrome extension: it checks the site from your own browser, where the protection already lets you through.

Contact

Questions or problems with the crawler: amirovlenarr@gmail.com.