domainrating.lol
List your site

Help for site owners

Allow our crawler through Cloudflare

When you add a site, domainrating.lol reads its home page once to fill in the listing. If Cloudflare answers that request with a bot challenge, we cannot read it: the challenge wants a browser that runs JavaScript and comes back with a cookie, and our reader is not one. We do not try to get around it. One narrow rule lets it through.

The user agent to allow

domainrating.lol metadata bot/1.0

What the crawler does

User agent
Exactly “domainrating.lol metadata bot/1.0”, nothing before or after it.
What it asks for
One GET of the address you submit, with Accept: text/html. It follows up to five redirects and gives up after eight seconds.
What it reads
The page’s title, description, og:image and favicon, from the HTML. It runs no JavaScript, keeps no cookies and never logs in.
What it expects back
A 200 with Content-Type: text/html and your page’s own markup. A challenge page, a 403 or a redirect loop is not your site.
When it comes back
When you submit, and about once a month to refresh the listing’s image.
Where it connects from
Our servers, to public addresses only. It refuses private and internal networks before and after every redirect.

A user agent is not a password

Anyone can send this exact string. A rule that matches it lets in everybody who copies it, so it must only ever skip a challenge on pages that are already public. Never use it to reach admin areas, logins, previews, APIs or anything behind authentication, and never to skip a block or a rate limit you rely on.

Cloudflare: one custom rule

A WAF custom rule with the Skip action lets a matching request past the features you pick, and nothing else.

  1. In the Cloudflare dashboard, open the Security rules page for your domain and choose Create rule › Custom rules.
  2. Name it something you will recognise later, such as “domainrating.lol crawler”.
  3. Edit the expression and paste this. It matches only our exact user agent, only GET requests, and only your home page:
    (http.user_agent eq "domainrating.lol metadata bot/1.0" and http.request.method eq "GET" and http.request.uri.path eq "/")

    If the address you submit is not your home page, or your home page redirects (to /en/, say), use that path instead of /.

  4. Under Choose action, select Skip, then tick only what is challenging the crawler:
    • All Super Bot Fight Mode rules, if Super Bot Fight Mode challenges automated traffic.
    • All remaining custom rules, if one of your own custom rules issues the challenge. Keep this rule above that one.
    • The Browser Integrity Check or Security Level products, if those are what challenge it.

    Leave rate limiting and managed rules alone unless they are the cause. Keep Log matching requests on, so you can see the rule work.

  5. Deploy the rule.

Bot Fight Mode is different. The free plan’s Bot Fight Mode cannot be skipped by custom rules: Skip, Allow and Bypass have no effect on it. If that is what challenges us, the choices Cloudflare gives are turning it off (Security settings, filtered by Bot traffic) or Super Bot Fight Mode on a paid plan, which does honour Skip. Or publish with your own details, below.

Cloudflare’s guides: the Skip action, what it can skip and Bot Fight Mode.

Any other firewall or WAF

Every provider names it differently, so check its own documentation. The rule you want has the same shape everywhere:

  • Match the user agent exactly, not “contains bot”, and only GET requests for the public page you submit.
  • Exempt it from JavaScript or CAPTCHA challenges and bot scoring only. Keep every block, rate limit and authentication check.
  • Order it before the rule that challenges, if your provider runs rules in order.
  • Never let it reach anything private: the string is public on this page.

Test it

Run this from a terminal, with your own address in place of your-site.com. It prints the response headers our crawler would get:

curl -sS -o /dev/null -D - -A "domainrating.lol metadata bot/1.0" -H "Accept: text/html" https://your-site.com/

You want HTTP/2 200 (or HTTP/1.1 200) and content-type: text/html. If you still see cf-mitigated: challenge, Cloudflare is still challenging it: check which feature in your security events, and skip that one. A test from your own network is a guide, not a guarantee, since Cloudflare may treat our servers’ addresses differently.

Then submit the site again.

Remove it later

Delete or disable the rule on the same Security rules page whenever you like. Your listing stays where it is, but the monthly image refresh will meet the challenge again, so its image and favicon may stop updating.

If you cannot change the firewall

Publish it with your own details instead. Submit the site while signed in; when we detect the challenge on that exact address, the form offers Publish with my details. The listing goes live straight away with the title, subtitle, description and category you typed, and nothing from the challenge page, so it has no favicon or image until our crawler can read your site.

It starts locked, like every new listing. Unlock it the usual way from your dashboard: pay for an unlock, or place our badge on your site and verify it.