I Don't Always Test My Code But When I Do I Production Shirt T-Shirt

If playing around with blocking bots here is one that will block some legit and friendly bots and some bad ones too. This will do nothing for headless chrome acting as a bot. Only use this on silly hobby sites, do not use in production.

In the main site config redirect anyone not using HTTP/2.0. GoogleBot still doesnt use HTTP/2.0 so this will block Google. Bing is OK though. One could instead use variables to make this multi-condition and make exceptions for their CIDR blocks. Obviously point auth. to the same IP and ensure you have a cert for it or a wildcard cert.

    # in main TLS site config:
    # replace apex with your domain and tld with its tld.
    if ($server_protocol != HTTP/2.0) 
{ return 302 https://auth.apex.tld$request_uri; }

Then in your "auth" domain use the same config as the main site minus the redirect but then add basic authentication. Anyone not using HTTP/2.0 can still access the site if they know the right username/password. If you get a lot of bots then have an init script copy the password file into /dev/shm and reference it from there in NGinx to avoid the disk reads.

    # then in the auth.apex.tld config.
    auth_delay 2s;
    location / 
{
    auth_basic "i_heart_bots"; auth_basic_user_file /etc/nginx/.pw;
    }

This will block some API command line tools, most bots good or bad, some scanning tools. Some bots will give up prior to 2 seconds so you will get a status 499 instead of 401 in the access logs. Only do this on silly hobby sites. Do not use in production. Only people wearing a T-Shirt like this one [1] may do this in production.

One may be surprised to find that most bots use old libraries that are not HTTP/2.0 enabled. When they catch up we can replace this logic using HTTP/3.0 and UDP. Beyond that we can force people to win a game of tic-tac-toe.

[1] - https://www.amazon.com/Dont-Always-Test-Production-Shirt/dp/...