We had one site that was struggling under the strain of AI bots (the admin became almost unusable); The hosting support confirmed they were getting hit massively by AI bots. They ended up implementing apache level blocks. Then we had 2 other sites (running on Digital Ocean droplets) which started to get hit and were crashing, hitting 100% CPU. I have since installed WireRequestBlocker on those 2 sites (and had to up the memory, CPU and storage) On one of them we are now seeing a lot of activity, and IPs being blocked, and have been running smoothly since.
In the meantime i'm trying out the ai.txt which can be generated here:
https://site.spawning.ai/spawning-ai-txt
Has anyone else tried it?