AI crawlers, yes... you know
AIs are training, and I'm somehow glad to provide content. The thing is, some of these crawlers are creating trouble because, as you might guess, the crawler isn't that intelligent.
I've read a lot about blocking AI crawlers, or crawlers in general, and some ideas just don't work. The two things to take into account are: if the crawler honors your settings, there is no need to block it; and if the opposite is true, no matter what settings you have, it won't read them.
In these examples, we are taking the easy route: you have a good sitemap, and the web is fast enough to respond to normal pages. This leaves us with the rest: heavy pages with pagination and multiple user-functionality links that were never intended for crawlers. If this is you, then these examples will help
The website settings
There are two places where you educate crawlers website wide: robots.txt and sitemap.xml (or whatever you named it). See them as filtering what's not allowed (robots.txt) and asking the crawler what to visit (sitemap.xml).
Now we'll make a simple supposition, you don't need the crawler to paginate, because you already provide the links in the sitemap. A very simple, and functional robots.txt will look like this:
robots.txt we tell all User-agent to wait 10 seconds between requests, no crawler reads this, but yeah, why not trying? We also tell them not to follow links with parameters, this actually works, just remember to provide the links in the sitemap. And finally, not to crawl the user login page, in the example in /user.
User-agent: *
Crawl-delay: 10
Disallow: /*?
Disallow: /user
Sitemap: https://www.yourdomain.com/sitemap.xml
Elephant auto-generates the sitemap.xml, including links for projects, challenges, publications, convocations and other Elephant Entities. If you have to create the file by hand, you'll probably need to read this www.sitemaps.org.
The page settings
Both robots.txt and sitemap.xml are website wide. Now we'll see what we can do in pages, and also what is unnecessary because simply doesn't work.
robots.txt, pages with pagination, user functionality and the like. The purpose is to tell the crawler to index the content, but not to follow links.
<html ...>
<head ...>
<meta name="robots" content="index,nofollow">
...
<a ... rel="nofollow">
The malicious crawlers
Of course, malicious crawlers will simply ignore any configuration. At this point you need a system to detect and block.
robots.txt. Personally, I think it's a waste of time, they will simply ignore it, and if you block far too much, the social links (card formattings) will suffer from this.
Some possible solutions are:
Fail2Ban is probably the best one. Requires the location of your access logs, the ones the HTTP server generates from requests. Temporarily bans IPs based on their insistence (what crawlers do). A drawback would be, it could also block educated/non-dangerous crawlers. How to fine-tune Fail2Ban settings is up to your experience. Fail2Ban is available in most Linux distributions, so installing won't be a problem. Read more on Fail2Ban.
Firewall rules is a drastic method to block IPs. Not my favorite because requires maintenance, and you will end up blocking entire subnets. The main problem is that the IPs are temporarily used by uneducated crawlers, plus they can change to another subnet faster than the time you take to block them.
Firewall and self-made application is a good solution, and may work as **Fail2Ban*, which gives a reason not to reinvent the wheel.
Self-made HTTP filter is the Elephant solution, but involves developing. Not difficult if you work with subnets, but still needs coding. I won't enter into obvious details, but if you need help, contact me
