Jump to content

Need help blocking a crawler


pwired
 Share

Recommended Posts

Hi,

One of my clients website has a lot of traffic due to a chinese crawler,

every day constantly up to 1000 visits.

It´s the agressive Baidu crawler coming from this ip range:

180.76.5.x
180.76.6.x

Besides the page crawling, it is really nervous to see how many attempts this crawler

makes on login.php

However this client has no business and no clients in china.

He wants me to block this crawler. I tried with robots.txt

but this crawler does not seem to respect what is configured

in robots.txt

I need something more powerfull with a .htaccess file.

Has anyone an example .htaccess file how to block

this ip range and do I put this in the root public.html ?

Link to comment
Share on other sites

  • 6 years later...
1 hour ago, wbmnfktr said:

In case you are using tools like Xenu, Screamingfrog and some others you might want to remove those from the lists.

Indeed.  I use Ahrefs for example to crawl my own site.  The tool I posted blocks it by default.

Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
 Share

  • Recently Browsing   0 members

    • No registered users viewing this page.
×
×
  • Create New...