Jump to content

Recommended Posts

Presentation
Originaly developped by Jeff Starr, Blackhole is a security plugin which trap bad bots, crawlers and spiders in a virtual black hole.
Once the bots (or any virtual user!) visit the black hole page, they are blocked and denied access for your entire site.
This helps to keep nonsense spammers, scrapers, scanners, and other malicious hacking tools away from your site, so you can save precious server resources and bandwith for your good visitors.

Β 

How It Works
You add a rule to your robots.txt that instructs bots to stay away. Good bots will obey the rule, but bad bots will ignore it and follow the link... right into the black hole trap. Once trapped, bad bots are blocked and denied access to your entire site.


The main benefits of Blackhole include:

Quote

Stops leeches, scanners, and spammers
Saves server resources for humans and good bots
Improves traffic quality and overall site security

Β Bots have one chance to obey your site’s robots.txt rules. Failure to comply results in immediate banishment.

Β 

Features

  • Disable Blackhole for logged in users
  • Optionally redirect all logged-in users
  • Send alert email message
  • Customize email message
  • Choose a custom warning message for bad bots
  • Show a WHOIS Lookup informations
  • Choose a custom blocked message for bad bots
  • Choose a custom HTTP Status Code for blocked bots
  • Choose which bots are whitelisted or not

Β 
Instructions

  1. Install the module
  2. Create a new page and assign to this page the template "blackhole"
  3. Create a new template file "blackhole.php" and call the module $modules->get('Blackhole')->blackhole();
  4. Add the rule to your robot.txt
  5. Call the module from your home.php template $modules->get('Blackhole')->blackhole();

Β Bye bye bad bots!


Downloads

Β 

Screen

blackhole.gif.8360604767dfcff7430cf4d317a11b94.gif

Β 


Β Enjoy :neckbeard:

Edited by flydev
module directory link
  • Like 15

Share this post


Link to post
Share on other sites

"Sounds" useful :)Β Thanks for sharing and caring!

  • Like 2

Share this post


Link to post
Share on other sites

Nice module, thanks for sharing.

I wonder though how effective it really is reading the last two sections "caveat emptor" and "blackhole whitelist":

https://perishablepress.com/blackhole-bad-bots/#blackhole-whitelist

Quote

Whitelisting these user agents ensures that anything claiming to be a major search engine is allowed open access. The downside is that user-agent strings are easily spoofed, so a bad bot could crawl along and say, β€œHey look, I’m teh Googlebot!” and the whitelist would grant access. It is possible to verify the true identity of each bot, but doing so consumes significant resources and could overload the server. Avoiding that scenario, the Blackhole errs on the side of caution: it’s better to allow a few spoofs than to block any of the major search engines.

Β 

  • Like 3

Share this post


Link to post
Share on other sites

To get a "quote" how useful it maybe for a specific site, log all (search bots) user agents for a while.Β 

  • Like 2

Share this post


Link to post
Share on other sites

Its a shared host.

9 hours ago, flydev said:

what say <?php echo ini_get('allow_url_fopen'); ?> ?

It says true.

For the moment I have disabled this module because the loading time of the page increases significantly.

Share this post


Link to post
Share on other sites
1 hour ago, Juergen said:

For the moment I have disabled this module because the loading time of the page increases significantly.

You can disable the WHOIS lookup in the module's config.

Β 

whois.thumb.png.6bad2c335a8ab2d189d46a4db0f339b8.png

  • Like 2

Share this post


Link to post
Share on other sites

I have installed it again but now I have only included the module in the blackhole.php (not on the home or other page) only to see if it works. It works now, but the loading time of the page is approx. 21 seconds!!!!

I have added a hidden link in my site to the blackhole.php and if I click on it my IP will be stored in the DAT file - works well. In the mail that I got afterwards there was a hint about a Port problem:

Whois Lookup:

Timed-out connecting to $server (port 43).

I am on a shared host so it seems that this port is not free. The strange thing is that I have disabled the Who is Lookup in my settings of the module

Screenshot(8).png.9ef7ce303425b5bc8ff54ec9cdf2ba76.png

Best regards JΓΌrgen

  • Thanks 1

Share this post


Link to post
Share on other sites

Thanks you @Juergen .

About the port 43, its common that this port is blocked by default and - depending on the hosting provider - can be configured trough the panel provided.

59 minutes ago, Juergen said:

The strange thing is that I have disabled the Who is Lookup in my settings of the module

Will look at it this afternoon as I am deploying this module a on a production site. Stay tuned, thanks again mate.

  • Like 1

Share this post


Link to post
Share on other sites

Works like a charm now! Would be great if the hard coded url of the "contact the administrator" page could be selected out of PW pages.

Thanks for the update!!!

Edit: It would be better if you add multilanguage support to the custom message textareas :)

Β 

Β 

Edited by Juergen
  • Like 1

Share this post


Link to post
Share on other sites
3 hours ago, flydev said:

I will try to do it, I never played with modules and multilanguage

Its not so important, because only bad bots will see it and probably no humans (I hope so). By the way 2 bots from China were caught in the trap - works!!!:)

  • Like 1
  • Haha 1

Share this post


Link to post
Share on other sites

Good and funny !

Β 

13 hours ago, Juergen said:

because only bad bots will see it and probably no humans

For example, on the site I deployed the module, it is a custom dashboard with sensible informations, I had to take care of hand crafted request which could retrieve data from other users. When this behavior is detected, the user is logged out, the role login-disabled is assigned and then the user is redirected into the blackhole to be banned.

Β 

public function SecureParks() {
        if($this->input->post->park) {
            $ids = explode('-', $this->sanitizer->pageName($this->input->post->park));
            $userroles = $this->getParkRoles();
            $userhaveright = $this->searchForParkId($ids[2], $userroles);
            if ($userhaveright === null) {
                $this->user->addRole('login-disabled');
                $this->user->save();
                $this->session->logout();
                $this->session->redirect($this->pages->get('/blackhole/')->url); // :)
            }
        }
    }

Β 

  • Like 2

Share this post


Link to post
Share on other sites

Just a thought:

I think it would be nice to store the banned IPs also in a logfile, so you have them in one place with the other protocols.

Fe:

$log->save('blackhole', 'Banned IP')

You can also add fe a checkbox in the module settings to offer enabling and disabling of this feature.

What do you think? Might this be useful for others too?

  • Like 2

Share this post


Link to post
Share on other sites

I was also thinking to add a new feature from where we could monitor 302/404 HTTP code and redirect the "guest" into the blackhole.

For example, all those try :

  • /phpMyAdmin/scripts/_setup.php
  • /w00tw00t.at.ISC.SANS.DFind:)
  • /blog/wp-login.php
  • /wp-login.php
  • etc.

will be banned.

I still don't know if I code all the feature or if I should hook into Jumplinks from @Mike Rockett.

Share this post


Link to post
Share on other sites
3 minutes ago, flydev said:
  • /blog/wp-login.php
  • /wp-login.php

I also have a lot of these requests in my 404 logger protocol :(.

I think if there is module that can handle it - use it.Β  Check if the module is installed first. If not output a message that this feature is only available if Jumplinks is installed.

I dont have Jumplinks installed and I dont know how well it works, but before starting to code from the beginning I would try to use an existing solution first.

Share this post


Link to post
Share on other sites

I use Redirect gone ... in .htaccess

Redirect gone /wp-login.php

for all that stuff. (First I log 404s for a period, than I add those candidates to the .htaccess, before ProcessWires entries!!)

I think it is better to not invoke PW for this stuff, (lesser overhead on the server!), instead use apache custom error page(s).

410_wp-login_php.thumb.jpg.86905f9ab46e4529163d4bc51d3df7e3.jpg

47ms is fast! :)

Β 

PS: 410 is better than 404, as I also use this for SearchEngineRequests that try to reach URLs that do not exist since 10 years or so. Normally the SEs should flush their cache on 410 returns.

Edited by horst
  • Like 2
  • Thanks 1

Share this post


Link to post
Share on other sites

In all honesty, I think that Jumplinks is better suited to site migrations. Black holes should either be covered by a specifically-built module, or by htaccess/vhost config...

  • Like 1
  • Thanks 1

Share this post


Link to post
Share on other sites

Ok guys, I get what you mean, so what about a module with this flow ?

  1. monitor and log HTTP error code for a period
  2. if an entry / request is superior of N then
  3. backup .htaccess file (versioning it)
  4. add new entries to the .htaccess file

Β 

Does it make sense or I should let the user manage their .htaccess file manually with a FAQ or something ?

  • Like 2

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

  • Recently Browsing   0 members

    No registered users viewing this page.

  • Similar Content

    • By MoritzLost
      This is a new module that provides a simple solution to clearing all your cache layers at once, and an extensible interface to perform various cache-related actions.
      The simple motivation behind this module was that I was tired of manually clearing caches in several places after deploying a change on a live site. The basic purpose of this module is a simple Clear all caches link in the Setup menu which clears out all caches, no matter where they hide. You can customize what exactly the module does through it's configuration menu:
      Expire or delete all cache entries in the database, or selectively clear caches by namespace ($cache API) Clear the the template render cache. Clear out specific folders inside your site's cache directory (/site/assets/cache) Refresh version strings for static assets to bust client-side browser caches (this requires some setup, see the full documentation for details). This is the basic function of the module. However, you can also add different cache management action through the API and execute them through the module's interface. For this advanced usage, the module provides:
      An interface to see all available cache actions and execute them. A system log and logging output on the module page to see verify what the module is doing. A CacheControlTools class with utility functions to clear out different caches. An API to add cache actions, execute them programmatically and even modify the default action. Permission management, allowing you granular control over which user roles can execute which actions. The complete documentation can be found in the module's README.
      Beta release
      Note that I consider this a Beta release. Since the module is relatively aggressive in deleting some caches, I would advise you to install in on a test environment before using it on a live site.
      Let me know if you're getting any errors, have trouble using the module or if you have suggestions for improvement!
      In particular, can someone let me know if this module causes any problems with the ProCache module? I don't own or use it, so I can't check. As far as I can tell, ProCache uses a folder inside the cache directory to cache static pages, so my module should be able to clear the ProCache site cache as well, I'd appreciate it if someone can test that for me.
      Future plans
      If there is some interest in this, I plan to expand this to a more general cache management solution. I particular, I would like to add additional cache actions. Some ideas that came to mind:
      Warming up the template render cache for publicly accessible pages. Removing all active user sessions. Let me know if you have more suggestions!
      Links
      https://github.com/MoritzLost/ProcessCacheControl ProcessCacheControl in the Module directory

    • By joshua
      This module is (yet another) way for implementing a cookie management solution.
      Of course there are several other possibilities:
      - https://processwire.com/talk/topic/22920-klaro-cookie-consent-manager/
      - https://github.com/webmanufaktur/CookieManagementBanner
      - https://github.com/johannesdachsel/cookiemonster
      - https://www.oiljs.org/
      - ... and so on ...
      In this module you can configure which kind of cookie categories you want to manage:

      You can also enable the support for respecting the Do-Not-Track (DNT) header to don't annoy users, who already decided for all their browsing experience.
      Currently there are four possible cookie groups:
      - Necessary (always enabled)
      - Statistics
      - Marketing
      - External Media
      All groups can be renamed, so feel free to use other cookie group names. I just haven't found a way to implement a "repeater like" field as configurable module field ...
      When you want to load specific scripts ( like Google Analytics, Google Maps, ...) only after the user's content to this specific category of cookies, just use the following script syntax:
      <script type="optin" data-type="text/javascript" data-category="statistics" data-src="/path/to/your/statistic/script.js"></script> <script type="optin" data-type="text/javascript" data-category="marketing" data-src="/path/to/your/mareketing/script.js"></script> <script type="optin" data-type="text/javascript" data-category="external_media" data-src="/path/to/your/external-media/script.js"></script> <script type="optin" data-type="text/javascript" data-category="marketing">console.log("Inline scripts are also working!");</script> The type has to be "optin" to get recognized by PrivacyWire, the data-attributes are giving hints, how the script shall be loaded, if the data-category is within the cookie consents of the user. These scripts are loaded asynchronously after the user made the decision.
      If you want to give the users the possibility to change their consent, you can use the following Textformatter:
      [[privacywire-choose-cookies]] It's planned to add also other Textformatters to opt-out of specific cookie groups or delete the whole consent cookie.
      You can also add a custom link to output the banner again with a link / button with following class:
      <a href="#" class="privacywire-show-options">Show Cookie Options</a> <button class="privacywire-show-options">Show Cookie Options</button> This module is still in development, but we already use it on several production websites.
      You find it here: https://github.com/blaueQuelle/privacywire/tree/master
      Download: https://github.com/blaueQuelle/privacywire/archive/master.zip
      I would love to hear your feedback πŸ™‚
      Edit: Updated URLs to master tree of git repo
      Β 
    • By David Karich
      Admin Page Tree Multiple Sorting
      ClassName: ProcessPageListMultipleSorting
      Extend the ordinary sort of children of a template in the admin page tree with multiple properties. For each template, you can define your own rule. Write each template (template-name) in a row, followed by a colon and then the additional field names for sorting.
      Example:Β All children of the template "blog" to be sorted in descending order according to the date of creation, then descending by modification date, and then by title. Type:
      blog: -created, -modified, titleΒ  Installation
      Copy the files for this module to /site/modules/ProcessPageListMultipleSorting/ In admin: Modules > Check for new modules. Install Module "Admin Page Tree Multible Sorting". Alternative in ProcessWire 2.4+
      Login to ProcessWire backend and go to Modules Click tab "New" and enter Module Class Name: "ProcessPageListMultipleSorting" Click "Download and Install" Β  Compatibility Β  I have currently tested the module only under PW 2.6+, but think that it works on older versions too. Maybe someone can give a feedback. Β  Β  Download Β  PW-Repo:Β http://modules.processwire.com/modules/process-page-list-multiple-sorting/ GitHub:Β https://github.com/FlipZoomMedia/Processwire-ProcessPageListMultipleSorting Β  Β  I hope someone can use the module. Have fun and best regards, David
    • By dimitrios
      Hello,
      this module can publish content of a Processwire page on a Facebook page, triggered by saving the Processwire page.
      To set it up, configure the module with a Facebook app ID, secret and a Page ID. Following is additional configuration on Facebook for developers:
      Minimum Required Facebook App configuration:
      on Settings -> Basics, provide the App Domains, provide the Site URL, on Settings -> Advanced, set the API version (has been tested up to v3.3), add Product: Facebook Login, on Facebook Login -> Settings, set Client OAuth Login: Yes, set Web OAuth Login: Yes, set Enforce HTTPS: Yes, add "https://www.example.com/processwire/page/" to field Valid OAuth Redirect URIs. This module is configurable as follows:
      Templates: posts can take place only for pages with the defined templates. On/Off switch: specify a checkbox field that will not allow the post if checked. Specify a message and/or an image for the post.
      Usage
      edit the desired PW page and save; it will post right after the initial Facebook log in and permission granting. After that, an access token is kept.
      Β 
      Download
      PW module directory: http://modules.processwire.com/modules/auto-fb-post/ Github: https://github.com/kastrind/AutoFbPost Β  Note: Facebook SDK for PHP is utilized.


Γ—
Γ—
  • Create New...