PW 3.0.214 – Core updates

[ryan]

This week ProcessWire 3.0.214 is on the dev branch. Relative to 3.0.213 this version has 16 new commits which include the addition of 3 new pull requests, 6 issue fixes, a new WireNumberTools utility class, and improvements to various other classes. 

A newly added $files->size($path) method was added which returns the total size of the given file or directory. When given a directory, it returns the size of all files in that directory, recursively. Improvements were also made to ProcessWire's log classes (WireLog and FileLog) with new methods for deleting or pruning all log files at once. This version also fixes an issue with the front-end page editor (PageFrontEdit) when used with InputfieldTinyMCE. For more details on these updates and more see the dev branch commit log.

Something else I've been working on this weekend is a vulnerability scanner blocker and throttler. I don't know if this is an issue for every site, or if it's because this is an open source project site, but we seem to get a lot of vulnerability scanner bots hitting the site. Sometimes they hit the site pretty hard (with hundreds of thousands of requests) and our AWS cluster servers and databases must scale to meet the demand, using more resources, and thus increasing cost. This is annoying, having to scale for a hyperactive vulnerability scanner rather than real traffic. And it always seems to happen in the middle of the night, when I'm not nearby to manually block it. So I'm working on a module that detects vulnerability scanner traffic patterns and then blocks or throttles requests from their IPs automatically. Once I've got it functioning smoothly here, I'll also plan to add it to ProDevTools board download thread in case it's useful to anyone else. Thanks for reading and have a great weekend! 
 

[szabesz]

34 minutes ago, ryan said:

I'll also plan to add it to ProDevTools board download thread in case it's useful to anyone else.

Sounds great for sure! Thank you in advance. 

[Pixrael]

@ryan maybe this could help: https://www.phpclasses.org/package/12821-PHP-Detect-and-block-spam-bots-from-accessing-sites.html

[wbmnfktr]

On 3/17/2023 at 8:14 PM, ryan said:

Something else I've been working on this weekend is a vulnerability scanner blocker and throttler. I don't know if this is an issue for every site, or if it's because this is an open source project site, but we seem to get a lot of vulnerability scanner bots hitting the site. Sometimes they hit the site pretty hard (with hundreds of thousands of requests) and our AWS cluster servers and databases must scale to meet the demand, using more resources, and thus increasing cost. This is annoying, having to scale for a hyperactive vulnerability scanner rather than real traffic. And it always seems to happen in the middle of the night, when I'm not nearby to manually block it. So I'm working on a module that detects vulnerability scanner traffic patterns and then blocks or throttles requests from their IPs automatically.

I face this issue on about 3/4 of all of my projects (client/personal/side) for several years now. Depending on the niche and how well known and established a project is in the search results and elsewhere... the amount of bot/"vulnerability scanner" traffic is impressive. From low 1000s/month to tens-of-thousands/day.

Real estate, finance, health, insurance... and some other niches (upcoming topics and trends) are somehow prone to those "visits".

I looked for solutions in the forum in the past, got real nice options and solutions, tried them, yet most of the time I went with Cloudflare as CDN and they blocked out most of those visits and even blocked out whole countries therefore. Sure it's another $20/month but for bigger projects necessary.

Thought of mine: Look up headers and all possible markers for Scrapebox, Xenu, GSA in addition to your lookups... as those are some of the most used engines/bots/tools to generate bot traffic "regular" users play with (as they are super affordable) to find sources for backlinks, guestbook links, spam comments and such. There are way more out there but they don't even make a dent in your logs/statistics.

Not to forget tools like ahrefs, Sistrix, Ubersuggest... and way way more. While those last mentioned tools are just for auditing, you might even want to use, all others should be... blocked in some kind or another. Or maybe there should be a list of filters looking for matches in some kind.

There is yet another one: Screamingfrog (which is acutally a real white hat tool for auditing - I use it almost every day on a lot of domains) yet it can cause a lot of traffic due to audits. I'm not sure if there is a marker for ScreamingFrog to whitelist it on a project/domain but it really just scans the site. For a good reason.

In case you want to test ScreamingFrog against your filters... you will find it and can use it for free in respectable way. It's an awesome tool we should be able to whitelist!

To anybody that uses that upcoming module in the future, PLEASE let me know your experience.
Could be cheaper for my wallet by far!

[netcarver]

@wbmnfktr Have you given Jeffrey Star's 7g firewall a try? Very simple install straight into your .htaccess file.

[wbmnfktr]

1 hour ago, netcarver said:

Have you given Jeffrey Star's 7g firewall a try? Very simple install straight into your .htaccess file.

I did indeed. And was quite happy in some instances.
To be honest... I don't know if it was the 7g or another version.
Somehow the 7g sounds unfamiliar at the moment. I might say it was 5g or so. Not sure at the moment.
But for sure it was from perishablepress.com.

It was recommened quite a few times. Yet it wasn't always working back in the days in all instances and therefore I went on and looked for more solutions and ended somewhere around Cloudflare and some other CDNs.

Nowadays I almost always (if possible) sign up and stay there at Cloudflare.
I really love what you get there.

 

[ryan]

On this side, I don't really find the spambots or seo bots to be much of an issue, so I mostly ignore them unless they get too aggressive. It's instead the vulnerability scanners that tend to be the issue here. They are fine when they are throttled. But when they are unthrottled (as is usually the case), they eat up a lot of resources.

Here's just one basic example: a vulnerability scanner might send through thousands (or tens of thousands) of URL variations looking for SQL files that it can grab, with dozens of different names each, like db.sql, database.sql, backup.sql, [domain].sql, database-[domain].sql, db-[domain.sql], [domain]-db.sql, and so on and on and on. Then add all the extension variations, .sql, .sql.gz, .sql.tar, .sql.tar.gz, and then add every URL with a trailing slash in the site as the prefix path for every check. So just a scan for SQL files in-the-open might account for tens of thousands of requests. And it'll try to do them all in a very short period of time, making the server like ours scale to meet the demand. Yet this is just an example of one vulnerability check out of thousands that it'll do. Once a vulnerability scanner gets started, it'll run for potentially days. But I usually block them well before that. Once I get an email from AWS about things scaling, I watch the logs pretty closely and then start blocking IPs. But the goal is to have the module just block them automatically. 

What the module does is that it allows you to define suspicious patterns in GET or POST requests, or user agent strings (and it comes with several patterns to start). For example, you might have patterns to match things like wp-login.php, those SQL request variations mentioned above, requests for .py, .cfm, .rb, .exe files, or others that you don't use on the server, requests containing SQL commands in the query string... these are just obvious examples.  Then it lets you define a number of strikes till the IP is out. So for every pattern match, the IP gets a strike. So if I set it to "3 strikes and you are out" then once it gets 3 pattern matches, the IP is blocked for a period of time, also defined with the module. If additional strikes occur while an IP is blocked, the block time gets reset so it starts over, ensuring it's always blocked that set amount of time from the last strike. 

[bedak]

After upgrading from 3.0.213 to 3.0.214 (using PHP 8.0.28) I get the error message "ProcessLogger: Unknown log: <protocol-name>" when clicking on a protocol-name. And the size-column and the changed-column showing the same values for all listed protocols.

If I replace the method `getLogs()`  in /wire/core/WireLog.php by `getLogs()`  of the version of 3.0.213 everything is ok again...
Can anyone confirm this errpneous behavior. Thanx! 

[matjazp]

https://github.com/processwire/processwire-issues/issues/1707

[gebeer]

13 hours ago, bedak said:

After upgrading from 3.0.213 to 3.0.214 (using PHP 8.0.28) I get the error message "ProcessLogger: Unknown log: <protocol-name>" when clicking on a protocol-name. And the size-column and the changed-column showing the same values for all listed protocols.

If I replace the method `getLogs()`  in /wire/core/WireLog.php by `getLogs()`  of the version of 3.0.213 everything is ok again...
Can anyone confirm this errpneous behavior. Thanx! 

Same behaviour here wirh PW 3.0.214 on PHP8.1

[gornycreative]

Yes I just noticed it also - the columns are correct and Tracy picks up the entries, but when I click a log page name I get the error message. Can confirm all files exist.

PW 3.0.214 PHP 8.1

[bernhard]

@gornycreative the issue is already fixed on the latest dev:

There has just not been a version bump since then.