Limit folder creation / re order folders

[blacksrv]

Hello to all, 

I'm facing some issues with folders being created by PW, I have about 1 million pages, and each time a new page is created it takes about 10s to complete the save.

is possible to change the current behavior to create / re order the folders created to be inside a subfolder by date / month / day ?

Any other Ideas?

 

Thank you!

[louisstephens]

Just curious, what do you have that is creating folders, image/file fields?

[blacksrv]

@louisstephens, Users with a single image field, and it does not matter if it's empty, the folder is created anyway.

[BillH]

Trying to manage folders yourself would be seriously hard work and difficult, as discussed in this post:

If a high proportion of your pages don't have images, and the total number of images is well under half the number of pages, you could take a one-image-per-page approach, with each image held in sub-page.

A couple of modules that might help with this are:

https://modules.processwire.com/modules/repeater-images/

https://modules.processwire.com/modules/process-visual-page-selector/

 

[bernhard]

Some background info: https://github.com/processwire/processwire-issues/issues/579

Quote

ryancramerdesign commented on 30 Apr 2018

If you've got a files or images field on the page, then it's going to create a directory for it, whether you've uploaded files to it or not. Likewise, if anything has asked for the path to where the page can store files via PagefilesManager, it'll make sure there's a directory where files can be stored. Though if you don't have any files field on a page, chances are there won't be a directory for it. What was changed (years ago) is that PW used to maintain directories even for pages that had no files fields, which it no longer does, unless something makes an API call asking for a location to store files for the page.

Quote

ryancramerdesign commented on 30 Apr 2018

If there's any kind of files field on a page, then there needs to be an associated directory present as well. If that weren't the case, any code that deals with files (whether core or 3rd party, modules, etc.) would have to ask for the path for files, then check if it exists, and then create it manually if it doesn't. As it is now, when code asks for the path for files, that code can then assume it can write to that path. If that weren't the case, plenty of things would break. To avoid having directories created, avoid putting files fields on templates that don't need them. No directory will be created if the page doesn't have any files fields, unless some other code is asking for files access. It is okay to delete empty directories if you want to, but note that PW will re-create them so long as there is a files field present. So you'd want to remove files fields on templates where you don't need them before removing empty directories.

Are you sure that the 10s delay is coming from the huge amount of folders?

[teppo]

Sounds like $config->pagefileExtendedPaths would make sense here. Note, though, that I'm not at all certain what will happen if you enable this on an existing site with million pages — if you have the chance to create a copy of the site and test there first, that'd be a good idea. Otherwise be sure to take backups of both your site's files and the database before enabling this feature.

[blacksrv]

@BillH, I hadn't thought of that alternative, however, that would leave all the files in 1 folder, I don't know if it would work.

@bernhard, not 100% sure, I'm currently emptying the empty folders, however it's not a permanent fix. Any idea how I could prove if it is indeed the folders? I/O monitor does not show any peak loads.

@teppo I didn't know about this, I'll set up a local copy and do the test, thanks.

[bernhard]

4 minutes ago, blacksrv said:

@bernhard, not 100% sure, I'm currently emptying the empty folders, however it's not a permanent fix. Any idea how I could prove if it is indeed the folders? I/O monitor does not show any peak loads.

Not sure, but you could create a test site using the blank or default site profile, then create 10k, 100k, ... pages and see if that has any impact on the page save timings.