Problem with assets/files folder

[ryan]

I just pushed an update to the source that should prevent it from creating directories for pages that don't need them.

Next up: going to make a module that removes empty directories, for those that want it.

[Pete]

If I could like your post multiple times I would

[ryan]

Thanks Pete. Here's a link to the new module that removes empty directories from /site/assets/files/:

http://modules.processwire.com/modules/page-clean-empty-dirs/

[apeisa]

Great news! Does it create folder when first image is added or when page with file field is created?

[ryan]

It gets created the first time a file or image field is accessed on the page (even if it's an empty file/image field). If the page has no file/image fields, then the directory won't get created.

If you create a page with the API and don't populate the file/image field, the directory won't get created at that time. Though if you create a page in the PW admin, you'll likely see its directory created when the page is created (since ProcessPageEdit iterates all the page's fields when it saves). Ultimately you'll still have empty directories, but they will only be for pages that actually have the file-based fields to need such directories.

[Adam Kiss]

So it still generates empty directory for each page, that has either image or file field? why is that? can't the folders be created/deleted each Page::save based on total number of files for page?

[Pete]

I think I agree that it would be better to do it the way Adam suggests, but I can see the problem in that anyone creating cusom modules that handle files/images will have to know to call a function to create the directory in the first place.

The more I actually think about my sites though, every page with an images field has an image anyway. There are scenarios where you wouldn't though such as blogs (you could have thousands and thousands of pages there and maybe only half of them with an image) and other uses like that.

For a CMS for general sites the field would usually have at least one file but it's important to note that some uses for PW could have need of the field whilst not using it on as high a percentage of them so that's where creating a file only when it's required makes more sense.

[Adam Kiss]

For a CMS for general sites the field would usually have at least one file but it's important to note that some uses for PW could have need of the field whilst not using it on as high a percentage of them so that's where creating a file only when it's required makes more sense.

Yeah, and there is still the thing that PW calls itself CMF (framework) - and if you need to have optional file in huge volume of data, you'd need to hack around via having separate 'file' pages and connecting them via 'page' field, for instance... So if it's possible to create/delete folders dynamically based on files count, that would be ideal.

(Note: I did this. I had an application [albeit microapplication, with only 1k pages], which had exactly this setting - optional file fields, and only about 3% of pages had any file)

[ryan]

So it still generates empty directory for each page, that has either image or file field? why is that?

Because when PW presents you with an upload field, it wants to know where the file is going to go.

There's more to it than that too. Every page has an optional delegate object called a PagefilesManager. That class manages all disk-based files for a Page and it doesn't know anything about who's using it (whether a Fieldtype, Inputfield, API access, etc.). This is a good thing. And it's only called on demand. But when it's called, it makes sure there is a place for that page on the drive. Ultimately, it's not just having an file/image field that causes the directory to be created, it's the actual calling upon the file/image field, which ensures they have a home. A prerequisite for a file-based asset is a place to put it. This limits the dependencies and complexity between these systems and it's the way I think it should be.

can't the folders be created/deleted each Page::save based on total number of files for page?

They could be, but that's trading processor time for disk space. Disk space is cheaper.

I also think that if you are putting file/image fields on huge quantities of pages that don't need them, you are adding unnecessary overhead to runtime that's a far bigger affect to performance than some empty directories on your server.

If you've got tens of thousands of pages with file fields on them that you edit in the admin, and you don't want them to have directories, then run the new empty dirs module once in awhile.

[er314]

So, if I understand correctly, there is no more the filesystem-related 32k-64k pages limit when pages don't actually embed an image or a file,

however this limit still applies as soon as one reach 32k-64k pages containing an image or a file, correct ?

[ryan]

That's correct. If your file system has a limit like that, and you expect to have that large of a quantity of pages (which I don't think we've seen anyone do yet) then you'd probably want to only put file/image fields on the templates where you need them. If you have 60k+ pages, all with file/image fields, chances are you aren't editing them often, so you could also use this module to remove the directories.

[Wanze]

Interesting discussion here...

The ext4 64k limit is becoming an issue for me with a large project.

I have a product template with an product image and additional files.

Currently, there are > 92'000 products in the old database.

Pw will be used to manage the products in the backend and also as an online shop for clients.

There will be 10 up to 100 new products per day, so the amount of data is growing constantly

I was playing with the PageFilesManager class and modified the ::_path() method:

//$publicPath = $path . ((int) $page->id) . '/';
// Make subfolders per ID
$folders = str_split("{$page->id}");
$publicPath = $path . implode('/', $folders) . '/';

This returns for page with ID 1730 a path like "/var/www/pw/site/assets/files/1/7/3/0/"

However, when I try to publish a new product page, I'm getting the error message

Session: destinationPath is empty or does not exist (image)

I didn't found where Pw should create this directory, I was first looking at the ProcessPageAdd and also in the Pages::save() method,

but no luck yet. The error is throwed by the InputfieldFile::processInput(), which checks if this path exists.

First: Would it be possible to create a solution like this?

Second: Does anyone know when/where Pw does create the folder in /site/assets/files/ ? 

Cheers

[adrian]

Hey Wanze,

Looks like it is here:

https://github.com/ryancramerdesign/ProcessWire/blob/a8024bb49785370aa2e3867bd683094663cfeabf/wire/core/PagefilesManager.php#L135

with $this->path() coming from:

https://github.com/ryancramerdesign/ProcessWire/blob/a8024bb49785370aa2e3867bd683094663cfeabf/wire/core/PagefilesManager.php#L166

The createPath function is calling this:

https://github.com/ryancramerdesign/ProcessWire/blob/a8024bb49785370aa2e3867bd683094663cfeabf/wire/core/Functions.php#L184

I guess it might need the mkdir recursive switch set to true in the wireMkdir function.

Is that what you are looking for, or am I missing your point? I think maybe I am

[ryan]

Wanze, Adrian is right with the _createPath() function he linked to. It maybe that you could solve it entirely simply by making it recursive, as he suggested. Actually, it might be good to just have the wireMkdir() function support creating a path rather than just a single directory... lets update that one in the core instead. The wireRmdir() already supports a $recursive argument to it, so wireMkdir() should too. 

[Wanze]

@adrian

Thanks for your help! You were right about the creatPath method, I was able to solve my problem

It maybe that you could solve it entirely simply by making it recursive, as he suggested. Actually, it might be good to just have the wireMkdir() function support creating a path rather than just a single directory... lets update that one in the core instead. The wireRmdir() already supports a $recursive argument to it, so wireMkdir() should too. 

 

Thanks ryan.

I solved it by making wireMkdir recursive with the additional argument. In the PagefilesManager class I did the following updates:

// Change _createPath() to support recursive directories
protected function _createPath($path) {
  if(is_dir($path)) return true; 
  return wireMkdir($path, true); 
}

// Add new method
static public function makeCustomPath(Page $page) {
  // Examples: ID=1780, path = /site/assets/files/17/80/
  //           ID=19814, path = /site/assets/files/19/81/4/
  //           ID=205478, path = /site/assets/files/20/54/78/
  $tmpPath = '';
  $digits = str_split("{$page->id}");
  foreach ($digits as $k => $digit) {
    $tmpPath .= $digit;
    if (($k+1) % 2 == 0) $tmpPath .= '/';
  }
  if (substr($tmpPath, -1) != '/') $tmpPath .= '/';
  return $tmpPath;
}

// Custom path needs to be set in the _path() and url() methods

static public function _path(Page $page) {
  //...
  //$publicPath = $path . ((int) $page->id) . '/'; 	
  $tmpPath = self::makeCustomPath($page);
  $publicPath = $path . $tmpPath;
  //...
}

public function url() {
  //return $this->config->urls->files . $this->page->id . '/';
  return $this->config->urls->files . self::makeCustomPath($this->page);
}

Should be able to test this with many data soon, I'll report back here for those interested.

[Martijn Geerts]

I like this approach. Thank you for sharing !

[Raymond Geerts]

Is this something that is build in the core at this moment or in the future? Or is there a module that handles this? I vaguely recall somebody mentioning this, but since i can not find the discussion anymore i doubt if i'm right about that.

[kongondo]

Is this something that is build in the core at this moment or in the future? Or is there a module that handles this? I vaguely recall somebody mentioning this, but since i can not find the discussion anymore i doubt if i'm right about that.

Yes; this is now (or soon will be) resolved in the core

https://github.com/ryancramerdesign/ProcessWire/issues/432

Not sure if it has been committed yet...

[Raymond Geerts]

Thanks kongondo for the link to github. Now i remember where i read it and why i couldnt find it on the forum

Great news though that this will be in the core soon

[RyanJ]

Just reached approximately 550,000 pages with PW , which all have an image field, but only about 10% are used. I am still in experimental state with this project, so many things will most likely change. I am also learning beyond the basics as I go, what a nice small project to learn with right?

My first signs of trouble was page cache. I realized it fills the hard drive quickly, so I had to disable that.

From what I understand processwire no longer creates the directory for image/file fields on page creation automatically? However, if the page is visited it does. Understandably Ryan made clear that this is necessary for any file or image field to have a place to live.  So I am clarifying that at this point my only option is to run the Clean Empty Page File Dir Module every so often. I am assuming I can do this with the Lazy Cron module? If so, I would like it to run every 30 minutes on page view. 

What is the best way to get the two modules to interact? Could I add to the PageCleanEmptyDirs module add something like the below in the public init function?

$this->addHook('LazyCron::every30Minutes', $this, 'PageCleanEmptyDirs');

or is there another approach I should take? Any suggestions is appreciated.

[Martijn Geerts]

Take a look at $config->pagefileExtendedPaths, if you want to enable this, put that one to your /site/config.php and set it to true.

[RyanJ]

Hi Martigin,

Thanks for the link with the comments. I was reading Wanze and Ryan's conversation, but did not completely understand it. That cleared things up a bit. My question though is that I am still left with the empty directory files for the images fields that do not contain an image correct?

I am inexperienced in OOP, but am going to try and play around more with the PageCleanEmptyDirs module and see if I can get it to run with the lazy cron by adding the snippet above, but so far no luck.

[suntrop]

If you create a page with the API and don't populate the file/image field, the directory won't get created at that time. 

I just created a page and its folder is created anyway. Is that feature gone? Using PW 2.5.17

I'm planing a site where users can create content (saved as PW pages). If it doesn't flop, there is a big chance I need more folders than my managed server is able to handle (they say it is good to keep files/folders under 20K per directory).

Any news on $config->pagefileExtendedPaths? Is it save to use it on a live site? I am worried to rebuild my app after 12 months, because there are too many users