filesystem 32k limit reached

[Frank Vèssia]

Hi guys,

i reached the filesystem limit 32000 for subfolders on my server and the server support told me this limit cannot be increased. Right now i suspended all uploads by users, any idea how to solve this? Thanks

[WillyC]

http://modules.processwire.com/modules/page-clean-empty-dirs/

[Frank Vèssia]

thanks WillyC, I installed and for "now" the problem is solved, but soon I will have the same issue, there is no way to use multiple directories for "/files/" folder?

[adrian]

You should have a good read of this thread:

https://processwire.com/talk/topic/992-problem-with-assetsfiles-folder/

[Frank Vèssia]

I've read the post but i didn't come up with any solution...

[MindFull]

thanks WillyC, I installed and for "now" the problem is solved, but soon I will have the same issue, there is no way to use multiple directories for "/files/" folder?

From this, I'm gathering that most of your directories in /files actually contain either images or files.  And though you just cleared out the empties, you're going to be hitting your max again soon. You need a way to circumvent Apache's file limit within any directory, like have a /file2 directory where new pages can store things. Correct?

[Wanze]

I've read the post but i didn't come up with any solution...

Have you checked my solution? It works. I have a site with > 200.000 pages each containing images and files.

One big disadvantage is that I needed to hack two core files. I'll open a GitHub issue with a proposal how this could

be implemented with hooks.

If you take a solution like mine, you'd need to write a script which creates the subfolders and moves the files for your existing site.

So for example:

files/1603/image.jpg

must be translated to:
files/16/03/image.jpg

[Frank Vèssia]

@wanze I would like to avoid hack core files but if there are no other solutions I'll do it, thanks. Maybe if you can send me details of what files did you modifiy and where exactly I would appreciate that.

[Wanze]

Hi Sevarf2,

Here are my notes to the two files that were patched.

The PagefilesManager::url() method is hookable so theoretically there is no patch needed. But since I needed to hack that file anyway,

I did not use a hook.

Just ask here if you have any questions. I think it would be nice if the developer could define a specific strategy for saving the files (with Hooks). I will open an Issue on GitHub and provide a proposal to ryan.

Edit: I do not use pagefileSecure feature, and actually have no idea if it would work with those "hacks"

Patches:

/wire/core/functions.php

function wireMkdir($path, $recursive = false) {
     //if(!is_dir($path)) return false;
     $chmodDir = wire('config')->chmodDir;
     if ($recursive && $chmodDir) {
          return @mkdir($path, octdec($chmodDir), true);
     }
     if(!@mkdir($path)) return false;
     if($chmodDir) chmod($path, octdec($chmodDir));
     return true;
} 

/wire/core/PagefilesManager.php

     static public function makeCustomPath(Page $page) {
         
          // Make subfolders per ID - always two digits (or one per folder)
          // Example: ID=1780, path = /site/assets/files/17/80/
          //          ID=19814, path = /site/assets/files/19/81/4/
          //          ID=205478, path = /site/assets/files/20/54/78/
          $tmpPath = '';
          $digits = str_split("{$page->id}");
          foreach ($digits as $k => $digit) {
               $tmpPath .= $digit;
               if (($k+1) % 2 == 0) $tmpPath .= '/';
          }
          if (substr($tmpPath, -1) != '/') $tmpPath .= '/';
          return $tmpPath;
              
     }

===

     protected function _createPath($path) {
          if(is_dir($path)) return true;
          return wireMkdir($path, true);
     }  

===

     static public function _path(Page $page) {
                   
          $config = wire('config');
          $path = $config->paths->files;
         
          //$publicPath = $path . ((int) $page->id) . '/';     
          $tmpPath = self::makeCustomPath($page);
          $publicPath = $path . $tmpPath;
                   
          $securePrefix = $config->pagefileSecurePathPrefix;
          if(!strlen($securePrefix)) $securePrefix = self::defaultSecurePathPrefix;

          // securePath has the page ID preceded with a prefix which PW's htaccess blocks http requests to
          //$securePath = $path . $securePrefix . ((int) $page->id) . '/';
          $securePath = $path . $securePrefix . $tmpPath;
         
          // we track this just in case the prefix was newly added to config.php, this prevents
          // losing track of the original directories
          //$securePath2 = $path . self::defaultSecurePathPrefix . ((int) $page->id) . '/';
          $securePath2 = $path . self::defaultSecurePathPrefix . $tmpPath; 

          [.. continue method...]

     }

===

public function ___url() {
  return $this->config->urls->files . self::makeCustomPath($this->page); 
}

[Wanze]

FYI: https://github.com/ryancramerdesign/ProcessWire/issues/432

[Frank Vèssia]

thanks @Wanze, I'm going to implement this!

[dfunk006]

IMO, an alternative to the filesystem where files can be saved externally (like Amazon S3 or something) would really be helpful in a scenario like this.

https://processwire.com/talk/topic/6046-alternate-filesystem/

[apeisa]

dfunk, I don't think you are going to help getting S3 support by posting about it on every possible topic. One or two mentions should be enough.

[dfunk006]

dfunk, I don't think you are going to help getting S3 support by posting about it on every possible topic. One or two mentions should be enough.

Oops! Sorry about that. Didn't realise I was crossing the line there. I shall be more cognisant about that from now! 

[apeisa]

Well, I don't think you crossed any official lines here. Just felt awkward reading same story from almost every topic.

[diogo]

(let's make it even more awkward for apeisa )

Sevarf2, a very similar question was asked here https://processwire.com/talk/topic/5961-filesystem-limit-32000-reached/

[netcarver]

@Wanze

I quite like the idea of having a hookable method that would allow consistent path manipulation. If it could rewrite the entire path for a resource then perhaps it could even be used to support things like image delivery from a dedicated sub-domain etc.

Edit: Looks like Teppo & Ryan had a similar idea over here.

Edited April 4, 2014 by netcarver

[Frank Vèssia]

@diogo, you linked this same post...

[diogo]

Sorry Sevarf, I was playing with Apeisa, didn't intend to confuse you. Ignore me

[Frank Vèssia]

@wanze, i just implemented your hack and it works pretty good, thanks Did you also created some script for moving old files to the new directories? 

[Wanze]

I don't have a migration script, because I've implemented this before starting the project

But should not be hard, I would do a command line script, bootstrap ProcessWire and then code something like this:

<?php
// Bootstrap Pw

// Make function wireMove, function mostly copied from wireCopy function
function wireMove($src, $dst) {

	if(substr($src, -1) != '/') $src .= '/';
	if(substr($dst, -1) != '/') $dst .= '/';

	$dir = opendir($src);
	if(!$dir) return false; 
	if(!wireMkdir($dst, true)) return false;

	while(false !== ($file = readdir($dir))) {
		if($file == '.' || $file == '..') continue;
	        rename($src . $file, $dst . $file);
		$chmodFile = wire('config')->chmodFile;
		if($chmodFile) @chmod($dst . $file, octdec($chmodFile));
	}

	closedir($dir);
	return true;
}

// Start migration - Loop your pages
// You could split by templates and use limit/start to get this done in multiple iterations. 
// Limit depends on your server memory 

$pages = wire('pages')->find('template=shibby,limit=2000');
foreach ($pages as $page) {
  $old_path = wire('config')->paths->files . $page->id . '/';
  $new_path = PagefilesManager::makeCustomPath($page);
  if (wireMove($old_path, $new_path)) {
    @unlink($old_path);
  }
}

Please only use on a copy of your installation, there will for sure be bugs as I've written it in the browser.

You will need to make some modifications, but maybe it helps you as a starting point.

Cheers

[Frank Vèssia]

Great starting point, thanks.

[Frank Vèssia]

@wanze I just made a correction here:

$new_path = wire('config')->paths->files . PagefilesManager::makeCustomPath($page);

 

makeCustomPath returns relative directory and the rename didn't work. The rest works pretty good. Thanks again.

[apeisa]

Btw, there seems to be no folder limit on ext4 filesystem. We are running pw installations with well over 100000 folders already.

[Frank Vèssia]

Hi guys, I'm updating this discussion. After a while using @wanze solution to avoid the 32k limit of subfolders I'm thinking of upgrade PW to the latest version (from current 2.3) because since middle 2014 I didn't make any update because of this core custom modification.
Now PW also solves this issue with the config option $config->pagefileExtendedPaths.
Is there any chance the pw solution is the same of wanze so updating pw don't cause any issue to my current folders structure?