Jump to content
Sevarf2

filesystem 32k limit reached

Recommended Posts

Hi guys,

i reached the filesystem limit 32000 for subfolders on my server and the server support told me this limit cannot be increased. Right now i suspended all uploads by users, any idea how to solve this? Thanks

Share this post


Link to post
Share on other sites

thanks WillyC, I installed and for "now" the problem is solved, but soon I will have the same issue, there is no way to use multiple directories for "/files/" folder?

Share this post


Link to post
Share on other sites

I've read the post but i didn't come up with any solution...

Share this post


Link to post
Share on other sites

thanks WillyC, I installed and for "now" the problem is solved, but soon I will have the same issue, there is no way to use multiple directories for "/files/" folder?

From this, I'm gathering that most of your directories in /files actually contain either images or files.  And though you just cleared out the empties, you're going to be hitting your max again soon. You need a way to circumvent Apache's file limit within any directory, like have a /file2 directory where new pages can store things. Correct?

Share this post


Link to post
Share on other sites

I've read the post but i didn't come up with any solution...

Have you checked my solution? It works. I have a site with > 200.000 pages each containing images and files.

One big disadvantage is that I needed to hack two core files. I'll open a GitHub issue with a proposal how this could

be implemented with hooks.

If you take a solution like mine, you'd need to write a script which creates the subfolders and moves the files for your existing site.

So for example:

files/1603/image.jpg

must be translated to:
files/16/03/image.jpg
  • Like 3

Share this post


Link to post
Share on other sites

@wanze I would like to avoid hack core files but if there are no other solutions I'll do it, thanks. Maybe if you can send me details of what files did you modifiy and where exactly I would appreciate that.

Share this post


Link to post
Share on other sites

Hi Sevarf2,

Here are my notes to the two files that were patched.

The PagefilesManager::url() method is hookable so theoretically there is no patch needed. But since I needed to hack that file anyway,

I did not use a hook.

Just ask here if you have any questions. I think it would be nice if the developer could define a specific strategy for saving the files (with Hooks). I will open an Issue on GitHub and provide a proposal to ryan.

Edit: I do not use pagefileSecure feature, and actually have no idea if it would work with those "hacks" ;)

Patches:

/wire/core/functions.php

function wireMkdir($path, $recursive = false) {
     //if(!is_dir($path)) return false;
     $chmodDir = wire('config')->chmodDir;
     if ($recursive && $chmodDir) {
          return @mkdir($path, octdec($chmodDir), true);
     }
     if(!@mkdir($path)) return false;
     if($chmodDir) chmod($path, octdec($chmodDir));
     return true;


/wire/core/PagefilesManager.php

     static public function makeCustomPath(Page $page) {
         
          // Make subfolders per ID - always two digits (or one per folder)
          // Example: ID=1780, path = /site/assets/files/17/80/
          //          ID=19814, path = /site/assets/files/19/81/4/
          //          ID=205478, path = /site/assets/files/20/54/78/
          $tmpPath = '';
          $digits = str_split("{$page->id}");
          foreach ($digits as $k => $digit) {
               $tmpPath .= $digit;
               if (($k+1) % 2 == 0) $tmpPath .= '/';
          }
          if (substr($tmpPath, -1) != '/') $tmpPath .= '/';
          return $tmpPath;
              
     }

===

     protected function _createPath($path) {
          if(is_dir($path)) return true;
          return wireMkdir($path, true);
     }  

===

     static public function _path(Page $page) {
                   
          $config = wire('config');
          $path = $config->paths->files;
         
          //$publicPath = $path . ((int) $page->id) . '/';     
          $tmpPath = self::makeCustomPath($page);
          $publicPath = $path . $tmpPath;
                   
          $securePrefix = $config->pagefileSecurePathPrefix;
          if(!strlen($securePrefix)) $securePrefix = self::defaultSecurePathPrefix;

          // securePath has the page ID preceded with a prefix which PW's htaccess blocks http requests to
          //$securePath = $path . $securePrefix . ((int) $page->id) . '/';
          $securePath = $path . $securePrefix . $tmpPath;
         
          // we track this just in case the prefix was newly added to config.php, this prevents
          // losing track of the original directories
          //$securePath2 = $path . self::defaultSecurePathPrefix . ((int) $page->id) . '/';
          $securePath2 = $path . self::defaultSecurePathPrefix . $tmpPath; 

          [.. continue method...]

     }

===

public function ___url() {
  return $this->config->urls->files . self::makeCustomPath($this->page); 
}

  • Like 3

Share this post


Link to post
Share on other sites

thanks @Wanze, I'm going to implement this!

  • Like 1

Share this post


Link to post
Share on other sites

dfunk, I don't think you are going to help getting S3 support by posting about it on every possible topic. One or two mentions should be enough.

  • Like 1

Share this post


Link to post
Share on other sites

dfunk, I don't think you are going to help getting S3 support by posting about it on every possible topic. One or two mentions should be enough.

Oops! Sorry about that. Didn't realise I was crossing the line there. I shall be more cognisant about that from now!  :)

Share this post


Link to post
Share on other sites

Well, I don't think you crossed any official lines here. Just felt awkward reading same story from almost every topic.

Share this post


Link to post
Share on other sites

@Wanze

I quite like the idea of having a hookable method that would allow consistent path manipulation. If it could rewrite the entire path for a resource then perhaps it could even be used to support things like image delivery from a dedicated sub-domain etc.

Edit: Looks like Teppo & Ryan had a similar idea over here.

Edited by netcarver
  • Like 2

Share this post


Link to post
Share on other sites

Sorry Sevarf, I was playing with Apeisa, didn't intend to confuse you. Ignore me :P

Share this post


Link to post
Share on other sites

@wanze, i just implemented your hack and it works pretty good, thanks :) Did you also created some script for moving old files to the new directories?  :-[

Share this post


Link to post
Share on other sites

I don't have a migration script, because I've implemented this before starting the project :)

But should not be hard, I would do a command line script, bootstrap ProcessWire and then code something like this:

<?php
// Bootstrap Pw

// Make function wireMove, function mostly copied from wireCopy function
function wireMove($src, $dst) {

	if(substr($src, -1) != '/') $src .= '/';
	if(substr($dst, -1) != '/') $dst .= '/';

	$dir = opendir($src);
	if(!$dir) return false; 
	if(!wireMkdir($dst, true)) return false;

	while(false !== ($file = readdir($dir))) {
		if($file == '.' || $file == '..') continue;
	        rename($src . $file, $dst . $file);
		$chmodFile = wire('config')->chmodFile;
		if($chmodFile) @chmod($dst . $file, octdec($chmodFile));
	}

	closedir($dir);
	return true;
}

// Start migration - Loop your pages
// You could split by templates and use limit/start to get this done in multiple iterations. 
// Limit depends on your server memory 

$pages = wire('pages')->find('template=shibby,limit=2000');
foreach ($pages as $page) {
  $old_path = wire('config')->paths->files . $page->id . '/';
  $new_path = PagefilesManager::makeCustomPath($page);
  if (wireMove($old_path, $new_path)) {
    @unlink($old_path);
  }
}

Please only use on a copy of your installation, there will for sure be bugs as I've written it in the browser.

You will need to make some modifications, but maybe it helps you as a starting point.

Cheers

  • Like 5

Share this post


Link to post
Share on other sites

@wanze I just made a correction here:

$new_path = wire('config')->paths->files . PagefilesManager::makeCustomPath($page);
 

makeCustomPath returns relative directory and the rename didn't work. The rest works pretty good. Thanks again.

Share this post


Link to post
Share on other sites

Btw, there seems to be no folder limit on ext4 filesystem. We are running pw installations with well over 100000 folders already.

  • Like 3

Share this post


Link to post
Share on other sites

Hi guys, I'm updating this discussion. After a while using @wanze solution to avoid the 32k limit of subfolders I'm thinking of upgrade PW to the latest version (from current 2.3) because since middle 2014 I didn't make any update because of this core custom modification.
Now PW also solves this issue with the config option $config->pagefileExtendedPaths.
Is there any chance the pw solution is the same of wanze so updating pw don't cause any issue to my current folders structure?

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

  • Recently Browsing   0 members

    No registered users viewing this page.

×
×
  • Create New...