Jump to content

filesystem 32k limit reached


Sevarf2
 Share

Recommended Posts

Hi guys,

i reached the filesystem limit 32000 for subfolders on my server and the server support told me this limit cannot be increased. Right now i suspended all uploads by users, any idea how to solve this? Thanks

Link to comment
Share on other sites

thanks WillyC, I installed and for "now" the problem is solved, but soon I will have the same issue, there is no way to use multiple directories for "/files/" folder?

From this, I'm gathering that most of your directories in /files actually contain either images or files.  And though you just cleared out the empties, you're going to be hitting your max again soon. You need a way to circumvent Apache's file limit within any directory, like have a /file2 directory where new pages can store things. Correct?

Link to comment
Share on other sites

I've read the post but i didn't come up with any solution...

Have you checked my solution? It works. I have a site with > 200.000 pages each containing images and files.

One big disadvantage is that I needed to hack two core files. I'll open a GitHub issue with a proposal how this could

be implemented with hooks.

If you take a solution like mine, you'd need to write a script which creates the subfolders and moves the files for your existing site.

So for example:

files/1603/image.jpg

must be translated to:
files/16/03/image.jpg
  • Like 3
Link to comment
Share on other sites

@wanze I would like to avoid hack core files but if there are no other solutions I'll do it, thanks. Maybe if you can send me details of what files did you modifiy and where exactly I would appreciate that.

Link to comment
Share on other sites

Hi Sevarf2,

Here are my notes to the two files that were patched.

The PagefilesManager::url() method is hookable so theoretically there is no patch needed. But since I needed to hack that file anyway,

I did not use a hook.

Just ask here if you have any questions. I think it would be nice if the developer could define a specific strategy for saving the files (with Hooks). I will open an Issue on GitHub and provide a proposal to ryan.

Edit: I do not use pagefileSecure feature, and actually have no idea if it would work with those "hacks" ;)

Patches:

/wire/core/functions.php

function wireMkdir($path, $recursive = false) {
     //if(!is_dir($path)) return false;
     $chmodDir = wire('config')->chmodDir;
     if ($recursive && $chmodDir) {
          return @mkdir($path, octdec($chmodDir), true);
     }
     if(!@mkdir($path)) return false;
     if($chmodDir) chmod($path, octdec($chmodDir));
     return true;


/wire/core/PagefilesManager.php

     static public function makeCustomPath(Page $page) {
         
          // Make subfolders per ID - always two digits (or one per folder)
          // Example: ID=1780, path = /site/assets/files/17/80/
          //          ID=19814, path = /site/assets/files/19/81/4/
          //          ID=205478, path = /site/assets/files/20/54/78/
          $tmpPath = '';
          $digits = str_split("{$page->id}");
          foreach ($digits as $k => $digit) {
               $tmpPath .= $digit;
               if (($k+1) % 2 == 0) $tmpPath .= '/';
          }
          if (substr($tmpPath, -1) != '/') $tmpPath .= '/';
          return $tmpPath;
              
     }

===

     protected function _createPath($path) {
          if(is_dir($path)) return true;
          return wireMkdir($path, true);
     }  

===

     static public function _path(Page $page) {
                   
          $config = wire('config');
          $path = $config->paths->files;
         
          //$publicPath = $path . ((int) $page->id) . '/';     
          $tmpPath = self::makeCustomPath($page);
          $publicPath = $path . $tmpPath;
                   
          $securePrefix = $config->pagefileSecurePathPrefix;
          if(!strlen($securePrefix)) $securePrefix = self::defaultSecurePathPrefix;

          // securePath has the page ID preceded with a prefix which PW's htaccess blocks http requests to
          //$securePath = $path . $securePrefix . ((int) $page->id) . '/';
          $securePath = $path . $securePrefix . $tmpPath;
         
          // we track this just in case the prefix was newly added to config.php, this prevents
          // losing track of the original directories
          //$securePath2 = $path . self::defaultSecurePathPrefix . ((int) $page->id) . '/';
          $securePath2 = $path . self::defaultSecurePathPrefix . $tmpPath; 

          [.. continue method...]

     }

===

public function ___url() {
  return $this->config->urls->files . self::makeCustomPath($this->page); 
}

  • Like 3
Link to comment
Share on other sites

dfunk, I don't think you are going to help getting S3 support by posting about it on every possible topic. One or two mentions should be enough.

Oops! Sorry about that. Didn't realise I was crossing the line there. I shall be more cognisant about that from now!  :)

Link to comment
Share on other sites

@Wanze

I quite like the idea of having a hookable method that would allow consistent path manipulation. If it could rewrite the entire path for a resource then perhaps it could even be used to support things like image delivery from a dedicated sub-domain etc.

Edit: Looks like Teppo & Ryan had a similar idea over here.

Edited by netcarver
  • Like 2
Link to comment
Share on other sites

I don't have a migration script, because I've implemented this before starting the project :)

But should not be hard, I would do a command line script, bootstrap ProcessWire and then code something like this:

<?php
// Bootstrap Pw

// Make function wireMove, function mostly copied from wireCopy function
function wireMove($src, $dst) {

	if(substr($src, -1) != '/') $src .= '/';
	if(substr($dst, -1) != '/') $dst .= '/';

	$dir = opendir($src);
	if(!$dir) return false; 
	if(!wireMkdir($dst, true)) return false;

	while(false !== ($file = readdir($dir))) {
		if($file == '.' || $file == '..') continue;
	        rename($src . $file, $dst . $file);
		$chmodFile = wire('config')->chmodFile;
		if($chmodFile) @chmod($dst . $file, octdec($chmodFile));
	}

	closedir($dir);
	return true;
}

// Start migration - Loop your pages
// You could split by templates and use limit/start to get this done in multiple iterations. 
// Limit depends on your server memory 

$pages = wire('pages')->find('template=shibby,limit=2000');
foreach ($pages as $page) {
  $old_path = wire('config')->paths->files . $page->id . '/';
  $new_path = PagefilesManager::makeCustomPath($page);
  if (wireMove($old_path, $new_path)) {
    @unlink($old_path);
  }
}

Please only use on a copy of your installation, there will for sure be bugs as I've written it in the browser.

You will need to make some modifications, but maybe it helps you as a starting point.

Cheers

  • Like 5
Link to comment
Share on other sites

@wanze I just made a correction here:

$new_path = wire('config')->paths->files . PagefilesManager::makeCustomPath($page);
 

makeCustomPath returns relative directory and the rename didn't work. The rest works pretty good. Thanks again.

Link to comment
Share on other sites

  • 1 year later...

Hi guys, I'm updating this discussion. After a while using @wanze solution to avoid the 32k limit of subfolders I'm thinking of upgrade PW to the latest version (from current 2.3) because since middle 2014 I didn't make any update because of this core custom modification.
Now PW also solves this issue with the config option $config->pagefileExtendedPaths.
Is there any chance the pw solution is the same of wanze so updating pw don't cause any issue to my current folders structure?

Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
 Share

  • Recently Browsing   0 members

    • No registered users viewing this page.
×
×
  • Create New...