delete orphaned files/images from site/assets/files

[benbyf]

imgs being created is the issue. So for every width(), height(), size(), function used in a template or in the editor creates an image, and other modules might make more. Change that image (say upload a different one) and now you have maybe one image and 3 variations that now dont need to be there in files as it isn't used anywhere in pw and therefore within the DB.

Be nice to have a module that reads /files and compares against anything in pw and deletes if you want it too.

[gebeer]

OK, I see. Yes, would be really nice to have such a module.

I believe this post refers to how we can also delete image variations.

[ceberlin]

I think too, that actually a general maintenance module would be great.

There are some maintenance tips and pieces spread over this forum. About cleaning unneeded assets dirs, and so on, cleaning up databases, there even is some extendable monitoring + maintenance module already there - without much functionality yet, but anyway.

(For WordPress there is a modle that can force rebuilding image variations/thumbs. I found that useful in some situations. I am missing this sort of technology here, too.)

some examples:

https://processwire.com/talk/topic/4437-delete-orphaned-filesimages-from-siteassetsfiles/

https://processwire.com/talk/topic/9383-search-and-replace/?hl=search

http://modules.processwire.com/modules/process-diagnostics/ (can cleanup databases)

[hellomoto]

I have a script derived from this thread that used to work but now I'm trying to run it on a dev branch project and I get this error message:

Quote

Fatal error:  Call to undefined function wire() in freshthumbs.php on line 4

Here's the code:

<pre><?php
ini_set('max_execution_time', 60 * 5); // 5 minutes, increase as needed
include("./index.php");
$dir = new DirectoryIterator(wire('config')->paths->files);

foreach ($dir as $file) {
    if ($file->isDot() || !$file->isDir()) {
        continue;
    }
    $id = $file->getFilename();
    if (!ctype_digit("$id")) {
        continue;
    }
    $page = wire('pages')->get((int) $id);
    if (!$page->id) {
        echo "Orphaned directory: " . wire('config')->urls->files . "$id/" . $file->getBasename() . "\n";
        continue;
    }
    // determine which files are valid for the page
    $valid = array();
    foreach ($page->template->fieldgroup as $field) {
        if ($field->type instanceof FieldtypeFile) {
            foreach ($page->get($field->name) as $file) {
                $valid[] = $file->basename;
                if ($field->type instanceof FieldtypeImage) {
                    foreach ($file->getVariations() as $f) {
                        //$valid[] = $f->basename;
                    }
                }
                // keep thumbnails:
                /*
                if ($field->type instanceof FieldtypeCropImage) {
                    $crops = $field->getArray();
                    $crops = $crops['thumbSetting'];
                    $crops_a = explode("\n", $crops); // ie. thumbname,200,200 (name,width,height)
                    foreach ($crops_a as $crop) {
                        $crop = explode(",", $crop);
                        $prefix = wire('sanitizer')->name($crop[0]);
                        $valid[] = $prefix . "_" . $file->basename;
                    }
                }*/
            }
        }
    }
    // now find all the files present on the page
    // identify those that are not part of our $valid array
    $d = new DirectoryIterator($page->filesManager->path);
    foreach ($d as $f) {
        if ($f->isDot() || !$f->isFile()) {
            continue;
        }
        if (!in_array($f->getFilename(), $valid)) {
            echo "Orphaned file: " . wire('config')->urls->files . "$id/" . $f->getBasename() . "\n";
//             unlink($f->getPathname());
        }
    }
    wire('pages')->uncache($page); // just in case we need the memory
}

I tried preceding the wire in mention with a $ and also with Processwire/ respectively. Processwire/wire returns this: 

Quote

Notice:  Use of undefined constant Processwire - assumed 'Processwire' in freshthumbs.php on line 4

Fatal error:  Call to undefined function wire() in freshthumbs.php on line 4

and $wire :

Quote

Fatal error:  Function name must be a string in freshthumbs.php on line 4

Why is this?

[flydev]

@hellomoto on dev branch you need to add the namespace. On top of your file, write:

<?php namespace ProcessWire;

[...]

and you will call DirectoryIterator like this :

$it = new \DirectoryIterator(...);

 

[hellomoto]

That was it, thanks!

[Robin S]

4 hours ago, flydev said:

on dev branch you need to add the namespace.

Thanks @flydev, can you (or anyone) explain why it's necessary to manually add the Processwire namespace to this file? I thought the file compiler was supposed to take care of this automatically. If not, how do we know which files need to have the namespace added manually and which are compiled automatically?

[LostKobrakai]

Only template files and modules are compiled. Files bootstrapping processwire won't trigger any compilation.

[Robin S]

Got it, thanks @LostKobrakai

[Pravin]

Why is it throwing out the following error?

Spoiler

Warning: Unknown: It is not safe to rely on the system's timezone settings. You are *required* to use the date.timezone setting or the date_default_timezone_set() function. In case you used any of those methods and you are still getting this warning, you most likely misspelled the timezone identifier. We selected the timezone 'UTC' for now, but please set date.timezone to select your timezone. in /home/pleasefi/public_html/clients/csdc-ganesh/clean-files.php on line 2

Fatal error: Namespace declaration statement has to be the very first statement in the script in /home/pleasefi/public_html/clients/csdc-ganesh/clean-files.php on line 2

 

 

 

 

 

 

 

[Pravin]

clean-file.php this is the code

    <pre>
    <?php namespace Processwire;
    ini_set('max_execution_time', 1200); // 5 minutes, increase as needed
    include("./index.php"); 
    $dir = new DirectoryIterator(wire('config')->paths->files); 
     
    foreach($dir as $file) {
      if($file->isDot() || !$file->isDir()) continue; 
      $id = $file->getFilename();
      if(!ctype_digit("$id")) continue;  
      $page = wire('pages')->get((int) $id);
      if(!$page->id) {
        echo "Orphaned directory: " . wire('config')->urls->files . "$id/" . $file->getBasename() . "\n";
        continue; 
      }
      // determine which files are valid for the page
      $valid = array();
      foreach($page->template->fieldgroup as $field) {    
        if($field->type instanceof FieldtypeFile) {
          foreach($page->get($field->name) as $file) {
            $valid[] = $file->basename; 
            if($field->type instanceof FieldtypeImage) {
              foreach($file->getVariations() as $f) {
                $valid[] = $f->basename; 
              }
            }
          }
        }
      } 
      // now find all the files present on the page
      // identify those that are not part of our $valid array
      $d = new DirectoryIterator($page->filesManager->path); 
      foreach($d as $f) {
        if($f->isDot() || !$f->isFile()) continue; 
        if(!in_array($f->getFilename(), $valid)) {
          echo "Orphaned file: " . wire('config')->urls->files . "$id/" . $f->getBasename() . "\n";                               
          // unlink($f->getPathname()); 
        }
      }
      wire('pages')->uncache($page); // just in case we need the memory
    }
    ?>
    </pre>
  

 

[flydev]

1 hour ago, Pravin said:

Why is it throwing out the following error?

  Reveal hidden contents

Warning: Unknown: It is not safe to rely on the system's timezone settings. You are *required* to use the date.timezone setting or the date_default_timezone_set() function. In case you used any of those methods and you are still getting this warning, you most likely misspelled the timezone identifier. We selected the timezone 'UTC' for now, but please set date.timezone to select your timezone. in /home/pleasefi/public_html/clients/csdc-ganesh/clean-files.php on line 2

Fatal error: Namespace declaration statement has to be the very first statement in the script in /home/pleasefi/public_html/clients/csdc-ganesh/clean-files.php on line 2

 

 

 

 

 

 

 

@Pravin

You need to remove the html code - first (<pre>) and last lines (</pre>) in your clean-file.php.

[evan]

For some reason the script doesn't find anything for me when checking for the correct Fieldtype.  $field->type instanceof FieldtypeImage and $field->type instanceof FieldtypeFile never returns true for any of my Image fields, but $field->type->name == "FieldtypeImage" does.  My working validation loop looks like this:

  $valid = array();
  foreach($page->template->fieldgroup as $field) {   
    foreach($page->get($field->name) as $file) {
      $valid[] = $file->basename;
      if($field->type->name == "FieldtypeImage") {
        foreach($file->getVariations() as $f) {
          $valid[] = $f->basename; 
        }
      }
    }
  }

I'm running PW 3.0.42, PHP v5.5.27.

[aagd]

*** post removed by author, problem was already solved above ***

[Jozsef]

When using the Processwire namespace I also needed to add a leading backslash to the DirectoryIterator class so it can access the global namespace, in case someone needs to use it with the latest PW and PHP versions.

<pre>
<?php namespace Processwire;

ini_set('max_execution_time', 60*5); // 5 minutes, increase as needed
include("./index.php"); 
// Add a leading back slash to the class so it can access the global namespace
$dir = new \DirectoryIterator(wire('config')->paths->files); 
 
foreach($dir as $file) {
  if($file->isDot() || !$file->isDir()) continue; 
  $id = $file->getFilename();
  if(!ctype_digit("$id")) continue;  
  $page = wire('pages')->get((int) $id);
  if(!$page->id) {
    echo "Orphaned directory: " . wire('config')->urls->files . "$id/" . $file->getBasename() . "\n";
    continue; 
  }
  // determine which files are valid for the page
  $valid = array();
  foreach($page->template->fieldgroup as $field) {    
    if($field->type instanceof FieldtypeFile) {
      foreach($page->get($field->name) as $file) {
        $valid[] = $file->basename; 
        if($field->type instanceof FieldtypeImage) {
          foreach($file->getVariations() as $f) {
            $valid[] = $f->basename; 
          }
        }
      }
    }
  } 
  // now find all the files present on the page
  // identify those that are not part of our $valid array
  // Add a leading back slash to the class so it can access the global namespace
  $d = new \DirectoryIterator($page->filesManager->path); 
  foreach($d as $f) {
    if($f->isDot() || !$f->isFile()) continue; 
    if(!in_array($f->getFilename(), $valid)) {
      echo "Orphaned file: " . wire('config')->urls->files . "$id/" . $f->getBasename() . "\n";                               
      // unlink($f->getPathname()); 
    }
  }
  wire('pages')->uncache($page); // just in case we need the memory
}
?>
</pre>

 

[Carsten]

On 9/8/2013 at 2:13 PM, ryan said:

I sometimes end up with orphaned files as a result of doing mass imports during development. My code won't be quite right the first time around and I'll end up with extra and/or duplicated files. At least that was the case this last week. It was on a pretty large scale, so not something I wanted to clean up manually. Here's how I cleaned them out. Place this in a file in your site root called clean-files.php and then load it in your browser. 

/clean-files.php

 

Hi,

I tried this because of troubles with orphaned files and repeater data and so on (see Orphaned repeater data?)

I created the file clean-files.php in the root and pasted the code as given.

The result is...

Warning:  session_set_save_handler(): Session save handler cannot be changed after headers have already been sent in /var/www/html/wire/core/WireSessionHandler.php on line 57
Warning:  session_name(): Session name cannot be changed after headers have already been sent in /var/www/html/wire/core/Session.php on line 291
Warning:  ini_set(): Session ini settings cannot be changed after headers have already been sent in /var/www/html/wire/core/Session.php on line 294
Warning:  ini_set(): Session ini settings cannot be changed after headers have already been sent in /var/www/html/wire/core/Session.php on line 295
Warning:  ini_set(): Session ini settings cannot be changed after headers have already been sent in /var/www/html/wire/core/Session.php on line 296
Warning:  ini_set(): Session ini settings cannot be changed after headers have already been sent in /var/www/html/wire/core/Session.php on line 297
Warning:  ini_set(): Session ini settings cannot be changed after headers have already been sent in /var/www/html/wire/core/Session.php on line 309
Fatal error:  Uncaught Error: Call to undefined function wire() in /var/www/html/clean-files.php:7
Stack trace:
#0 {main}
  thrown in /var/www/html/clean-files.php on line 7
Warning:  Cannot modify header information - headers already sent by (output started at /var/www/html/clean-files.php:1) in /var/www/html/wire/core/WireHttp.php on line 1688

I also tried it with namespace ProcessWire

 

[dragan]

@Carsten The echo '<pre>' part comes too early. Change the first few lines to:

<?php namespace Processwire;
ini_set('max_execution_time', 60 * 5); // 5 minutes, increase as needed
include("./index.php");
echo '<pre>'; // rest below unchanged:
// Add a leading back slash to the class so it can access the global namespace
$dir = new \DirectoryIterator(wire('config')->paths->files);

Tried the script like this and it works just fine (PW 3.0.205)

[Carsten]

2 hours ago, dragan said:

@Carsten The echo '<pre>' part comes too early. Change the first few lines to:

<?php namespace Processwire;
ini_set('max_execution_time', 60 * 5); // 5 minutes, increase as needed
include("./index.php");
echo '<pre>'; // rest below unchanged:
// Add a leading back slash to the class so it can access the global namespace
$dir = new \DirectoryIterator(wire('config')->paths->files);

Tried the script like this and it works just fine (PW 3.0.205)

@dragan

Thank you, now orphaned directories are shown but not from repeaters.

I deleted all pages and repeaters from pages and there still are some folders which are not shown as orphaned.

I tried it with manually creating a folder e.g. "9999" and it was shown as orphaned. Then I created manually a file in a not orphaned folder and the file was shown as orphaned. So far so good.

But these repeater stuff doesn't disapear and I don't know how to delete that stuff

 

id: 3538
name: 1667137707-9667-6

ProcessWire\RepeaterPage Object
(
    [id] => 3538
    [name] => 1667137707-9667-6
    [parent] => /ppww/repeaters/for-field-155/for-page-0/
    [template] => repeater_blog_repeater
    [name1110] => 
    [status1110] => 1
    [data] => Array
        (
            [name1110] => 
            [status1110] => 1
        )

)

[Carsten]

16 hours ago, Carsten said:

@dragan

Thank you, now orphaned directories are shown but not from repeaters.

I deleted all pages and repeaters from pages and there still are some folders which are not shown as orphaned.

I tried it with manually creating a folder e.g. "9999" and it was shown as orphaned. Then I created manually a file in a not orphaned folder and the file was shown as orphaned. So far so good.

But these repeater stuff doesn't disapear and I don't know how to delete that stuff

 

id: 3538
name: 1667137707-9667-6

ProcessWire\RepeaterPage Object
(
    [id] => 3538
    [name] => 1667137707-9667-6
    [parent] => /ppww/repeaters/for-field-155/for-page-0/
    [template] => repeater_blog_repeater
    [name1110] => 
    [status1110] => 1
    [data] => Array
        (
            [name1110] => 
            [status1110] => 1
        )

)

I did not solve it, so I made tests with Fieldset (Page). There also is a Repeaters section but with my fieldset.

 

It's not what I wanted but the Fieldsets are deleted when I delete a page manually or via api so that there's no waste with orphaned  folders and files.

The stuff with the orphaned files and folders and entries in admin/Repeaters/ ... still exists and could not be detected with Ryans script (clean-files.php).

Could this then be a bug?

Thank you for your help

Carsten