Delete 100 000+ pages

[Roberts R]

Last night I run into issue. I was trying to empty my trash that have 140 000+ pages init. All pages have repeater fields. When I pressed "Empty" on trash menu I wasn't able to use back-end for 2h+ (I kept refreshing page when It time-outed). After PW process was done  30k pages were deleted. I'm still left with 110k pages to remove.

Is there a way to safely remove all pages but with increased speed? Thanks

[diogo]

You can use from the API:

$pages->emptyTrash();

It's even better if you can do it by bootstraping PW from the terminal.

[arjen]

You can use wireshell as well. This is really handy. Just give the following command and grab a coffee 

wireshell page:emptytrash

I'm not really sure if it will delete everything in one run, but I deleted 60k pages the other day without problems. Really fast too and the main advantage is that you are not tied to a PHP session.

[Roberts R]

I don't own the system. And server admins most likely will not setup things for me. So will have to run a script to do :

$pages->emptyTrash();

Thanks for your support.

 

[Jonathan Lahijani]

I have a site that I oftentimes sync the live database to my dev database (I made a bash script to automate the process).  I generally like to "clean" the database once it has been copied over to my dev machine, which involves running a script that deletes 15,000 pages (orders in my case), among other things.  Doing this using $page->delete() in a script (which I'm running through the command line for added performance), takes about 30 minutes which is painful.

I thought it through further and I came up with the following relatively simple script that can achieve the same result in a few seconds!  It achieves this speed by running MySQL delete queries directly, bypassing PW's API.

Here it is (modify accordingly):

<?php namespace ProcessWire; ?>
<?php
include(dirname(__FILE__).'/index.php'); // bootstrap pw

$order_ids = $pages->findIDs('parent=/orders/,template=order');
if(!count($order_ids)) exit;
$order_ids = implode(",", $order_ids);

$t = $templates->get("order");
foreach($t->fields as $f) {
  if( $f->type!="FieldtypeFieldsetOpen" && $f->type!="FieldtypeFieldsetClose" ) {
    $table = "field_".$f->name;
    $database->query("DELETE FROM $table WHERE pages_id IN ($order_ids)");
  }
}
$database->query("DELETE FROM pages WHERE id IN ($order_ids)");

 

[teppo]

On 10/11/2017 at 7:33 AM, Jonathan Lahijani said:

It achieves this speed by running MySQL delete queries directly, bypassing PW's API.

Kind of pointing out the obvious here, but that could be asking for trouble. A number of modules (including core ones, such as Repeaters) perform cleanup etc. with hooks. Unless you're really careful, you could easily leave a ton of orphaned data behind with this method.

[netcarver]

In the past, I've had to delete multiple tens of thousands of records (pages) and did it by pushing the ids of the pages to be deleted into a WireQueue and then, at regular intervals (think cron-driven), pull out small batches of them to delete. To completely empty the queue takes a long time, but the admin interface remains responsive and you simply need to check the number of items remaining in the queue to know how far through the process you are.