☁️ Duplicator: Backup and move sites

[jploch]

On 3/19/2019 at 7:58 AM, flydev said:

You can download this dev version on GitHub :  https://github.com/flydev-fr/Duplicator/archive/dev.zip or at the end of this post.

This worked for me. Thank you for the update!

[arjen]

I have Duplicator running smoothly on several installs. Today I felt I needed to upgrade to the dev version (sometimes you feel crazy ?), but I'm running into this line:

if($zip->open($zipfile) !== true) throw new WireException("Unable to re-open ZIP: $zipfile");

I'm not sure how to debug this. I'm running PHP 7.2.17. ZipArchive has been installed. The Duplicator log also doesn't show any errors. Any pointers how to debug this?

[Autofahrn]

18 minutes ago, arjen said:

The Duplicator log also doesn't show any errors. Any pointers how to debug this?

Did you check the additional logfile written into the backup directory? That should at least show "CLOSING ZIP" and "OPENING ZIP" along with the filename.

You may check, if that file actually exists on your server. If it does exist and the re-open fails its probably best to modify the code to get the error, something like:

	$res = $zip->open($zipfile);
	if($res !== true) throw new WireException("Unable to re-open ZIP ({$zipfile}): {$res}");

https://www.php.net/manual/en/ziparchive.open.php#117339

Anyway, you may set the flush margin (DUP_ZIP_FLUSH_MBYTES in Duplicator.module) to a higher value to disable this close&re-open mechanism.

[arjen]

Thanks for getting back on this.

The file doesn't exist on the server. I currently am testing on another install (same server same php version) and here it's working fine (as in it creates the zip flawless), but the logs in the ProcessModule says:

2019-05-12 13:06:32:  - package build failed.
2019-05-12 13:06:32:  - package build failed,  doesn't exist

When I compare the verbose logs between the two installs I notice this:

Install which fails tries to open after closing:

Quote

0.0s: CLOSING ZIP: /**/public/site/assets/backups/2019-05-11_13-52-12.package2.zip
1.4s: OPENING ZIP: /**/public/site/assets/backups/2019-05-11_13-52-12.package2.zip
Done in 1.41s

Install which succeeds only closes:

Quote

12.4s: CLOSING ZIP: /**/public/site/assets/backups/2019-05-12_13-06-08.package2.zip
Done in 22.43s

I will investigate further.

[Autofahrn]

The second install is probably working on a smaller site (less than 200 MBytes of data), so the re-opening mechanism is not triggered at all.

I guess you edited the pathnames, since the /**/ looks weird to me and I'm missing the domain name.

1 hour ago, arjen said:

2019-05-12 13:06:32:  - package build failed.
2019-05-12 13:06:32:  - package build failed,  doesn't exist 

Does your domain name contains a dash? Then this fix should help:

 

[entschleunigung]

hello,

maybe the topic has already been discussed, but unfortunately I didn't find anything to it.
i have the problem that my site has to run under windows and iis, but Duplicator doesn't create a complete archive. the database can be archived, that seems to be complete, but the files don't. the archive always has only 1 KB.

does anyone have an idea?

 

[wbmnfktr]

If the job needs to be done just download all files manually or try Site Profile Exporter.

Some tipps and details on how to migrate manually can be found here: 

https://processwire.com/docs/security/migration/

[flydev]

18 hours ago, entschleunigung said:

hello,

maybe the topic has already been discussed, but unfortunately I didn't find anything to it.
i have the problem that my site has to run under windows and iis, but Duplicator doesn't create a complete archive. the database can be archived, that seems to be complete, but the files don't. the archive always has only 1 KB.

does anyone have an idea?

 

I ran Duplicator on a Windows Server 2012, IIS and PHP 7.2.18 without issue, which version of Duplicator have you tested ?

[arjen]

On 5/12/2019 at 2:50 PM, Autofahrn said:

Does your domain name contains a dash?

No, the domain name doesn't have a dash, but the file path does. These look like: /homes/username-tld/sites/domain.tld/. Perhaps the dash in the file path is the problem? I'll check out your fix later on. Thanks!

[Autofahrn]

19 minutes ago, arjen said:

Perhaps the dash in the file path is the problem?

Don't think so. The fix only retrieves the timestamp from filename differently, it does not look at the path.

Edit: just saw:

Quote

package build failed,  doesn't exist

There normally should be a filename between "failed," and "doesn't". Did you specify something in the name field?

[entschleunigung]

1 hour ago, flydev said:

I ran Duplicator on a Windows Server 2012, IIS and PHP 7.2.18 without issue, which version of Duplicator have you tested ?

hi fly,

I use the following environment:

Windows Server 2016
Version     1.2.9 
PHP Version 7.2.7

Duplicator Log:

13.05.2019 14:29:20:  - job finished in 1.102115sec
13.05.2019 14:29:20:  - package saved in local folder: 2019-05-13_14-29-19-xyz.package.zip
13.05.2019 14:29:20:  - package built successfully in 1.097146sec

as I said before, the database archive works perfectly, the rest unfortunately doesn't.

 

thank you

[flydev]

1 hour ago, entschleunigung said:

Version     1.2.9 

Please upgrade to the dev version 1.3.10-ATO here :  https://github.com/flydev-fr/Duplicator/tree/dev

 

Steps :

- Rename the Duplicator folder to .Duplicator

- Download the dev version and extract the folder in the modules directory

- Rename Duplicator-dev to Duplicator

- Go to Modules > Refresh

 

It should works.

[entschleunigung]

48 minutes ago, flydev said:

It should works

thank you fly, 

thank you. now it works, but only with one exception. the package manager doesn't show the backups on the overview page, but you can find them under /site/assets/backups. i also tested different browsers, always with the same result. i also deleted the compiled files under module, no changes. 

and the logs:

14.05.2019 12:55:29:  - package build failed.
14.05.2019 12:55:29:  - package build failed,  doesn't exist

no stress, for me this is a small beauty error, i can live with it as long as the backups are made. but i don't mind if the overview page would work ?

thx

 

 

[Autofahrn]

7 minutes ago, entschleunigung said:

and the logs:

14.05.2019 12:55:29:  - package build failed.
14.05.2019 12:55:29:  - package build failed,  doesn't exist

so same issue as @arjen observed, quite strange.

@entschleunigung, did you verify the backups are complete? Do they have a reasonable size?

[entschleunigung]

8 minutes ago, Autofahrn said:

did you verify the backups are complete? Do they have a reasonable size?

hi,

at first sight the archives look good: about 500 MB.
i was able to unpack the archive, everything seems to be in the right place ?

 

[flydev]

On 4/14/2019 at 8:43 PM, Autofahrn said:

Btw, @flydev, do you plan to include the suggested domain name fix into ProcessDuplicator::getPackagesDetails to support dashes in domain names?

Done. pushed the update in Duplicator 1.3.11-ATO.

 

5 hours ago, entschleunigung said:

but only with one exception. the package manager doesn't show the backups on the overview page

Does your domain name contain a dash char ?  If yes, try the latest update (v1.3.11-ATO).

[arjen]

I've updated to the latest dev and unfortunately it is not working. 

2019-05-14 20:39:11:  - package build failed.
2019-05-14 20:39:11:  Unable to re-open ZIP: /home/xxxxxx-nl/webapps/xxxxxx-nl/public/site/assets/backups/2019-05-14_20-39-09-xxxxxx.nl.package2.zip

I tried upping DUP_ZIP_FLUSH_MBYTES and played with other settings, but no luck. I'm unsure why. For now I'll keep running the previous version. Thanks for the hard work on this module.

[entschleunigung]

16 hours ago, flydev said:

Does your domain name contain a dash char ?  If yes, try the latest update (v1.3.11-ATO).

yes, that's the case with me. i have now installed the last update and everything works fine. existing backups now appear in the overview.

thank you
 

[Autofahrn]

14 hours ago, arjen said:

I tried upping DUP_ZIP_FLUSH_MBYTES and played with other settings, but no luck. I'm unsure why.

Well, I guess your site contains a lot of data so the re-opening still is triggered. I'm really interested in the returned error number, did you try to add the suggested code change?

Otherwise you may simply disable the whole block in duplicator.module:

		if(false
		&& ($fragmentBytes >= self::DUP_ZIP_FLUSH_MBYTES*1024*1024))
			{
			$zipLog->verbose("CLOSING ZIP: {$zipfile}");
			$zip->close();
			$zipLog->verbose("OPENING ZIP: {$zipfile}");
//			if($zip->open($zipfile) !== true) throw new WireException("Unable to re-open ZIP: $zipfile");
			$res = $zip->open($zipfile);
			if($res !== true) throw new WireException("Unable to re-open ZIP ({$zipfile}): {$res}");
			$zipLog->verbose("OPENED ZIP: {$zipfile}");
			$fragmentBytes = 0;
			set_time_limit(300);
			}

@flydev, maybe we should have DUP_ZIP_FLUSH_MBYTES configurable with that option disabled when not set.

[flydev]

5 hours ago, Autofahrn said:

@flydev, maybe we should have DUP_ZIP_FLUSH_MBYTES configurable with that option disabled when not set.

Just pushed an update - v1.3.12-ATO - the quantity of megabytes saved before an archive flush is now configurable. When the value is empty or equal to 0 then the option is disabled.

 

5 hours ago, Autofahrn said:

if(false

@Autofahrn what is this check ? I don't have it in my code ?

Edited May 15, 2019 by flydev
link to the dev branch

[arjen]

7 hours ago, Autofahrn said:

Well, I guess your site contains a lot of data so the re-opening still is triggered. I'm really interested in the returned error number, did you try to add the suggested code change?

Sorry mate, missed this one. It is error 9.

2019-05-15 19:00:26: Unable to re-open ZIP (/home/xxxxxx-nl/webapps/xxxxxx-nl/public/site/assets/backups/2019-05-15_19-00-24-xxxxxx.nl.package2.zip): 9

When I disable that code block I get:

2019-05-15 19:04:00:  - package build failed.
2019-05-15 19:04:00:  - an error occured during package build.
2019-05-15 19:04:00:  - an error occured while building the ProcessWire structure: /home/xxxxxx-nl/webapps/xxxxxx-nl/public/site/assets/backups/2019-05-15_19-03-56-xxxxxx.nl.package2.zip root: /home/xxxxxx-nl/webapps/xxxxxx-nl/public/ Array (     [filename] => 2019-05-15_19-03-56-xxxxxx.nl.package2.zip     [folder] => /home/xxxxxx-nl/webapps/xxxxxx-nl/public/site/assets/backups/     [exclude] => Array         (             [0] => /site/assets/cache/WireTempDir             [1] => /site/assets/backups             [2] => /site/assets/cache             [3] => /site/assets/sessions             [4] => %\.\d+x\d+\.[^/]+$%             [5] =>          )      [extension] => Array         (         )      [allowHidden] => 1 )
2019-05-15 19:03:56:  - package build failed.
2019-05-15 19:03:56:  - an error occured during package build.
2019-05-15 19:03:56:  - an error occured while building the ProcessWire structure: /home/xxxxxx-nl/webapps/xxxxxx-nl/public/site/assets/backups/2019-05-15_19-03-52-xxxxxx.nl.package2.zip root: /home/xxxxxx-nl/webapps/xxxxxx-nl/public/ Array (     [filename] => 2019-05-15_19-03-52-xxxxxx.nl.package2.zip     [folder] => /home/xxxxxx-nl/webapps/xxxxxx-nl/public/site/assets/backups/     [exclude] => Array         (             [0] => /site/assets/cache/WireTempDir             [1] => /site/assets/backups             [2] => /site/assets/cache             [3] => /site/assets/sessions             [4] => %\.\d+x\d+\.[^/]+$%             [5] =>          )      [extension] => Array         (         )      [allowHidden] => 1 )

Thanks again! I currently use 1.3.11.

[Autofahrn]

4 minutes ago, arjen said:

It is error 9.

ok, "no such file". No surprise, that re-opening and the package build fail. So I wonder why creation of the zip does not already throw an exception.

Do you see that file on your server?

[Autofahrn]

3 hours ago, flydev said:

what is this check ? I don't have it in my code

the if(false is my php replacement for #if 0 ?

[arjen]

12 hours ago, Autofahrn said:

Do you see that file on your server?

Nope, that's the strange part. The other installs run the same PW versions, same file permissions, same PHP version. 

[Autofahrn]

I've just re-read the previous posts and see that CLOSING ZIP actually happens after 0.0 seconds of operation. This is pretty unlikely since I expect some time for directory enumeration and packaging the first part. Can you send me that verbose log via PM?