Jump to content
flydev

Duplicator: Backup and move sites

Recommended Posts

I have Duplicator running smoothly on several installs. Today I felt I needed to upgrade to the dev version (sometimes you feel crazy 🙂), but I'm running into this line:

if($zip->open($zipfile) !== true) throw new WireException("Unable to re-open ZIP: $zipfile");

I'm not sure how to debug this. I'm running PHP 7.2.17. ZipArchive has been installed. The Duplicator log also doesn't show any errors. Any pointers how to debug this?

  • Like 1

Share this post


Link to post
Share on other sites
18 minutes ago, arjen said:

The Duplicator log also doesn't show any errors. Any pointers how to debug this?

Did you check the additional logfile written into the backup directory? That should at least show "CLOSING ZIP" and "OPENING ZIP" along with the filename.

You may check, if that file actually exists on your server. If it does exist and the re-open fails its probably best to modify the code to get the error, something like:

	$res = $zip->open($zipfile);
	if($res !== true) throw new WireException("Unable to re-open ZIP ({$zipfile}): {$res}");

https://www.php.net/manual/en/ziparchive.open.php#117339

Anyway, you may set the flush margin (DUP_ZIP_FLUSH_MBYTES in Duplicator.module) to a higher value to disable this close&re-open mechanism.

  • Like 2

Share this post


Link to post
Share on other sites

Thanks for getting back on this.

The file doesn't exist on the server. I currently am testing on another install (same server same php version) and here it's working fine (as in it creates the zip flawless), but the logs in the ProcessModule says:

2019-05-12 13:06:32:  - package build failed.
2019-05-12 13:06:32:  - package build failed,  doesn't exist

When I compare the verbose logs between the two installs I notice this:

Install which fails tries to open after closing:

Quote

0.0s: CLOSING ZIP: /**/public/site/assets/backups/2019-05-11_13-52-12.package2.zip
1.4s: OPENING ZIP: /**/public/site/assets/backups/2019-05-11_13-52-12.package2.zip
Done in 1.41s

Install which succeeds only closes:

Quote

12.4s: CLOSING ZIP: /**/public/site/assets/backups/2019-05-12_13-06-08.package2.zip
Done in 22.43s

I will investigate further.

Share this post


Link to post
Share on other sites

The second install is probably working on a smaller site (less than 200 MBytes of data), so the re-opening mechanism is not triggered at all.

I guess you edited the pathnames, since the /**/ looks weird to me and I'm missing the domain name.

1 hour ago, arjen said:

2019-05-12 13:06:32:  - package build failed.
2019-05-12 13:06:32:  - package build failed,  doesn't exist 

Does your domain name contains a dash? Then this fix should help:

 

  • Like 2

Share this post


Link to post
Share on other sites

hello,

maybe the topic has already been discussed, but unfortunately I didn't find anything to it.
i have the problem that my site has to run under windows and iis, but Duplicator doesn't create a complete archive. the database can be archived, that seems to be complete, but the files don't. the archive always has only 1 KB.

does anyone have an idea?


 

Share this post


Link to post
Share on other sites
18 hours ago, entschleunigung said:

hello,

maybe the topic has already been discussed, but unfortunately I didn't find anything to it.
i have the problem that my site has to run under windows and iis, but Duplicator doesn't create a complete archive. the database can be archived, that seems to be complete, but the files don't. the archive always has only 1 KB.

does anyone have an idea?


 

I ran Duplicator on a Windows Server 2012, IIS and PHP 7.2.18 without issue, which version of Duplicator have you tested ?

Share this post


Link to post
Share on other sites
On 5/12/2019 at 2:50 PM, Autofahrn said:

Does your domain name contains a dash?

No, the domain name doesn't have a dash, but the file path does. These look like: /homes/username-tld/sites/domain.tld/. Perhaps the dash in the file path is the problem? I'll check out your fix later on. Thanks!

Share this post


Link to post
Share on other sites
19 minutes ago, arjen said:

Perhaps the dash in the file path is the problem?

Don't think so. The fix only retrieves the timestamp from filename differently, it does not look at the path.

Edit: just saw:

Quote

package build failed,  doesn't exist

There normally should be a filename between "failed," and "doesn't". Did you specify something in the name field?

Share this post


Link to post
Share on other sites
1 hour ago, flydev said:

I ran Duplicator on a Windows Server 2012, IIS and PHP 7.2.18 without issue, which version of Duplicator have you tested ?

hi fly,

I use the following environment:

Windows Server 2016
Version     1.2.9 
PHP Version 7.2.7

Duplicator Log:

13.05.2019 14:29:20:  - job finished in 1.102115sec
13.05.2019 14:29:20:  - package saved in local folder: 2019-05-13_14-29-19-xyz.package.zip
13.05.2019 14:29:20:  - package built successfully in 1.097146sec

as I said before, the database archive works perfectly, the rest unfortunately doesn't.

 

thank you

Share this post


Link to post
Share on other sites
1 hour ago, entschleunigung said:

Version     1.2.9 

Please upgrade to the dev version 1.3.10-ATO here :  https://github.com/flydev-fr/Duplicator/tree/dev

 

Steps :

- Rename the Duplicator folder to .Duplicator

- Download the dev version and extract the folder in the modules directory

- Rename Duplicator-dev to Duplicator

- Go to Modules > Refresh

 

It should works.

  • Like 2
  • Thanks 1

Share this post


Link to post
Share on other sites
48 minutes ago, flydev said:

It should works

thank you fly, 

thank you. now it works, but only with one exception. the package manager doesn't show the backups on the overview page, but you can find them under /site/assets/backups. i also tested different browsers, always with the same result. i also deleted the compiled files under module, no changes. 

and the logs:

14.05.2019 12:55:29:  - package build failed.
14.05.2019 12:55:29:  - package build failed,  doesn't exist

no stress, for me this is a small beauty error, i can live with it as long as the backups are made. but i don't mind if the overview page would work 🙂

thx

 

 

  • Like 1

Share this post


Link to post
Share on other sites
7 minutes ago, entschleunigung said:

and the logs:


14.05.2019 12:55:29:  - package build failed.
14.05.2019 12:55:29:  - package build failed,  doesn't exist

so same issue as @arjen observed, quite strange.

@entschleunigung, did you verify the backups are complete? Do they have a reasonable size?

  • Like 2

Share this post


Link to post
Share on other sites
8 minutes ago, Autofahrn said:

did you verify the backups are complete? Do they have a reasonable size?

hi,

at first sight the archives look good: about 500 MB.
i was able to unpack the archive, everything seems to be in the right place 🙂

 

  • Like 2

Share this post


Link to post
Share on other sites
On 4/14/2019 at 8:43 PM, Autofahrn said:

Btw, @flydev, do you plan to include the suggested domain name fix into ProcessDuplicator::getPackagesDetails to support dashes in domain names?

Done. pushed the update in Duplicator 1.3.11-ATO.

 

5 hours ago, entschleunigung said:

but only with one exception. the package manager doesn't show the backups on the overview page

Does your domain name contain a dash char ?  If yes, try the latest update (v1.3.11-ATO).

  • Like 2
  • Thanks 1

Share this post


Link to post
Share on other sites

I've updated to the latest dev and unfortunately it is not working. 

2019-05-14 20:39:11:  - package build failed.
2019-05-14 20:39:11:  Unable to re-open ZIP: /home/xxxxxx-nl/webapps/xxxxxx-nl/public/site/assets/backups/2019-05-14_20-39-09-xxxxxx.nl.package2.zip

I tried upping DUP_ZIP_FLUSH_MBYTES and played with other settings, but no luck. I'm unsure why. For now I'll keep running the previous version. Thanks for the hard work on this module.

  • Like 2

Share this post


Link to post
Share on other sites
16 hours ago, flydev said:

Does your domain name contain a dash char ?  If yes, try the latest update (v1.3.11-ATO).

yes, that's the case with me. i have now installed the last update and everything works fine. existing backups now appear in the overview.

thank you
 

  • Like 2

Share this post


Link to post
Share on other sites
14 hours ago, arjen said:

I tried upping DUP_ZIP_FLUSH_MBYTES and played with other settings, but no luck. I'm unsure why.

Well, I guess your site contains a lot of data so the re-opening still is triggered. I'm really interested in the returned error number, did you try to add the suggested code change?

Otherwise you may simply disable the whole block in duplicator.module:

		if(false
		&& ($fragmentBytes >= self::DUP_ZIP_FLUSH_MBYTES*1024*1024))
			{
			$zipLog->verbose("CLOSING ZIP: {$zipfile}");
			$zip->close();
			$zipLog->verbose("OPENING ZIP: {$zipfile}");
//			if($zip->open($zipfile) !== true) throw new WireException("Unable to re-open ZIP: $zipfile");
			$res = $zip->open($zipfile);
			if($res !== true) throw new WireException("Unable to re-open ZIP ({$zipfile}): {$res}");
			$zipLog->verbose("OPENED ZIP: {$zipfile}");
			$fragmentBytes = 0;
			set_time_limit(300);
			}

@flydev, maybe we should have DUP_ZIP_FLUSH_MBYTES configurable with that option disabled when not set.

  • Like 2

Share this post


Link to post
Share on other sites
Posted (edited)
5 hours ago, Autofahrn said:

@flydev, maybe we should have DUP_ZIP_FLUSH_MBYTES configurable with that option disabled when not set.

Just pushed an update - v1.3.12-ATO - the quantity of megabytes saved before an archive flush is now configurable. When the value is empty or equal to 0 then the option is disabled.

 

5 hours ago, Autofahrn said:

if(false

@Autofahrn what is this check ? I don't have it in my code 😅

Edited by flydev
link to the dev branch
  • Like 2

Share this post


Link to post
Share on other sites
7 hours ago, Autofahrn said:

Well, I guess your site contains a lot of data so the re-opening still is triggered. I'm really interested in the returned error number, did you try to add the suggested code change?

Sorry mate, missed this one. It is error 9.

2019-05-15 19:00:26: Unable to re-open ZIP (/home/xxxxxx-nl/webapps/xxxxxx-nl/public/site/assets/backups/2019-05-15_19-00-24-xxxxxx.nl.package2.zip): 9

When I disable that code block I get:

2019-05-15 19:04:00:  - package build failed.
2019-05-15 19:04:00:  - an error occured during package build.
2019-05-15 19:04:00:  - an error occured while building the ProcessWire structure: /home/xxxxxx-nl/webapps/xxxxxx-nl/public/site/assets/backups/2019-05-15_19-03-56-xxxxxx.nl.package2.zip root: /home/xxxxxx-nl/webapps/xxxxxx-nl/public/ Array (     [filename] => 2019-05-15_19-03-56-xxxxxx.nl.package2.zip     [folder] => /home/xxxxxx-nl/webapps/xxxxxx-nl/public/site/assets/backups/     [exclude] => Array         (             [0] => /site/assets/cache/WireTempDir             [1] => /site/assets/backups             [2] => /site/assets/cache             [3] => /site/assets/sessions             [4] => %\.\d+x\d+\.[^/]+$%             [5] =>          )      [extension] => Array         (         )      [allowHidden] => 1 )
2019-05-15 19:03:56:  - package build failed.
2019-05-15 19:03:56:  - an error occured during package build.
2019-05-15 19:03:56:  - an error occured while building the ProcessWire structure: /home/xxxxxx-nl/webapps/xxxxxx-nl/public/site/assets/backups/2019-05-15_19-03-52-xxxxxx.nl.package2.zip root: /home/xxxxxx-nl/webapps/xxxxxx-nl/public/ Array (     [filename] => 2019-05-15_19-03-52-xxxxxx.nl.package2.zip     [folder] => /home/xxxxxx-nl/webapps/xxxxxx-nl/public/site/assets/backups/     [exclude] => Array         (             [0] => /site/assets/cache/WireTempDir             [1] => /site/assets/backups             [2] => /site/assets/cache             [3] => /site/assets/sessions             [4] => %\.\d+x\d+\.[^/]+$%             [5] =>          )      [extension] => Array         (         )      [allowHidden] => 1 )

Thanks again! I currently use 1.3.11.

  • Like 3

Share this post


Link to post
Share on other sites
4 minutes ago, arjen said:

It is error 9.

ok, "no such file". No surprise, that re-opening and the package build fail. So I wonder why creation of the zip does not already throw an exception.

Do you see that file on your server?

  • Like 2

Share this post


Link to post
Share on other sites
3 hours ago, flydev said:

what is this check ? I don't have it in my code

the if(false is my php replacement for #if 0 😉

  • Like 2

Share this post


Link to post
Share on other sites
12 hours ago, Autofahrn said:

Do you see that file on your server?

Nope, that's the strange part. The other installs run the same PW versions, same file permissions, same PHP version. 

  • Like 1

Share this post


Link to post
Share on other sites

I've just re-read the previous posts and see that CLOSING ZIP actually happens after 0.0 seconds of operation. This is pretty unlikely since I expect some time for directory enumeration and packaging the first part. Can you send me that verbose log via PM?

  • Like 1

Share this post


Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.


  • Recently Browsing   0 members

    No registered users viewing this page.

  • Similar Content

    • By jaro
      This module (github) does with site/assets/files what Ryan's DatabaseBackups module does with the database:
      Backup site/assets Download ZIP archive Upload ZIP archive Restore site/assets Motivation: This module can be the missing part for projects with content backup responsibility on the client's side: The client will be able to download DB and assets/files snapshots through the backend without filesystem access, thus backing up all content themselves.
      Release state alpha – do not use in production environments.
      Credits for the nice UI go to @ryan – I reused most of it and some other code from the DatabaseBackups module.
    • By David Karich
      ProcessWire InputfieldRepeaterMatrixDuplicate
      Thanks to the great ProModule "RepeaterMatrix" I have the possibility to create complex repeater items. With it I have created a quite powerful page builder. Many different content modules, with many more possible design options. The RepeaterMatrix module supports the cloning of items, but only within the same page. Now I often have the case that very design-intensive pages and items are created. If you want to use a content module on a different page (e.g. in the same design), you have to rebuild each item manually every time.
      This module extends the commercial ProModule "RepeaterMatrix" by the function to duplicate repeater items from one page to another page. The condition is that the target field is the same matrix field from which the item is duplicated. This module is currently understood as proof of concept. There are a few limitations that need to be considered. The intention of the module is that this functionality is integrated into the core of RepeaterMatrix and does not require an extra module.
      Check out the screencast
      What the module can do
      Duplicate a repeater item from one page to another No matter how complex the item is Full support for file and image fields Multilingual support Support of Min and Max settings Live synchronization of clipboard between multiple browser tabs. Copy an item and simply switch the browser tab to the target page and you will immediately see the past button Support of multiple RepeaterMatrix fields on one page Configurable which roles and fields are excluded Duplicated items are automatically pasted to the end of the target field and set to hidden status so that changes are not directly published Automatic clipboard update when other items are picked Automatically removes old clipboard data if it is not pasted within 6 hours Delete clipboard itself by clicking the selected item again Benefit: unbelievably fast workflow and content replication What the module can't do
      Before an item can be duplicated in its current version, the source page must be saved. This means that if you make changes to an item and copy this, the old saved state will be duplicated Dynamic loading is currently not possible. Means no AJAX. When pasting, the target page is saved completely No support for nested repeater items. Currently only first level items can be duplicated. Means a repeater field in a repeater field cannot be duplicated. Workaround: simply duplicate the parent item Dynamic reloading and adding of repeater items cannot be registered. Several interfaces and events from the core are missing. The initialization occurs only once after the page load event Changelog
      1.0.4
      Bug fix: Various bug fixes and improvements in live synchronization Bug fix: Items are no longer inserted when the normal save button is clicked. Only when the past button is explicitly clicked Feature: Support of multiple repeater fields in one page Feature: Support of repeater Min/Max settings Feature: Configurable roles and fields Enhancement: Improved clipboard management Enhancement: Documentation improvement Enhancement: Corrected few typos #1 1.0.3
      Feature: Live synchronization Enhancement: Load the module only in the backend Enhancement: Documentation improvement 1.0.2
      Bug fix: Various bug fixes and improvements in JS functions Enhancement: Documentation improvement Enhancement: Corrected few typos 1.0.1
      Bug fix: Various bug fixes and improvements in the duplication process 1.0.0
      Initial release Support this module
      If this module is useful for you, I am very thankful for your small donation: Donate 5,- Euro (via PayPal – or an amount of your choice. Thank you!)
      Download this module
      > Github: https://github.com/FlipZoomMedia/InputfieldRepeaterMatrixDuplicate
      > PW module directory: – soon –
    • By NorbertH
      Is there a hook to do something right after cloning a page ?
      I want the page to be saved right after cloning it either from the button in the tree or from a lister, because saving the page triggers several calculations that are not triggered by just cloning the page.
       
      Thanks in advance !
    • By John W.
      Question 1
      I recently installed PW 3.0.62 for a new site and also have sites running older version of PW 3.x.
      Can I export the database on an older version of PW 3.x and import it to PW 3.0.62 without any issues?
       
      Question 2
      (This is kind of alternative to the above for long term use - and maybe a better solution...)
      On  the sites I've previously built I have templates (home, basic-page, contact) and fields that I commonly use, such as business_name, phone_1.  The last site I built is running PW 3.0.42.  I was considering cloning this into a local site and running the upgrade module to bring it up to PW 3.0.62. From there on out when I start I new project I could just run the PW upgrade module, copy the folder to the location for the new project and duplicate the database using the new projects name.

      So basically, I'll always keep a "blank slate" site that I can just run the PW upgrade on, then duplicate into a new project. This would cut down on the work and time spent having to re-create these common fields, that I use. From there, I would just add fields, templates, etc, specific for the new website project.

      Is this a sound approach to speed up development?

       
×
×
  • Create New...