Jump to content
flydev 👊🏻

☁️ Duplicator: Backup and move sites

Recommended Posts

Thanks @flydev 👊🏻 for your hard work on this. I installed on a Debian server and had a few issues which I have managed to fix. I submitted a bit of a messed up PR - I think we were both working at the same time.

All my tests were in native mode

1) The server must have the "zip" package installed - mine didn't which caused me some grief for a while. I suggest looking for this and warning if not installed.

2) I had to change the mysqldump command quite a bit - you can see in the PR, but key things were --single-transaction and specifying the DB and removing the --skip-lock-tables 

3) Had to chmod the .sh script to 744 so it was executable, but obviously this may not work on all servers depending on the owner etc.

4) Had to change the directory and the exec call to actually execute the .sh file - it was failing to run otherwise.

With those changes everything seems to work great, although I did notice that after running a duplication via the Process module, it no longer reloads the page when done so you don't see the new package unless you manually reload when it's finished.

Thanks again!

 

  • Like 2
  • Thanks 1

Share this post


Link to post
Share on other sites

Thanks you very much for this report @adrian

1 hour ago, adrian said:

1) The server must have the "zip" package installed - mine didn't which caused me some grief for a while. I suggest looking for this and warning if not installed.

Already on the todo list.

1 hour ago, adrian said:

2) I had to change the mysqldump command quite a bit - you can see in the PR, but key things were --single-transaction and specifying the DB and removing the --skip-lock-tables 

I will check the PR after this message, but this can depend on the user end database schema used. We will have on the next version multitude options to customize the behavior of mysqldump. I was thinking too to give a textbox to enter some custom swicth for the command-line before proceeding.

1 hour ago, adrian said:

3) Had to chmod the .sh script to 744 so it was executable, but obviously this may not work on all servers depending on the owner etc.

Thanks, I will make some checks available to the destination dir of the script.

1 hour ago, adrian said:

4) Had to change the directory and the exec call to actually execute the .sh file - it was failing to run otherwise.

I am sure this is because you have safe_mode enabled, if you can confirm me that, it would be awesome.

  

1 hour ago, adrian said:

With those changes everything seems to work great, although I did notice that after running a duplication via the Process module, it no longer reloads the page when done so you don't see the new package unless you manually reload when it's finished.

Yes it's for debugging purpose and to avoid the page reloading - just set the var $debug to false in ProcessDuplicator.js, 

 

Thanks again, going to check the PR.

Edited by flydev 👊🏻
last answer
  • Like 2

Share this post


Link to post
Share on other sites

Just another thing, the next minor version will contain an encryption setting, trying to make Duplicator a bit GDPR compliant.

  • Like 3

Share this post


Link to post
Share on other sites
4 hours ago, flydev 👊🏻 said:

I am sure this is because you have safe_mode enabled, if you can confirm me that, it would be awesome.

Are you talking about PHP's "safe_mode" ? I am thinking not because it hasn't been available since php 5.4. Sorry is there another safe_mode setting I'm not thinking about?

  • Like 1

Share this post


Link to post
Share on other sites

@flydev 👊🏻 @adrian

Thanks for all your had work on this invaluable module.

I've updated a few sites to the latest Duplicator version. But have been running into an issue when I have AWS Backups enabled.

When AWS Backups is enabled and going to Setup -> Duplicator, I get a Call to a member function getTimestamp() on bool Error.

See attached screenshot.

The site core and modules are all up to date as of today (Jan 14, 2020).

Using php 7.3, got the same error w/ php 7.2.

Amazon libraries installed w/ Composer.

Screen Shot 2020-01-14 at 2.11.26 PM.png

  • Thanks 1

Share this post


Link to post
Share on other sites

I just tested it and the process ended without error 🤦🏻‍♂️

756401006_Annotation2020-01-15005128.png.fd7ab0d0ded9f69d833b7630497a63c9.png

 

What's the version of your Duplicator ?

Edit1:

Ok got it. Will push a fix.

Edit2: 

You can update the module to the version 1.3.15

Edited by flydev 👊🏻
fix - new version
  • Like 1

Share this post


Link to post
Share on other sites

Hi Flydev,

Fantastic, thanks for the quick fix.

I'm no longer getting the error.

However, I do see an invalid timestamp display in the Created column when viewing the list of backups for my AWS backups.

 

Screen Shot 2020-01-14 at 6.40.31 PM.png

  • Like 1

Share this post


Link to post
Share on other sites
15 hours ago, GhostRider said:

However, I do see an invalid timestamp display in the Created column when viewing the list of backups for my AWS backups.

I fixed the issue (thanks for reporting it) so you could access the process module again, but I don't have idea at this moment where is or what's the error.

Could you try to make others AWS backups with different website and post the result ? Meanwhile I will re-read the code to see if I can catch something.

  • Like 2

Share this post


Link to post
Share on other sites

Screenshot added of another site giving the same "Invalid Timestamp" under the created column.

The "Invalid Timestamp" message is only for the AWS backups.

It's not really an issue as the database name has a timestamp.

Everything else is working great.

Screen Shot 2020-01-15 at 7.27.44 PM.png

  • Thanks 1

Share this post


Link to post
Share on other sites

Hi Flydev,

I have been thinking about one feature request if you are taking any. I don't want to add more work, but perhaps something to think about.

On the Duplicator settings page, I have used the "Files and folders excluded from" checkbox (very handy) and also the "Custom excluded files and paths'.

I'm wondering if there is any way of excluding any custom file directories, rather than file extensions?

Here are a few examples:

1. After adding the AWS library via composer, Duplicator is backing up the Composer directory called vendor (at the same level as wire, files etc...). This file has become quite large over time.

2. We also at times create directories at the same level for legacy files when building a new client sites, where the legacy files need to still be accessible. Our latest client had a very large media folder nearly 7GB. ie /media/

3. When using ProCache, Proche adds two folders in the assets directory called ProCache-###### and pwpc. The ProCache-#### file is a collection of all the static versions of the pages on the site, which can also become quite large depending on the site.

With most of these directories, we do have a good idea what file extensions are being used and can exclude them in the 'Custom excluded files and paths' section.

However, with the Composer vendor directory, there are so many file extensions being used with a very deep nested group of folders. This makes for a very large list of possibilities to exclude.

With a few of our sites, our backups went from ~90MB to over 700MB.

 

It would be great to hear your thoughts on hiding directories/folders when you have time.

Below is a list of some of the file paths we struggle with.

 

/vendor or any custom directory at this level

/site/assets/ProCache-###

/site/assets/pwpw/*

 

Thanks

  • Like 1

Share this post


Link to post
Share on other sites

@GhostRider, what's wrong with the 'Custom excluded files and paths' box? To exclude a path simply enter it without a file pattern like:

/site/assets/pwpw/
/vendorOrAnyCustomDirectoryAtThisLevel/

To exclude the ProCache stuff using a regular expression should do:

%/ProCache-[^/]+$%  # Ignore ProCache directory

To exclude image variants you may try this:

%\.\d+x\d+\.[^/]+$%  # Ignore variants created by PW

  • Like 2

Share this post


Link to post
Share on other sites

My apologies, this was user error on my part.

I was using the full server paths, rather than the site path.

/home/accountname/public_html/site/vendor/ rather than /site/vendor/

@Autofahrn thanks for your post.

 

  • Like 2

Share this post


Link to post
Share on other sites
8 hours ago, GhostRider said:

I have been thinking about one feature request if you are taking any. I don't want to add more work, but perhaps something to think about.

No problem, as Andreas said, everything you requested is already implemented. If you have other features in mind, do not hesitate guys 💪

Also, with the next version, you will be able to backup the big site with Duplicator. I can make backup of a 1.6GO website without issue. It can be already tested - it's on the dev branch on Github.


 

Spoiler

Meanwhile, I am developing the documentation site of Duplicator under ProcessWire and VueJS. Note aside, this profile will be available for the community..

dup-Annot.thumb.png.14fea43120153157e373ef1ce96ac103.png

 

 

Edited by flydev 👊🏻
Infos
  • Like 4

Share this post


Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.


  • Recently Browsing   0 members

    No registered users viewing this page.

  • Similar Content

    • By David Karich
      ProcessWire InputfieldRepeaterMatrixDuplicate
      Thanks to the great ProModule "RepeaterMatrix" I have the possibility to create complex repeater items. With it I have created a quite powerful page builder. Many different content modules, with many more possible design options. The RepeaterMatrix module supports the cloning of items, but only within the same page. Now I often have the case that very design-intensive pages and items are created. If you want to use a content module on a different page (e.g. in the same design), you have to rebuild each item manually every time.
      This module extends the commercial ProModule "RepeaterMatrix" by the function to duplicate repeater items from one page to another page. The condition is that the target field is the same matrix field from which the item is duplicated. This module is currently understood as proof of concept. There are a few limitations that need to be considered. The intention of the module is that this functionality is integrated into the core of RepeaterMatrix and does not require an extra module.
      Check out the screencast
      What the module can do
      Duplicate a repeater item from one page to another No matter how complex the item is Full support for file and image fields Multilingual support Support of Min and Max settings Live synchronization of clipboard between multiple browser tabs. Copy an item and simply switch the browser tab to the target page and you will immediately see the past button Support of multiple RepeaterMatrix fields on one page Configurable which roles and fields are excluded Duplicated items are automatically pasted to the end of the target field and set to hidden status so that changes are not directly published Automatic clipboard update when other items are picked Automatically removes old clipboard data if it is not pasted within 6 hours Delete clipboard itself by clicking the selected item again Benefit: unbelievably fast workflow and content replication What the module can't do
      Before an item can be duplicated in its current version, the source page must be saved. This means that if you make changes to an item and copy this, the old saved state will be duplicated Dynamic loading is currently not possible. Means no AJAX. When pasting, the target page is saved completely No support for nested repeater items. Currently only first level items can be duplicated. Means a repeater field in a repeater field cannot be duplicated. Workaround: simply duplicate the parent item Dynamic reloading and adding of repeater items cannot be registered. Several interfaces and events from the core are missing. The initialization occurs only once after the page load event Changelog
      1.0.4
      Bug fix: Various bug fixes and improvements in live synchronization Bug fix: Items are no longer inserted when the normal save button is clicked. Only when the past button is explicitly clicked Feature: Support of multiple repeater fields in one page Feature: Support of repeater Min/Max settings Feature: Configurable roles and fields Enhancement: Improved clipboard management Enhancement: Documentation improvement Enhancement: Corrected few typos #1 1.0.3
      Feature: Live synchronization Enhancement: Load the module only in the backend Enhancement: Documentation improvement 1.0.2
      Bug fix: Various bug fixes and improvements in JS functions Enhancement: Documentation improvement Enhancement: Corrected few typos 1.0.1
      Bug fix: Various bug fixes and improvements in the duplication process 1.0.0
      Initial release Support this module
      If this module is useful for you, I am very thankful for your small donation: Donate 5,- Euro (via PayPal – or an amount of your choice. Thank you!)
      Download this module
      > Github: https://github.com/FlipZoomMedia/InputfieldRepeaterMatrixDuplicate
      > PW module directory: https://modules.processwire.com/modules/inputfield-repeater-matrix-duplicate/
    • By jaro
      This module (github) does with site/assets/files what Ryan's DatabaseBackups module does with the database:
      Backup site/assets Download ZIP archive Upload ZIP archive Restore site/assets Motivation: This module can be the missing part for projects with content backup responsibility on the client's side: The client will be able to download DB and assets/files snapshots through the backend without filesystem access, thus backing up all content themselves.
      Release state alpha – do not use in production environments.
      Credits for the nice UI go to @ryan – I reused most of it and some other code from the DatabaseBackups module.
    • By NorbertH
      Is there a hook to do something right after cloning a page ?
      I want the page to be saved right after cloning it either from the button in the tree or from a lister, because saving the page triggers several calculations that are not triggered by just cloning the page.
       
      Thanks in advance !
    • By John W.
      Question 1
      I recently installed PW 3.0.62 for a new site and also have sites running older version of PW 3.x.
      Can I export the database on an older version of PW 3.x and import it to PW 3.0.62 without any issues?
       
      Question 2
      (This is kind of alternative to the above for long term use - and maybe a better solution...)
      On  the sites I've previously built I have templates (home, basic-page, contact) and fields that I commonly use, such as business_name, phone_1.  The last site I built is running PW 3.0.42.  I was considering cloning this into a local site and running the upgrade module to bring it up to PW 3.0.62. From there on out when I start I new project I could just run the PW upgrade module, copy the folder to the location for the new project and duplicate the database using the new projects name.

      So basically, I'll always keep a "blank slate" site that I can just run the PW upgrade on, then duplicate into a new project. This would cut down on the work and time spent having to re-create these common fields, that I use. From there, I would just add fields, templates, etc, specific for the new website project.

      Is this a sound approach to speed up development?

       
×
×
  • Create New...