☁️ Duplicator: Backup and move sites

[adrian]

Thanks @flydev ?? for your hard work on this. I installed on a Debian server and had a few issues which I have managed to fix. I submitted a bit of a messed up PR - I think we were both working at the same time.

All my tests were in native mode

1) The server must have the "zip" package installed - mine didn't which caused me some grief for a while. I suggest looking for this and warning if not installed.

2) I had to change the mysqldump command quite a bit - you can see in the PR, but key things were --single-transaction and specifying the DB and removing the --skip-lock-tables 

3) Had to chmod the .sh script to 744 so it was executable, but obviously this may not work on all servers depending on the owner etc.

4) Had to change the directory and the exec call to actually execute the .sh file - it was failing to run otherwise.

With those changes everything seems to work great, although I did notice that after running a duplication via the Process module, it no longer reloads the page when done so you don't see the new package unless you manually reload when it's finished.

Thanks again!

 

[flydev]

Thanks you very much for this report @adrian

1 hour ago, adrian said:

1) The server must have the "zip" package installed - mine didn't which caused me some grief for a while. I suggest looking for this and warning if not installed.

Already on the todo list.

1 hour ago, adrian said:

2) I had to change the mysqldump command quite a bit - you can see in the PR, but key things were --single-transaction and specifying the DB and removing the --skip-lock-tables 

I will check the PR after this message, but this can depend on the user end database schema used. We will have on the next version multitude options to customize the behavior of mysqldump. I was thinking too to give a textbox to enter some custom swicth for the command-line before proceeding.

1 hour ago, adrian said:

3) Had to chmod the .sh script to 744 so it was executable, but obviously this may not work on all servers depending on the owner etc.

Thanks, I will make some checks available to the destination dir of the script.

1 hour ago, adrian said:

4) Had to change the directory and the exec call to actually execute the .sh file - it was failing to run otherwise.

I am sure this is because you have safe_mode enabled, if you can confirm me that, it would be awesome.

  

1 hour ago, adrian said:

With those changes everything seems to work great, although I did notice that after running a duplication via the Process module, it no longer reloads the page when done so you don't see the new package unless you manually reload when it's finished.

Yes it's for debugging purpose and to avoid the page reloading - just set the var $debug to false in ProcessDuplicator.js, 

 

Thanks again, going to check the PR.

Edited January 13, 2020 by flydev ??
last answer

[flydev]

Just another thing, the next minor version will contain an encryption setting, trying to make Duplicator a bit GDPR compliant.

[adrian]

4 hours ago, flydev ?? said:

I am sure this is because you have safe_mode enabled, if you can confirm me that, it would be awesome.

Are you talking about PHP's "safe_mode" ? I am thinking not because it hasn't been available since php 5.4. Sorry is there another safe_mode setting I'm not thinking about?

[flydev]

Yes I had in mind safe_mode_exec_dir but I forgot that it was removed.

[GhostRider]

@flydev ?? @adrian

Thanks for all your had work on this invaluable module.

I've updated a few sites to the latest Duplicator version. But have been running into an issue when I have AWS Backups enabled.

When AWS Backups is enabled and going to Setup -> Duplicator, I get a Call to a member function getTimestamp() on bool Error.

See attached screenshot.

The site core and modules are all up to date as of today (Jan 14, 2020).

Using php 7.3, got the same error w/ php 7.2.

Amazon libraries installed w/ Composer.

[flydev]

I just tested it and the process ended without error ??‍♂️

 

What's the version of your Duplicator ?

Edit1:

Ok got it. Will push a fix.

Edit2: 

You can update the module to the version 1.3.15

Edited January 15, 2020 by flydev ??
fix - new version

[GhostRider]

Hi Flydev,

Fantastic, thanks for the quick fix.

I'm no longer getting the error.

However, I do see an invalid timestamp display in the Created column when viewing the list of backups for my AWS backups.

 

[flydev]

15 hours ago, GhostRider said:

However, I do see an invalid timestamp display in the Created column when viewing the list of backups for my AWS backups.

I fixed the issue (thanks for reporting it) so you could access the process module again, but I don't have idea at this moment where is or what's the error.

Could you try to make others AWS backups with different website and post the result ? Meanwhile I will re-read the code to see if I can catch something.

[GhostRider]

Screenshot added of another site giving the same "Invalid Timestamp" under the created column.

The "Invalid Timestamp" message is only for the AWS backups.

It's not really an issue as the database name has a timestamp.

Everything else is working great.

[GhostRider]

Hi Flydev,

I have been thinking about one feature request if you are taking any. I don't want to add more work, but perhaps something to think about.

On the Duplicator settings page, I have used the "Files and folders excluded from" checkbox (very handy) and also the "Custom excluded files and paths'.

I'm wondering if there is any way of excluding any custom file directories, rather than file extensions?

Here are a few examples:

1. After adding the AWS library via composer, Duplicator is backing up the Composer directory called vendor (at the same level as wire, files etc...). This file has become quite large over time.

2. We also at times create directories at the same level for legacy files when building a new client sites, where the legacy files need to still be accessible. Our latest client had a very large media folder nearly 7GB. ie /media/

3. When using ProCache, Proche adds two folders in the assets directory called ProCache-###### and pwpc. The ProCache-#### file is a collection of all the static versions of the pages on the site, which can also become quite large depending on the site.

With most of these directories, we do have a good idea what file extensions are being used and can exclude them in the 'Custom excluded files and paths' section.

However, with the Composer vendor directory, there are so many file extensions being used with a very deep nested group of folders. This makes for a very large list of possibilities to exclude.

With a few of our sites, our backups went from ~90MB to over 700MB.

 

It would be great to hear your thoughts on hiding directories/folders when you have time.

Below is a list of some of the file paths we struggle with.

 

/vendor or any custom directory at this level

/site/assets/ProCache-###

/site/assets/pwpw/*

 

Thanks

[Autofahrn]

@GhostRider, what's wrong with the 'Custom excluded files and paths' box? To exclude a path simply enter it without a file pattern like:

/site/assets/pwpw/
/vendorOrAnyCustomDirectoryAtThisLevel/

To exclude the ProCache stuff using a regular expression should do:

%/ProCache-[^/]+$%  # Ignore ProCache directory

To exclude image variants you may try this:

%\.\d+x\d+\.[^/]+$%  # Ignore variants created by PW

[GhostRider]

My apologies, this was user error on my part.

I was using the full server paths, rather than the site path.

/home/accountname/public_html/site/vendor/ rather than /site/vendor/

@Autofahrn thanks for your post.

 

[flydev]

8 hours ago, GhostRider said:

I have been thinking about one feature request if you are taking any. I don't want to add more work, but perhaps something to think about.

No problem, as Andreas said, everything you requested is already implemented. If you have other features in mind, do not hesitate guys ?

Also, with the next version, you will be able to backup the big site with Duplicator. I can make backup of a 1.6GO website without issue. It can be already tested - it's on the dev branch on Github.

 

Spoiler

Meanwhile, I am developing the documentation site of Duplicator under ProcessWire and VueJS. Note aside, this profile will be available for the community..

 

 

Edited January 17, 2020 by flydev ??
Infos

[shogun]

UPDATE: Solved. I had gulp and the browsersync node module running while I packaged up which added a weird script in the installer.php page. I found it and removed it and re-ran. All good now.

When I upload the package and installer and go to the installer.php file on the server i get this error:

Parse error: syntax error, unexpected '__bs_script__' (T_STRING), expecting ',' or ';' in /...n/installer.php on line 334

Edited January 29, 2020 by shogun
SOLVED

[shogun]

I love this module! I'm wondering though, how to make this easier as we build out a site's cms.

Is there a way for when I'm working locally and I create new fields or have some new CMS content updates locally, is there a way to sync up or easily push this to the cloned remote site's database without having to create a whole new backup, upload, create db all over again on the second server?

[d'Hinnisdaël]

9 hours ago, shogun said:

Is there a way for when I'm working locally and I create new fields or have some new CMS content updates locally, is there a way to sync up or easily push this to the cloned remote site's database without having to create a whole new backup, upload, create db all over again on the second server?

Have you tried the Migrations module? It's a bit of overhead to create migrations classes instead of using the admin UI, but it's nice to have reproducible template and field changes across all environments.

Then you'd only need to mirror or sync /sites/assets/files/ and /site/templates/ somehow.

 

[saschapi]

Hi Guys,

I'm trying to get Dublicator up and running as a automated backup solution. While it is working as expected if I trigger it manually, I'm still figuring out the automation. But this will come ?

Anyway. I wanted to report that if you delete packages via the Package Manager, the corresponding logs do not get deleted. This results in Packages shown in the summary on the config page. (see attachment). This might be a wanted behaviour but confused me. ? This could be fixed by deleting the logs manually via FTP.

Cheers Sascha

[flydev]

HI @saschapi 

Thanks for reporting this issue, I also noticed it and fixed it in the dev version. 

The next stable version (which come with native backups) should be pushed next week.

  

On 2/14/2020 at 10:23 AM, saschapi said:

I'm still figuring out the automation.

Just make a scheduled task which point to the CRON Duplicator's helper file. If you need further help, just ask it here ?

 

On 1/29/2020 at 3:22 AM, shogun said:

I love this module! I'm wondering though, how to make this easier as we build out a site's cms.

Hello @shogun ,  I have the same requirement and I am using what @d'Hinnisdaël suggested. 

I use Migration with a self made GUI and Duplicator for daily backups.
 

Edited February 17, 2020 by flydev ??
more answer.

[B3ta]

On 2/17/2020 at 7:14 PM, flydev ?? said:

The next stable version (which come with native backups) should be pushed next week.

 

Any news on the new stable version? Currently i'm getting this error when I'm trying to activate the module. 

[flydev]

Hi @B3ta

I had too much work this week so its planned for the next tuesday.

Can you upload again the screenshot please ? I can't see it. Thanks you.

[B3ta]

Thank you for the work you put into building this plugin and for the great news!  

23 hours ago, flydev ?? said:

Can you upload again the screenshot please?

Here is the error i'm getting once I try to activate the dropbox backup option. 

[flydev]

@B3ta

I don't have the same code on this line 32, look at the repo there : https://github.com/flydev-fr/Duplicator/blob/v1.3.14/Classes/DupLogs.php#L32

 

I tested Dropbox before posting and its working (even if it look like there is a small issue with the timestamp).

FYI, the latest version is 1.3.14 and you should update your Duplicator I think, let me know ?

[Frank Vèssia]

Hello,
I have and issue with php 7.4 about a deprecated function get_magic_quotes_gpc() used inside the installer.php...any quick fix ? Thanks in advance

[Autofahrn]

Since get_magic_quotes_gpc returns FALSE from PHP 5.4, the installer line

$value = get_magic_quotes_gpc() ? stripslashes($_POST[$field]) : $_POST[$field];

safely could be replaced with

$value = $_POST[$field];

(assuming you are not installing on a legacy platform)