☁️ Duplicator: Backup and move sites

[flydev]

@adrian  @Claus fixes pushed - thanks you ?

[Claus]

Awesome, thank you! I am getting ‘Error reported by web service: That module is not currently tracked by the modules directory’ when I try to update the status in ‘processwire/module/edit?name=ProcessDuplicator’

I was not in the correct module… sorry.

[kalimati]

One of my websites using Processwire 3.0.184 and Duplicator 1.4.19. It was creating backups till 6 months back. I tried again today and although it starts the process it cannot complete it. It does not throw any error or display any message, just stops. The incomplete zip filw can be seen in the /site/assets/backups directory with a name like 2022-03-22_18-57-17-websitename.com.package2.zip.DOmR6A.

Any idea what could be going wrong?

[flydev]

@kalimati could you please give me more information by checking :

1. generated log if it exist located in `site/assets/backups`
2. ProcessWire error log
3. PHP(-fpm) error log

then reported back here please. Thanks.

 

Edit: 

• Are you aware of an eventual hosting provider upgrade ?
• Did the website and/or database as grow since last backup ?

[kalimati]

Thanks for your response, flydev.

Here are the three error logs but I don't see any clue in it.

Duplicator Logs

Quote

22 hours ago
2022-03-22 18:57:05     kalimati-jfitch     setup/duplicator/?action=backup_now     - Backup using standard mode
22 hours ago
2022-03-22 18:57:05     kalimati-jfitch     setup/duplicator/?action=backup_now     Backup Database
6 months ago
2021-10-04 20:19:03     kalimati-jfitch     setup/duplicator/?action=backup_now     - job finished in 51.702124sec
6 months ago
2021-10-04 20:19:03     kalimati-jfitch     setup/duplicator/?action=backup_now     - package saved in local folder: 2021-10-04_20-18-18-machinerylubricationindia.com.package2.zip
6 months ago
2021-10-04 20:19:03     kalimati-jfitch     setup/duplicator/?action=backup_now     - package built successfully in 51.701014sec

Processwire error log

Quote

4 days ago
2022-03-20 03:43:51     guest     /blog/think-affordable-artificial-blast/     Fatal Error: Uncaught Error: Call to undefined function displayAds() in /site/templates/_main.php:79 Stack trace:
1. /wire/core/TemplateFile.php(330): require()
2. /wire/core/Wire.php(394): TemplateFile->___render()
3. /wire/core/WireHooks.php(823): Wire->_callMethod('___render', Array)
4. /wire/core/Wire.php(465): WireHooks->runHooks(Object(TemplateFile), 'render', Array)
5. /wire/modules/PageRender.module(536): Wire->__call('render', Array)
6. /wire/core/Wire.php(397): PageRender->___renderPage(Object(HookEvent))
7. /wire/core/WireHooks.php(823): Wire->_callMethod('___renderPage', Array)
8. /wire/core/Wire.php(465): WireHooks->runHooks(Object(PageRende
Line 79 of /site/templates/_main.php
4 days ago
2022-03-20 03:43:43     guest     /blog/phase-data-extended-transaction/     Fatal Error: Uncaught Error: Call to undefined function displayAds() in /site/templates/_main.php:79 Stack trace:
1. /wire/core/TemplateFile.php(330): require()
2. /wire/core/Wire.php(394): TemplateFile->___render()
3. /wire/core/WireHooks.php(823): Wire->_callMethod('___render', Array)
4. /wire/core/Wire.php(465): WireHooks->runHooks(Object(TemplateFile), 'render', Array)
5. /wire/modules/PageRender.module(536): Wire->__call('render', Array)
6. /wire/core/Wire.php(397): PageRender->___renderPage(Object(HookEvent))
7. /wire/core/WireHooks.php(823): Wire->_callMethod('___renderPage', Array)
8. /wire/core/Wire.php(465): WireHooks->runHooks(Object(PageRende
Line 79 of /site/templates/_main.php

PHP Error log

Quote

 

[29-Dec-2020 20:44:00 Asia/Kolkata] PHP Fatal error:  Exception: SQLSTATE[HY000] [1045] Access denied for user 'pw_mli'@'localhost' (using password: YES) (in /home2/machinm3/public_html/mli2/wire/core/ProcessWire.php line 494)

#0 /home2/machinm3/public_html/mli2/wire/core/ProcessWire.php(302): ProcessWire\ProcessWire->load(Object(ProcessWire\Config))
#1 /home2/machinm3/public_html/mli2/index.php(52): ProcessWire\ProcessWire->__construct(Object(ProcessWire\Config))
#2 {main} in /home2/machinm3/public_html/mli2/index.php on line 64
[29-Dec-2020 20:44:00 Asia/Kolkata] PHP Warning:  Cannot modify header information - headers already sent by (output started at /home2/machinm3/public_html/mli2/index.php:64) in /home2/machinm3/public_html/mli2/wire/core/WireShutdown.php on line 306
[29-Dec-2020 20:44:52 Asia/Kolkata] PHP Fatal error:  Exception: SQLSTATE[HY000] [1045] Access denied for user 'pw_mli2'@'localhost' (using password: YES) (in /home2/machinm3/public_html/mli2/wire/core/ProcessWire.php line 494)

 

I am not aware of any eventual hosting provider upgrade.

Yes the database has grown since last backup.

[flydev]

Mmh from what I read, the issue doesn't appear to come from Duplicator but from an issue in /site/templates/_main.php line 79 on the call of displayAds which doesn't exist in Duplicator.

 

[kalimati]

Thanks, that was it.

Actually that was being generated on sample pages (hidden) from a profile that I had left in the tree. displayAds was not supposed to function there. I trashed the pages and Duplicator worked.

But curious as to how a code on a template should affect Duplicator.

 

[flydev]

3 hours ago, kalimati said:

But curious as to how a code on a template should affect Duplicator.

This is because Duplicator, as ProcessWire module, rely on ProcessWire code and thus, the PHP interpreter.

Then after saying that, you will remember that in PHP you have basically four (4) errors - notice, warning, parse and fatal error, and you will also remember that a notice give you a hint generally for an undefined variable but do not stop the code, the second tell you that you will end up scratching your head in the future but again do not stop the code. The third tell you that PHP can't parse the code and logically stop the code, as well as the last - the same as you got in - clearly say that the error is fatal and stop the code. At this point, your whole app is broken.

If we were using an external tool to make the package/backup, we would not encounter any error. This is the main answer of your question. Hope it help ??

 

[flydev]

?

An issue was reported on github about the FTP functionality which was not working on PHP-8.1, which is fixed in the dev branch (Duplicator v1.4.24).

As I didn't tested to backup a thing on GoogleDrive and Amazon S3, if you spot other issues, please feel free to report it in this same thread or fill a github issue. It might also be the time to implement something related to GDPR (package encryption) which I think was already discussed somewhere. Thanks you.

[flydev]

Another update (v1.4.25) with a new setting available to customize the permission set on duplicator.sh when using the Native Mode feature. Thanks @netcarver

It also delete the corresponding log file.

 

ℹ️ The module can be upgraded again trough ProcessWireUpgrade, I was only updating a static var forgetting the module version field and then the module version was not reflected in the modules directory.. ?‍♂️

[flydev]

After reading again Steve's messages, I added a new setting (enabled by default) which if set, delete orphaned logs files from the folder.

Update to v1.4.26 to get that.

[James Morris]

On 9/13/2022 at 11:29 PM, flydev said:

As I didn't tested to backup a thing on GoogleDrive and Amazon S3, if you spot other issues, please feel free to report it in this same thread or fill a github issue. 

 

I am getting an error of 'Class "DUP_Logs" not found' when upgrading to the latest version (1.4.26) when using the Amazon S3 option.  The provided SDK is on place, all was working fine before the upgrade.  Reverting back to 1.4.0 solved the problem.  Here is the full call stack for reference:

 

Thanks for a great module!

EDIT: I should note that the site is on PHP 8 and the latest stable version of PW

[flydev]

I will add a new answer once I have made some tests, comments are welcome.

https://github.com/flydev-fr/Duplicator/issues/37

[protro]

Thank you for this terrific module. Sorry if basic question: What is best methodology for simply updating the database on a production server if the rest of the site content / processwire core have already been deployed?

[flydev]

@protro thanks ?

If you have already your site content, you have two simple solutions:

A. Using Duplicator:

1. build a package
2. remove all files and folders from the package2.zip and keep only the .sql.zip inside
3. upload the installer.php file and the modified package2.zip on the root directory
4. navigate to https://yoursite.com/installer.php and follow the instructions

The modified package2.zip file content ?

 

B. Using ProcessDatabaseBackups

1. install the module
2. upload or select a db file
3. restore it

⚠️ As always, make a backup of everything before proceeding

 

[flydev]

Incoming update of Duplicator v1.5

---

Encryption is being implemented, if you want to give a try, the feature is available on the branch feature-encryption :

-  https://github.com/flydev-fr/Duplicator/tree/feature-encryption

More infos and Instructions can be found by reading or running the script generate-keys.php on a terminal.

---

I shared infos with Steve, I also want to share it with you :

 => Test on a Package of 687MB on PHP8.1

Quote

2023-03-22 21:30:55    guest   http://processwire.sek/http404/  Backup Database
2023-03-22 21:30:55    guest   http://processwire.sek/http404/  - Backup using native tools
2023-03-22 21:31:14    guest   http://processwire.sek/http404/  - package built successfully in 18.953901sec
2023-03-22 21:31:14    guest   http://processwire.sek/http404/  - package encryption starting...
2023-03-22 23:19:16    guest   http://processwire.sek/http404/  - package encryption success, took 6482.159817sec
2023-03-22 23:19:16    guest   http://processwire.sek/http404/  - package decryption staring...
2023-03-23 01:58:47    guest   http://processwire.sek/http404/  - package decryption success, took 9571.181766sec
2023-03-23 01:58:47    guest   http://processwire.sek/http404/  - job finished in 16072.312503sec

Do not take care of performance info given on this post, see ?

[szabesz]

40 minutes ago, flydev said:

 => Test on a Package of 687MB on PHP8.1

Under 4 hours 27 mins 52 secs? Protecting data is very expensive for sure. It's just that lawmakers don't give a damn about that because it's not they who need to pay the price.

Anyway, thank you for advancing Duplicator!

[flydev]

4 hours ago, szabesz said:

Under 4 hours 27 mins 52 secs?

Yes, but was the whole process, encrypting the 687MB and decrypting it. Encrypting took nearly 1h48. On my side, it's not usable as I have some databases of 7GB or even 11GB, and a custom tool handle it, but of course, it's only available on private server where we have the hand on it. I will launch this afternoon a test on a Windows Server with Xeon CPU and 32GB of RAM, let's see..

The issue of processing time will be tackled, but only on a private server. Anyway, having a huge database or file size on a shared server should not be too frequent (I have no stats to look at..). 

 

Do not take care of performance info given on this post, see ?

[flydev]

Then I must say that the bad performances was due to my machine and/or setup (caddy/php-fpm, thats is quite bad and sad on Windows).

On the Windows Server, performance are ? !

A backup of a setup of 2.11GB, once compressed, 801MB, the whole process took 107sec ✅ ??

• encryption: 2.5sec
• decryption: 3.5sec

Quote

2023-03-23 16:17:10    root    /backend/setup/duplicator/?action=backup_now    Backup Database
2023-03-23 16:17:10    root    /backend/setup/duplicator/?action=backup_now    - Backup using native tools
2023-03-23 16:18:50    root    /backend/setup/duplicator/?action=backup_now    - package built successfully in 100.684342sec
2023-03-23 16:18:50    root    /backend/setup/duplicator/?action=backup_now    - package encryption starting...
2023-03-23 16:18:53    root    /backend/setup/duplicator/?action=backup_now    - package encryption success, took 2.431822sec
2023-03-23 16:18:53    root    /backend/setup/duplicator/?action=backup_now    - package decryption starting...
2023-03-23 16:18:56    root    /backend/setup/duplicator/?action=backup_now    - package decryption success, took 3.465724sec
2023-03-23 16:18:56    root    /backend/setup/duplicator/?action=backup_now    - package saved in local folder: 2023-03-23_16-17-29-test.me.package2.zip
2023-03-23 16:18:56    root    /backend/setup/duplicator/?action=backup_now    - orphaned log files not cleaned (disabled by settings)
2023-03-23 16:18:56    root    /backend/setup/duplicator/?action=backup_now    - job finished in 106.586344sec

Not bad ?

 

[netcarver]

Nice :)

[adrian]

Hi @flydev - thanks so much for your work on this. I've just been testing it out and have a few questions:

1) I am seeing libsodium setup [FAIL] at the bottom of the private key section in the config settings. I have libsodium-dev and php-libsodium installed so not sure where there error is coming from. Any ideas?

2) The backup seems to work as expected, but I am seeing both .zip and .zip.enc files created. Are they both supposed to remain after the backup has completed running? 

3) Is is expected that the .enc version is about 2/5ths the size of the main .zip file?

Actually, I think I just answered #2 and #3 - turns out I got this error Fatal Error: Maximum execution time of 300 seconds exceeded (line 23 of /paragonie/sodium_compat/src/Core/ChaCha20.php) which I suppose means the encryption didn't complete. Do you think it would make sense to override the php time limit within this process to avoid this?

Thanks again!

[adrian]

Just looking into that libsodium error. I see that it's being generated by Halite's isLibsodiumSetupCorrectly() function, but when I call the components from that I get the following, which suggest it should be ok.

 

Actually, it's just a case of inverted logic in your code - note the ! where it shouldn't be.

    if (!\ParagonIE\Halite\Halite::isLibsodiumSetupCorrectly()) {
      $field->notes = __("☑️ `libsodium` setup [**OK**]");
    } else {
      $field->notes = __("⚠️ `libsodium` setup [**FAIL**]");
    }

 

[flydev]

Hi @adrian hope you are fine and thanks for testing.

1 hour ago, adrian said:

Actually, it's just a case of inverted logic in your code

Oh common! ? I introduced it when testing the ui and pushed as is - corrected.

On your second post, we can see that everything look ok to use it. Can you tell me if you are on a DigitalOcean solution or maybe an AWS one ?

Anyway, the cause of #3 is due to the max_execution_time limit. Can you tell me which options you use in Duplicator settings (cron, local, google) ? 

I will test with GoogleDrive right now to see what happen, I have not tested it with the encryption enabled. In local mode only, you should not hit the limit (but that's depend your server type), as the php-cli is by default not supposed to be interfered by this settings, only if set some where, and that's the reason we call cron.php from the task scheduler or from the terminal. As I have no control over the stream (Halite seal encryption use chunks on stream when using the File class), it will hit this limit on nearly each managed host (at least I think).

#2 There will be an options available to keep the zip or not. By default it will be deleted if encryption is enabled.

 

1 hour ago, adrian said:

Do you think it would make sense to override the php time limit within this process to avoid this?

I could make a settings available to adjust this, but again, we need to see from where the value is set. I bet it's the google mode. If not, I will try to find in you hosting provider documentation if there is a limit set on the php-cli. Checking and listening to your answer about this.

[adrian]

Currently on Digital Ocean (although will actually be moving the project to AWS shortly).

I just ran a backup from the CLI and in that situation (where the timeout doesn't come into affect), it seems to have worked as expected, with both the zip and zip.enc files having the same size, but I am still confused why the .zip is kept - shouldn't that be deleted after the creation of the .zip.enc file?

Actually, should both be exactly the same file size - does the encryption not result in some change?

I came across the timeout when trying to run a backup from ProcessDuplicator using the native DB / local only options - I still think this need to work.

[flydev]

16 minutes ago, adrian said:

I am still confused why the .zip is kept - shouldn't that be deleted after the creation of the .zip.enc file?

Yes sorry, I edited my post after posting it.

17 minutes ago, adrian said:

I came across the timeout when trying to run a backup from ProcessDuplicator using the native DB / local only options - I still think this need to work.

Yes ok, that's what I had in mind. Actually, with Halite, it shouldn't be possible. The only solution I have in mind after I read the C library yesterday, is the need to write another wrapper to be able to set max execution time limit in the process. This is what I meant on my previous post > As I have no control over the stream (Halite seal encryption use chunks on stream when using the File class)...