☁️ Duplicator: Backup and move sites

[flydev]

2 hours ago, adrian said:

Actually, should both be exactly the same file size - does the encryption not result in some change?

 

Missed your question, and the answer is not necessarily.

In our case, we use asymmetric file encryption and the lib run his seals operations on a resource implementing StreamInterface (PSR-7). Just to say that, without going into deep details, the methods read the file from a stream block by block, with a defined buffer size, and a minimum shortest cypher text of 100 bytes, then run the cipher algo (XChaCha20) on the blocks till the EOF. This mean that your file should have the same size. The encrypted file could have 100 bytes more than the source.

You can try to run in the terminal `du -b source_file` and `du -b encrypted_file` to compare them. By curiosity, you can also test it with openssl by running :

`openssl enc -d -chacha20 -K YOUR_PUBLIC_KEY -iv 0123456789abcdef0123456789abcdef -in SOURCE.zip > ENCRYPTED.zip`

(the difference between Chacha20 and XChacha20 is the last use an extended nonce, the result should be same size).

[flydev]

On 11/21/2022 at 8:49 PM, James Morris said:

I am getting an error of 'Class "DUP_Logs" not found' when upgrading to the latest version (1.4.26) when using the Amazon S3 option.  The provided SDK is on place, all was working fine before the upgrade.  Reverting back to 1.4.0 solved the problem.  Here is the full call stack for reference:

Fixed it, it will be released in Duplicator v1.5.0. I tested AWS, no problem found. I will release also the updated Duplicator-SDKs.

 

On 3/23/2023 at 8:47 PM, adrian said:

I am still confused why the .zip is kept - shouldn't that be deleted after the creation of the .zip.enc file?

It's committed.

[DV-JF]

Hey @flydev

thank you for providing this module, it's really helpful to me an safes a ton of time!

Just want to mention that there seems to be something wrong with the version number on PW:

Also when trying to update the module (obviously)

Cheers !

[flydev]

Thanks @DV-JF just fixed it to make things easier.

[Claus]

I’m getting this when I try to go to the Duplicator module:

Fatal Error: Uncaught Error: Call to undefined function shell_exec() in site/modules/Duplicator/Duplicator.module:1439

#0 [internal function]: Duplicator::getModuleConfigInputfields(Array)
#1 wire/core/Modules.php (4130): call_user_func(Array, Array)
#2 wire/core/Wire.php (419): Modules->___getModuleConfigInputfields('Duplicator', Object(InputfieldForm))
#3 wire/core/WireHooks.php (952): Wire->_callMethod('___getModuleCon...', Array)
#4 wire/core/Wire.php (484): WireHooks->runHooks(Object(Modules), 'getModuleConfig...', Array)
#5 wire/modules/Process/ProcessModule/ProcessModule.module (1680): Wire->__call('getModuleConfig...', Array)
#6 wire/modules/Process/ProcessModule/ProcessModule.module (1415): ProcessModule->renderEdit('Duplicator', Array)
#7 wire/core/Wire.php (413): ProcessModule->___executeEdit()
#8 wire/core/WireHooks.php (952): Wire->_callMethod('___executeEdit', Array)
#9 wire/core/Wire.php (484): WireHooks->runHooks(Object(ProcessModule), 'executeEdit', Array)
#10 wire/core/ProcessController.php (350): Wire->__call('executeEdit', Array)
#11 wire/core/Wire.php (413): ProcessController->___execute()
#12 wire/core/WireHooks.php (952): Wire->_callMethod('___execute', Array)
#13 wire/core/Wire.php (484): WireHooks->runHooks(Object(ProcessController), 'execute', Array)
#14 wire/core/admin.php (160): Wire->__call('execute', Array)
#15 wire/modules/AdminTheme/AdminThemeDefault/controller.php (13): require('/customers/a/0/...')
#16 site/templates/admin.php (15): require('/customers/a/0/...')
#17 wire/core/TemplateFile.php (328): require('/customers/a/0/...')
#18 wire/core/Wire.php (413): TemplateFile->___render()
#19 wire/core/WireHooks.php (952): Wire->_callMethod('___render', Array)
#20 wire/core/Wire.php (484): WireHooks->runHooks(Object(TemplateFile), 'render', Array)
#21 wire/modules/PageRender.module (575): Wire->__call('render', Array)
#22 wire/core/Wire.php (416): PageRender->___renderPage(Object(HookEvent))
#23 wire/core/WireHooks.php (952): Wire->_callMethod('___renderPage', Array)
#24 wire/core/Wire.php (484): WireHooks->runHooks(Object(PageRender), 'renderPage', Array)
#25 wire/core/WireHooks.php (1060): Wire->__call('renderPage', Array)
#26 wire/core/Wire.php (484): WireHooks->runHooks(Object(Page), 'render', Array)
#27 wire/modules/Process/ProcessPageView.module (184): Wire->__call('render', Array)
#28 wire/modules/Process/ProcessPageView.module (114): ProcessPageView->renderPage(Object(Page), Object(PagesRequest))
#29 wire/core/Wire.php (416): ProcessPageView->___execute(true)
#30 wire/core/WireHooks.php (952): Wire->_callMethod('___execute', Array)
#31 wire/core/Wire.php (484): WireHooks->runHooks(Object(ProcessPageView), 'execute', Array)
#32 index.php (55): Wire->__call('execute', Array)
#33 {main}
thrown (line 1439 of site/modules/Duplicator/Duplicator.module) 

This error message was shown because: you are logged in as a Superuser. Error has been logged.

 

[flydev]

Thanks @Claus, fixed, you can grab the new version.  Can you tell me which type of host and the name if not secret please ?  (use private messaging if you want) 

[Claus]

Thank you for the hotfix @flydev It’s working!

[adrian]

Hi @flydev - for remote DBs, I find I need this for native backups to work.

    $data = '
        # (1) set up all the mysqldump variables
        FILE=' . $this->options['backup']['filename'] . '
        DBSERVER=' . wire('config')->dbHost . '
        DATABASE=' . wire('config')->dbName . '
        USER=' . wire('config')->dbUser . '
        PASS=' . wire('config')->dbPass . '
        PORT=' . wire('config')->dbPort . '

        # (2) in case you run this more than once a day, remove the previous version of the file
        unalias rm     2> /dev/null
        rm ${FILE}     2> /dev/null
        rm ${FILE}.zip  2> /dev/null

        # (3) do the mysql database backup (dump)
        mysqldump --opt --protocol=TCP --user=${USER} --password=${PASS} --host=${DBSERVER} --port=${PORT} ${DATABASE} > ' . $cachePath . '${FILE}
        ';

The key things are setting the DBSERVER to $config->dbHost and adding the PORT

Also, wondering if you've had any thoughts on making the line in section #3 configurable in the module settings, because currently any module updates break native backups.

Thanks!

[Boost]

Hei @flydev

First of all, I want to say a big thank you for this awesome module!

I'm facing a little hiccup while trying to use Duplicator with XAMPP. The thing is, XAMPP installs everything in the /opt directory, which leads to a permissions issue when running mysqldump. I did try creating a symlink of mysqldump in /usr/local/bin, but unfortunately, it's still not working as expected. It seems like Duplicator can see mysqldump, but it's having trouble executing it. Here is the log. 

023-10-17 12:29:13:  - package build failed.
2023-10-17 12:29:13:  Error while running UnixNative Backup , err 2: []
2023-10-17 12:29:13:  - Backup using native tools
2023-10-17 12:29:13:  Backup Database
2023-10-17 11:32:56:  - job finished in 7.060878sec
2023-10-17 11:32:56:  - package saved in local folder: 2023-10-17_11-32-49-boost.local.package2.zip
2023-10-17 11:32:56:  - package built successfully in 7.06046sec
2023-10-17 11:32:49:  - Backup using standard mode

 

[flydev]

Hi, sorry for the delay guys.

@adrian I made it configurable, it should also check for carriage return when a custom script is given, please check releases/tag/v1.5.0 and tell me if you need something more.

Edit2: Rereading your post again, I just realized that I omitted a detail... it'll be ok on v1.5.1 sorry.

 

@Boost Thanks ? . You can check if mysqldump is avail. on the config page. And if needed, to be sure that mysqldump is found, you could run some tests, also make sure - with Duplicator 1.4.29 - that the binary zip is available. Duplicator v1.5.0 show you this information on the module configuration page, with some more friendly exec return code error when the script run (see link at the end of this post).

Some suggestions/debug steps if needed:

1. is mysql in env path and runnable? with normal user type `mysqldump` and `zip`
2. if #1 is ok, try with user/group assigned to your web server, eg. with www-data:

   sudo su www-data -s /bin/sh
   mysqldump
   zip

3. For `mysqldump` only - grab the dump script there and adjust variables in order to make a test
   1. cd /path/to/site/assets/cache
   2. sudo wget https://raw.githubusercontent.com/flydev-fr/Duplicator/v1.5.0/scripts/mysqldump.unix.sh
   3. sudo nano ./mysqldump.unix.sh and set correct or dummy values (or with vim, etc..)
   4. sudo chown www-data:www-data ./mysqldump.unix.sh
   5. sudo su www-data -s /bin/sh
   6. chmod +x ./mysqldump.unix.sh
   7. run a test: ./mysqldump.unix.sh

Anyway, you should grab the last version I just release before messing with it, because if the issue is that zip isn't available, it will fallback using WireZipFile. 

Edit: Just a word, I got this issue and it was zip that was missing (about exit code 2).

https://github.com/flydev-fr/Duplicator/blob/8d2ee3c0327baf4cee0128b12a0724d20dadfb53/Classes/BackupDatabase.php#L253-L270

Edited October 28, 2023 by flydev

[flydev]

@adrian with v1.5.1 you can write your dump script in the custom shell script config field like that:

#!/bin/sh

# Custom Remote Dump

# (1) Set up all the mysqldump variables
FILE=%%FILE%%
DBSERVER=%%SERVER%%
PORT=%%PORT%%
DATABASE=%%DATABASE%%
USER=%%USER%%
PASS=%%PASS%%
CACHEPATH=%%CACHEPATH%%

# Fix trailing slash in cache path if needed
CACHEPATH="${CACHEPATH%/}/"
OUTPUT="${CACHEPATH}${FILE}"

# (2) in case you run this more than once a day, remove the previous version of the file
unalias rm     2> /dev/null
rm ${FILE}     2> /dev/null
rm ${FILE}.zip  2> /dev/null

# (3) do the mysql database backup dump (remote)
mysqldump --opt --protocol=TCP --user=${USER} --password=${PASS} --host=${DBSERVER} --port=${PORT} ${DATABASE} > ${OUTPUT}

You can of course also set hardcoded values.

[adrian]

Thanks @flydev 

The only issue I noticed is that the supplied mysqldump.unix.sh looks like this:

#!/bin/sh

# (1) Set up all the mysqldump variables
FILE=%%FILE%%
DBSERVER=%%SERVER%%
DATABASE=%%DATABASE%%
USER=%%USER%%
PASS=%%PASS%%
CACHEPATH=%%CACHEPATH%%

# Fix trailing slash in cache path
CACHEPATH="${CACHEPATH%/}/"

OUTPUT="${CACHEPATH}${FILE}"

# (2) In case you run this more than once a day, remove the previous version of the file
# unalias rm     2> /dev/null
# rm ${FILE}     2> /dev/null
# rm ${FILE}.zip  2> /dev/null

# (3) Do the mysql database backup (dump)
#  - to log errors of the dump process to a file, add --log-error=mysqldump_dup_error.log
# (a) Use this command for a remote database. Add other options if need be.
# mysqldump --opt --protocol=TCP --user=${USER} --password=${PASS} --host=${DBSERVER} --port=${PORT} ${DATABASE} ${DATABASE} > ${OUTPUT}
# (b) Use this command for a database server on localhost. Add other options if need be.
mysqldump --routines --triggers --single-transaction --user=${USER} --password=${PASS} --databases ${DATABASE} > ${OUTPUT}

Note the missing PORT from the variables in #1 and also the duplication of ${DATABASE} in #3a

Once I fixed those two things, it work great!

Thanks again for making this configurable - a really nice improvement!

[adrian]

Actually, the only other suggestion is to make it a real initial value for the textarea rather than a placeholder, because I actually had to go and open mysqldump.unix.sh to get a starting script I could then edit.

[flydev]

8 hours ago, adrian said:

Note the missing PORT from the variables in #1 and also the duplication of ${DATABASE} in #3a

Yes I saw that, but it was in the commented line so I didn't fixed it instantly, will push the fix.

 

8 hours ago, adrian said:

Actually, the only other suggestion is to make it a real initial value for the textarea rather than a placeholder, because I actually had to go and open mysqldump.unix.sh to get a starting script I could then edit.

Ok I see, in first instance I had in mind to put a button to get the template script copied in the clipboard in order to paste and modify it. I will push also the fix following your recommendation ?

 

I was also thinking of pushing a new feature that would allow you to export and import module settings on other installations, what do you think, it could help? 

[adrian]

9 hours ago, flydev said:

I was also thinking of pushing a new feature that would allow you to export and import module settings on other installations, what do you think, it could help? 

I always have https://processwire.com/modules/module-settings-import-export/ installed on all my sites, so no need from my end for it to be built into your module.

[James Morris]

On 3/26/2023 at 2:04 PM, flydev said:

On 11/21/2022 at 11:49 AM, James Morris said:

I am getting an error of 'Class "DUP_Logs" not found' when upgrading to the latest version (1.4.26) when using the Amazon S3 option.  The provided SDK is on place, all was working fine before the upgrade.  Reverting back to 1.4.0 solved the problem.  Here is the full call stack for reference:

Fixed it, it will be released in Duplicator v1.5.0. I tested AWS, no problem found. I will release also the updated Duplicator-SDKs.

It doesn't look like this fix made it into 1.5.0 or 1.5.1.  Would be great to have as we are hoping to upgrade to PHP 8.2 on all our sites soon with support for 8.0 ending.  Thanks!

[flydev]

@James Morris I had fixed it in on the feature-encryption branch, I will need to finish the feature and merge it, but meanwhile, can you try this version and let me know and confirm ?

there: https://github.com/flydev-fr/Duplicator/tree/feature-encryption

direct-dl: https://github.com/flydev-fr/Duplicator/archive/refs/heads/feature-encryption.zip

 

Edit: @James Morris please try instead the dev branch, It is based on the latest v1.5.1 and contain the S3 fixes.

dev branch: https://github.com/flydev-fr/Duplicator/tree/dev

direct download: https://github.com/flydev-fr/Duplicator/archive/refs/heads/dev.zip

[Flashmaster82]

Can´t install my site with the downloaded duplicated version. Duplicator 1.5.2. Processwire 3.0.229

Deprecated: strrpos(): Passing null to parameter #1 ($haystack) of type string is deprecated in E:\Wamp\www\..\installer.php on line 709

[flydev]

@Flashmaster82 can you give me details about the version you are trying to install and your php version please?

[Autofahrn]

Sounds like the session variables do not survive, maybe cookies disabled?

[protro]

Not sure if I am missing something, but it seems Duplicator may be skipping over images added via the TinyMCE text-editor in PW, because migrating from local to production, everything works great except for images added in a TinyMCE textarea field. Inspector shows the img site path twice ("https://my-site.com/my-site/site/assets/files/…") etc where /my-site/ is the proper local path but should be removed once migrating to production.

Do I need to make a Custom excluded files and paths declaration in the Duplicator plugin option ?
Many thanks!

[flydev]

Hi @protro

But does it work even if you upload the site manually? I am asking that because as the path is written in the body field raw data in the db, with or without duplicator, image paths should be corrupted in any way.

As example and based on your setup, the data of a body field with an image should look - on dev and prod - something similar to (can you confirm ?) :

<p>Sagittis turpis ad habitasse risus lorem phasellus platea imperdiet aenean, platea porta adipiscing dapibus velit eros purus malesuada laoreet vestibulum</p>
<p><img src="/my-site/site/assets/files/1/44849-the-image-2016.897x0-is.jpg" alt="" width="897" /></p>
<p>...</p>

 

[protro]

@flydev I confirm that is what the file path appears on dev and production, but on production i would get 404s on paths because they included the domain as a prefix to the path i.e. https://my-site/my-site/site/assets/files/1/44849-the-image-2016.897x0-is.jpg

I was able to do a hot fix on my prod server by using @ryan TextformatterFindReplace module and attaching it to the body textarea fields in question that had the images added via TinyMCE … and set the rule to e.g.

my-site/=

This fixed my issue, by converting the redundant domain name to an empty string.

[flydev]

Ok, so would you need a feature that would allow you to customize asset paths during a backup?

[Pavel Radvan]

Hi.
Your module is super and backups are working nice.
I have there error from VSCODE which is showing this:
${} string interpolation' is deprecated since PHP 8.2
It is in this file: \site\modules\Duplicator\Classes\ClientAmazonS3.php

Also there is date of update module on PW directory 2024/05/14 and I have version 1.5.2 which is latest but not from 14yh May this year...
Is the date of development version or where I could get actual version?
Is that error in new version solved ?
Thank you