☁️ Duplicator: Backup and move sites

[dotnetic]

@schwarzdesign Try deleting the LazyCron.cache file in dist\site\assets\cache, then it could work again.

[schwarzdesign]

On 8/11/2018 at 12:44 PM, jmartsch said:

@schwarzdesign Try deleting the LazyCron.cache file in dist\site\assets\cache, then it could work again.

@jmartsch Thanks for the suggestion, unfortunately that didn't help ...

[Peter Knight]

I notice on the Module settings that the username and password are displayed in plain text.

I've a few admins that I wouldn't like to be able to view these. Is masking these on the roadmap or even encrypting them?

Does this also mean that they're stored in the database in plain text?

Thanks

[Peter Knight]

Hi 

One more question

I have 2 FTP applications here successfully connecting to a destination server over FTPS SSL/TLS 

 

When I add the exact same credentials to Duplicator, I get the following

Quote

2018-09-06 15:00:43:  - job finished in 110.8802sec
2018-09-06 15:00:43:  [FTP] error: cannot upload file <2018-09-06_14-58-52-www.domain.co.uk.package.zip> on server.
2018-09-06 14:59:40:  FTP: starting upload of /home/sites/domain.co.uk/public_html/site/assets/backups/2018-09-06_14-58-52-www.domain.co.uk.package.zip
2018-09-06 14:59:40:  FTP: directory listing of Backups successfull.
2018-09-06 14:59:40:  FTP: retrieving directory listing of Backups...
2018-09-06 14:59:40:  FTP: directory listing of Sharey01 successfull.
2018-09-06 14:59:40:  FTP: retrieving directory listing of Sharey01...
2018-09-06 14:59:39:  FTP: logged in.
2018-09-06 14:59:39:  FTP: connection established.
2018-09-06 14:59:39:  FTP: connecting to myserver.com:21...
2018-09-06 14:59:39:  - package built successfully in 46.558444sec

So it looks like Duplicator is successfully building the package, logging in to the server and listing the directory files. It only fails to actually upload the ZIP

There are no size restrictions on the server and there are I have disconnected the other FTP apps to ensure that's not the issue either.

Any ideas why the upload might fail? This only happens on TLS/SSL and not under 'regular' unencrypted FTP

Thanks

 

[RDC]

Found a problem when the webiste is on a different port number (http:\\127.0.0.1:81) the backup fails. The logs list:

1 hour ago 2018-09-07 12:53:33	admin	/processwire/setup/duplicator/?action=backup_n…	- package build failed.
1 hour ago 2018-09-07 12:53:33	admin	/processwire/setup/duplicator/?action=backup_n…	- an error occured during package build.
1 hour ago 2018-09-07 12:53:33	admin	/processwire/setup/duplicator/?action=backup_n…	- an error occured while building the ProcessWire structure.

 

It is adding the colon(:) into the filename of the zips. I changed the code in DupUtil.php on line 48 to filter the colon(:) out and replace it with a period:

FROM:    $filename = $date . '-' . str_replace(' ', '_', $name); 

TO:         $filename = $date . '-' . str_replace(':', '.', str_replace(' ', '_', $name));

There may be a better place to put this in but it does the trick!

Thanks for the plugin - it's my goto for securing backups.

Yours truly

Robert

[Robin S]

@flydev, just trying this module for the first time today. Very nice! The subfolder exclusion options are coming in handy.

Do you think the main Duplicator.module should automatically install ProcessDuplicator.module? Otherwise if you install by entering the Duplicator classname then the Process module doesn't get installed and it's not immediately obvious why the link from the config screen to the Package Manager results in an "Unrecognized path" error.

[flydev]

@Robin S Thanks for trying it, glad you like it !  

The Process module is not really required because Duplicator could be, if needed, only run from CRON task. The link to he process module should had been removed if the process module was not present, just checking my todo list, it was into. 

 

A known issue when restoring a package :

On 6/21/2018 at 11:04 AM, flydev said:

On the current state of the module, it is usable for small website, I mean small database (I am aware of the issue which include the  parent directory in the compressed package which make the package non deployable by the installer).

So for restoring it, we need to manually move the site files out of the container dir.

 

About AmazonS3, @Rudy asked the ability to upload package on sub-bucket (I forgot this one, sorry Rudy!) : https://github.com/flydev-fr/Duplicator/issues/1

 

I will try to code theses corrections soon so the V1 will work as expected, but I really want to begin to work on the Duplicator V2 implementing encryption and handling gigabits sized databases. Want To Buy time or vacation ?

 

 

[Robin S]

11 hours ago, flydev said:

The Process module is not really required because Duplicator could be, if needed, only run from CRON task.

I see what you mean, although there isn't really any "cost" to a Process module being installed because it doesn't do anything unless you visit its page. So for the sake of new users trying to understand the different ways they can use Duplicator I still think it would be good to auto-install the Process module and CRON-only users can just ignore it, or uninstall it if they don't want to see the item in the Setup menu for some reason. But now that I have used Duplicator for a bit I could figure it out either way.

[Autofahrn]

Is there a chance to exclude files by regex?

I'm thinking about something like "\.\d+x\d+\." to exclude any image variant created from the API, for example.

[jploch]

On 4/19/2018 at 9:03 AM, Karl_T said:

I have encountered an issue for Duplicator ver 1.2.9.

 I am moving a site from localhost(windows 10) to Amazon Linux. I use RDS, which is a standalone server for database, so the dbHost is not localhost.

The issue happened in the installation process after entering the database data and proceed. There's an error telling config file cannot be found. Then I go to config.php, there are at least 4 places are using old information. They are dbHost, dbName, dbUser, dbPass and httpHosts. I think for some reasons the config.php did not update at all after the installation complete. After I manually update the fields, the site works fine now. This means database import is working fine. I wonder if it is the dbHost matters. It is a long URL ended with xxx.rds.amazonaws.com.

I had the same problem a couple of times now. For example when I try to install a backup of a live site on my localhost (Running on Mamp) or an a web server I always get the error config file cannot be found.
The issue happened in the installation process after entering the database data and proceed.
Maybe this is happening when the website is in another subdirectory and not the web root? 
The duplicator always makes a subdirectory when unzipping the files and then looks for the config in the root folder.
After I manually change the config file, the site was showing up and I could see all the content in the admin, but had some strange errors  on the frontend.

[Autofahrn]

I remember falling into the same trap. I didn't modify the generated config.php but simply moved the contents from the unzipped subdirectory one up before I submit the database information.

[jploch]

49 minutes ago, Autofahrn said:

I remember falling into the same trap. I didn't modify the generated config.php but simply moved the contents from the unzipped subdirectory one up before I submit the database information.

The installation worked that way. But Iam still getting errors on the frontend saying Warning: preg_replace(): JIT compilation failed: no more memory
Any Ideas, what is causing this? When I use the site profile exporter and install the profile everything is working fine.

[Autofahrn]

Sorry, no idea. Never saw an error like that.

[netcarver]

@jploch AFAIK, you can turn off PCRE JIT compilation if needed. Some info in the discussion here. 

[jploch]

I changed the PHP version to 5.6 and these errors are gone now.
However my markup regions seem not to be working. But when logged into the admin everything on the frontend works fine.
When logged out I get an error saying "no regions" and the site is messed up.
Iam about to give up on this and just use the site profile exporter for now ?

 

[flydev]

3 hours ago, jploch said:

The duplicator always makes a subdirectory when unzipping the files and then looks for the config in the root folder.

Sorry dude, I am aware of this issue but forgot to update the first post and the github doc. So yes, basically, In the last version of Duplicator, its necessary (but unwanted) to move the unzipped data to the parent dir..

And about your last issue, try what its said in the github issue linked by @netcarver and more precisely this one : https://github.com/composer/composer/issues/7836#issuecomment-452341683

And just saying, I have not tested the module with PHP 7.x and it might have some issues on this version.

 

2 hours ago, jploch said:

Iam about to give up on this and just use the site profile exporter for now ?

sorry for the inconveniences, I hope to get the time to finally work on the V2 ? but I promise you guys to take a look at the actual version and make a revision in the last 10 days or ask to moderators to ban me from the forum ! ? ?

 

[jploch]

15 minutes ago, flydev said:

And just saying, I have not tested the module with PHP 7.x and it might have some issues on this version.

Thats fine with me. If I use Mamp with PHP 5.6 this issue is solved.

15 minutes ago, flydev said:

sorry for the inconveniences, I hope to get the time to finally work on the V2 ? but I promise you guys to take a look at the actual version and make a revision in the last 10 days or ask to moderators to ban me from the forum ! ? ?

No worries! Keep up the good work! Looking forward to use this one day ?
I still have no clue what stops the markup regions from working though.
 

[flydev]

A small news about Duplicator.

There is a work in progress on a development version (v1.3.10-ATO) - nothing which can harm your server and could be already tested on different setup. You can download this dev version on GitHub :  https://github.com/flydev-fr/Duplicator/archive/dev.zip or at the end of this post.

The module still contain minor issues but the overall processing is faster than before and packages can be built and restored as well. Thanks to @Autofahrn for putting his hand in the module! Autofahrn also added a great feature which give the possibility to exclude file through regular expression. Also, image variations created by ProcessWire are now excluded by default.

 

 

 

Duplicator-dev-1.3.10-ATO.zip

[horst]

@flydev I just found this (https://www.phpclasses.org/package/10950-PHP-Compress-files-and-create-archives-in-Zip-format.html), maybe it is of interest for you?

 

zipfly-2018-12-21.zip

[flydev]

Thanks @horst !  

Checking that this week-end - thanks for the sample too ! ?

[ank]

When i try to move a existing website back to local development on VHX (VirtualHostX) i get a error message:

Couldn't find "config.php" in /var/www/webank.vhx.cloud/site/config.php.

the path isn't correct, "/var/www" does not apply here, can i change this ?

 

 

[Autofahrn]

That "/var/www" is the root path (within your filesystem) where your webserver places the content of your domains. This can only be changed when you point your server to some other folder.

I guess you are currently working with the release version (1.2.9) of Duplicator, right? That one expands the site files into a directory level below which needs to be moved manually during the install. Please try the V1.3.10 above which should solve this.

Btw, @flydev, do you plan to include the suggested domain name fix into ProcessDuplicator::getPackagesDetails to support dashes in domain names? If anyone else is using Duplicator on domains containing dashes, you may want to change these lines in aforementioned method (otherwise the package list stays empty):

					$parts = explode('-', $originalFilename);
					array_pop($parts);
					$tsstr = implode('-', $parts);

with this dirty hack:

					$tsstr = substr($originalFilename, 0, 19); //  2019-03-20_01-15-41

 

[ank]

On 4/14/2019 at 8:43 PM, Autofahrn said:

I guess you are currently working with the release version (1.2.9) of Duplicator, right? That one expands the site files into a directory level below which needs to be moved manually during the install. Please try the V1.3.10 above which should solve this.

Yes this solved my problem,

Thanks @Autofahrn!

 

[pwired]

Quote

I just found this (https://www.phpclasses.org/package/10950-PHP-Compress-files-and-create-archives-in-Zip-format.html), maybe it is of interest for you?

Hi,

I downloaded the script and tested the included test.php on my hoster and it is working very well with single files.

I would like to modify the included test.php script to zip a directory.
At the bottom of the included test.php it looks like this:

//$zipFile = new \zipFly\zipFly64(GTEST_DIR.'onfly'.$nr.'.zip');
$zipFile = new \zipFly\zipFly64();
$zipFile->setDebugMode(false);
$zipFile->setZipFeature(true, false);

$zipFile->create(GTEST_DIR.'onfly'.$nr.'.zip');

$zipFile->addFile(GTEST_DIR."zipFly/zipFly64.php", "zipFly64.php", \zipFly\zipFly64::METHOD_BZIP2);
$zipFile->addFile(GTEST_DIR."zipFly/parts/headers.php", "parts/headers.php", \zipFly\zipFly64::METHOD_DEFLATE);
$zipFile->addFromString("Zip64\nTest\n", 'test.txt');

$zipFile->close();

As you see it only uses $zipFile->addFile . . . .

Maybe something like this would do it.

 

// let's iterate
foreach ($files as $name => $file) {
	$filePath = $file->getRealPath();
	$zip->addFile($filePath);
}

 

 

[pwired]

I just compared and turns out that the pclzip library is vastly superior,
so it doesn't matter much anymore.