Jump to content
flydev

Duplicator: Backup and move sites

Recommended Posts

I notice on the Module settings that the username and password are displayed in plain text.

I've a few admins that I wouldn't like to be able to view these. Is masking these on the roadmap or even encrypting them?

Does this also mean that they're stored in the database in plain text?

Thanks

  • Like 1

Share this post


Link to post
Share on other sites

Hi 

One more question

I have 2 FTP applications here successfully connecting to a destination server over FTPS SSL/TLS 

 

When I add the exact same credentials to Duplicator, I get the following

Quote

2018-09-06 15:00:43:  - job finished in 110.8802sec
2018-09-06 15:00:43:  [FTP] error: cannot upload file <2018-09-06_14-58-52-www.domain.co.uk.package.zip> on server.
2018-09-06 14:59:40:  FTP: starting upload of /home/sites/domain.co.uk/public_html/site/assets/backups/2018-09-06_14-58-52-www.domain.co.uk.package.zip
2018-09-06 14:59:40:  FTP: directory listing of Backups successfull.
2018-09-06 14:59:40:  FTP: retrieving directory listing of Backups...
2018-09-06 14:59:40:  FTP: directory listing of Sharey01 successfull.
2018-09-06 14:59:40:  FTP: retrieving directory listing of Sharey01...
2018-09-06 14:59:39:  FTP: logged in.
2018-09-06 14:59:39:  FTP: connection established.
2018-09-06 14:59:39:  FTP: connecting to myserver.com:21...
2018-09-06 14:59:39:  - package built successfully in 46.558444sec

So it looks like Duplicator is successfully building the package, logging in to the server and listing the directory files. It only fails to actually upload the ZIP

There are no size restrictions on the server and there are I have disconnected the other FTP apps to ensure that's not the issue either.

Any ideas why the upload might fail? This only happens on TLS/SSL and not under 'regular' unencrypted FTP

Thanks

 

  • Like 1

Share this post


Link to post
Share on other sites

Found a problem when the webiste is on a different port number (http:\\127.0.0.1:81) the backup fails. The logs list:

1 hour ago
2018-09-07 12:53:33
admin /processwire/setup/duplicator/?action=backup_n… - package build failed.
1 hour ago
2018-09-07 12:53:33
admin /processwire/setup/duplicator/?action=backup_n… - an error occured during package build.
1 hour ago
2018-09-07 12:53:33
admin /processwire/setup/duplicator/?action=backup_n… - an error occured while building the ProcessWire structure.

 

It is adding the colon(:) into the filename of the zips. I changed the code in DupUtil.php on line 48 to filter the colon(:) out and replace it with a period:

FROM:    $filename = $date . '-' . str_replace(' ', '_', $name); 

TO:         $filename = $date . '-' . str_replace(':', '.', str_replace(' ', '_', $name));

There may be a better place to put this in but it does the trick!

Thanks for the plugin - it's my goto for securing backups.

Yours truly

Robert

  • Like 1

Share this post


Link to post
Share on other sites

@flydev, just trying this module for the first time today. Very nice! The subfolder exclusion options are coming in handy.

Do you think the main Duplicator.module should automatically install ProcessDuplicator.module? Otherwise if you install by entering the Duplicator classname then the Process module doesn't get installed and it's not immediately obvious why the link from the config screen to the Package Manager results in an "Unrecognized path" error.

  • Like 1
  • Thanks 1

Share this post


Link to post
Share on other sites

@Robin S Thanks for trying it, glad you like it !  

The Process module is not really required because Duplicator could be, if needed, only run from CRON task. The link to he process module should had been removed if the process module was not present, just checking my todo list, it was into. 

 

A known issue when restoring a package :

On 6/21/2018 at 11:04 AM, flydev said:

On the current state of the module, it is usable for small website, I mean small database (I am aware of the issue which include the  parent directory in the compressed package which make the package non deployable by the installer).

So for restoring it, we need to manually move the site files out of the container dir.

 

About AmazonS3, @Rudy asked the ability to upload package on sub-bucket (I forgot this one, sorry Rudy!) : https://github.com/flydev-fr/Duplicator/issues/1

 

I will try to code theses corrections soon so the V1 will work as expected, but I really want to begin to work on the Duplicator V2 implementing encryption and handling gigabits sized databases. Want To Buy time or vacation 🔥

 

 

  • Like 5

Share this post


Link to post
Share on other sites
11 hours ago, flydev said:

The Process module is not really required because Duplicator could be, if needed, only run from CRON task.

I see what you mean, although there isn't really any "cost" to a Process module being installed because it doesn't do anything unless you visit its page. So for the sake of new users trying to understand the different ways they can use Duplicator I still think it would be good to auto-install the Process module and CRON-only users can just ignore it, or uninstall it if they don't want to see the item in the Setup menu for some reason. But now that I have used Duplicator for a bit I could figure it out either way.

  • Like 2

Share this post


Link to post
Share on other sites

Is there a chance to exclude files by regex?

I'm thinking about something like "\.\d+x\d+\." to exclude any image variant created from the API, for example.

  • Like 1

Share this post


Link to post
Share on other sites
On 4/19/2018 at 9:03 AM, Karl_T said:

I have encountered an issue for Duplicator ver 1.2.9.

 I am moving a site from localhost(windows 10) to Amazon Linux. I use RDS, which is a standalone server for database, so the dbHost is not localhost.

The issue happened in the installation process after entering the database data and proceed. There's an error telling config file cannot be found. Then I go to config.php, there are at least 4 places are using old information. They are dbHost, dbName, dbUser, dbPass and httpHosts. I think for some reasons the config.php did not update at all after the installation complete. After I manually update the fields, the site works fine now. This means database import is working fine. I wonder if it is the dbHost matters. It is a long URL ended with xxx.rds.amazonaws.com.

I had the same problem a couple of times now. For example when I try to install a backup of a live site on my localhost (Running on Mamp) or an a web server I always get the error config file cannot be found.
The issue happened in the installation process after entering the database data and proceed.
Maybe this is happening when the website is in another subdirectory and not the web root? 
The duplicator always makes a subdirectory when unzipping the files and then looks for the config in the root folder.
After I manually change the config file, the site was showing up and I could see all the content in the admin, but had some strange errors  on the frontend.

  • Like 1

Share this post


Link to post
Share on other sites

I remember falling into the same trap. I didn't modify the generated config.php but simply moved the contents from the unzipped subdirectory one up before I submit the database information.

  • Like 2

Share this post


Link to post
Share on other sites
49 minutes ago, Autofahrn said:

I remember falling into the same trap. I didn't modify the generated config.php but simply moved the contents from the unzipped subdirectory one up before I submit the database information.

The installation worked that way. But Iam still getting errors on the frontend saying Warning: preg_replace(): JIT compilation failed: no more memory
Any Ideas, what is causing this? When I use the site profile exporter and install the profile everything is working fine.

  • Like 1

Share this post


Link to post
Share on other sites

I changed the PHP version to 5.6 and these errors are gone now.
However my markup regions seem not to be working. But when logged into the admin everything on the frontend works fine.
When logged out I get an error saying "no regions" and the site is messed up.
Iam about to give up on this and just use the site profile exporter for now 😞

 

  • Like 1

Share this post


Link to post
Share on other sites
3 hours ago, jploch said:

The duplicator always makes a subdirectory when unzipping the files and then looks for the config in the root folder.

Sorry dude, I am aware of this issue but forgot to update the first post and the github doc. So yes, basically, In the last version of Duplicator, its necessary (but unwanted) to move the unzipped data to the parent dir..

And about your last issue, try what its said in the github issue linked by @netcarver and more precisely this one : https://github.com/composer/composer/issues/7836#issuecomment-452341683

And just saying, I have not tested the module with PHP 7.x and it might have some issues on this version.

 

2 hours ago, jploch said:

Iam about to give up on this and just use the site profile exporter for now 😞

sorry for the inconveniences, I hope to get the time to finally work on the V2 🤞 but I promise you guys to take a look at the actual version and make a revision in the last 10 days or ask to moderators to ban me from the forum ! 🤐 😅

 

  • Like 2

Share this post


Link to post
Share on other sites
15 minutes ago, flydev said:

And just saying, I have not tested the module with PHP 7.x and it might have some issues on this version.

Thats fine with me. If I use Mamp with PHP 5.6 this issue is solved.

15 minutes ago, flydev said:

sorry for the inconveniences, I hope to get the time to finally work on the V2 🤞 but I promise you guys to take a look at the actual version and make a revision in the last 10 days or ask to moderators to ban me from the forum ! 🤐 😅

No worries! Keep up the good work! Looking forward to use this one day 🙂
I still have no clue what stops the markup regions from working though.
 

  • Thanks 1

Share this post


Link to post
Share on other sites

A small news about Duplicator.

There is a work in progress on a development version (v1.3.10-ATO) - nothing which can harm your server and could be already tested on different setup. You can download this dev version on GitHub :  https://github.com/flydev-fr/Duplicator/archive/dev.zip or at the end of this post.

The module still contain minor issues but the overall processing is faster than before and packages can be built and restored as well. Thanks to @Autofahrn for putting his hand in the module! Autofahrn also added a great feature which give the possibility to exclude file through regular expression. Also, image variations created by ProcessWire are now excluded by default.

 

 

 

Duplicator-dev-1.3.10-ATO.zip

  • Like 5

Share this post


Link to post
Share on other sites

Thanks @horst !  

Checking that this week-end - thanks for the sample too ! 👍

  • Like 3

Share this post


Link to post
Share on other sites

When i try to move a existing website back to local development on VHX (VirtualHostX) i get a error message:

Couldn't find "config.php" in /var/www/webank.vhx.cloud/site/config.php.

the path isn't correct, "/var/www" does not apply here, can i change this ?

 

 

Share this post


Link to post
Share on other sites

That "/var/www" is the root path (within your filesystem) where your webserver places the content of your domains. This can only be changed when you point your server to some other folder.

I guess you are currently working with the release version (1.2.9) of Duplicator, right? That one expands the site files into a directory level below which needs to be moved manually during the install. Please try the V1.3.10 above which should solve this.

Btw, @flydev, do you plan to include the suggested domain name fix into ProcessDuplicator::getPackagesDetails to support dashes in domain names? If anyone else is using Duplicator on domains containing dashes, you may want to change these lines in aforementioned method (otherwise the package list stays empty):

					$parts = explode('-', $originalFilename);
					array_pop($parts);
					$tsstr = implode('-', $parts);

with this dirty hack:

					$tsstr = substr($originalFilename, 0, 19); //  2019-03-20_01-15-41

 

  • Like 2

Share this post


Link to post
Share on other sites
On 4/14/2019 at 8:43 PM, Autofahrn said:

I guess you are currently working with the release version (1.2.9) of Duplicator, right? That one expands the site files into a directory level below which needs to be moved manually during the install. Please try the V1.3.10 above which should solve this.

Yes this solved my problem,

Thanks @Autofahrn!

 

  • Like 1

Share this post


Link to post
Share on other sites
Quote

I just found this (https://www.phpclasses.org/package/10950-PHP-Compress-files-and-create-archives-in-Zip-format.html), maybe it is of interest for you?

Hi,

I downloaded the script and tested the included test.php on my hoster and it is working very well with single files.

I would like to modify the included test.php script to zip a directory.
At the bottom of the included test.php it looks like this:

//$zipFile = new \zipFly\zipFly64(GTEST_DIR.'onfly'.$nr.'.zip');
$zipFile = new \zipFly\zipFly64();
$zipFile->setDebugMode(false);
$zipFile->setZipFeature(true, false);

$zipFile->create(GTEST_DIR.'onfly'.$nr.'.zip');

$zipFile->addFile(GTEST_DIR."zipFly/zipFly64.php", "zipFly64.php", \zipFly\zipFly64::METHOD_BZIP2);
$zipFile->addFile(GTEST_DIR."zipFly/parts/headers.php", "parts/headers.php", \zipFly\zipFly64::METHOD_DEFLATE);
$zipFile->addFromString("Zip64\nTest\n", 'test.txt');

$zipFile->close();

As you see it only uses $zipFile->addFile . . . .

Maybe something like this would do it.

 

// let's iterate
foreach ($files as $name => $file) {
	$filePath = $file->getRealPath();
	$zip->addFile($filePath);
}

 

 

Share this post


Link to post
Share on other sites

I just compared and turns out that the pclzip library is vastly superior,
so it doesn't matter much anymore.

 

Share this post


Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.


  • Recently Browsing   0 members

    No registered users viewing this page.

  • Similar Content

    • By David Karich
      ProcessWire InputfieldRepeaterMatrixDuplicate
      Thanks to the great ProModule "RepeaterMatrix" I have the possibility to create complex repeater items. With it I have created a quite powerful page builder. Many different content modules, with many more possible design options. The RepeaterMatrix module supports the cloning of items, but only within the same page. Now I often have the case that very design-intensive pages and items are created. If you want to use a content module on a different page (e.g. in the same design), you have to rebuild each item manually every time.
      This module extends the commercial ProModule "RepeaterMatrix" by the function to duplicate repeater items from one page to another page. The condition is that the target field is the same matrix field from which the item is duplicated. This module is currently understood as proof of concept. There are a few limitations that need to be considered. The intention of the module is that this functionality is integrated into the core of RepeaterMatrix and does not require an extra module.
      Check out the screencast
      What the module can do
      Duplicate a repeater item from one page to another No matter how complex the item is Full support for file and image fields Multilingual support Support of Min and Max settings Live synchronization of clipboard between multiple browser tabs. Copy an item and simply switch the browser tab to the target page and you will immediately see the past button Support of multiple RepeaterMatrix fields on one page Configurable which roles and fields are excluded Duplicated items are automatically pasted to the end of the target field and set to hidden status so that changes are not directly published Automatic clipboard update when other items are picked Automatically removes old clipboard data if it is not pasted within 6 hours Delete clipboard itself by clicking the selected item again Benefit: unbelievably fast workflow and content replication What the module can't do
      Before an item can be duplicated in its current version, the source page must be saved. This means that if you make changes to an item and copy this, the old saved state will be duplicated Dynamic loading is currently not possible. Means no AJAX. When pasting, the target page is saved completely No support for nested repeater items. Currently only first level items can be duplicated. Means a repeater field in a repeater field cannot be duplicated. Workaround: simply duplicate the parent item Dynamic reloading and adding of repeater items cannot be registered. Several interfaces and events from the core are missing. The initialization occurs only once after the page load event Changelog
      1.0.4
      Bug fix: Various bug fixes and improvements in live synchronization Bug fix: Items are no longer inserted when the normal save button is clicked. Only when the past button is explicitly clicked Feature: Support of multiple repeater fields in one page Feature: Support of repeater Min/Max settings Feature: Configurable roles and fields Enhancement: Improved clipboard management Enhancement: Documentation improvement Enhancement: Corrected few typos #1 1.0.3
      Feature: Live synchronization Enhancement: Load the module only in the backend Enhancement: Documentation improvement 1.0.2
      Bug fix: Various bug fixes and improvements in JS functions Enhancement: Documentation improvement Enhancement: Corrected few typos 1.0.1
      Bug fix: Various bug fixes and improvements in the duplication process 1.0.0
      Initial release Support this module
      If this module is useful for you, I am very thankful for your small donation: Donate 5,- Euro (via PayPal – or an amount of your choice. Thank you!)
      Download this module
      > Github: https://github.com/FlipZoomMedia/InputfieldRepeaterMatrixDuplicate
      > PW module directory: – soon –
    • By NorbertH
      Is there a hook to do something right after cloning a page ?
      I want the page to be saved right after cloning it either from the button in the tree or from a lister, because saving the page triggers several calculations that are not triggered by just cloning the page.
       
      Thanks in advance !
    • By John W.
      Question 1
      I recently installed PW 3.0.62 for a new site and also have sites running older version of PW 3.x.
      Can I export the database on an older version of PW 3.x and import it to PW 3.0.62 without any issues?
       
      Question 2
      (This is kind of alternative to the above for long term use - and maybe a better solution...)
      On  the sites I've previously built I have templates (home, basic-page, contact) and fields that I commonly use, such as business_name, phone_1.  The last site I built is running PW 3.0.42.  I was considering cloning this into a local site and running the upgrade module to bring it up to PW 3.0.62. From there on out when I start I new project I could just run the PW upgrade module, copy the folder to the location for the new project and duplicate the database using the new projects name.

      So basically, I'll always keep a "blank slate" site that I can just run the PW upgrade on, then duplicate into a new project. This would cut down on the work and time spent having to re-create these common fields, that I use. From there, I would just add fields, templates, etc, specific for the new website project.

      Is this a sound approach to speed up development?

       
    • By John W.
      I just posted a question about moving PW from a resting domain to a production domain, after consideration I figure I would create a new post for a second question I have.
      I noticed the coming feature of client size image resizing. This will be really helpful, as I have some clients on a Satellite connection. I'm telling you, them trying to upload 15MB images, or them getting them straight on batch resizers (or Photoshop) to resize before upload is a real chore. Needless to say, Ryan has been doing an excellent job addressing this.
      My question, is that I have several clients on PW 2.x thru 3.x. Is there a really good guide on steps that I need to take to update the core to take advantage of the features of new releases? Are their perils to look out for?
      Yeah, after developing several sites, I feel a bit silly about asking this one, but, the docs doesn't seem to be clear in a "general" sense of upgrading the core across all versions.
      Thanks everyone for your time and assistance.
×
×
  • Create New...