☁️ Duplicator: Backup and move sites

[flydev 👊🏻 1,859]

 

Current version: 1.3.14-ATO

You can find the module on Github and in the modules directory :

• https://modules.processwire.com/modules/duplicator/
• https://github.com/flydev-fr/Duplicator/archive/master.zip

 

Quote

To restore a website, just upload the package and the installer, then go to yourwebsite.com/installer.php then follow the instructions.

PLEASE, do not forget - after the end of the process - to delete the installer.php file and your package 

 

Screenshots / Features

• Dir and files exclusion

 

• CRON job

 

• Advanced settings 

 

• Local and Cloud storage

 

Changelog :

Spoiler

[Unreleased]

• new logging mechanism
• auto-deploy thought FTP/FTPS
• installer optional deletion

[1.3.14-ATO] - 2020-01-06

ATO DEVELOPMENT version

Fixed

• Fix: undefined variable ($zipFilename)

[1.3.13-ATO] - 2020-01-06

ATO DEVELOPMENT version

Added

• Supporting ~ in local path to prefix input with current webroot ($config->paths->root)

Fixed

• Fix: Removing colon from default package name (port in URL)
• Fix: Custom filename not used
• Fix: Archive function not returning ZIP (lead to fail of FTP etc.)

[1.3.12-ATO] - 2019-05-15

ATO DEVELOPMENT version

Added

• Duplicator: added a config option to set the quantity of megabytes saved before an archive flush

[1.3.11-ATO] - 2019-05-14

ATO DEVELOPMENT version

Fixed

• ProcessDuplicator: support dash char in domain name

[1.3.10-ATO] - 2019-03-18

ATO DEVELOPMENT version

Added

• Added regexp exclusion feature
• Added checkbox for ProcessWire image variations exclusion (checked by default)

Fixed

• fix installer.php

Changed

• zipData revised
• package structure: SQL dump included in site structure zip
• exclusion feature: extensions are now case-insensitive

[1.2.9] - 2018-04-09

Added

• added Dropbox v2 support (composer support only)
• added Package Manager shortcut to Duplicator

[1.1.9] - 2018-04-09

Fixed

• fix ProcessDuplicator: log file existence check on initialization

[1.1.8] - 2018-04-09

Changed

• ProcessDuplicator UI

Fixed

• fix Duplicator: ZIP file (zipData) work properly on Windows and Unix

[1.1.7] - 2017-12-12

Changed

• changed check for cron triggers from ready() to init()
• changed support links in both modules

[1.1.6] - 2017-12-09

Added

• added js debug mode

Fixed

• fix Duplicator: zipData() RecursiveFilterIterator check for unreadable dir
• fix Duplicator: getLocalFolderInfo() check if the destination path exist
• fix ProcessDuplicator: corrected the trash-icon action link
• fix ProcessDuplicator: packages stored on the cloud are shown in the table even if LocalFodler is not used
• various fix in DUP_Util class

[1.1.5] - 2017-12-07

Fixed

• fix ProcessDuplicator: checking for mod_userdir is not needed anymore
• fix: packages are retrieved, blended and re-ordered by timestamp

[1.1.4] - 2017-12-07

Fixed

• fix ProcessDuplicator: now it work with Apache mod_userdir

[1.1.3] - 2017-12-07

Added

• ___cronJob() is now hookable.

Fixed

• fix FTP class were DEFINE() was replaced by Constants in earlier version
• fix HookEvent in cronJob()

[1.1.2] - 2017-12-04

Added

• added support for running cron jobs wihtout external module

[1.0.2] - 2017-12-03

Fixed

• fix the module's version number for the ProcessWire modules directory

Removed

• removed Dropbox support due to the deprecation of the API v1 since september 2017 - link

[1.0.1] - 2017-12-01

Fixed

• the installer parse correctly the config.php file on windows machine
• small typo in ProcessDuplicator

[1.0.0] - 2017-11-30

Added

• check if ProcessDuplicator module is installed then display the link in the local folder overview to the manager if yes.

Fixed

• changed all _() call to __() for translation string

 

Edited January 12, 2020 by flydev 👊🏻
Release of the v1.3.14

[Harmen 58]

Great work! I think this module could be very handy. And I think if your module has other options than @rot's module you can release it. But if it is exactly the same, I don't know what solution would fit this situation.

[arjen 1,381]

Hey flydev,

Looks great. How far have you developed this?

The other module is at the very least beta software. I've got it running on several websites and it is running quite fast and smoothly. On the other hand I think there could be improvements. Perhaps you could join forces? Your modules both have the same goal. If you need help, I'm happy to help in any way.

[szabesz 2,846]

58 minutes ago, arjen said:

If you need help, I'm happy to help in any way.

+1  Since @rot does not seem to have enough time to work on his module (he can correct me anytime...  ), I think it is quite reasonable for you to finish and maintain this module of yours. I would also be happy to use it, and share my findings about how it performs, etc.. meaning "testing it". 

[adrian 12,063]

It's weird, I was also about to start on something very similar, but also just recently noticed @rot's module, and now yours. @rot has been quiet here for a couple of months, so it would be good if he could confirm his intentions with updating his module. If you are close to being functional, it would be great to test both to figure out the differences in feature set.

[flydev 👊🏻 1,859]

Currently the module is functional, I use it on every site and it work quite well, no error since November.  At this moment only ProcessWire >3.0.0 is supported but I was planning to release it being compatible with 2.7.

 

What you can do with the module (working features) :

• You can modify package's settings :  name of final filename, ignore some folders, set a maximum number of packages, remove older packages and trigger the backup function with LazyCron or PwCron.
• Save the package in a local folder
• Upload package to Dropbox
• Upload package to GoogleDrive
• Remove packages from those cloud services (depending on the configuration set)

*A package is a ZIP file bundled with a database dump and a full website directory dump (site, wire, etc)

 

I think I will finish the first public version and we'll see what happens as I tested the @arjen's fork of @rot's module and didn't managed to get it working as expected. I also find my module more "simple" to use; I mean you set options, check some checkbox, set a cronjob, click submit and let it run.

Also I saw some good features like emailing the logs on failure that could be added easily. I'll look at the module a little closer this afternoon.

 

Thanks for your input guys.

[adrian 12,063]

Thanks for the update @flydev - I am definitely very excited to try this - thanks for the hard work!

[arjen 1,381]

Quote

I also find my module more "simple" to use.

Thanks @flydev I think you are right about this. If you need any testing help, please let me know.

[flydev 👊🏻 1,859]

A small preview of uploading package to Dropbox and GoogleDrive at the same time :

 

 

In this screencast I am using my own connection, thus the upload process is a bit slow.

 

 

[szabesz 2,846]

Thanks for sharing the preview! A few questions:

• Will it be possible to do a manual backup without changing the module's Event trigger? It is not practical to change it back-and-forth just to do it manually once in a while. Or am I missing something?
• Is a timestamp like this possible: 2017-01-28_13-55-17 (year-month-day-hour-minute-second)? This can be alphabetically sorted so we also get the order arranged chronologically at the same time.
• Are you planning to implement any sort of "installer functionality" to simplify deployment/restore from the packages a little bit?

[flydev 👊🏻 1,859]

5 minutes ago, szabesz said:

Will it be possible to do a manual backup without changing the module's Event trigger? It is not practical to change it back-and-forth just to do it manually once in a while. Or am I missing something?

Good point. How about a simple button "Backup Now" to do it ?

 

8 minutes ago, szabesz said:

Is a timestamp like this possible: 2017-01-28_13-55-17 (year-month-day-hour-minute-second)? This can be alphabetically sorted so we also get the order arranged chronologically at the same time.

This is a thing in which I have to brainstorm; But definitely yes. I am looking for a good solution to name the packages files. Thanks here.

 

16 minutes ago, szabesz said:

Are you planning to implement any sort of "installer functionality" to simplify deployment/restore from the packages a little bit?

It was stipulated on my first post, this will be the best part of the module  

This one is the hardest part to implement in my point of view so it will come in a second time but ASAP! What I mean by ASAP - I will finish to implement Amazon upload  and the s/FTP as this last feature is required to jump to the installer one.

[szabesz 2,846]

• "How about a simple button "Backup Now" to do it ?" - Sure
• "I am looking for a good solution to name the packages files." - Normally I name them like this: 2017-01-28_13-55-17-anything-i-find-descriptive.zip

Since you dubbed the module "Duplicator" you also implied that it will also be able to help in installing so I just wanted to make sure I got that right.

Thanks again for the awesome work!

[Macrura 3,082]

This is going to be amazing! - here is a wishlist.., not sure if these can be gradually implemented..

- backup from scheduled cron job
- exclude by file extension, for example exclude all .mp3, or .jpg etc;  /alternately if the above is not possible, then the existing functionality of excluding directory would be ok (like exclude the site/assets/files in case that folder is multiple GB; we often just sync that folder with an S3 directory to keep a backup)...
- protect access to the module screen by role/user

[szabesz 2,846]

9 hours ago, Macrura said:

- backup from scheduled cron job

This is already implemented as far as I can see in the preview gif. The module requires PwCron and in the Event trigger dropdown there are time intervals to choose from. This is where an "OnSubmit" option is also included, but that one needs to be turned into a standalone button. @flydev That reminds me that you will probably need to handle the situation when a scheduled cron job will try to run while a manual backup is running (and vice versa...) I guess it is not a good idea to let both run at the same time.

Edit: but I might be mistaken about the two jobs running parallel being a problem... I was just guessing here.

Edited January 29, 2017 by szabesz

[flydev 👊🏻 1,859]

As @szabesz already spotted it, scheduled cron job is already implemented using PwCron. About the Event-trigger dropdown, its meant to work with LazyCron but I had in mind to give a try to interact with PwCron directly from the interface of Duplicator.

For the situation where a package will be built manually at the same time as a scheduled cron job, there is a File Locking technique implemented, so it should not be an issue.

 

On 29/01/2017 at 0:08 AM, Macrura said:

- exclude by file extension, for example exclude all .mp3, or .jpg etc;  /alternately if the above is not possible, then the existing functionality of excluding directory would be ok (like exclude the site/assets/files in case that folder is multiple GB; we often just sync that folder with an S3 directory to keep a backup)...

It should not be a problem - In the todo list !  Thanks you !

 

On 29/01/2017 at 0:08 AM, Macrura said:

- protect access to the module screen by role/user

Any clue how I could start that ?

 

22 hours ago, szabesz said:

This is where an "OnSubmit" option is also included, but that one needs to be turned into a standalone button.

 

 

[kongondo 7,381]

24 minutes ago, flydev said:

Any clue how I could start that ?

Something like the below in your module's init() method. You can of course change it to  lookup a role or roles or some user IDs [if(!in_array()...].

public function init() {	
	if ($this->permissions->get('duplicator')->id && !$this->user->hasPermission('duplicator'))
		 throw new WirePermissionException($this->_('Duplicator: You have no permission to use this module.'));
}

 

[Robin S 7,397]

1 hour ago, flydev said:

On 29/01/2017 at 0:08 PM, Macrura said:

- protect access to the module screen by role/user

Any clue how I could start that ?

If the module is a Process module you can create and require a custom permission in the module info (that is, getModuleInfo(), etc).

See the ProcessHello module for an example: https://github.com/ryancramerdesign/ProcessHello/blob/9c1aa18eb40d069c7fb227f09badddc90f0b3276/ProcessHello.info.php#L33-L39

[flydev 👊🏻 1,859]

Just finished improving the exclude feature. You can now exclude multiple dir, files, or excluding files by extension.

 

 

Edited January 31, 2017 by flydev
typo

[szabesz 2,846]

Pretty cool. I especially like the shortcuts of logs/sessions/cache  

[elabx 865]

When can we get our hands on this? Where do I throw my money?!

[flydev 👊🏻 1,859]

As I have just finished implementing the FTP transfer protocol, I hope it will be available for beta testing this week-end . I have to implement Amazon before. I am going to buy an Amazon account tonight and implement this feature tomorrow.

 

 

[flydev 👊🏻 1,859]

A small news for this morning, the AmazonS3 feature is implemented on the module (its a pain when you don't know about a SDK).

Also please check the name of the final package as it was suggested by @szabesz.  Thanks again for the suggestion, its better and easily manageable.

 

Edit:

For information, all SDKs are installed via composer. Is it a problem for you ?

 

 

 

 

Edited February 5, 2017 by flydev
infos

[arjen 1,381]

Thanks for your work => it looks very good!

On 2/5/2017 at 9:52 AM, flydev said:

For information, all SDKs are installed via composer. Is it a problem for you ?

Will this work together with ProcessWireUpgrade? This will make updates through more easily I presume.

[szabesz 2,846]

3 minutes ago, arjen said:

Will this work together with ProcessWireUpgrade?

+1 Experienced developers should not have problem learning the basics of Composer, however, using the ProcessWireUpgrade module is easier and even site owners can use it out of the box, so generally I prefer relying on it instead of Composer. This is a module that should support ProcessWireUpgrade as well, I think.

[flydev 👊🏻 1,859]

Mmm the module itself will be, but updating the SDKs with ProcessWireUpgrade (if that what you mean) will be hard if I am not missing something with this module.

What I am thinking after some tests, and in particular about shared hosting,  its to put on Github the vendor directory and let the user download and upload it to the their server if they want to use third party service storage. This approach only work (as the module itself) for ProcessWire 2.8 and 3.x. And if some people want simplicity on their virtual/private server, they just have to run a simple command line - composer update - in the root directory. Really, composer is nothing more than a tool to manage dependencies, and installing/using it require only one command line or a simple exe setup on Windows.

Anyway, I will try to find a compromise and/or to figure how to manage theses dependencies in the module directory itself. If I can manage this, the module will be compatible with ProcessWire 2.7.