Jump to content
flydev

Duplicator: Backup and move sites

Recommended Posts

Duplicator-logo-v2-Dlong.thumb.png.8898ec8cda79046779b10d8527fc0def.png

 

There you can download the module : https://modules.processwire.com/modules/duplicator/
 

Quote

The documentation is incomplete, you have for now to explore the settings on your own. The install instructions are written but the SDKs are not uploaded yet, which mean that AmazonS3, Dropbox and GoogleDrive are only available by installing them through composer. I will upload them tomorrow but they are not recently tested.

Anyway, there is no requirements to run Duplicator as is. Just install Duplicator and Duplicator - Packages Manager then go to Setup > Duplicator and build your package by clicking the button. If everything run smooth, you will see the package listed and you will be able to download it directly and also the installer.

To restore a website, just upload the package and the installer then go to yourwebsite.com/installer.php then follow the instructions.

PLEASE, do not forget - after the end of the process - to delete the installer.php file and your package (this will be fixed in the v0.0.45).

 

It should work with default theme and the UIkit theme.

A dedicated thread will be created in the modules forum soon. Do not hesitate to share here your results by providing your hosting type / provider / config, etc. :)

 

---

Introduction

I was not aware of this module/topic and spend some time developing a module to backup, clone and move sites to a local folder or even GoogleDrive and Dropbox.

The idea begun when I read about this Wordpress's plugin. I was planning to add features like s/FTP, Amazon Cloud, deployment installer and others great features from Wordpress Duplicator and release it for free to the community.

Now that I see this @rot's module , I am wondering if it would be benefit to the community to have a second solution and so, if I should finish it.. ?

---

Features / Wishlist :

  • Save package to local folderimplemented
  • Save package to Amazon AWS : implemented
  • Save package to Dropboximplemented
  • Save package to GoogleDriveimplemented
  • Save package to FTP/FTPSimplemented
  • Manual backup with a standalone buttonimplemented
  • Backup from scheduled cron jobimplemented
  • Exclude by file extension / filename / directoryimplemented
  • File naming convention (Y-m-d_H-i-s-description.ext)   implemented
  • Protect access to the module screen by role/user   implemented
  • Installer / deployment functionality   implemented

---

2017-02-07: Starting the CLOSED BETA TESTING.

2017-11-28: Module release - version 0.0.44 at https://github.com/flydev-fr/Duplicator

 

 

screenshots:

1.png

2.png

 

Edited by flydev
Release of the v1.3.11-ATO DEV
  • Like 21
  • Thanks 1

Share this post


Link to post
Share on other sites

Great work! I think this module could be very handy. And I think if your module has other options than @rot's module you can release it. But if it is exactly the same, I don't know what solution would fit this situation.

  • Like 2

Share this post


Link to post
Share on other sites

Hey flydev,

Looks great. How far have you developed this?

The other module is at the very least beta software. I've got it running on several websites and it is running quite fast and smoothly. On the other hand I think there could be improvements. Perhaps you could join forces? Your modules both have the same goal. If you need help, I'm happy to help in any way.

  • Like 4

Share this post


Link to post
Share on other sites
58 minutes ago, arjen said:

If you need help, I'm happy to help in any way.

+1 :) Since @rot does not seem to have enough time to work on his module (he can correct me anytime... ;) ), I think it is quite reasonable for you to finish and maintain this module of yours. I would also be happy to use it, and share my findings about how it performs, etc.. meaning "testing it". 

  • Like 4

Share this post


Link to post
Share on other sites

It's weird, I was also about to start on something very similar, but also just recently noticed @rot's module, and now yours. @rot has been quiet here for a couple of months, so it would be good if he could confirm his intentions with updating his module. If you are close to being functional, it would be great to test both to figure out the differences in feature set.

  • Like 4

Share this post


Link to post
Share on other sites

Currently the module is functional, I use it on every site and it work quite well, no error since November.  At this moment only ProcessWire >3.0.0 is supported but I was planning to release it being compatible with 2.7.

 

What you can do with the module (working features) :

  • You can modify package's settings :  name of final filename, ignore some folders, set a maximum number of packages, remove older packages and trigger the backup function with LazyCron or PwCron.
  • Save the package in a local folder
  • Upload package to Dropbox
  • Upload package to GoogleDrive
  • Remove packages from those cloud services (depending on the configuration set)

*A package is a ZIP file bundled with a database dump and a full website directory dump (site, wire, etc)

 

I think I will finish the first public version and we'll see what happens as I tested the @arjen's fork of @rot's module and didn't managed to get it working as expected. I also find my module more "simple" to use; I mean you set options, check some checkbox, set a cronjob, click submit and let it run.

Also I saw some good features like emailing the logs on failure that could be added easily. I'll look at the module a little closer this afternoon.

 

Thanks for your input guys.

  • Like 8

Share this post


Link to post
Share on other sites

Thanks for the update @flydev - I am definitely very excited to try this - thanks for the hard work!

  • Like 3

Share this post


Link to post
Share on other sites
Quote

I also find my module more "simple" to use.

Thanks @flydev I think you are right about this. If you need any testing help, please let me know.

  • Like 2

Share this post


Link to post
Share on other sites

A small preview of uploading package to Dropbox and GoogleDrive at the same time :

 

duplicator-preview.gif

 

In this screencast I am using my own connection, thus the upload process is a bit slow.

 

 

  • Like 6

Share this post


Link to post
Share on other sites

Thanks for sharing the preview! A few questions:

  • Will it be possible to do a manual backup without changing the module's Event trigger? It is not practical to change it back-and-forth just to do it manually once in a while. Or am I missing something?
  • Is a timestamp like this possible: 2017-01-28_13-55-17 (year-month-day-hour-minute-second)? This can be alphabetically sorted so we also get the order arranged chronologically at the same time.
  • Are you planning to implement any sort of "installer functionality" to simplify deployment/restore from the packages a little bit?
  • Like 2

Share this post


Link to post
Share on other sites
5 minutes ago, szabesz said:

Will it be possible to do a manual backup without changing the module's Event trigger? It is not practical to change it back-and-forth just to do it manually once in a while. Or am I missing something?

Good point. How about a simple button "Backup Now" to do it ?

 

8 minutes ago, szabesz said:

Is a timestamp like this possible: 2017-01-28_13-55-17 (year-month-day-hour-minute-second)? This can be alphabetically sorted so we also get the order arranged chronologically at the same time.

This is a thing in which I have to brainstorm; But definitely yes. I am looking for a good solution to name the packages files. Thanks here.

 

16 minutes ago, szabesz said:

Are you planning to implement any sort of "installer functionality" to simplify deployment/restore from the packages a little bit?

It was stipulated on my first post, this will be the best part of the module :) 

This one is the hardest part to implement in my point of view so it will come in a second time but ASAP! What I mean by ASAP - I will finish to implement Amazon upload  and the s/FTP as this last feature is required to jump to the installer one.

  • Like 4

Share this post


Link to post
Share on other sites
  • "How about a simple button "Backup Now" to do it ?" - Sure :)
  • "I am looking for a good solution to name the packages files." - Normally I name them like this: 2017-01-28_13-55-17-anything-i-find-descriptive.zip

Since you dubbed the module "Duplicator" you also implied that it will also be able to help in installing so I just wanted to make sure I got that right.

Thanks again for the awesome work!

  • Like 3

Share this post


Link to post
Share on other sites

This is going to be amazing! - here is a wishlist.., not sure if these can be gradually implemented..

- backup from scheduled cron job
- exclude by file extension, for example exclude all .mp3, or .jpg etc;  /alternately if the above is not possible, then the existing functionality of excluding directory would be ok (like exclude the site/assets/files in case that folder is multiple GB; we often just sync that folder with an S3 directory to keep a backup)...
- protect access to the module screen by role/user

  • Like 5

Share this post


Link to post
Share on other sites
9 hours ago, Macrura said:

- backup from scheduled cron job

This is already implemented as far as I can see in the preview gif. The module requires PwCron and in the Event trigger dropdown there are time intervals to choose from. This is where an "OnSubmit" option is also included, but that one needs to be turned into a standalone button. @flydev That reminds me that you will probably need to handle the situation when a scheduled cron job will try to run while a manual backup is running (and vice versa...) I guess it is not a good idea to let both run at the same time.

Edit: but I might be mistaken about the two jobs running parallel being a problem... I was just guessing here.

Edited by szabesz
  • Like 3

Share this post


Link to post
Share on other sites

As @szabesz already spotted it, scheduled cron job is already implemented using PwCron. About the Event-trigger dropdown, its meant to work with LazyCron but I had in mind to give a try to interact with PwCron directly from the interface of Duplicator.

For the situation where a package will be built manually at the same time as a scheduled cron job, there is a File Locking technique implemented, so it should not be an issue.

 

On 29/01/2017 at 0:08 AM, Macrura said:

- exclude by file extension, for example exclude all .mp3, or .jpg etc;  /alternately if the above is not possible, then the existing functionality of excluding directory would be ok (like exclude the site/assets/files in case that folder is multiple GB; we often just sync that folder with an S3 directory to keep a backup)...

It should not be a problem - In the todo list !  Thanks you !

 

On 29/01/2017 at 0:08 AM, Macrura said:

- protect access to the module screen by role/user

Any clue how I could start that ?

 

22 hours ago, szabesz said:

This is where an "OnSubmit" option is also included, but that one needs to be turned into a standalone button.

duplicator-preview.gif

 

 

  • Like 2

Share this post


Link to post
Share on other sites
24 minutes ago, flydev said:

Any clue how I could start that ?

Something like the below in your module's init() method. You can of course change it to  lookup a role or roles or some user IDs [if(!in_array()...].

public function init() {	
	if ($this->permissions->get('duplicator')->id && !$this->user->hasPermission('duplicator'))
		 throw new WirePermissionException($this->_('Duplicator: You have no permission to use this module.'));
}

 

  • Like 2

Share this post


Link to post
Share on other sites
1 hour ago, flydev said:
On 29/01/2017 at 0:08 PM, Macrura said:

- protect access to the module screen by role/user

Any clue how I could start that ?

If the module is a Process module you can create and require a custom permission in the module info (that is, getModuleInfo(), etc).

See the ProcessHello module for an example: https://github.com/ryancramerdesign/ProcessHello/blob/9c1aa18eb40d069c7fb227f09badddc90f0b3276/ProcessHello.info.php#L33-L39

  • Like 2

Share this post


Link to post
Share on other sites

Just finished improving the exclude feature. You can now exclude multiple dir, files, or excluding files by extension.

 

ignore feature.png

 

Edited by flydev
typo
  • Like 6

Share this post


Link to post
Share on other sites

As I have just finished implementing the FTP transfer protocol, I hope it will be available for beta testing this week-end :-X. I have to implement Amazon before. I am going to buy an Amazon account tonight and implement this feature tomorrow.

 

Capture d’écran 2017-02-02 à 17.56.29.png

 

  • Like 10

Share this post


Link to post
Share on other sites

A small news for this morning, the AmazonS3 feature is implemented on the module (its a pain when you don't know about a SDK).

Also please check the name of the final package as it was suggested by @szabesz.  Thanks again for the suggestion, its better and easily manageable.

 

Edit:

For information, all SDKs are installed via composer. Is it a problem for you ?

 

duplicator-preview.gif

 

 

 

Edited by flydev
infos
  • Like 6
  • Thanks 1

Share this post


Link to post
Share on other sites

Thanks for your work => it looks very good!

On 2/5/2017 at 9:52 AM, flydev said:

For information, all SDKs are installed via composer. Is it a problem for you ?

Will this work together with ProcessWireUpgrade? This will make updates through more easily I presume.

  • Like 5

Share this post


Link to post
Share on other sites
3 minutes ago, arjen said:

Will this work together with ProcessWireUpgrade?

+1 Experienced developers should not have problem learning the basics of Composer, however, using the ProcessWireUpgrade module is easier and even site owners can use it out of the box, so generally I prefer relying on it instead of Composer. This is a module that should support ProcessWireUpgrade as well, I think.

  • Like 5

Share this post


Link to post
Share on other sites

Mmm the module itself will be, but updating the SDKs with ProcessWireUpgrade (if that what you mean) will be hard if I am not missing something with this module.

What I am thinking after some tests, and in particular about shared hosting,  its to put on Github the vendor directory and let the user download and upload it to the their server if they want to use third party service storage. This approach only work (as the module itself) for ProcessWire 2.8 and 3.x. And if some people want simplicity on their virtual/private server, they just have to run a simple command line - composer update - in the root directory. Really, composer is nothing more than a tool to manage dependencies, and installing/using it require only one command line or a simple exe setup on Windows.

Anyway, I will try to find a compromise and/or to figure how to manage theses dependencies in the module directory itself. If I can manage this, the module will be compatible with ProcessWire 2.7.

  • Like 4

Share this post


Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.


  • Recently Browsing   0 members

    No registered users viewing this page.

  • Similar Content

    • By David Karich
      ProcessWire InputfieldRepeaterMatrixDuplicate
      Thanks to the great ProModule "RepeaterMatrix" I have the possibility to create complex repeater items. With it I have created a quite powerful page builder. Many different content modules, with many more possible design options. The RepeaterMatrix module supports the cloning of items, but only within the same page. Now I often have the case that very design-intensive pages and items are created. If you want to use a content module on a different page (e.g. in the same design), you have to rebuild each item manually every time.
      This module extends the commercial ProModule "RepeaterMatrix" by the function to duplicate repeater items from one page to another page. The condition is that the target field is the same matrix field from which the item is duplicated. This module is currently understood as proof of concept. There are a few limitations that need to be considered. The intention of the module is that this functionality is integrated into the core of RepeaterMatrix and does not require an extra module.
      Check out the screencast
      What the module can do
      Duplicate a repeater item from one page to another No matter how complex the item is Full support for file and image fields Multilingual support Support of Min and Max settings Live synchronization of clipboard between multiple browser tabs. Copy an item and simply switch the browser tab to the target page and you will immediately see the past button Support of multiple RepeaterMatrix fields on one page Configurable which roles and fields are excluded Duplicated items are automatically pasted to the end of the target field and set to hidden status so that changes are not directly published Automatic clipboard update when other items are picked Automatically removes old clipboard data if it is not pasted within 6 hours Delete clipboard itself by clicking the selected item again Benefit: unbelievably fast workflow and content replication What the module can't do
      Before an item can be duplicated in its current version, the source page must be saved. This means that if you make changes to an item and copy this, the old saved state will be duplicated Dynamic loading is currently not possible. Means no AJAX. When pasting, the target page is saved completely No support for nested repeater items. Currently only first level items can be duplicated. Means a repeater field in a repeater field cannot be duplicated. Workaround: simply duplicate the parent item Dynamic reloading and adding of repeater items cannot be registered. Several interfaces and events from the core are missing. The initialization occurs only once after the page load event Changelog
      1.0.4
      Bug fix: Various bug fixes and improvements in live synchronization Bug fix: Items are no longer inserted when the normal save button is clicked. Only when the past button is explicitly clicked Feature: Support of multiple repeater fields in one page Feature: Support of repeater Min/Max settings Feature: Configurable roles and fields Enhancement: Improved clipboard management Enhancement: Documentation improvement Enhancement: Corrected few typos #1 1.0.3
      Feature: Live synchronization Enhancement: Load the module only in the backend Enhancement: Documentation improvement 1.0.2
      Bug fix: Various bug fixes and improvements in JS functions Enhancement: Documentation improvement Enhancement: Corrected few typos 1.0.1
      Bug fix: Various bug fixes and improvements in the duplication process 1.0.0
      Initial release Support this module
      If this module is useful for you, I am very thankful for your small donation: Donate 5,- Euro (via PayPal – or an amount of your choice. Thank you!)
      Download this module
      > Github: https://github.com/FlipZoomMedia/InputfieldRepeaterMatrixDuplicate
      > PW module directory: https://modules.processwire.com/modules/inputfield-repeater-matrix-duplicate/
    • By jaro
      This module (github) does with site/assets/files what Ryan's DatabaseBackups module does with the database:
      Backup site/assets Download ZIP archive Upload ZIP archive Restore site/assets Motivation: This module can be the missing part for projects with content backup responsibility on the client's side: The client will be able to download DB and assets/files snapshots through the backend without filesystem access, thus backing up all content themselves.
      Release state alpha – do not use in production environments.
      Credits for the nice UI go to @ryan – I reused most of it and some other code from the DatabaseBackups module.
    • By NorbertH
      Is there a hook to do something right after cloning a page ?
      I want the page to be saved right after cloning it either from the button in the tree or from a lister, because saving the page triggers several calculations that are not triggered by just cloning the page.
       
      Thanks in advance !
    • By John W.
      Question 1
      I recently installed PW 3.0.62 for a new site and also have sites running older version of PW 3.x.
      Can I export the database on an older version of PW 3.x and import it to PW 3.0.62 without any issues?
       
      Question 2
      (This is kind of alternative to the above for long term use - and maybe a better solution...)
      On  the sites I've previously built I have templates (home, basic-page, contact) and fields that I commonly use, such as business_name, phone_1.  The last site I built is running PW 3.0.42.  I was considering cloning this into a local site and running the upgrade module to bring it up to PW 3.0.62. From there on out when I start I new project I could just run the PW upgrade module, copy the folder to the location for the new project and duplicate the database using the new projects name.

      So basically, I'll always keep a "blank slate" site that I can just run the PW upgrade on, then duplicate into a new project. This would cut down on the work and time spent having to re-create these common fields, that I use. From there, I would just add fields, templates, etc, specific for the new website project.

      Is this a sound approach to speed up development?

       
×
×
  • Create New...