Jump to content
rot

Pre-Release: Remote Backup

Recommended Posts

I spent way too much of my spare time with trying to produce an overly complex site backup module. Anyway - it is here in a pre-release state. I somehow have to get rid of the monster.

Features:

Use Storage Providers

There are two base classes for Storage modules and three reference implementations:

  • Remote Storage Driver
    This is a baseclass for construcing plug-in modules that allow to send data to a remote storage.
    You need to extend all abstract functions: connect, disconnect, upload and getConfigFieldset

    Implemented Examples
    • Storage Mail
      Sends a backup as mail attachment. If the file size exceeds a set limit it will get split. It uses PHPMailer library as
      WireMail does not support attachments.

      @todo: For now this mails all in a single smtp session - maybe thats not so safe?
  • Remote Directory Driver
    This is a baseclass for construcing plug-in modules that allow to send data to a remote storage and list and delete old files.
    You need to extend all abstract functions: connect, disconnect, upload, find, size, mdate, delete and getConfigFieldset.
    Implemented Examples
    • Storage FTP
      Allows to connect to an ftp server and upload, list and delete files.
      Uses standard php ftp functions.
    • Storage Google Drive
      Allows to connect to google drive server and upload, list and delete files. Uses the php google api.
      You have to create a Service account with the google developers console and add the key file to the plugin directory (or another directory if you specify a relative or absolute path to that file).
      s. https://developers.google.com/identity/protocols/OAuth2ServiceAccount#creatinganaccount
      I don't use the OAuth token process because it is not more secure. Once there is a renew token (which is necessary to avoid user interaction) it is as powerful and insecure as a keyfile. It is just more complex as it needs a callback url for registering.

      @todo? In case you can prove otherwise I will implement the callback registration.

Run from the web or the command line

It's allways better to have a regular cron job running. But sometimes you might need webcron

  • Command Line
    You just need to call backup.php with the id of a backup job and it will be run
  • Web Cron
    There is a token that starts the backup job from the web if passed as a url parameter.
    You can specify whether you want logging the http stream or not. 
    You can also specify whether you want a job to be repeated within a certain timespan. This is for using unreliable webcron services by hitting the backup multiple times.
    @todo Consider integration of cron.pw
    @todo I use the init function of an automatically loaded module as a hook. This seems a bit strange. Is there better ways to do that?
     

Log to mail, file and admin

You can recieve logs by mail (on success or failure), log to a file and see log in a an admin page:

post-3268-0-56445800-1429030079_thumb.pn

Configure

I built a admin interface that - besides the log viewer - features a list of jobs:

post-3268-0-81485900-1429030644_thumb.pn

 

and an editor for the job (which is too extensive to be described in detail):

post-3268-0-26791900-1429030651_thumb.pn

 

Dicussion

I am not too sure on how to solve the issues indicated with @todo.

My main concern are the hooking (init of an autoload module for the moment) and locking (none, no singleton for the moment).

As for hooking I only know of the alternative of using a page where one would have (afaik) to use a special template as the admin template is secured or hook into the security functions (which would probably call for a singleton module).

Concerning the locking issue I think it might be good if the Admin Class would lock if it is updateing something. For the moment this is the same class that runs the backups thus it would also lock the admin if there is a backup running. And it would lock the whole site if it is on autoload (as I use the init hook).

Lastly I should reconsider the logging and maybe try to better integrate it with processwire logging.

I would appreciate comments and suggestionsn on these issues.

I appreciate your test results. Don't be took frutsrated if something goes wrong, this is at an early stage but afaik it should be running.

Please find the modulle on:
https://github.com/romanseidl/remote-backup

  • Like 14

Share this post


Link to post
Share on other sites

I use my own log file(s) by using the LogFile class. Is that all there is? There is this $this->message and $this->error functions. Maybe one should implement them too. At the moment I only implement log() and it does not forward to the Wire baseclass logger.

I will look at LazyCron Thx.

For the moment I seem to lose the database connection on long runninng jobs which is a problem because I want to save the results to the database:

Exception: SQLSTATE[HY000]: General error: 2006 MySQL server has gone away (in /home/.sites/306/site1268/web/wire/core/Modules.php line 2416)
This error message was shown because you are logged in as a Superuser. Error has been logged.xception: 
SQLSTATE[HY000]: General error: 2006 MySQL server has gone away (in /home/.sites/306/site1268/web/wire/core/Modules.php line 2416)
This error message was shown because you are logged in as a Superuser. Error has been logged.

Any ideas on how to avoid that?

Share this post


Link to post
Share on other sites

Hi @rot,

a real big monster! :)

Regarding to lost connections, you also should think about querying max_execution_time on start of a backup job.

And regardless if you have unlimited time, maybe better to run those jobs in batches?

Which results do you want to store to the database? Summary or each single action?

Share this post


Link to post
Share on other sites

These message and error functions are coming from the Notice class and are not related to the logs, but you can set the flag Notice::log or Notice::logOnly, so they get logged to the messages or errors log. You could extend this to add your own notices, which will be logged to your log file and show up as notice. The FileLog.php is essentially what the api has to offer. I can't see what more you're expecting from it. It's about writing log-messages to a file. 

Share this post


Link to post
Share on other sites
a real big monster! :)

Regarding to lost connections, you also should think about querying max_execution_time on start of a backup job.

And regardless if you have unlimited time, maybe better to run those jobs in batches?

Which results do you want to store to the database? Summary or each single action?

max_execution_time is set (by calling set_time_limit() - it probably only works if the php setup allows so - ill have to check). You can set that in the admin:)

post-3268-0-04055600-1429038003_thumb.pn

The Script runs until it is nearly finished. Then I want to save the result (info the job was successful and the log) to the database. So its probably what you call a "summary".

So maybe this is a seperate database timeout? Does processwire open a mysql connection for every request? Maybe that connection just dies after some minutes of doing nothing.

These message and error functions are coming from the Notice class and are not related to the logs, but you can set the flag Notice::log or Notice::logOnly, so they get logged to the messages or errors log. You could extend this to add your own notices, which will be logged to your log file and show up as notice. The FileLog.php is essentially what the api has to offer. I can't see what more you're expecting from it. It's about writing log-messages to a file. 

There can always be more :) Like e.g. class based loggin in a LOG4J style. But more is not always better.

At the moment I don't log to files by default (which might be a bad Idea considering the database timeouts...) but I used to log to an own log file.

  • Like 1

Share this post


Link to post
Share on other sites

The problem was a database timeout.

I fixed the timout problem by using the following to reconnect if needed.

    $data = $backup->run($web);
    //force reconnect if database has been lost due to long job runtime
    try {
            $this->setJobData($id, $data);
    } catch (Exception $e) {
        $this->modules->wire('database')->closeConnection();
            $this->setJobData($id, $data);
    }

Not the most beautiful solution.

I would prefer to have a $db->ping() method as there was with mysqli.

Maybe that would be a good extension to the DBO?

I created a pull request:

https://github.com/ryancramerdesign/ProcessWire/pull/1097

Its not dead important but I think it would be nice.

  • Like 1

Share this post


Link to post
Share on other sites

Concerning the locking issue: Is there even something like locking? Or am I just confused? As thre is no middleware (confuses me :)) there should be no problem with setting such a module to singular and still serve multiple requests in paralell. All locking there is is in the db. Or am I wrong?

I studied $modules->getModule() and it shows that :

  • $modules->get tries to find the module in the WireArray of $modules that should contain all installed modules.
  • Those who are on autoload have been initialized.
  • If it is NOT singular it gets recreated. And if it has not been initialized (because it is not on autoload or not singular) it get initialized.

This seem to imply that a module that is on autoload and not singular produces an autoload instance and then a new one for each $modules->get().

LazyCron is autoload but not singular and it hooks onto:

$this->addHookAfter('ProcessPageView::finished', $this, 'afterPageView');

Maybe it is not important that it is not singular as LazyCron is should not be directly called but hooked:

$this->addHook('LazyCron::every30Minutes', $this, 'myHook'); 

Also concerning the hook I looked at "Jumplinks" and this hooks to "Page not Found" which seems pretty nice for something that should try not to slow down the regular page view process:

$this->addHookBefore('ProcessPageView::pageNotFound', $this, 'scanAndRedirect', array('priority' => 10));

Funny enough I could not find out what Jumplinks is doing in terms of liefecycle. Probably it is using the getMolduleInfo defaults and thus it is not singular (but is it autoload? i suppose it has to be)

Share this post


Link to post
Share on other sites

I have also been considering to use a file for this kind of thing.

Probably it would be the easiest way also to provide some kind of "safe transactions" as for my case this is about long running processes that might die somewhere in the making. But I could also write that to the db as I am using that anyway to get process config info.

So I would have to set a start flag (in the db or to a file) and consider the job as dead if it doesn't reply within a certain amount of time.

  • Like 1

Share this post


Link to post
Share on other sites

EasyCron is a Web Cron service. A Web Cron repeatedly requests a user provided URL at a given interval and processes the result of the request.[/font][/color]

Edited by adrian
Removed link - I don't think that service is much help for this module.

Share this post


Link to post
Share on other sites

@rot, thanks a lot for this module!

I've tested both the FTP and Google Drive options for a site of mine hosted on a Digital Ocean VPS, and it worked flawlessly. It took less than a minute to create the backup files and upload the 650 MBs ZIP to Google Drive, which is quite impressive. I must say that setting up the Google Drive option was hard due to the lack of documentation, but since it's a pre-release, that's expected. 

  • Like 1

Share this post


Link to post
Share on other sites

Hi @rot,

Thanks for this module. Are you still using this? I'm wondering because I couldn't find it in the module repository.

It works really great. I've got some minor fixes (i.e. errors on 3.x due to namespaces, errors since my PW install is in a sub-folder) for which I will send a Pull Request.

Share this post


Link to post
Share on other sites

Hi arjen!

Yes, I am still using the module but with an old processwire install.  I just never published it to the repository.

Just send me the pull requests and I will recheck and finally publish the module. I did't find the time to do so when I first did it and then I just forgot.

It would be a waste if it works fine and others don't find it.

  • Like 3

Share this post


Link to post
Share on other sites
3 hours ago, rot said:

It would be a waste if it works fine and others don't find it.

Definitely. I was looking for something else in the forum and stumbled upon your module. It does exactly what another backup script (PHPBU) is doing, but now from within the admin panel.

  • Like 2

Share this post


Link to post
Share on other sites

Hi I just installed your module on pw version 3.0.36.

Immediately after installation, on the Admin > Remote Backup page, I get this error:

Notice: Undefined variable: out in /srv/users/serverpilot/apps/missions-pw/public/site/assets/cache/FileCompiler/site/modules/ProcessRemoteBackup/ProcessRemoteBackup.module on line 158

any ideas what may be happening? 

Share this post


Link to post
Share on other sites

Hi @rastographics, this module won't work out of the box with PW 3.x. You need to do some changes to make it work. I don't know which one from memory, but I got a copy running locally at home. I might find some time tonight to update the code and send @rot the Pull Request I promised.

  • Like 2

Share this post


Link to post
Share on other sites

No problem. I want to install this on several sites we are running. Looks like it has to be done this weekend. Will get back to you.

  • Like 1

Share this post


Link to post
Share on other sites

So, finally got some time to get back to setting up new backup services and it turns out from > 3.0.42 (latest stable) fixes the Filecompiler issues ;) I think Ryan made some changes regarding the file compiler a while ago. 

For now I've changed:

  1. FTP with TLS/SSL support
  2. Dynamically loading of ProcessWire bases on the current folder

I have sent a pull request. 

  • Like 3

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

  • Recently Browsing   0 members

    No registered users viewing this page.

  • Similar Content

    • By jploch
      Hey folks,
      for a module (a pagebuilder based on PageTable) I need to save some settings as JSON. The values are saved for each page table item (a pw page). It's working well, but I am looking for ways to improve the structure I have. As I'm not that experienced with JSON, maybe someone more experienced can take a look and tell me if my approach is good practice. 

      My goal is to make all the items accessible by page id, without looping over them (using objects instead of arrays):
      // access from template with pw page var $jsonObject->items->{$page}->cssClass; Her is an example of my JSON structure:
      { "items": { "3252": { "id": "3252", "cssClass": "pgrid-main", "breakpoints": { "base": { "css": { "grid-column-end": "auto", "grid-row-end": "auto", "grid-column-start": "auto", "grid-row-start": "auto", "align-self": "auto", "z-index": "auto", "padding-left": "60px", "padding-right": "60px", "padding-top": "60px", "padding-bottom": "60px", "background-color": "rgb(255, 255, 255)", "color": "rgb(0, 0, 0)" }, "size": "@media (min-width: 576px)", "name": "base" } } }, "3686": { "id": "3686", "cssClass": "test_global", "breakpoints": { "base": { "css": { "grid-column-end": "-1", "grid-row-end": "span 1", "grid-column-start": "1", "grid-row-start": "auto", "align-self": "auto", "z-index": "auto", "padding-left": "0px", "padding-right": "0px", "padding-top": "0px", "padding-bottom": "0px", "background-color": "rgba(0, 0, 0, 0)", "color": "rgb(0, 0, 0)" }, "size": "@media (min-width: 576px)", "name": "base" } } }, "3687": { "id": "3687", "cssClass": "block_editor-3687", "breakpoints": { "base": { "css": { "grid-column-end": "span 2", "grid-row-end": "span 1", "grid-column-start": "auto", "grid-row-start": "auto", "align-self": "auto", "z-index": "auto", "padding-left": "0px", "padding-right": "0px", "padding-top": "0px", "padding-bottom": "0px", "background-color": "rgba(0, 0, 0, 0)", "color": "rgb(0, 0, 0)" }, "size": "@media (min-width: 576px)", "name": "base" } } }, "3696": { "id": "3696", "cssClass": "block_editor-3696", "breakpoints": { "base": { "css": { "grid-column-end": "span 2", "grid-row-end": "span 1", "grid-column-start": "auto", "grid-row-start": "auto", "align-self": "auto", "z-index": "auto", "padding-left": "0px", "padding-right": "0px", "padding-top": "0px", "padding-bottom": "0px", "background-color": "rgba(0, 0, 0, 0)", "color": "rgb(0, 0, 0)" }, "size": "@media (min-width: 576px)", "name": "base" } } } }, "breakpointActive": "base", "breakpointActiveSize": "@media (min-width: 576px)" }  
    • By jploch
      Fieldtype Page Table Grid
      This is a sneak preview of a side project I've been working on for quite some time now. A lot of work and thought has gone into this, so I will most likely release this as a commercial module at some point in the near future. 

      As a designer (and developer) I get the appeal of a WYSIWYG editor. After playing around with some WYSIWYG page builder tools, I always felt something was wrong about them. So I decided to build my own PW version based on PageTable.

      Here is a small demo (using AdminThemeCanvas, but its working with other admin themes as well) :
      There is also a complete website that I built for a friend of mine using this module and some custom blocks.
      Concept
      This fieldtype shares a lot of features with PageTableExtended: it's also an extension of PageTable and renders the block templates in the backend and frontend (native PW templates and fields). You can also add your own css via module settings.
      The difference is, this fieldtype also gives you the ability to rearrange and resize elements in a visual way as well as enable inline editing for text, ckeditor and file fields. Similar (and promising) attempts have been made, but I wanted something based on native CSS grid instead of a CSS framework...so I built my own version. Most CSS frameworks are based on flexbox, which is great for layouting elements horizontally. With CSS grid, you can place elements horizontally and vertically, allowing for layouts that were not previously possible with CSS. Similar to webflow, this fieldtype uses javascript (in the backend) to let you manipulate CSS grid in a visual way to design fully responsive websites (or parts of them). It should still be possible to include a CSS framework if you like (just add the classes to your block markup and include the CSS via module settings).
      The CSS grid layout manipulations are saved in a single field as a JSON array and used to generate a dynamic stylesheet that you simply include in your main template (no inline styles). The styles are saved within the breakpoint you select and cascade down to smaller breakpoints. That means you can specify just the basic breakpoint and adjust other breakpoints if needed. The exception is the mobile breakpoint which will display everything in one column as a default (you can change the layout here too).
      The fieldtype also comes with an optional style panel to manipulate some additional CSS properties directly on the page. You can customize the panel or disable it completely from the module settings (and just use a CSS file that you include via module settings). The style panel is based on inputfields (nothing is saved to the database). This means that you just have to install the module and all fields are available to all blocks automatically (this can be customized). It also has the benefit that your installation is not flooded with fields; this module only installs one field.
      Don't want to give your customer all that power? Design features can be disabled for certain roles. The grid editor role can just edit the content and use the inline editing feature to edit content quickly. You can then also grant access individually to the style panel, resize or drag functionality.
      Features
      Blocks are just pages Blocks are defined by native PW templates and fields Manipulate CSS grid in a visual way to design fully responsive websites (or parts of them) Design features can be disabled for certain roles Inline editing of text, ckeditor and file fields The layout is 100% CSS grid (very small css file size) Simply drag and resize to manipulate grid items directly inside the backend Manipulate grid columns and rows directly on the page (use any number of columns you want) All style manipulations are saved as JSON and used to generate a dynamic stylesheet that you just include in your main template (no inline styles) Nested groups/grids (child pages of nested blocks are created under group parent) Global blocks work with page reference field (changes on one page, changes all blocks on all pages) Manual and auto placement of grid items Define custom icons for your blocks via native template settings (template -> advanced -> icon) Option to load lazysizes in the backend to enable lazy loading of assets with class lazyload Works with all default and ui-kit based admin themes If you have any questions or feedback, let me know.
    • By bernhard
      I built this module because I needed a versatile solution to replace tags and simple if-blocks in some E-Mails and PDF documents.
      If you only need to replace static tags (no if-conditions), then you can use default PW api and need no module:
      $str = "My favourite color is {color}."; $texttools = $sanitizer->getTextTools(); echo $texttools->populatePlaceholders($str, ['color' => 'red']); // output: My favourite color is red. Usage:
      See the two example Files in the folder /replacements

       
      Methods:
      replacementsTable()
      Renders an overview of all available replacements (see the example in the Module's config file:
       
      Create new Replacements:
      Simply copy the sample file and adopt to your needs.
       
      Download:
      https://gitlab.com/baumrock/RockReplacer
    • By bernhard
      DEPRECATED
      I'm using this module in several projects, but it will likely not see any updates in the future. I'm not happy with it and I'm looking for ways to develop better solutions. RockTabulator was my first try, but I'm also not 100% happy with that. The tabulator library is great, but my module implementation is not. I hope to get a good solution soon, but it will be a lot of work...
      ---
      Some of you might have followed the development of this module here: https://processwire.com/talk/topic/15524-previewdiscussion-rockdatatables/ . It is the successor of "RockDataTables" and requires RockFinder to get the data for the grid easily and efficiently. It uses the open source part of agGrid for grid rendering.
       
      WHY?
      ProcessWire is awesome for creating all kinds of custom backend applications, but where it is not so awesome in my opinion is when it comes to listing this data. Of course we have the built in page lister and we have ListerPro, but none of that solutions is capable of properly displaying large amounts of data, for example lists of revenues, aggregations, quick and easy sorts by the user, instant filter and those kind of features. RockGrid to the rescue 😉 
       
      Features/Highlights:
      100k+ rows Instant (client side) filter, search, sort (different sort based on data type, eg "lower/greater than" for numbers, "contains" for strings) extendable via plugins (available plugins at the moment: fullscreen, csv export, reload, batch-processing of data, column sum/statistics, row selection) all the agGrid features (cell renderers, cell styling, pagination, column grouping etc) vanilla javascript, backend and frontend support (though not all plugins are working on the frontend yet and I don't plan to support it as long as I don't need it myself)  
      Limitations:
      While there is an option to retrieve data via AJAX the actual processing of the grid (displaying, filtering, sorting) is done on the client side, meaning that you can get into troubles when handling really large datasets of several thousands of rows. agGrid should be one of the most performant grid options in the world (see the official example page with a 100k row example) and does a lot to prevent problems (such as virtual row rendering), but you should always have this limitation in mind as this is a major difference to the available lister options that do not have this limitation.
      Currently it only supports AdminThemeUikit and I don't plan to support any other admin theme.
       
      Download: https://gitlab.com/baumrock/FieldtypeRockGrid
      Installation: https://gitlab.com/baumrock/RockGrid/wikis/Installation
      Quikckstart: https://gitlab.com/baumrock/RockGrid/wikis/quickstart
      Further instructions: https://gitlab.com/baumrock/RockGrid/wikis/quickstart#further-instructions
      German Translation File: site--modules--fieldtyperockgrid--fieldtyperockgrid-module-php.json
      Changelog: https://gitlab.com/baumrock/FieldtypeRockGrid/raw/master/changelog.md
       
      Module status: alpha, License: MIT
      Note that every installation and uninstallation sends an anonymous google analytics event to my google analytics account. If you don't want that feel free to remove the appropriate lines of code before installation/uninstallation.
       
      Contribute:
      You can contribute to the development of this and other modules or just say thank you by
      testing, reporting issues and making PRs at gitlab liking this post buying me a drink: paypal.me/baumrock/5 liking my facebook page: facebook.com/baumrock hiring me for pw work: baumrock.com  
      Support: Please note that this module might not be as easy and plug&play as many other modules. It needs a good understanding of agGrid (and JavaScript in general) and it likely needs some looks into the code to get all the options. Please understand that I can not provide free support for every request here in the forum. I try to answer all questions that might also help others or that might improve the module but for individual requests I offer paid support (please contact me via PM).
       
      Use Cases / Examples:
      Colored grid cells, Icons, Links etc. The Grid also has a "batcher" feature built in that helps communicating with the server via AJAX and managing resource intensive tasks in batches:

      Filters, PW panel links and instant reload on panel close:

      You can combine the grid with a chart library like I did with the (outdated) RockDataTables module:

    • By Paul Greinke
      Hi there. I wrote a custom module for one of my projects. In fact I maybe want to use my module in other projects too. In order to be variable and customizable  I need to implement some custom hooks into my module. So I can afterwards hook into the my functions in order to modify them to match the needs of the new project.
      I tried simply defining functions with the '__' prefix. But that did not work. I'm imagining something like the following:
      <?php class MyClass { public function ___someFunction() { // Do something } } // ready.php $this->addHookBefore('MyClass::someFunction', function($event) { // some customization }); Is there a way to accomplish that? 
×
×
  • Create New...