Jump to content
jaro

AutoExportTemplatesAndFields enables continuous integration of template and field configuration changes

Recommended Posts

Hello,

in one of my current projects, we have a test and production stage with their own ProcessWire installations, and we regularly want to migrate template/field configuration to the next stage without migrating content as well. ProcessWire supports this via the import/export functionality in the admin GUI. However we (and some others) would like to do this as part of an automated process. There seem to have been some discussion on this topic and two existing modules that get pretty near our requirements:

However, they try to solve more then we need and for mindplay-dk's module, development seems to have discontinued. At that point I decided to build a more simple module with reduced scope that basically just makes ProcessWire's import/export automation-ready.

Thanks in advance for trying this out and any feedback!

About this Module

CAUTION: This module is incompatible with repeater fields and ProFields.

The module enables you to transfer template and field configuration from one processwire installation to another in an automated way.

  • Changes to templates or fields are exported to the file system immediately (so they can be added to source control together with changes in the template files)
  • The import script is designed to run from the command line (so it can be invoked from your build/deployment tool). On invocation in import mode,
    • a DB backup is created
    • template/field changes from the persist directory are imported into the DB
  • In restore mode, the import script can restore any of the DB backups previously saved

How to Use

  1. Make sure the module is installed in the source and destination ProcessWire environments.
  2. After installation of the module, you should check if the module's default persistDirectory configuration setting fits your requirements. The module will automatically export all fields, fieldgroups, and templates to JSON files in the directory specified by that setting. Note that the same setting is used by the import script as well, so if you change it, make sure you change it in all affected ProcessWire environments.
  3. The JSON files can be transferred to the destination ProcessWire environment.
  4. Running the import script from the command line will import template and field data in the destination ProcessWire environment.

Manual Installation

Caution: Beta software - do not use in a production environment without prior testing and regular back-ups.

Caution: This module is incompatible with repeater fields and ProFields.

In case automatic installation does not work, get the code at https://github.com/jaromic/pw-autoexport-tf/ and follow the instructions in README.md for manual installation.

Manual Uninstall

Delete the following files and directories from your module directory:

AutoExportTemplatesAndFields/
AutoExportTemplatesAndFields.module
 Download

Install from the module directory, clone the repository via GitHub, or download ZIP.

Edited by jaro
Statement about incompatibility added
  • Like 15

Share this post


Link to post
Share on other sites

Hi and welcome to the forums.

This sounds interesting. I definetly will try it out, but not soon. Maybe in a week or two.

The DB-backup integration looks good. :)

Do you have used it with namespaced PW version 3 too?

Share this post


Link to post
Share on other sites

Thanks, horst :)

Using that version in our project, I have developed and tested for 2.6 only so far. It does not use namespaces, so 3.x will probably not load the module. I will adapt and test with 2.7 and 3.x as soon as my time allows, hope to get to that within the next two weeks too.

  • Like 1

Share this post


Link to post
Share on other sites

Now, that's some nice first post to have in the forums, so first of all welcome here.

I just wanted to add a word of caution here. The export functionality should work well for core fieldtypes, but 3rd party fieldtypes might not export/import their settings correctly. Especially for more complex (db-wise) fieldtypes like Table or maybe Matrix it's often not so simple to export to a text format. E.g. FieldtypeOptions didn't export it's options for quite some time. 

Also I'd be cautious with calling modules discontinued, which is, at least for my module, not the truth. Not seeing a weekly changelog is not necessarily a bad thing. 

  • Like 2

Share this post


Link to post
Share on other sites

Thanks, and sorry, I just corrected that statement about your module.

Even if it cannot handle all field types that some other modules contribute, I hope I'll be able to improve my module to the point where it reliably warns about field types causing problems.

Share this post


Link to post
Share on other sites

@jaro @LostKobrakai

<dream mode>

Not sure if this is a good idea or not, but could this module be made to automatically export migrations that are compatible with Benjamin's migrations module? This would allow his module's interface to be used on the importing side to select which migrations to import.

</dream mode>

Edited to add: Now I've actually tried this module, I think the dream outlined above is more of a delusion. As it stands, the module does seem like a very neat way to keep the state of the fields and templates under version control along with the template files; nice work, @jaro!

Share this post


Link to post
Share on other sites
On 3.9.2016 at 1:59 PM, netcarver said:

Not sure if this is a good idea or not, but could this module be made to automatically export migrations that are compatible with Benjamin's migrations module? This would allow his module's interface to be used on the importing side to select which migrations to import.

Sorry it took me a while to answer, and thanks for the input and for trying it out!

The module is all about ProcessWire's built-in import/export functionality, merely exposing and automating it. For the moment I'd rather have it become good and stable in this than doing both formats mediocrely. For example, there are still issues with some common field types, and I want to look into special cases during import that are handled specially by the GUI when doing it graphically.

On 12.7.2016 at 5:01 PM, horst said:

Do you have used it with namespaced PW version 3 too?

In the meantime I did some tests with ProcessWire versions 2 and 3. The module itself worked without any changes both in v2 and v3 installations (thanks to ProcessWire's namespace magic), but I had to do differentiate in the import script. v1.0.1 contains that and some small fixes.

  • Like 1

Share this post


Link to post
Share on other sites

Hi,

I Installed the "AutoExportTemplatesAndFields" and when creating repeater fields the module was affecting the fields editing.

I had two repeater fields: "repeater1" and "repeater2", the "repeater1" should repeat the "images1" field and the "repeater2" should repeat the "images2",

the problem was each time I edited one of them and change the repeated field the other assumed that modification. Example: editing the "repeater1" and change the field to "image3" the "repeater2" instead of having the "image2" has the field to be repeated assumed the field "image3".

After uninstall the "AutoExportTemplatesAndFields" module I could edit each repeater and assign different fields to them.

I hope you can find the issue it is a great module to use.

Thank you.

  • Like 3

Share this post


Link to post
Share on other sites
On 1.4.2017 at 5:31 PM, vmo said:

I had two repeater fields [...] the problem was each time I edited one of them and change the repeated field the other assumed that modification. [...] I hope you can find the issue it is a great module to use.

Hi, thanks for that bug report, and I'm glad you like that module! I hope I'll have some time to look into this soon. I have added it to the issues on github for now.

  • Like 1

Share this post


Link to post
Share on other sites
On 4/6/2017 at 2:25 AM, jaro said:

Hi, thanks for that bug report, and I'm glad you like that module! I hope I'll have some time to look into this soon. I have added it to the issues on github for now.

Hi i ran into the same issue und it cost me much time to find the culprit of it. Please at least post a big warning in the readme for users to be aware before installation since i suspect this can affect many users who have installed pro fields

  • Like 1

Share this post


Link to post
Share on other sites
3 hours ago, ICF Church said:

Hi i ran into the same issue und it cost me much time to find the culprit of it. Please at least post a big warning in the readme for users to be aware before installation since i suspect this can affect many users who have installed pro fields

@ICF Church README has been updated. Thanks for your feedback.

  • Like 1

Share this post


Link to post
Share on other sites

This issue hit me a while back too with the RepeaterMatrix field. I ended up posting about it in the Profields support board but, embarrassingly, forgot to let you know about it at the time, jaro. My apologies to all for the oversight.

 

Share this post


Link to post
Share on other sites

The issue, which I'd like to call Agent Smith Bug, that caused a Repeater/RepeaterMatrix field copying itself onto other same type fields while erasing the contents of the fields happened to me inversely without warning. Fortunately, it happened on a site while it's in development. I installed the module way before while browsing for new modules in the directory, and I forgot it was ever installed. It was later I upgraded the core many times, and available modules with the ProcessWireUpgrade module and later I installed the ProFields.

I think we need a special cross-checking notification system, just like a product changes its price or availability in Amazon's shopping cart. One way it was achieved, while you're installing a module it shows a warning for the system version. Right there, a system should notify us with a critical warning that module x would possibly break the behavior of module y. In my case, it would be great if the ProFields installation notified me. But how can it know? (Reminds me the Project Insight scene of Winter Soldier) As humans, we can't check every issue, every board entry.  Hence we need a database, testers, modifiers, structural changes, etc. Too many resources, scratch this idea.

Share this post


Link to post
Share on other sites
On 1.4.2017 at 5:31 PM, vmo said:

I had two repeater fields: "repeater1" and "repeater2", the "repeater1" should repeat the "images1" field and the "repeater2" should repeat the "images2",

the problem was each time I edited one of them and change the repeated field the other assumed that modification. Example: editing the "repeater1" and change the field to "image3" the "repeater2" instead of having the "image2" has the field to be repeated assumed the field "image3".

 

On 2.11.2017 at 1:26 PM, adoxus said:

The issue, which I'd like to call Agent Smith Bug, that caused a Repeater/RepeaterMatrix field copying itself onto other same type fields while erasing the contents of the fields happened to me inversely without warning.

 

On 10.10.2017 at 1:34 PM, netcarver said:

This issue hit me a while back too with the RepeaterMatrix field.

 

On 9.10.2017 at 11:38 AM, ICF Church said:

Hi i ran into the same issue und it cost me much time to find the culprit of it.

 

Thanks guys for all the feedback.

I looked into this a little, and I was able to reproduce an instance of the problem using VMO's description with two repeater fields 'rep1' and 'rep2' and two text fields 'text1' and 'text2'. The assignment of text fields to repeater fields got mixed up if and only if the AutoExportTemplatesAndFieldsModule was installed.

I could narrow down the problem onto my use of ProcessWire's $field->getExportData() inside a hook 'after' ProcessPageView::finished. The problem is triggered when this function is called on the repeater fields in that hook, so the function seems to have side effects. I have added a minimal showcase for this called 'BreakRepeaterFields module' to the git repository, see module info summary for usage.

To sum this up: I am pretty confident there is no bug in my code directly causing the faulty behaviour. On the other hand one could not say that there is a bug in ProcessWire either, as my module is using an internal method which is not part of the documented API.

Best I can do for now is to declare this module incompatible with repeater fields and ProFields.

  • Thanks 1

Share this post


Link to post
Share on other sites

Thanks for looking into it @jaro!

Is there no possibility to check weather a field is / is used inside, a repeater and then treat it differently (or omitting it)? Since this solves a rather critical weakness of Processwire (imho), maybe @ryan can help out with this?

Share this post


Link to post
Share on other sites
7 minutes ago, noelboss said:

Thanks for looking into it @jaro!

Is there no possibility to check weather a field is / is used inside, a repeater and then treat it differently (or omitting it)? Since this solves a rather critical weakness of Processwire (imho), maybe @ryan can help out with this?

Fields could be omitted by type, but the output of this module would then not be a complete snapshot of templates and fields any more, which was its purpose in the first place. Do you think, such an incomplete snapshot (without repeater fields and similar) could still be useful for repeater field users?

Treating problematic fields differently could solve the problem, but the module would most probably have to be equipped with code for each problematic field type.

Yes of course, one or two hints from @ryan could be very helpful now :)

Share this post


Link to post
Share on other sites

I think a partial export is still better then breaking an entire system ;) So even if it does not do anything, it's still better to if-repeater-return than to do harm imho. You could still log a warning that there have been unsupported fields that need manual exporting so the user knows what fields need to be exported…

Now, as for me, i just had to manually copy a field (not repeater) that I otherwise could have just exported - imported… (which I now did via the db directly)…

Also, shouldn't it be possible to narrow down the problem even further now that we know what causes the problem? What happens if you use "getTableData()" directly?

 

  • Like 1

Share this post


Link to post
Share on other sites
50 minutes ago, noelboss said:

I think a partial export is still better then breaking an entire system ;) So even if it does not do anything, it's still better to if-repeater-return than to do harm imho. You could still log a warning that there have been unsupported fields that need manual exporting so the user knows what fields need to be exported…

So far, I haven't heard of any case where using this module "breaks an entire system". However, you are claiming that it "nuked [your] system" in your article as well (linking here). I must say I'm really surprised to read about what sounds far worse than the problems mentioned so far by you or anyone else in this thread. Can you please provide some details on what happened there? Thanks.

Share this post


Link to post
Share on other sites

Hi Jaro, sorry for the long delay.

Now, nuked is probably a strong word, sorry if that offended you, that was not my intention and I can change that.

Now to as what happened: I stored some Configuration (like meta-data, logo, global site info etc) of my site in a Repeater Matrix structure (would rather user functional fields but they don't support all kinds of fields like images etc.) – and I also used several other repeaters on my site. Now whenever I made changes to any of them, the other Repeaters where changed as well, deleting my work I put into creating them and also the data that each of the already existing repeaters held. Thus breaking my site in some cases (where it dependet on configurations beeing present) and losing work, data stored as well as work I put into structuring things.

For example: I created a repeater for Social Media Links and started filling these in. After that I would create another Repeater for Multiple Information Junks. – Now my Configuration Repeater would lose its data, its setup and the same happened with the Social Media Links repeater. I would lose its data plus its setup. I had to recreate the Configuration Repeater and the Social Media Repeater and input the data again.

Until i found the culprit, I lost quite some hours of development work. And it was not obvious from where the problem stemed. At first I thought Repeater was broken, or my setup was wrong (I'm fairly new to PW).

In my humble opinion, dataloss is probably the worst that could happen to a system. A PHP Error, no Problem, remove the module, fix the code. Loss of files – they are under GIT, some work might be lost but not much. But data loss, even more important if it's structural – thats really bad. Things get worse if it's not obvious what causes this data to disappear.

I think, all of this is the case here. Hope this explains it a bit better. If I can be of any help in solving this issue, let me know. BTW, you said

On 11/6/2017 at 10:10 AM, jaro said:

but the output of this module would then not be a complete snapshot of templates and fields any more

with this issue, this is not the case anyway because the module can not handle the repeaters correctly and thus also not export its config corretly, thus the output is not a complete snapshot in the first place ;)

 

Share this post


Link to post
Share on other sites

Sorry for the even longer delay – as always, the weeks before christmas are the busiest of the year.

On 9.11.2017 at 11:09 AM, noelboss said:

Now, nuked is probably a strong word, sorry if that offended you, that was not my intention and I can change that.

No worries, I don't feel offended personally, it's just bad PR and that it kind of looks like some careless coding of this module caused data loss. I'd appreciate if you could reword or just remove the link.

Thanks for the detailed explanation.

On 9.11.2017 at 11:09 AM, noelboss said:

with this issue, this is not the case anyway because the module can not handle the repeaters correctly and thus also not export its config corretly, thus the output is not a complete snapshot in the first place ;)

I agree, but that's not how I meant it anyway, I was referring to the partial export vs. clean fail design choice.

Sticking to the second, I have added a blacklist of known incompatible fieldtypes (containing only repeater so far) for now so the module will not only skip incompatible fields, but refuse to export all fields with a clear error message in that case.

  • Like 1

Share this post


Link to post
Share on other sites

Hi @jaro

Thanks for the update. Would it be possible to still have a partial export (after checking a confirmation or using module settings)?

I have removed the links :)

  • Like 1

Share this post


Link to post
Share on other sites

Hi @jaro,

I've been having a look at the Repeater issue and the BreakRepeaterFields demonstration.

Interestingly, the issue is not that fields are copied from one repeater to another. It is that the module interferes with the listing of fields when the repeater field settings are viewed in ProcessField, causing the fields of one repeater to appear in the AsmSelect of another repeater. So if you check the database the fields associated with each repeater is actually correct despite the incorrect listing shown in ProcessField. But if the user then saves the field in ProcessField it is at this point that the associated fields become incorrect.

Could you explain why the module needs to hook ProcessPageView::finished? This method fires on every page view, front-end and back-end. But most of these page views will have no connection to field settings or template settings changing. Instead of hooking this method, couldn't you hook only methods that are related to field and template settings? So some method in ProcessField and ProcessTemplate?

And also, the module is looping over all fields on every page view. Could you focus in on just the field that is being edited? So when field1 is edited in ProcessField, just export the data for field1 rather than looping over every field. This would avoid the situation where the data for repeater2 becomes involved when editing repeater1.

I should say that I've just had a brief look at the issue so far, so there might be good reasons why the module does what it does.

  • Like 4
  • Thanks 1

Share this post


Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.


  • Recently Browsing   0 members

    No registered users viewing this page.

  • Similar Content

    • By Robin S
      A community member raised a question and I thought a new sanitizer method for the purpose would be useful, hence...
      Sanitizer Transliterate
      Adds a transliterate method to $sanitizer that performs character replacements as defined in the module config. The default character replacements are based on the defaults from InputfieldPageName, but with uppercase characters included too.
      Usage
      Install the Sanitizer Transliterate module.
      Customise the character replacements in the module config as needed.
      Use the sanitizer on strings like so:
      $transliterated_string = $sanitizer->transliterate($string);
       
      https://github.com/Toutouwai/SanitizerTransliterate
      https://modules.processwire.com/modules/sanitizer-transliterate/
       
    • By dimitrios
      Hello,
      this module can publish content of a Processwire page on a Facebook page, triggered by saving the Processwire page.
      To set it up, configure the module with a Facebook app ID, secret and a Page ID. Following is additional configuration on Facebook for developers:
      Minimum Required Facebook App configuration:
      on Settings -> Basics, provide the App Domains, provide the Site URL, on Settings -> Advanced, set the API version to 2.10, add Product: Facebook Login, on Facebook Login -> Settings, set Client OAuth Login: Yes, set Web OAuth Login: Yes, set Enforce HTTPS: Yes, add "http://www.example.com/processwire/page/" to field Valid OAuth Redirect URIs. This module is configurable as follows:
      Templates: posts can take place only for pages with the defined templates. On/Off switch: specify a checkbox field that will not allow the post if checked. Specify a message and/or an image for the post.
      Usage
      edit the desired PW page and save; it will post right after the initial Facebook log in and permission granting. After that, an access token is kept.
       
      Download
      PW module directory: http://modules.processwire.com/modules/auto-fb-post/ Github: https://github.com/kastrind/AutoFbPost   Note: Facebook SDK for PHP is utilized.


    • By thomasaull
      I created a little helper module to trigger a CI pipeline when your website has been changed. It's quite simple and works like this: As soon as you save a page the module sets a Boolean via a pages save after hook. Once a day via LazyCron the module checks if the Boolean is set and sends a POST Request to a configurable Webhook URL.
      Some ideas to extend this:
      make request type configurable (GET, POST) make the module trigger at a specified time (probably only possible with a server cronjob) trigger manually Anything else? If there's interest, I might put in some more functionality. Let me know what you're interested in. Until then, maybe it is useful for a couple of people 🙂
      Github Repo: https://github.com/thomasaull/CiTrigger
    • By Robin S
      I created this module a while ago and never got around to publicising it, but it has been outed in the latest PW Weekly so here goes the support thread...
      Unique Image Variations
      Ensures that all ImageSizer options and focus settings affect image variation filenames.

      Background
      When using methods that produce image variations such as Pageimage::size(), ProcessWire includes some of the ImageSizer settings (height, width, cropping location, etc) in the variation filename. This is useful so that if you change these settings in your size() call a new variation is generated and you see this variation on the front-end.
      However, ProcessWire does not include several of the other ImageSizer settings in the variation filename:
      upscaling cropping, when set to false or a blank string interlace sharpening quality hidpi quality focus (whether any saved focus area for an image should affect cropping) focus data (the top/left/zoom data for the focus area) This means that if you change any of these settings, either in $config->imageSizerOptions or in an $options array passed to a method like size(), and you already have variations at the requested size/crop, then ProcessWire will not create new variations and will continue to serve the old variations. In other words you won't see the effect of your changed ImageSizer options on the front-end until you delete the old variations.
      Features
      The Unique Image Variations module ensures that any changes to ImageSizer options and any changes to the focus area made in Page Edit are reflected in the variation filename, so new variations will always be generated and displayed on the front-end.
      Installation
      Install the Unique Image Variations module.
      In the module config, set the ImageSizer options that you want to include in image variation filenames.
      Warnings
      Installing the module (and keeping one or more of the options selected in the module config) will cause all existing image variations to be regenerated the next time they are requested. If you have an existing website with a large number of images you may not want the performance impact of that. The module is perhaps best suited to new sites where image variations have not yet been generated.
      Similarly, if you change the module config settings on an existing site then all image variations will be regenerated the next time they are requested.
      If you think you might want to change an ImageSizer option in the future (I'm thinking here primarily of options such as interlace that are typically set in $config->imageSizerOptions) and would not want that change to cause existing image variations to be regenerated then best to not include that option in the module config after you first install the module.
       
      https://github.com/Toutouwai/UniqueImageVariations
      https://modules.processwire.com/modules/unique-image-variations/
    • By Sebi
      I've created a small module which lets you define a timestamp after which a page should be accessible. In addition you can define a timestamp when the release should end and the page should not be accessable any more.
      ProcessWire-Module: http://modules.processwire.com/modules/page-access-releasetime/
      Github: https://github.com/Sebiworld/PageAccessReleasetime
      Usage
      PageAccessReleasetime can be installed like every other module in ProcessWire. Check the following guide for detailed information: How-To Install or Uninstall Modules
      After that, you will find checkboxes for activating the releasetime-fields at the settings-tab of each page. You don't need to add the fields to your templates manually.
      Check e.g. the checkbox "Activate Releasetime from?" and fill in a date in the future. The page will not be accessable for your users until the given date is reached.
      If you have $config->pagefileSecure = true, the module will protect files of unreleased pages as well.
      How it works
      This module hooks into Page::viewable to prevent users to access unreleased pages:
      public function hookPageViewable($event) { $page = $event->object; $viewable = $event->return; if($viewable){ // If the page would be viewable, additionally check Releasetime and User-Permission $viewable = $this->canUserSee($page); } $event->return = $viewable; } To prevent access to the files of unreleased pages, we hook into Page::isPublic and ProcessPageView::sendFile.
      public function hookPageIsPublic($e) { $page = $e->object; if($e->return && $this->isReleaseTimeSet($page)) { $e->return = false; } } The site/assets/files/ directory of pages, which isPublic() returns false, will get a '-' as prefix. This indicates ProcessWire (with activated $config->pagefileSecure) to check the file's permissions via PHP before delivering it to the client.
      The check wether a not-public file should be accessable happens in ProcessPageView::sendFile. We throw an 404 Exception if the current user must not see the file.
      public function hookProcessPageViewSendFile($e) { $page = $e->arguments[0]; if(!$this->canUserSee($page)) { throw new Wire404Exception('File not found'); } } Additionally we hook into ProcessPageEdit::buildForm to add the PageAccessReleasetime fields to each page and move them to the settings tab.
      Limitations
      In the current version, releasetime-protected pages will appear in wire('pages')->find() queries. If you want to display a list of pages, where pages could be releasetime-protected, you should double-check with $page->viewable() wether the page can be accessed. $page->viewable() returns false, if the page is not released yet.
      If you have an idea how unreleased pages can be filtered out of ProcessWire selector queries, feel free to write an issue, comment or make a pull request!
×
×
  • Create New...