Jump to content

Recommended Posts

Great progress! I haven't tried the new features yet (maybe this is covered already) but it might be handy to have it spit out a list of any template files it moves. That would be to remind us to manually move over any include files or other resources those files depend on.

  • Like 1
Link to post
Share on other sites

Hi again everyone - I have spent quite a lot of time on this the last few days and it is now much more robust and feature rich.

In an effort to encourage some testing from you guys, I have attached an export of Ryan's blog profile that you can install into an existing PW site.

  1. Download the attached zip file
  2. Grab the latest version of Page Tree Migrator from Github (https://github.com/adrianbj/ProcessPageTreeMigrator) and install.
  3. Make sure your templates directory is writable by the server
  4. Run the module from the your Setup menu
  5. Choose "Import" and continue to Step 2
  6. Choose the Parent Page - choose "Home" which will install the blog as a first level child of your homepage.
  7. Make sure "Everything, including all data pages" is selected for Components to Import.
  8. From Data Source, choose "Zip File Upload", browse to the blog.zip file.
  9. Click "Upload and Create Content" and wait a few seconds.

That's it - you should have a fully functioning blog set up at: Home > Blog

It comes complete with the same test entries, images, gallery, tags, categories etc that Ryan's original one does, although I did add in one extra photo, embedded into an RTE and resized - the photo comes from Joss' Foundation 5 profile.

The one thing you'll notice is that the embedded Youtube link doesn't work, because the Video embedder module isn't installed. Page Tree Migrator will install core modules as needed, but obviously can't install other ones - yet :)

You'll notice that the blog is installed with its CSS and script files in a subdirectory of the templates folder to avoid an overwriting of existing files.

Please test and let me know what you think - Thanks!


  • Like 9
Link to post
Share on other sites

I've got two notices and a fatal error from your latest version on GitHub:

Notice: Trying to get property of non-object in D:\wamp\www\echofinance\site\modules\ProcessPageTreeMigrator.module on line 879
Notice: Trying to get property of non-object in D:\wamp\www\echofinance\site\modules\ProcessPageTreeMigrator.module on line 1049
Fatal error: Call to a member function getArray() on a non-object in D:\wamp\www\echofinance\site\modules\ProcessPageTreeMigrator.module on line 1052
Link to post
Share on other sites

Hey Pete - sorry about that.

Sounds like you are exporting. The issue is that you haven't selected anything in the "Parent Page" or "Components to Export" option. My bad for forgetting to provide a friendly error instead of failing, but if you choose something in that step, you should be fine!

I'll fix it in the next version.

Let me know how you go :)

Link to post
Share on other sites

Cutting edge php required? :)

Parse Error: syntax error, unexpected '[' (line 830 of /site/modules/ProcessPageTreeMigrator-master/ProcessPageTreeMigrator.module) 
Link to post
Share on other sites

Oops - sorry about that Antti, and for the delayed response - was out enjoying some unseasonable sunshine!

I just pushed a quick fix that should work for older versions of php. I need to set up a 5.3 testing environment.

EDIT: Pete - the export process should now also provide friendly errors if a required option is not selected.

Edited by adrian
Link to post
Share on other sites

Thanks Marty - glad to hear it is working for someone other than me - I was beginning to wonder :)

Also, I just set up php 5.3 testing environment and everything seems to be working there now too, so hopefully I won't be wasting Pete and Antti's time when they try again!

But please keep the bug reports and feature suggestions coming!

Link to post
Share on other sites

If it helps my MAMP setup is running PHP 5.4.25

Actually there was a one things I noticed that didn't get packaged up in the zip I exported - and that was an include I used to show my blog categories. It might just be the way I've set it up but I'll put together something for you to look at later.

Link to post
Share on other sites

Thanks for the report - I was developing it on 5.5 and have now tested on 5.3, so hopefully we are covered now :)

It makes sense that your include file was not added to the zip. Hopefully you noticed the message after the import to that effect - it mentions that you need to copy any .inc and other supporting files yourself. At the moment the migrator will only add template php files to the zip that are tied to the templates used in the pages that are exported. I have actually contemplated scanning through the template files looking for include, include_once, require, require_once and grabbing those files as well. I'll have a think about it and see if I think it makes sense to do that.

The blog profile zip I attached a few posts back was manually modified to contain all the associated inc, js, and css files, but everything else was direct from the automatic export process.

Link to post
Share on other sites

I figured that might be the case as I hadn't logged that inc as a template at all - there was no real need to. I didn't see a message though - or maybe I missed it. I'll take a look next time I try it, which will be soon.

Link to post
Share on other sites

Another pretty major update today. Here's the list of changes:

  1. Friendly error handling when modules/fieldtypes aren't installable on the destination site which also prevents any actions until you install the required modules.
  2. Support for migrating page data for all (hopefully) custom fieldtypes. I have tested with cropImage, MapMarker, Dimensions, and Phone. Let me know if you find one that doesn't work. Also, if you are playing around with MapMarker, be sure to make the changes shown in this PR: https://github.com/ryancramerdesign/FieldtypeMapMarker/pull/4/files - seems like it might only affect very recent devs of PW and hopefully Ryan will commit that change soon anyway.
  3. New Import Types: Append, Overwrite, Replace. Explained fully in the module, but the key new functionality is replace, which effectively allows you to edit the content in a section (Page Tree) of a site (say on your dev server), including changes to field settings, which fields are in the template (including repeater fields and their subfields), and the page data in the fields, and all these changes will be replicated on the destination site. This is very new, so let me know if you find something I have missed.

Remember, there is one happy customer already ....

Spectacular. I just moved a blog tree to another dev site which has saved me a good hour of setup time. Thank you!

But I am looking for more :)

If you want a quick demo, don't forget you can try installing Ryan's blog profile as a child of your Home using the zip attached to this post.

  • Like 1
Link to post
Share on other sites

@adrian, I've been testing this a bit and getting this after export:

Notice: Trying to get property of non-object in /site/modules/ProcessPageTreeMigrator/ProcessPageTreeMigrator.module on line 1038

Warning: Invalid argument supplied for foreach() in /site/modules/ProcessPageTreeMigrator/ProcessPageTreeMigrator.module on line 1038

Not sure if this affects the import process, though. At least pages get imported properly, but there is something a bit strange there too; see attached screenshot. "Following template files" seems to suggest that something was (supposed to be) migrated there too, but now it looks like somethings missing below that headline?

Earlier this template had repeater field on it and I was seeing quite a bit more errors, so I removed the repeater field in case that was causing it somehow. Errors did actually go away, but only partly -- as you can see, some still remain.

I'm running 2.4 master branch on a local MAMP installation with PHP 5.5.10. You can see exported data here.


  • Like 1
Link to post
Share on other sites

Happy yet demanding :) I can't wait to test this update.

This Replace feature has me thinking (which I'll test soon). I know it's probably outside of the scope of what you're doing but how difficult would it be to periodically replace a page tree from one site to another? I'm thinking in terms of a site sharing a blog or news section. I know I could use Pages Web Service to show post lists but I'm thinking of shifting entire posts between sites regularly.

Link to post
Share on other sites

Hey teppo,

Thank you for the testing. I can replicate that same export error. I hadn't come across it before, but looking at your export JSON, I noticed that the page field you included has a parent_id of "home". I never anticipated that. The module will actually find all page field parent/child trees and add them to the export, so if for example you had a pagefield that linked to a categories/category tree, then it would grab that info, export it and import it on the new site, even though it is not directly under the parent you chose when exporting.

I think the module actually handles this scenario quite well (despite the error which is an easy fix) - so if you do choose home, it won't try to export the entire home tree, but it will use the children of home on the new site, once you do the import, which is what happened for me when I imported your JSON. Does that make sense?

EDIT: Actually that 1038 error is because your page field doesn't have a "Template of Selectable Pages" set. I have always been testing with page fields with both "Parent of ..." and "Template of..." defined in the Input tab. I'll have to make sure there aren't any further complications here.

The message about importing template files should be hidden in this case, since it is clear you used the copy/paste approach. Obviously template files (*.php) can only be migrated with the "Save" output format. I'll take care of hiding that when it's not appropriate.

Now for the confusing one, the errors with repeater fields. I haven't had any issues with those in quite some time now so I am curious on the exact setup. Would you mind testing one again and send me the errors you're getting and also the JSON, and if they're not being included in the export, perhaps you could sketch out their structure.

One thing I am curious about on the repeater front - what option are you choosing for "Components to Export"? I have been doing almost all my recent testing with "Everything", so I am wondering if that might have something to do with it. I notice that none of your page fields have any content.

Thanks again, really appreciate this!

BTW, I have attached an export of a page tree that shows repeater and page fields working, along with images and RTE embedded images. Be sure to import "Everything" and choose "Replace". It should go into any parent, but I have been testing mostly with Home as the import's root is "About". It will replace your About (if you have one) and also install a US states page tree for the state selector field. There are a couple of repeaters at the bottom of the gallery child page.


  • Like 1
Link to post
Share on other sites

I've also been exporting "everything", though (like you said) the copy method obviously can't literally export everything.. :)

These are the errors I get in the same situation, with repeater field added. Ready items for this repeater is set to 3 and you can view the final JSON output here.

Notice: Trying to get property of non-object in /site/modules/ProcessPageTreeMigrator/ProcessPageTreeMigrator.module on line 1403

Warning: Invalid argument supplied for foreach() in /site/modules/ProcessPageTreeMigrator/ProcessPageTreeMigrator.module on line 1403

Notice: Trying to get property of non-object in /site/modules/ProcessPageTreeMigrator/ProcessPageTreeMigrator.module on line 1038

Warning: Invalid argument supplied for foreach() in /site/modules/ProcessPageTreeMigrator/ProcessPageTreeMigrator.module on line 1038

Warning: Invalid argument supplied for foreach() in /site/modules/ProcessPageTreeMigrator/ProcessPageTreeMigrator.module on line 1026

Notice: Trying to get property of non-object in /site/modules/ProcessPageTreeMigrator/ProcessPageTreeMigrator.module on line 1403

Warning: Invalid argument supplied for foreach() in /site/modules/ProcessPageTreeMigrator/ProcessPageTreeMigrator.module on line 1403

Notice: Trying to get property of non-object in /site/modules/ProcessPageTreeMigrator/ProcessPageTreeMigrator.module on line 1403

Warning: Invalid argument supplied for foreach() in /site/modules/ProcessPageTreeMigrator/ProcessPageTreeMigrator.module on line 1403

Notice: Trying to get property of non-object in /site/modules/ProcessPageTreeMigrator/ProcessPageTreeMigrator.module on line 1403

Warning: Invalid argument supplied for foreach() in /site/modules/ProcessPageTreeMigrator/ProcessPageTreeMigrator.module on line 1403

Testing your zip file didn't go exactly as planned, though that was probably user error; didn't select "replace", tried to "append" with parent set to "home" and as a result I got "Integrity constraint violation: 1062 Duplicate entry 'about-1' for key 'name_parent_id'".

Slightly weird thing here is that when trying this again with "replace", I got the same error, though this might be because something was broken earlier. Might have to setup another test site to see what's actually going on here -- if this works for you, there's most likely something wrong with my test site.

One specific thing I'm not really comfortable with, though I'm not entirely sure if that's possible to avoid either, is that even though the import fails things like fields and templates are still added and/or kept. This could potentially leave the site in a kind of a strange state, with some old content and some new. It'd be cool if failed import "cleaned up the mess", so to speak :)

Another thing that probably should be handled somehow are possible permission errors:

Warning: copy(/site/templates///basic-page.php): failed to open stream: Permission denied in /site/modules/ProcessPageTreeMigrator/ProcessPageTreeMigrator.module on line 449

Warning: copy(/site/templates///gallery.php): failed to open stream: Permission denied in /site/modules/ProcessPageTreeMigrator/ProcessPageTreeMigrator.module on line 449

The reason for these is obvious and (again) I should've known better, but then again, the module could also check for necessary permissions beforehand -- or perhaps just interrupt the import the moment this happens and notify user to make sure that permissions need to be tweaked?

.. and by the way, those template paths with three forward slashes look a bit strange, but that's far from fatal :)

PS. I actually enjoy testing this module and see a lot of potential in it. As a matter of fact it's already proven it's worth in some of our projects, saving me quite literally hours of work, on more than one occasion. If it sounds like I'm only seeing bad things here, that couldn't be further from the truth :)

  • Like 1
Link to post
Share on other sites

Thanks again teppo - awesomely helpful!

So I think I have taken care of everything from your reports:

  1. If there are no template php files in the import, then the message is not displayed to avoid confusion.
  2. Write permissions are now checked on the destination templates directory (if there are template files to be imported), before anything happens and the user is alerted so they can fix and start again.
  3. Also fixed the extra forward slashes :)
  4. I have fixed the line 1403 errors for repeaters - that error only occurred when there were no entries for a repeater on a page which is why I hadn't seen it before. I am not getting the line 1026 error at all, so please let me know if that one still shows up for you.
  5. Fixed the line 1038 error - now the module will work without the page field's Template of Selectable Pages set. Along with this I have also put a limitation on automatically migrating content for page fields - it won't automatically export the entire page tree if a page field's Parent of Selectable Pages is Home (id#1, just in case the name was changed), and also only if the Parent of Selectable Pages is set - so, it won't potentially try to migrate pages from all over the page tree, which might happen if some is using Template, or even worse a Custom Selector or Custom PHP option.

Not sure why you are getting that integrity constraint violation. I know I have broken many a PW install while testing this module, so maybe if you could start with a fresh one and let me know, that would be great. By the way - I tested that files.zip with "Append" and it works as expected without errors, so I do think something is likely amiss at your end, but please let me know what you find.

Actually, I have managed to replicate something similar to this - it's amazing what you can sometimes miss due to the way you are doing something. Anyway, I am on it - it is related to the addition of the new REPLACE option, but I am having trouble tracking it down fully tonight, so it might have to wait another day or until I can find some more time, but I'll post again when that is fixed. In the meantime, so long as you delete the About tree from your install before importing, that zip should work!

Great thought on the issue of a failed import and the cleanup idea. Ideally I'd love for there to be no reason for a failed import, so maybe better checking for potential conflicts before anything happens is the easiest way. Undoing changes / cleanup might be tricky!

Really appreciate hearing that this has saved you lots of time - I would love this to be a "must have" PW module once some more kinks are ironed out.

Thanks again.

Edited by adrian
Link to post
Share on other sites

Happy yet demanding :) I can't wait to test this update.

This Replace feature has me thinking (which I'll test soon). I know it's probably outside of the scope of what you're doing but how difficult would it be to periodically replace a page tree from one site to another? I'm thinking in terms of a site sharing a blog or news section. I know I could use Pages Web Service to show post lists but I'm thinking of shifting entire posts between sites regularly.

Hey Marty, anything for you :)

Do you mean automatically or manually on a regular basis? Manually will be no problem at all - that is what this is designed to do (once I get a for more of the REPLACE kinks sorted out). If you want automatic replacement, I am sure we could use a cron job to handle both the export and import - the key thing would be to save the zip on the server, instead of downloading it to local machine. Then get the zip from the destination server and trigger the import. All sounds do-able. That said, if it is just content we are talking about and no fields or templates would change, then I wonder if an import from a web service feed might not be a better approach - this could even bring across new images.

I'll sleep on it!

Link to post
Share on other sites

Turns out I really broke some keys things when I added the import actions functionality (for append, overwrite, replace). I need sleep now and won't be able to get to this again for at least a day, maybe two. So, the most stable version right now is probably:


Of course this doesn't include the fixes for the PHP warnings that teppo found, nor the directory writing checks etc.

Anyway, I'll post again when the import actions additions are working properly.

Link to post
Share on other sites

Ok, I have fixed another round of bugs, including one weird one getting an "Unknown column 'field_images.modified'" error when importing a page tree into a brand new install before ever having viewed a page on the site. Getting off track, but the issue was due to the need to trigger getDatabaseSchema on the images field which adds the modified and created DB fields. 

The things I've done is add optional support for migrating createdUser/modifiedUser and created/modified datetime for pages. I am using quiet save which is supposed to support createdUser, modifiedUser, and created, but it isn't supporting modifiedUser for me at the moment. I might resort to SQL, or I might decide it's actually best to just change the created versions and leave the modified to change to the user running the import and the current time.

On the user migration - if a user doesn't exist it will be created and assigned its roles from the source installation as defined in the JSON. Also, if the needed roles aren't available, they will be created too.

Let me know how it goes for you!

  • Like 1
Link to post
Share on other sites

On a roll - after a chat with Joss, I have added a new option on export that allows you to select any additional required helper files such as .inc, .css, .js, etc. These selected files will be packaged in the zip for the migration.

This option is only displayed if you choose "save" because obviously not relevant to "copy" where it's only the JSON that can be exported.

Note that there is no need to select normal template files as these are still included in the export automatically.

  • Like 1
Link to post
Share on other sites

Another pretty major update!

There is now an "Edit Imported Content" option during the import process that lets you determine exactly which fields and pages (from the JSON) get imported.

This should come in especially handy when importing the one of the Shared JSON packages. I have added a new one that is a good test for this, called "person" (maybe needs a more detailed name), that creates a person template with a complete range of contact/address fields. So, on import you can easily choose only the fields you want from the package.

I think this should be really handy for distributing standard templates, but with easy customization for each use case.

Link to post
Share on other sites

Adrian, this module sounds more and more useful everytime. Next thing, you will be creating a "recording" mode that collects all changes that happen in PW during a certain period and makes it possible to export them as json to make late changes on published sites really easy! (did that sound like a suggestion?)

  • Like 2
Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
  • Recently Browsing   0 members

    No registered users viewing this page.

  • Similar Content

    • By monollonom
      (once again I was surprised to see a work of mine pop up in the newsletter, this time without even listing the module on PW modules website 😅. Thx @teppo !)
      Github: https://github.com/romaincazier/FieldtypeQRCode
      Modules directory: https://processwire.com/modules/fieldtype-qrcode/
      This is a simple module I made so a client could quickly grab a QR Code of the page's url in the admin.
      There's not much to it for now, but if need be you can output anything using a hook:
      $wire->addHookAfter("FieldtypeQRCode::getQRText", function($event) { $event->return = "Your custom text"; }) You can also output the QR code on your front-end by calling the field:
      echo $page->qr_code_field; The module uses the PHP library QR Code Generator by Kazuhiko Arase. When looking for a way to generate a QR Code in PW I came across @ryan's integration in his TFA module. I'm not very familiar with fieldtype/inputfield module development so I blindly followed @bernhard (great) tutorial and his BaseFieldtypeRuntime. At some point I'll take a deeper look to make a module on my own.
      Some ideas for improvements :
      add the ability to choose what to ouput : page's url / editUrl / file(s) / image(s) / ... allow to output multiple QR codes ?
    • By Chris Bennett
      Inspired by @bernhard's excellent work on the new customisable LESS CSS getting rolled into the core soon, I thought I would offer up the module for beta testing, if it is of interest to anyone.

      It takes a different approach to admin styling, basically using the Cascade part of CSS to over-ride default UiKit values.
      Values are stored in ModuleConfig Module creates a separate AdminThemeTweaker Folder at root, so it can link to AdminThemeTweaker.php as CSS AdminThemeTweaker.php reads the module values, constructs the CSS variables then includes the CSS framework Can be switched on and off with a click. Uninstall removes everything, thanks to bernhard's wonderful remove dir & contents function.
      It won't touch your core. It won't care if stuff is upgraded. You won't need to compile anything and you don't need to touch CSS unless you want to.

      It won't do much at all apart from read some values from your module config, work out the right CSS variables to use (auto contrast based on selected backgrounds) and throw it on your screen.
      You can configure a lot of stuff, leave it as it comes (dark and curvy), change two main colors (background and content background) or delve deep to configure custom margins, height of mastheads, and all manner of silly stuff I never use.

      Have been developing it for somewhere around 2 years now. It has been (and will continue to be) constantly tweaked over that time, as I click on something and find something else to do.
      That said, it is pretty solid and has been in constant use as my sole Admin styling option for all of those 2 years.

      If nothing else, it would be great if it can provide any assistance to @bernhard or other contributor's who may be looking to solve some of the quirkier UiKit behavior.
      Has (in my opinion) more robust and predictable handling of hidden Inputfields, data-colwidths and showIf wrappers.
      I am very keen to help out with that stuff in any way I can, though LESS (and any css frameworks/tools basically) are not my go.
      I love CSS variables and banging-rocks-together, no-dependency CSS you can write with notepad.


    • By opalepatrick
      I see old posts saying that repeaters are not the way to go in Custom Process Modules. If that is the case, when using forms (as I am trying to do) how would one tackle things like repeat contact fields where there can be multiple requirements for contact details with different parameters? (Like point of contact, director, etc) or even telephone numbers that have different uses?
      Just for background I am creating a process module that allows me to create types of financial applications in the admin area (no need to publish any of this, pure admin) that require a lot of personal or company information.
      Maybe I am thinking about this incorrectly?
    • By HMCB
      I ran across a reference to IftRunner module. The post was 6 years ago. I cant find it in available modules. Has it been pulled?
    • By tcnet
      PageViewStatistic for ProcessWire is a module to log page visits of the CMS. The records including some basic information like IP-address, browser, operating system, requested page and originate page. Please note that this module doesn't claim to be the best or most accurate.
      One of the biggest advantage is that this module doesn't require any external service like Google Analytics or similar. You don't have to modify your templates either. There is also no Javascript or image required.
      There is only one disadvantage. This module doesn't record visits if the browser loads the page from its browser cache. To prevent the browser from loading the page from its cache, add the following meta tags to the header of your page:
      <meta http-equiv="Cache-Control" content="no-cache, no-store, must-revalidate" /> <meta http-equiv="Pragma" content="no-cache" /> <meta http-equiv="Expires" content="0" /> How to use
      The records can be accessed via the Setup-menu of the CMS backend. The first dropdown control changes the view mode.

      Detailed records
      View mode "Detailed records" shows all visits of the selected day individually with IP-address, browser, operating system, requested page and originate page. Click the update button to see new added records.

      Cached visitor records
      View modes other than "Detailed records" are cached visitor counts which will be collected on a daily basis from the detailed records. This procedure ensures a faster display even with a large number of data records. Another advantage is that the detailed records can be deleted while the cache remains. The cache can be updated manually or automatically in a specified time period. Multiple visits from the same IP address on the same day are counted as a single visitor.

      Upgrade from older versions
      Cached visitor counts is new in version 1.0.8. If you just upgraded from an older version you might expire a delay or even an error 500 if you display cached visitor counts. The reason for this is that the cache has to be created from the records. This can take longer if your database contains many records. Sometimes it might hit the maximally execution time. Don't worry about that and keep reloading the page until the cache is completely created.
      Special Feature
      PageViewStatistic for ProcessWire can record the time a visitor viewed the page. This feature is deactivated by default. To activate open the module configuration page and activate "Record view time". If activated you will find a new column "S." in the records which means the time of view in seconds. With every page request, a Javascript code is inserted directly after the <body> tag. Every time the visitor switches to another tab or closes the tab, this script reports the number of seconds the tab was visible. The initial page request is recorded only as a hyphen (-).

      You can access the module settings by clicking the Configuration button at the bottom of the records page. The settings page is also available in the menu: Modules->Configure->ProcessPageViewStat.
      This module uses the IP2Location database from: http://www.ip2location.com. This database is required to obtain the country from the IP address. IP2Location updates this database at the begin of every month. The settings of ProcessPageViewStat offers the ability to automatically download the database monthly. Please note, that automatically download will not work if your webspace doesn't allow allow_url_fopen.
      This module uses DragScroll. A JavaScript available from: http://github.com/asvd/dragscroll. Dragscroll adds the ability in view mode "Day" to drag the records horizontally with the mouse pointer.
      This module uses the PHP class parseUserAgentStringClass available from: http://www.toms-world.org/blog/parseuseragentstring/. This class is required to filter out the browser type and operating system from the server request.
  • Create New...