Module Migrator

[SteveB]

Great progress! I haven't tried the new features yet (maybe this is covered already) but it might be handy to have it spit out a list of any template files it moves. That would be to remind us to manually move over any include files or other resources those files depend on.

[adrian]

Great idea Steve - I'll do that for sure. 

Done and committed!

Edited March 2, 2014 by adrian

[adrian]

Hi again everyone - I have spent quite a lot of time on this the last few days and it is now much more robust and feature rich.

In an effort to encourage some testing from you guys, I have attached an export of Ryan's blog profile that you can install into an existing PW site.

1. Download the attached zip file
2. Grab the latest version of Page Tree Migrator from Github (https://github.com/adrianbj/ProcessPageTreeMigrator) and install.
3. Make sure your templates directory is writable by the server
4. Run the module from the your Setup menu
5. Choose "Import" and continue to Step 2
6. Choose the Parent Page - choose "Home" which will install the blog as a first level child of your homepage.
7. Make sure "Everything, including all data pages" is selected for Components to Import.
8. From Data Source, choose "Zip File Upload", browse to the blog.zip file.
9. Click "Upload and Create Content" and wait a few seconds.

That's it - you should have a fully functioning blog set up at: Home > Blog

It comes complete with the same test entries, images, gallery, tags, categories etc that Ryan's original one does, although I did add in one extra photo, embedded into an RTE and resized - the photo comes from Joss' Foundation 5 profile.

The one thing you'll notice is that the embedded Youtube link doesn't work, because the Video embedder module isn't installed. Page Tree Migrator will install core modules as needed, but obviously can't install other ones - yet

You'll notice that the blog is installed with its CSS and script files in a subdirectory of the templates folder to avoid an overwriting of existing files.

Please test and let me know what you think - Thanks!

blog.zip

[cstevensjr]

I have attached an export of Ryan's blog profile that you can install into an existing PW site.

I had to re-read your last post! This is fantastic.  Thanks.

[Pete]

I've got two notices and a fatal error from your latest version on GitHub:

Notice: Trying to get property of non-object in D:\wamp\www\echofinance\site\modules\ProcessPageTreeMigrator.module on line 879
Notice: Trying to get property of non-object in D:\wamp\www\echofinance\site\modules\ProcessPageTreeMigrator.module on line 1049
Fatal error: Call to a member function getArray() on a non-object in D:\wamp\www\echofinance\site\modules\ProcessPageTreeMigrator.module on line 1052

[adrian]

Hey Pete - sorry about that.

Sounds like you are exporting. The issue is that you haven't selected anything in the "Parent Page" or "Components to Export" option. My bad for forgetting to provide a friendly error instead of failing, but if you choose something in that step, you should be fine!

I'll fix it in the next version.

Let me know how you go

[apeisa]

Cutting edge php required?

Parse Error: syntax error, unexpected '[' (line 830 of /site/modules/ProcessPageTreeMigrator-master/ProcessPageTreeMigrator.module) 

[adrian]

Oops - sorry about that Antti, and for the delayed response - was out enjoying some unseasonable sunshine!

I just pushed a quick fix that should work for older versions of php. I need to set up a 5.3 testing environment.

EDIT: Pete - the export process should now also provide friendly errors if a required option is not selected.

Edited March 11, 2014 by adrian

[Marty Walker]

Spectacular. I just moved a blog tree to another dev site which has saved me a good hour of setup time. Thank you!

[adrian]

Thanks Marty - glad to hear it is working for someone other than me - I was beginning to wonder

Also, I just set up php 5.3 testing environment and everything seems to be working there now too, so hopefully I won't be wasting Pete and Antti's time when they try again!

But please keep the bug reports and feature suggestions coming!

[Marty Walker]

If it helps my MAMP setup is running PHP 5.4.25

Actually there was a one things I noticed that didn't get packaged up in the zip I exported - and that was an include I used to show my blog categories. It might just be the way I've set it up but I'll put together something for you to look at later.

[adrian]

Thanks for the report - I was developing it on 5.5 and have now tested on 5.3, so hopefully we are covered now

It makes sense that your include file was not added to the zip. Hopefully you noticed the message after the import to that effect - it mentions that you need to copy any .inc and other supporting files yourself. At the moment the migrator will only add template php files to the zip that are tied to the templates used in the pages that are exported. I have actually contemplated scanning through the template files looking for include, include_once, require, require_once and grabbing those files as well. I'll have a think about it and see if I think it makes sense to do that.

The blog profile zip I attached a few posts back was manually modified to contain all the associated inc, js, and css files, but everything else was direct from the automatic export process.

[Marty Walker]

I figured that might be the case as I hadn't logged that inc as a template at all - there was no real need to. I didn't see a message though - or maybe I missed it. I'll take a look next time I try it, which will be soon.

[adrian]

Another pretty major update today. Here's the list of changes:

1. Friendly error handling when modules/fieldtypes aren't installable on the destination site which also prevents any actions until you install the required modules.
2. Support for migrating page data for all (hopefully) custom fieldtypes. I have tested with cropImage, MapMarker, Dimensions, and Phone. Let me know if you find one that doesn't work. Also, if you are playing around with MapMarker, be sure to make the changes shown in this PR: https://github.com/ryancramerdesign/FieldtypeMapMarker/pull/4/files - seems like it might only affect very recent devs of PW and hopefully Ryan will commit that change soon anyway.
3. New Import Types: Append, Overwrite, Replace. Explained fully in the module, but the key new functionality is replace, which effectively allows you to edit the content in a section (Page Tree) of a site (say on your dev server), including changes to field settings, which fields are in the template (including repeater fields and their subfields), and the page data in the fields, and all these changes will be replicated on the destination site. This is very new, so let me know if you find something I have missed.

Remember, there is one happy customer already ....

Spectacular. I just moved a blog tree to another dev site which has saved me a good hour of setup time. Thank you!

But I am looking for more

If you want a quick demo, don't forget you can try installing Ryan's blog profile as a child of your Home using the zip attached to this post.

[teppo]

@adrian, I've been testing this a bit and getting this after export:

Notice: Trying to get property of non-object in /site/modules/ProcessPageTreeMigrator/ProcessPageTreeMigrator.module on line 1038

Warning: Invalid argument supplied for foreach() in /site/modules/ProcessPageTreeMigrator/ProcessPageTreeMigrator.module on line 1038

Not sure if this affects the import process, though. At least pages get imported properly, but there is something a bit strange there too; see attached screenshot. "Following template files" seems to suggest that something was (supposed to be) migrated there too, but now it looks like somethings missing below that headline?

Earlier this template had repeater field on it and I was seeing quite a bit more errors, so I removed the repeater field in case that was causing it somehow. Errors did actually go away, but only partly -- as you can see, some still remain.

I'm running 2.4 master branch on a local MAMP installation with PHP 5.5.10. You can see exported data here.

[Marty Walker]

Happy yet demanding I can't wait to test this update.

This Replace feature has me thinking (which I'll test soon). I know it's probably outside of the scope of what you're doing but how difficult would it be to periodically replace a page tree from one site to another? I'm thinking in terms of a site sharing a blog or news section. I know I could use Pages Web Service to show post lists but I'm thinking of shifting entire posts between sites regularly.

[adrian]

Hey teppo,

Thank you for the testing. I can replicate that same export error. I hadn't come across it before, but looking at your export JSON, I noticed that the page field you included has a parent_id of "home". I never anticipated that. The module will actually find all page field parent/child trees and add them to the export, so if for example you had a pagefield that linked to a categories/category tree, then it would grab that info, export it and import it on the new site, even though it is not directly under the parent you chose when exporting.

I think the module actually handles this scenario quite well (despite the error which is an easy fix) - so if you do choose home, it won't try to export the entire home tree, but it will use the children of home on the new site, once you do the import, which is what happened for me when I imported your JSON. Does that make sense?

EDIT: Actually that 1038 error is because your page field doesn't have a "Template of Selectable Pages" set. I have always been testing with page fields with both "Parent of ..." and "Template of..." defined in the Input tab. I'll have to make sure there aren't any further complications here.

The message about importing template files should be hidden in this case, since it is clear you used the copy/paste approach. Obviously template files (*.php) can only be migrated with the "Save" output format. I'll take care of hiding that when it's not appropriate.

Now for the confusing one, the errors with repeater fields. I haven't had any issues with those in quite some time now so I am curious on the exact setup. Would you mind testing one again and send me the errors you're getting and also the JSON, and if they're not being included in the export, perhaps you could sketch out their structure.

One thing I am curious about on the repeater front - what option are you choosing for "Components to Export"? I have been doing almost all my recent testing with "Everything", so I am wondering if that might have something to do with it. I notice that none of your page fields have any content.

Thanks again, really appreciate this!

BTW, I have attached an export of a page tree that shows repeater and page fields working, along with images and RTE embedded images. Be sure to import "Everything" and choose "Replace". It should go into any parent, but I have been testing mostly with Home as the import's root is "About". It will replace your About (if you have one) and also install a US states page tree for the state selector field. There are a couple of repeaters at the bottom of the gallery child page.

files.zip

[teppo]

I've also been exporting "everything", though (like you said) the copy method obviously can't literally export everything..

These are the errors I get in the same situation, with repeater field added. Ready items for this repeater is set to 3 and you can view the final JSON output here.

Notice: Trying to get property of non-object in /site/modules/ProcessPageTreeMigrator/ProcessPageTreeMigrator.module on line 1403

Warning: Invalid argument supplied for foreach() in /site/modules/ProcessPageTreeMigrator/ProcessPageTreeMigrator.module on line 1403

Notice: Trying to get property of non-object in /site/modules/ProcessPageTreeMigrator/ProcessPageTreeMigrator.module on line 1038

Warning: Invalid argument supplied for foreach() in /site/modules/ProcessPageTreeMigrator/ProcessPageTreeMigrator.module on line 1038

Warning: Invalid argument supplied for foreach() in /site/modules/ProcessPageTreeMigrator/ProcessPageTreeMigrator.module on line 1026

Notice: Trying to get property of non-object in /site/modules/ProcessPageTreeMigrator/ProcessPageTreeMigrator.module on line 1403

Warning: Invalid argument supplied for foreach() in /site/modules/ProcessPageTreeMigrator/ProcessPageTreeMigrator.module on line 1403

Notice: Trying to get property of non-object in /site/modules/ProcessPageTreeMigrator/ProcessPageTreeMigrator.module on line 1403

Warning: Invalid argument supplied for foreach() in /site/modules/ProcessPageTreeMigrator/ProcessPageTreeMigrator.module on line 1403

Notice: Trying to get property of non-object in /site/modules/ProcessPageTreeMigrator/ProcessPageTreeMigrator.module on line 1403

Warning: Invalid argument supplied for foreach() in /site/modules/ProcessPageTreeMigrator/ProcessPageTreeMigrator.module on line 1403

Testing your zip file didn't go exactly as planned, though that was probably user error; didn't select "replace", tried to "append" with parent set to "home" and as a result I got "Integrity constraint violation: 1062 Duplicate entry 'about-1' for key 'name_parent_id'".

Slightly weird thing here is that when trying this again with "replace", I got the same error, though this might be because something was broken earlier. Might have to setup another test site to see what's actually going on here -- if this works for you, there's most likely something wrong with my test site.

One specific thing I'm not really comfortable with, though I'm not entirely sure if that's possible to avoid either, is that even though the import fails things like fields and templates are still added and/or kept. This could potentially leave the site in a kind of a strange state, with some old content and some new. It'd be cool if failed import "cleaned up the mess", so to speak

Another thing that probably should be handled somehow are possible permission errors:

Warning: copy(/site/templates///basic-page.php): failed to open stream: Permission denied in /site/modules/ProcessPageTreeMigrator/ProcessPageTreeMigrator.module on line 449

Warning: copy(/site/templates///gallery.php): failed to open stream: Permission denied in /site/modules/ProcessPageTreeMigrator/ProcessPageTreeMigrator.module on line 449

The reason for these is obvious and (again) I should've known better, but then again, the module could also check for necessary permissions beforehand -- or perhaps just interrupt the import the moment this happens and notify user to make sure that permissions need to be tweaked?

.. and by the way, those template paths with three forward slashes look a bit strange, but that's far from fatal

PS. I actually enjoy testing this module and see a lot of potential in it. As a matter of fact it's already proven it's worth in some of our projects, saving me quite literally hours of work, on more than one occasion. If it sounds like I'm only seeing bad things here, that couldn't be further from the truth

[adrian]

Thanks again teppo - awesomely helpful!

So I think I have taken care of everything from your reports:

1. If there are no template php files in the import, then the message is not displayed to avoid confusion.
2. Write permissions are now checked on the destination templates directory (if there are template files to be imported), before anything happens and the user is alerted so they can fix and start again.
3. Also fixed the extra forward slashes
4. I have fixed the line 1403 errors for repeaters - that error only occurred when there were no entries for a repeater on a page which is why I hadn't seen it before. I am not getting the line 1026 error at all, so please let me know if that one still shows up for you.
5. Fixed the line 1038 error - now the module will work without the page field's Template of Selectable Pages set. Along with this I have also put a limitation on automatically migrating content for page fields - it won't automatically export the entire page tree if a page field's Parent of Selectable Pages is Home (id#1, just in case the name was changed), and also only if the Parent of Selectable Pages is set - so, it won't potentially try to migrate pages from all over the page tree, which might happen if some is using Template, or even worse a Custom Selector or Custom PHP option.

Not sure why you are getting that integrity constraint violation. I know I have broken many a PW install while testing this module, so maybe if you could start with a fresh one and let me know, that would be great. By the way - I tested that files.zip with "Append" and it works as expected without errors, so I do think something is likely amiss at your end, but please let me know what you find.

Actually, I have managed to replicate something similar to this - it's amazing what you can sometimes miss due to the way you are doing something. Anyway, I am on it - it is related to the addition of the new REPLACE option, but I am having trouble tracking it down fully tonight, so it might have to wait another day or until I can find some more time, but I'll post again when that is fixed. In the meantime, so long as you delete the About tree from your install before importing, that zip should work!

Great thought on the issue of a failed import and the cleanup idea. Ideally I'd love for there to be no reason for a failed import, so maybe better checking for potential conflicts before anything happens is the easiest way. Undoing changes / cleanup might be tricky!

Really appreciate hearing that this has saved you lots of time - I would love this to be a "must have" PW module once some more kinks are ironed out.

Thanks again.

Edited March 16, 2014 by adrian

[adrian]

Happy yet demanding I can't wait to test this update.

This Replace feature has me thinking (which I'll test soon). I know it's probably outside of the scope of what you're doing but how difficult would it be to periodically replace a page tree from one site to another? I'm thinking in terms of a site sharing a blog or news section. I know I could use Pages Web Service to show post lists but I'm thinking of shifting entire posts between sites regularly.

Hey Marty, anything for you

Do you mean automatically or manually on a regular basis? Manually will be no problem at all - that is what this is designed to do (once I get a for more of the REPLACE kinks sorted out). If you want automatic replacement, I am sure we could use a cron job to handle both the export and import - the key thing would be to save the zip on the server, instead of downloading it to local machine. Then get the zip from the destination server and trigger the import. All sounds do-able. That said, if it is just content we are talking about and no fields or templates would change, then I wonder if an import from a web service feed might not be a better approach - this could even bring across new images.

I'll sleep on it!

[adrian]

Turns out I really broke some keys things when I added the import actions functionality (for append, overwrite, replace). I need sleep now and won't be able to get to this again for at least a day, maybe two. So, the most stable version right now is probably:

https://github.com/adrianbj/ProcessPageTreeMigrator/blob/019e006833ad970263533516aa7615b8c87af573/ProcessPageTreeMigrator.module

Of course this doesn't include the fixes for the PHP warnings that teppo found, nor the directory writing checks etc.

Anyway, I'll post again when the import actions additions are working properly.

[adrian]

Ok, I have fixed another round of bugs, including one weird one getting an "Unknown column 'field_images.modified'" error when importing a page tree into a brand new install before ever having viewed a page on the site. Getting off track, but the issue was due to the need to trigger getDatabaseSchema on the images field which adds the modified and created DB fields. 

The things I've done is add optional support for migrating createdUser/modifiedUser and created/modified datetime for pages. I am using quiet save which is supposed to support createdUser, modifiedUser, and created, but it isn't supporting modifiedUser for me at the moment. I might resort to SQL, or I might decide it's actually best to just change the created versions and leave the modified to change to the user running the import and the current time.

On the user migration - if a user doesn't exist it will be created and assigned its roles from the source installation as defined in the JSON. Also, if the needed roles aren't available, they will be created too.

Let me know how it goes for you!

[adrian]

On a roll - after a chat with Joss, I have added a new option on export that allows you to select any additional required helper files such as .inc, .css, .js, etc. These selected files will be packaged in the zip for the migration.

This option is only displayed if you choose "save" because obviously not relevant to "copy" where it's only the JSON that can be exported.

Note that there is no need to select normal template files as these are still included in the export automatically.

[adrian]

Another pretty major update!

There is now an "Edit Imported Content" option during the import process that lets you determine exactly which fields and pages (from the JSON) get imported.

This should come in especially handy when importing the one of the Shared JSON packages. I have added a new one that is a good test for this, called "person" (maybe needs a more detailed name), that creates a person template with a complete range of contact/address fields. So, on import you can easily choose only the fields you want from the package.

I think this should be really handy for distributing standard templates, but with easy customization for each use case.

[diogo]

Adrian, this module sounds more and more useful everytime. Next thing, you will be creating a "recording" mode that collects all changes that happen in PW during a certain period and makes it possible to export them as json to make late changes on published sites really easy! (did that sound like a suggestion?)