Module Migrator

[adrian]

Ahh, that makes total sense - I am sure it is because all your fields become objects because of the two language versions eg:

"body": {
                    "data": "",
                    "data1029": ""
                },

rather than just simple strings.

Not sure how difficult this will be to fix - hopefully simple

[apeisa]

Yep, that is. Probably some type sniffing and in case of object you can safely assume it will be multilang field. I have done much with those in API level, but I know soma is the man with those (like many others). Probably pretty simple, I think the number is language id, and without number it is default lang.

[Marty Walker]

Hi Adrian,

With your latest version I just get blank pages on my PW 2.3.0 install when I try to export anything. My server PHP version is 5.3.25. Perhaps I need a recent dev version?

There's something up with my install there I think. Tried it locally and it works perfectly. This will be a huge time saver Mr Jones. Thank you.

[adrian]

That's great news Marty

Unfortunately I haven't managed to make any time to work on this today and I am away for the next couple of days, but I hope next week to solve the multilanguage issue that Antti discovered - I have a plan and should be quite simple so long as the receiving PW installation has the needed languages installed - I should be able to issue a warning if it doesn't.

I also have some ideas on how to make migration of images and files work too and maybe even RTE embedded images. Not sure how important migration of this sort of content is though - maybe it is adding too much complexity. Do you have any thoughts on number 3 of the road map above: http://processwire.com/talk/topic/4420-page-list-migrator/?p=43582

BTW, I have spent many days in your part of the world when I was a keen climber - beautiful place!

[Marty Walker]

There's a fair percentage of the population up here who are climbers. While I don't count myself as one of them I'm happy to photograph the mad bastards :

http://martywalker.com.au/blue-mountains/blackheath/climber/

[ryan]

Awesome job Adrian! I can't wait to use this one. I'd be using it right now if I weren't holding a baby in one arm and typing with the other. But I did browse the code and thought it looked great. 

[adrian]

Thanks Ryan,

I hope I can make some time in the next couple of days to get a few more things sorted out with this. I am excited to get this release worthy!

[adrian]

Hey Antti,

I have just committed an update that I think fixes the multi-language issues. I know there are two different ways to do different language versions of a field, but from my understanding, this should work with both.

Would you mind checking on your end and see if the import is now working for you.

Thanks!

[apeisa]

Currently it gives me an error:

Parse error: syntax error, unexpected '[' in /.../site/modules/ProcessPageTreeMigrator/ProcessPageTreeMigrator.module on line 666 

And the code there is:

$newArray = [];

So it requires php 5.4. Just changed it to array() and now it works.

Importing full data still doesn't work, this time it seems to be repeaters:

FieldtypeRepeater: Created Repeater Page Parent: /processwire/repeaters/for-field-98/for-page-1067/

TemplateFile: Method RepeaterPageArray::setLanguageValue does not exist or is not callable in this context

For me export & import of templates and fields is the main selling point of this module (not the data).

[adrian]

Ah, sorry - been on 5.4 for so long I forgot about this.

Yes, repeaters aren't supported yet. For the moment I'll add that to the list of fieldtypes to ignore so at least there won't be an error.

Btw, the last update also now handles RTE links that have been processed by the PageLinkAbstractor module.

EDIT: Committed version with those small fixes. NB that the check to ignore repeaters is currently only on the export, so you would need to export again to check.

[adrian]

A small update - I have added support for import options so now you can decide whether you want to import just the fields and templates, or the entire page tree at the import stage, regardless of what is in the exported JSON file. Of course you can't import the page tree if it wasn't exported, but you can import just the fields and templates from a JSON file that contains the entire page tree.

I have also added support for importing page trees directly from the repo at: https://github.com/adrianbj/ProcessWirePageLists Just choose the "Shared JSON Packages" option when importing. 

I haven't tackled the repeater field etc issues yet, hopefully soon.

Will also be adding more shared packages soon and would love any contributions

See the attached screenshot showing the direct page tree import using Ryan's awesome new admin theme

[apeisa]

great progress Adrian!

[owzim]

Hey adrian, this is great work, thanks a lot for the effort. I tested it with small chunks of datalists, selection values and so on, works like a charm.

On one specific page with FieldSetTabs I get this, after hitting the Export button in Step 2:

Error: Cannot break/continue 1 level (line 641 of /pwtesting/site/modules/ProcessPageTreeMigrator/ProcessPageTreeMigrator.module) 

This chunk seems to be responsible:

	if($type == 'fields'){

	    if($page->type instanceof FieldtypeFieldsetOpen) continue;

	    $data = array(

	        'name' => $page->name,
	        'label' => $page->label,
	        'description' => $page->description,
	        'template' => $template_name,
	        'flags' => $page->flags,
	        'type' => "{$page->type}",

	    );

	}

Also this might not be relevant yet, but on a page on which I use soma's new alpha module for ContentBlocks after exporting I get this:

Error: Call to a member function get() on a non-object (line 734 of /pwtesting/site/modules/ProcessPageTreeMigrator/ProcessPageTreeMigrator.module) 

I'm on PW 2.3.3.3

Perhaps it would be useful for users to see in the github readme what's already working and what not working yet. Perhaps a little roadmap as well.

Thanks again, this will be a huge timesaver and the more modular approach for migrating little logical/data chunks rather than whole profiles will come in very handy when developing locally and on server parallely.

[adrian]

Thanks ozwim for the testing and bug reports. I'll see what I can do about getting these sorted out. I haven't tested with fieldsets at all yet. As for the error on line 734 - it is a language module related thing. I had seen that once before - should be easy enough to sort out.

Good idea for adding details to the github readme on what's working etc - I have a feeling it will be an increasing list for a while, before it starts getting smaller, but I am sure we'll get there

[adrian]

Ok, I have pushed another version to Github. v0.0.8 fixes the fieldsettab issue that Ozwim reported and introduces support for migrating repeater fields. It handles the creation of everything required on the source PW installation for the new repeater fields to work.

Next on the roadmap is full support for page fields - currently only the page field itself is migrated, but not the selectable pages fields and templates. Then it is on to migrating the page content for repeater and page fields

I would really appreciate any help testing out this new version. 

EDIT: Just updated the readme on github to detail what does and doesn't work, along with a rough roadmap - please let me know if your testing shows that I have any working/not working mistakes - I am sure there are some

[adrian]

Ok another update this morning - I have added support for full migration of page fields - it now automatically handles creation of the selectable pages fields, templates, and the pages themselves.

The one thing I haven't really figured out how to tackle is what parent to put these pages under. At the moment they end up under the same parent page that you choose when starting an import. This is probably not ideal, but otherwise it could result in the need to create a whole other lot of parent parents/templates/fields that you don't really need on the destination PW site. This way everything needed gets migrated but nothing more and you can easily move them into a new parent (like a tools parent) if needed. Anyone have any better ideas? You might need to run a test to see what I mean.

Again, this will probably need lots of testing so thanks in advance - there is a lot going on to make this all work!

[adrian]

Another update - things are getting closer!

I have just added support for migration of the selected values of page fields when migrating "everything, including all data pages".

Only repeater field page data (the fields, templates etc are already working) to go, and then of course the inevitable bugs

EDIT:

Just found a bug in the page field data migration - hopefully solve this soon.

Repeater data is working now though - not on github yet, but I will push that too once I have the page field issue sorted out.

Edited October 31, 2013 by adrian

[adrian]

Getting stoked

The latest version on github now supports migration of repeater field page content and fixes the problems with migrating page field content.

That should take care of migrating the fields, templates, and page content for all the field types, not including page content for file and image fields as obviously these can't, or more accurately, shouldn't be included in a json file.

Multi-language fields should migrate perfectly so long as you have the required language support modules and languages installed on the destination PW installation. Let me know if you have any troubles here - I did see some weird errors during early testing - I hope they are all fixed now.

Please note: that most of my testing of late has been with the "Everything, including all data pages" option on export and import. I need to go back and test the other options soon, but I'll probably do that when I start implementing the series of checkboxes allowing you to select exactly which fields, templates, and pages to be exported and imported.

Also, I have been testing with the latest dev version of PW (downloaded today). Not sure if anything is critical, but if you are getting errors, please try upgrading PW first.

[adrian]

Sorry for yet another post, but I just fixed a bug that prevented certain fields being exported and assigned to templates in some situations. Not the most elegant solution (repetitive declaration in the JSON file), but it should work for the moment until I have time to rework slightly.

Grab the latest version from Github for testing

[apeisa]

Adrian, thanks for your work on this. I am super busy at the moment, but I have great plans for this (reusing common templates etc..). Will definitely give this good ride soon.

[adrian]

Just pushed a minor fix to prevent 403 forbidden errors when hitting the github page lists repository when starting the import process. Apparently they were forcing a user-agent header for several months now, but didn't seem to affect this module till recently. Anyway, should be fixed now.

Sometime early in the new year I will sort out all the php notices/warnings in this module and add some more features for finer detailed selection of what you want to export/import.

[adrian]

Just committed a fairly major update to support full migration of files and images, including rewriting of the src tag to match the ID of the new page for images embedded into RTE fields. Additionally, the module now also migrates the template .php files.

Files/images/template files and the json structure/data file are exported in a zip file which is then imported into the destination PW install.

So, you can now select one parent page during export and it will migrate the following content for all child pages of the selected parent:

• All standard field types, including RTE, and decoding of links modified by the PageLinkAbstractor module and abstracting again on the destination PW install.
• File/Image/CropImage fields including the actual files/images/thumbnails and all other variations
• Repeater fields and all their required fields, templates, and all content, including files/images
• Page fields (and the pages, templates, and fields that make up their selectable pages)
• Multi-language versions of all field content
• Templates (including Access, Family, URL and other settings) and the template .php files. It even grabs the appropriate file if you are using the "Alternate Template Filename" setting. NB the templates directory on the destination PW installation must be writable for these to be imported.

 

So, you could build sections of content on a local dev PW installation, export it, and then with a couple of clicks import everything into the live PW installation.

 

Some outstanding issues that I hope to get to shortly:

• Need to support images inserted from a different page into an RTE field
• Rewrite any references to page ids, eg $pages->get(xxxx) in template .php files so they will be converted to the correct id on the destination installation.
• Need to look into the new core link abstractor that was added to PW 2.4 and see how to handle those links compared to the PageLinkAbstractor module.
• I still want to add finer control for determining exactly what components will be exported and imported.
• Generally needs lots more error checking for things like making sure required languages are installed on the destination PW install etc
• Need to add checks so that existing template php files are not overwritten (or give the option to choose)
• Might need to override PHP max_execution_time and other settings for larger exports and maybe chunk out zipping of all images to prevent memory issues on larger exports.

In its current state it is handling almost all my needs, but I would really like to have this robust enough for anyone to use, so if you have time please do some testing for me - THANKS!

At the moment, please only use it on two unimportant test PW installs!

Edited March 8, 2014 by adrian

[Marty Walker]

Far out! When I get a spare hour I'll give this a go.

[adrian]

Thanks Marty - I'd really appreciate that

[adrian]

Several of the outstanding issues and a couple of bug fixes have been taken care of this morning. Be sure to grab the latest from Github 

EDIT: Even more issues taken care of this afternoon. Updated in post #47 above.

Edited March 3, 2014 by adrian