Module Migrator

[NooseLadder]

Run again and find out the id of the assets/files folder where the zip is stored. It should match the id of the Migrator Admin > Setup > Migrator page id, but some reason @NooseLadder is seeing it in one of his normal page id folders. Seems like it's a windows/xampp issue.

Just to clarify, the zip file (files.zip) was correctly created in the /1062/ folder but the error mssg pointed to /1022/ which belongs to another page.

[kongondo]

.....................

@kongondo - thanks for offering. You are right, this thread is a behemoth - I will start a new one as soon as I am ready to release, but for now to test:

1. Grab the module from: https://github.com/adrianbj/ProcessMigrator and install
2. Make sure debug mode is on
3. Run the module (Setup > Migrator)
4. Choose Export
5. Choose a Parent Page - to keep things simple, maybe something like the default PW's About page
6. Export
7. Check that the downloaded zip file contains a data.json file
8. Let me know if you get any path related errors like @NooseLadder
9. If you're feeling energetic, comment out line 335: //unlink($zipFilename);
10. Run again and find out the id of the assets/files folder where the zip is stored. It should match the id of the Migrator Admin > Setup > Migrator page id, but some reason @NooseLadder is seeing it in one of his normal page id folders. Seems like it's a windows/xampp issue.

Thanks so much!

@Adrian,

Does what it says on the tin. No errors and the id of the assets/files folder where the zip is stored matches id of Migrator page..

[Ronnie]

Hi.

First off the work you guys have done with the Wordpress migration tool is amazing, unfortunalety I am not able to get it to work

I am running PW 2.4, I have tried to modules both in a WAMP environment and ISP provided environments, both in brand new PW installations and current ones.

I have loaded the blog.zip profile first, this works fine.

Then when I try to import the XML from wordpress (3.9.1) all I am getting is a message saying.

"Missing required ZIP or JSON source"

There is a possibility that I have missed something vital.

This are the basic steps that I take.

Install PW

Install module ProcessMigration

Install module MigratorWordpress

Migrating the blog.zip profile to /home/

Trying to migrate the wordpress.xml to /home/blog (root of blog recently created with blog.zip)

Any Ideas ?

Would really like to get this one working.

Thanks Ronnie

[adrian]

@Ronnie,

Sorry about that - it was a simple fix - please try the latest version on Github and let me know how it goes for you. You might be still be affected by an outstanding multi-language issue but hopefully I can have the fixed soon.

@tobaco

Still have some issues with the language error and also trying to improve the Replace option - need to sort out recursive deletion of existing pages to work better, but I have fixed a bunch of issues around reporting/automatic installation and proper configuring of Inputfield types that are needed by the imported fields - for example, the icon select field now properly detects the need for installing InputfieldSelectMultipleTransfer and then uses it once installed. This should work for all input field types now.

[adrian]

@Ronnie,

I should also mention that you do not need to import the blog.zip file. You can use Kongondo's Blog module (http://modules.processwire.com/modules/process-blog/) as the base, or just import your WP xml and manually create the templates to suit your needs.

If you decide to use the Blog module, here is a screenshot of the MigratorWordpress config settings that are required:

[adrian]

@tobaco - please try the latest version - I have made a lot of fixes and I now get a fully(?) working import of your json file!

Firstly, please run the export on your original site again first as I made a couple of fixes there. Then test on a clean PW install and use the REPLACE option so that your import will completely replace the existing pages from the default PW profile.

Please let me know if all the content appears to be populated as you expect - there are lots of fields with empty values in the json export, but I am not sure if that is correct or not, so I'll need your input on that.

There are still a couple of issues:

1. Using the replace method a second time results in out of memory errors - I think it's the recursive deletion of the content from that repeater field
2. I have temporarily disabled importing content for image fields inside repeaters as that is causing errors with your import. I am pretty sure images in repeaters worked fine in my previous testing, so I am not sure where that problem is yet.

Thanks again!

PS The latest version handles installation of all required language modules and gracefully warns of missing language packs and lets you install them as needed.

PPS - if you are doing lots of testing and need to be able to revert to a fresh install regularly, it pays to fire up PHPMyAdmin and create a clean copy of the DB after you have installed PW (or perhaps after you have also installed all your required modules) so that you can simply drop the actual DB and create a new copy from the backed up copy. You might be already aware of this approach, but it's a huge timesaver when testing things like this.

[Ronnie]

@Adrian

Thanks for the reply, unfortunately I am still experiencing the same issue.

I have used the latest versions from GitHub that you suggested.

When I turned on debug mode on my PW installation I see something that might be interesting.

I see some references to:
C:/Bitnami/processwire-2.4.0-3/apps/processwire/htdocs/site/assets/files/1043/migratorfiles

and

C:/Bitnami/processwire-2.4.0-3/apps/processwire/htdocs/site/assets/files/1043/migratorfiles\data.json

Attached file: 

The only thing that is present in that folder is the WP xml file.

C:\Bitnami\processwire-2.4.0-3\apps\processwire\htdocs\site\assets\files\1043\neossida.xml

No migratorfiles folder / data.json file

Attached file:

This file / folder is not present on my machine, I am not completely sure how this works but I quess that the plugin should transform the WP xml to a JSON file that is supposed to be used in the migration.

Maybe that would point you in some direction ? 

Thanks / Ronnie

[adrian]

Sorry Ronnie,

I have tested more thoroughly this time and the latest version of Migrator should fix the issue.

Thanks again for letting me know how it goes.

[Ronnie]

@Adrian.

Migrating WP sites like a boss now

Thanks for the fix, that seemed to do the trick, I have not tried it thoroughly yet, but I got past the previously issue and content is being created.

[adrian]

Great to hear

Now I just need to hear from @tobaco that everything is working for him with him and I'll be a happy camper - literally - I am heading camping for the rest of the week, but I'll follow up with any other issues here when I get back.

[muzzer]

Hey Adrian,

I've got a merging of two PWs to deal with and I was wondering if this was possible with Migrator.

PW Site One essentially contains client information (business data - names, addresses, descriptions etc)

PW Site Two contains invoicing information - a site developed to track user billing and payment for the businesses in PW Site One. 

What I would like is to merge the two sites so the Site Two pages would become a branch of the Site One Tree, eg /invoicing/siteTwo

However, it looks like Migrator is developed to migrate dev to live etc, so presumably page ids would be the same etc. What happens if a data page from Site Two has the same id as a invoicing page from Site One for example. Is this magically 'handled' or would a overwrite occur?

[adrian]

muzzer,

Your scenario is exactly what Migrator was designed to do. It does not migrate page IDs at all. Every reference to an id gets converted to a name/path on export and a new page id created on import to the new site.

I would still recommend testing the migration to a test PW install first until you have a full understanding of all the import settings and how they can affect the import. Also when you do it to the live site, please make sure to backup your database first. This module generally works as advertised, but there are some weird scenarios like what Macrura experienced with a page field that points to users that can cause problems. I will be fixing this shortly - I just need some more info from him.

Please let me know how it goes for you and if you think of any possible improvements in the way it works.

[adrian]

Super cool new feature just added - well at least I think it's cool

You can now automatically backup your entire site's database and templates directory during the import process. Backups are archived so you can go back to any previously created version whenever you want.

If something goes wrong during the import phase of the migration, or you simply imported the wrong json file, or you changed your mind, you can restore everything with the click of a button!

This database part of this functionality relies on Processwire's brand new WireDatabaseBackup class that Ryan committed today, so to use this functionality, go grab the latest dev version of PW and the latest version of this module.

Please let me know how it goes for you!

[kongondo]

Adrian, do you ever sleep?   This is awesome man! Quick question, maybe I missed it, maybe this (especially with the new additions) should now be a superadmin 's only toy? Just wondering....

[adrian]

Adrian, do you ever sleep?   This is awesome man! Quick question, maybe I missed it, maybe this (especially with the new additions) should now be a superadmin 's only toy? Just wondering....

The backup/restore stuff was easy thanks to Ryan

I haven't given the module a permission setting so I think that limits it to superuser accounts. Are you seeing otherwise?

Something a little weird I just noticed - if you try to enter the url to a Process module, eg: 

http://pwtest.dev/admin/setup/migrator/

when logged in as a user without the appropriate permission, you get this error: TemplateFile: Unrecognized path - maybe that's better than a meaningful error saying you don't have permission or something?

Anyway, I think this is locked down sufficiently to superusers?

[kongondo]

Aah, didn't know permissions work like that. If so, it is locked down OK...

[Jonathan Lahijani]

@adrian How's further development going on ProcessMigrator?  I was playing around with it yesterday and it seems it's still quite buggy.  I tested it about 20 times on fresh installs and got a few errors which I can discuss with you.

I would like to feature it in a WordPress vs. ProcessWire screencast.  Your plugin solves a huge pain point.

Let me know if I can help.

[adrian]

Hi @jlahijani,

I haven't done much on it for a while now.

I did think that most bugs were sorted by now though. The biggest remaining issue (I thought) was page fields and how to handle them if their content pages are in a page tree that is not part of the exported parent. At the moment they are exported and recreated on the destination site as a child of home, which obviously may not be ideal, so I think I need to add some additional logic to recreate the entire tree above the page field parent (if it doesn't exist on the destination site), although I can see this being problematic too. Since you've been testing lots, I'd love any thoughts you might have on the best way to handle this.

Also, please send through all the details of the errors you got while testing - it will be a huge help I am sure.

As for Wordpress vs Processwire - are you wanting to feature Migrator's use as an importer of WP content (using Nico's plugin), or just its ability for migration of PW content from one site to another? If the former, then I think that functionality should be pretty robust.

Anyway, please let me know what you have uncovered - it would be great to get this to the next level!

[Jonathan Lahijani]

Good to hear.  I will put together some info about the bugs.

I'm planning on featuring the ProcessMigrator plugin as a general exporter/importer of content from one PW site to another.  For example, let's say you have a development and production version of a site, and a new set of content needs to be created (on dev), then eventually launched (on prod).  Rather than re-creating that content manually on production, I want to show how your plugin can achieve that.

WordPress has an export tool, however it's very limited and doesn't work that well for the situation I described.  There are some plugins to enhance it but I haven't had great success with those either.  There's one commercial plugin which I haven't used but seems to be the ultimate solution to do what I described: https://crowdfavorite.com/ramp/

ProcessMigrator, in my eyes, would be the analog to RAMP.

[adrian]

One thing that would be very helpful is if you can provide the exported JSON (or full zip package if that is relevant) that is causing issues when you import - this will make it much easier to debug.

Thanks again!

[adrian]

I just want to drop a note here to publicly thank jlahijani and muzzer for lots of recent help getting several more bugs identified and fixed and at the same time mention that you should all grab the most recent version on Github (https://github.com/adrianbj/ProcessMigrator).

Thanks again guys!

[adrian]

WordPress has an export tool, however it's very limited and doesn't work that well for the situation I described.  There are some plugins to enhance it but I haven't had great success with those either.  There's one commercial plugin which I haven't used but seems to be the ultimate solution to do what I described: https://crowdfavorite.com/ramp/

 

ProcessMigrator, in my eyes, would be the analog to RAMP.

jlahijani - I finally had a look at the video for RAMP - it looks very cool, especially the automatic pushing from dev to production. Other than that, the big difference I noticed compared to Migrator is that currently Migrator can only export one page tree at a time and it has to be the entire tree of children below it. Sure you have the option to determine which pages get imported to the other site, but I think it would be a great enhancement to offer this selection during export as well. I am thinking the option to choose multiple pages and for each one (if it has children), determine whether to include the children or not. I think this approach would make it much easier to migrate a few newly developed pages, or several new sections, without the need to overwrite everything under an overarching  parent page. You don't actually have to do this with the existing Edit Imported Content option, but it is not as obvious or easy.

Anyway, it has given me some great ideas - thanks! 

[adrian]

Some more bug fixes - thanks again to muzzer! 

More interestingly though, during import you now have the option to automatically download and install any required missing fieldtypes/inputfields from the PW modules directory.

This is optional and I have noted that if you don't trust the source of the data you are importing, then you might want not want to enable this option.

It is off by default, in which case you'll receive notifications of what's missing so you can install manually before re-starting the import.

[Joss]

Hi, just thought I would have a go with this. I got this:

Error: Call to a member function size() on a non-object (line 2504 of /home/foodloversdiary/public_html/site/modules/ProcessMigrator/ProcessMigrator.module) 

So, it seems to have imported the categories and a load of tags, but only 5 posts and they have no content.

Any thoughts?

Edit: forgot to mention that it was with a Wordpress import

[adrian]

Hey Joss,

Thanks for testing this out. Looks like you have come across an issue with a very recently added function to scan through RTE fields and automatically generate resized versions of images that have been embedded.

Actually I am wondering - is the WP still still online and working? It needs to be so that the module can download the images from the site and add them to each of the pages in PW.

Any chance you could PM me the WP xml file so I can test?

I am heading out for the rest of the day, but I will look into it in the morning for you - I am sure it will be an easy fix.

PS, Please make sure you are always using the latest version of Migrator and also MigratorWordpress (my fork - Nico's is not up to date)