adrian

Migrator

Recommended Posts

Run again and find out the id of the assets/files folder where the zip is stored. It should match the id of the Migrator Admin > Setup > Migrator page id, but some reason @NooseLadder is seeing it in one of his normal page id folders. Seems like it's a windows/xampp issue.

Just to clarify, the zip file (files.zip) was correctly created in the /1062/ folder but the error mssg pointed to /1022/ which belongs to another page.

  • Like 1

Share this post


Link to post
Share on other sites

.....................

@kongondo - thanks for offering. You are right, this thread is a behemoth - I will start a new one as soon as I am ready to release, but for now to test:

  1. Grab the module from: https://github.com/adrianbj/ProcessMigrator and install
  2. Make sure debug mode is on
  3. Run the module (Setup > Migrator)
  4. Choose Export
  5. Choose a Parent Page - to keep things simple, maybe something like the default PW's About page
  6. Export
  7. Check that the downloaded zip file contains a data.json file
  8. Let me know if you get any path related errors like @NooseLadder
  9. If you're feeling energetic, comment out line 335: //unlink($zipFilename);
  10. Run again and find out the id of the assets/files folder where the zip is stored. It should match the id of the Migrator Admin > Setup > Migrator page id, but some reason @NooseLadder is seeing it in one of his normal page id folders. Seems like it's a windows/xampp issue.

Thanks so much!

@Adrian,

Does what it says on the tin. No errors and the id of the assets/files folder where the zip is stored matches id of Migrator page..

  • Like 2

Share this post


Link to post
Share on other sites
Hi.

First off the work you guys have done with the Wordpress migration tool is amazing, unfortunalety I am not able to get it to work :(

I am running PW 2.4, I have tried to modules both in a WAMP environment and ISP provided environments, both in brand new PW installations and current ones.

I have loaded the blog.zip profile first, this works fine.

Then when I try to import the XML from wordpress (3.9.1) all I am getting is a message saying.

"Missing required ZIP or JSON source"

There is a possibility that I have missed something vital.

This are the basic steps that I take.

Install PW

Install module ProcessMigration

Install module MigratorWordpress

Migrating the blog.zip profile to /home/

Trying to migrate the wordpress.xml to /home/blog (root of blog recently created with blog.zip)

Any Ideas ?

Would really like to get this one working.

Thanks Ronnie

Share this post


Link to post
Share on other sites

@Ronnie,

Sorry about that - it was a simple fix - please try the latest version on Github and let me know how it goes for you. You might be still be affected by an outstanding multi-language issue but hopefully I can have the fixed soon.

@tobaco

Still have some issues with the language error and also trying to improve the Replace option - need to sort out recursive deletion of existing pages to work better, but I have fixed a bunch of issues around reporting/automatic installation and proper configuring of Inputfield types that are needed by the imported fields - for example, the icon select field now properly detects the need for installing InputfieldSelectMultipleTransfer and then uses it once installed. This should work for all input field types now.

Share this post


Link to post
Share on other sites

@Ronnie,

I should also mention that you do not need to import the blog.zip file. You can use Kongondo's Blog module (http://modules.processwire.com/modules/process-blog/) as the base, or just import your WP xml and manually create the templates to suit your needs.

If you decide to use the Blog module, here is a screenshot of the MigratorWordpress config settings that are required:

post-985-0-90394500-1404404097_thumb.png

Share this post


Link to post
Share on other sites

@tobaco - please try the latest version - I have made a lot of fixes and I now get a fully(?) working import of your json file!

Firstly, please run the export on your original site again first as I made a couple of fixes there. Then test on a clean PW install and use the REPLACE option so that your import will completely replace the existing pages from the default PW profile.

Please let me know if all the content appears to be populated as you expect - there are lots of fields with empty values in the json export, but I am not sure if that is correct or not, so I'll need your input on that.

There are still a couple of issues:

  1. Using the replace method a second time results in out of memory errors - I think it's the recursive deletion of the content from that repeater field
  2. I have temporarily disabled importing content for image fields inside repeaters as that is causing errors with your import. I am pretty sure images in repeaters worked fine in my previous testing, so I am not sure where that problem is yet.

Thanks again!

PS The latest version handles installation of all required language modules and gracefully warns of missing language packs and lets you install them as needed.

PPS - if you are doing lots of testing and need to be able to revert to a fresh install regularly, it pays to fire up PHPMyAdmin and create a clean copy of the DB after you have installed PW (or perhaps after you have also installed all your required modules) so that you can simply drop the actual DB and create a new copy from the backed up copy. You might be already aware of this approach, but it's a huge timesaver when testing things like this.

  • Like 1

Share this post


Link to post
Share on other sites

@Adrian

Thanks for the reply, unfortunately I am still experiencing the same issue.

I have used the latest versions from GitHub that you suggested.

When I turned on debug mode on my PW installation I see something that might be interesting.

I see some references to:
C:/Bitnami/processwire-2.4.0-3/apps/processwire/htdocs/site/assets/files/1043/migratorfiles

and

C:/Bitnami/processwire-2.4.0-3/apps/processwire/htdocs/site/assets/files/1043/migratorfiles\data.json

Attached file: 

post-1232-0-46162200-1404732759_thumb.jp

The only thing that is present in that folder is the WP xml file.

C:\Bitnami\processwire-2.4.0-3\apps\processwire\htdocs\site\assets\files\1043\neossida.xml

No migratorfiles folder / data.json file

Attached file:

post-1232-0-95093300-1404732759_thumb.jp

This file / folder is not present on my machine, I am not completely sure how this works but I quess that the plugin should transform the WP xml to a JSON file that is supposed to be used in the migration.

Maybe that would point you in some direction ? 

Thanks / Ronnie

post-1232-0-46162200-1404732759_thumb.jp

post-1232-0-95093300-1404732759_thumb.jp

Share this post


Link to post
Share on other sites

Sorry Ronnie,

I have tested more thoroughly this time and the latest version of Migrator should fix the issue.

Thanks again for letting me know how it goes.

Share this post


Link to post
Share on other sites

@Adrian.

Migrating WP sites like a boss now :)

Thanks for the fix, that seemed to do the trick, I have not tried it thoroughly yet, but I got past the previously issue and content is being created.

  • Like 4

Share this post


Link to post
Share on other sites

Great to hear :)

Now I just need to hear from @tobaco that everything is working for him with him and I'll be a happy camper - literally :) - I am heading camping for the rest of the week, but I'll follow up with any other issues here when I get back.

  • Like 1

Share this post


Link to post
Share on other sites

Hey Adrian,

I've got a merging of two PWs to deal with and I was wondering if this was possible with Migrator.

PW Site One essentially contains client information (business data - names, addresses, descriptions etc)

PW Site Two contains invoicing information - a site developed to track user billing and payment for the businesses in PW Site One. 

What I would like is to merge the two sites so the Site Two pages would become a branch of the Site One Tree, eg /invoicing/siteTwo

However, it looks like Migrator is developed to migrate dev to live etc, so presumably page ids would be the same etc. What happens if a data page from Site Two has the same id as a invoicing page from Site One for example. Is this magically 'handled' or would a overwrite occur?

Share this post


Link to post
Share on other sites

muzzer,

Your scenario is exactly what Migrator was designed to do. It does not migrate page IDs at all. Every reference to an id gets converted to a name/path on export and a new page id created on import to the new site.

I would still recommend testing the migration to a test PW install first until you have a full understanding of all the import settings and how they can affect the import. Also when you do it to the live site, please make sure to backup your database first. This module generally works as advertised, but there are some weird scenarios like what Macrura experienced with a page field that points to users that can cause problems. I will be fixing this shortly - I just need some more info from him.

Please let me know how it goes for you and if you think of any possible improvements in the way it works.

  • Like 2

Share this post


Link to post
Share on other sites

Super cool new feature just added - well at least I think it's cool :)

You can now automatically backup your entire site's database and templates directory during the import process. Backups are archived so you can go back to any previously created version whenever you want.

If something goes wrong during the import phase of the migration, or you simply imported the wrong json file, or you changed your mind, you can restore everything with the click of a button!

This database part of this functionality relies on Processwire's brand new WireDatabaseBackup class that Ryan committed today, so to use this functionality, go grab the latest dev version of PW and the latest version of this module.

Please let me know how it goes for you!

post-985-0-07817100-1408509206_thumb.png

post-985-0-78763500-1408509237_thumb.png

post-985-0-67320700-1408509271_thumb.png

  • Like 15

Share this post


Link to post
Share on other sites

Adrian, do you ever sleep? :P  This is awesome man! Quick question, maybe I missed it, maybe this (especially with the new additions) should now be a superadmin 's only toy? Just wondering....

Share this post


Link to post
Share on other sites

Adrian, do you ever sleep? :P  This is awesome man! Quick question, maybe I missed it, maybe this (especially with the new additions) should now be a superadmin 's only toy? Just wondering....

The backup/restore stuff was easy thanks to Ryan :)

I haven't given the module a permission setting so I think that limits it to superuser accounts. Are you seeing otherwise?

Something a little weird I just noticed - if you try to enter the url to a Process module, eg: 

http://pwtest.dev/admin/setup/migrator/

when logged in as a user without the appropriate permission, you get this error: TemplateFile: Unrecognized path - maybe that's better than a meaningful error saying you don't have permission or something?

Anyway, I think this is locked down sufficiently to superusers?

Share this post


Link to post
Share on other sites

Aah, didn't know permissions work like that. If so, it is locked down OK...

Share this post


Link to post
Share on other sites

@adrian How's further development going on ProcessMigrator?  I was playing around with it yesterday and it seems it's still quite buggy.  I tested it about 20 times on fresh installs and got a few errors which I can discuss with you.

I would like to feature it in a WordPress vs. ProcessWire screencast.  Your plugin solves a huge pain point.

Let me know if I can help.

  • Like 1

Share this post


Link to post
Share on other sites

Hi @jlahijani,

I haven't done much on it for a while now.

I did think that most bugs were sorted by now though. The biggest remaining issue (I thought) was page fields and how to handle them if their content pages are in a page tree that is not part of the exported parent. At the moment they are exported and recreated on the destination site as a child of home, which obviously may not be ideal, so I think I need to add some additional logic to recreate the entire tree above the page field parent (if it doesn't exist on the destination site), although I can see this being problematic too. Since you've been testing lots, I'd love any thoughts you might have on the best way to handle this.

Also, please send through all the details of the errors you got while testing - it will be a huge help I am sure.

As for Wordpress vs Processwire - are you wanting to feature Migrator's use as an importer of WP content (using Nico's plugin), or just its ability for migration of PW content from one site to another? If the former, then I think that functionality should be pretty robust.

Anyway, please let me know what you have uncovered - it would be great to get this to the next level!

Share this post


Link to post
Share on other sites

Good to hear.  I will put together some info about the bugs.

I'm planning on featuring the ProcessMigrator plugin as a general exporter/importer of content from one PW site to another.  For example, let's say you have a development and production version of a site, and a new set of content needs to be created (on dev), then eventually launched (on prod).  Rather than re-creating that content manually on production, I want to show how your plugin can achieve that.

WordPress has an export tool, however it's very limited and doesn't work that well for the situation I described.  There are some plugins to enhance it but I haven't had great success with those either.  There's one commercial plugin which I haven't used but seems to be the ultimate solution to do what I described: https://crowdfavorite.com/ramp/

ProcessMigrator, in my eyes, would be the analog to RAMP.

  • Like 1

Share this post


Link to post
Share on other sites

One thing that would be very helpful is if you can provide the exported JSON (or full zip package if that is relevant) that is causing issues when you import - this will make it much easier to debug.

Thanks again!

Share this post


Link to post
Share on other sites

I just want to drop a note here to publicly thank jlahijani and muzzer for lots of recent help getting several more bugs identified and fixed and at the same time mention that you should all grab the most recent version on Github (https://github.com/adrianbj/ProcessMigrator).

Thanks again guys!

  • Like 5

Share this post


Link to post
Share on other sites

WordPress has an export tool, however it's very limited and doesn't work that well for the situation I described.  There are some plugins to enhance it but I haven't had great success with those either.  There's one commercial plugin which I haven't used but seems to be the ultimate solution to do what I described: https://crowdfavorite.com/ramp/

 

ProcessMigrator, in my eyes, would be the analog to RAMP.

jlahijani - I finally had a look at the video for RAMP - it looks very cool, especially the automatic pushing from dev to production. Other than that, the big difference I noticed compared to Migrator is that currently Migrator can only export one page tree at a time and it has to be the entire tree of children below it. Sure you have the option to determine which pages get imported to the other site, but I think it would be a great enhancement to offer this selection during export as well. I am thinking the option to choose multiple pages and for each one (if it has children), determine whether to include the children or not. I think this approach would make it much easier to migrate a few newly developed pages, or several new sections, without the need to overwrite everything under an overarching  parent page. You don't actually have to do this with the existing Edit Imported Content option, but it is not as obvious or easy.

Anyway, it has given me some great ideas - thanks! 

  • Like 1

Share this post


Link to post
Share on other sites

Some more bug fixes - thanks again to muzzer! 

More interestingly though, during import you now have the option to automatically download and install any required missing fieldtypes/inputfields from the PW modules directory.

This is optional and I have noted that if you don't trust the source of the data you are importing, then you might want not want to enable this option.

It is off by default, in which case you'll receive notifications of what's missing so you can install manually before re-starting the import.

  • Like 5

Share this post


Link to post
Share on other sites

Hi, just thought I would have a go with this. I got this:

Error: Call to a member function size() on a non-object (line 2504 of /home/foodloversdiary/public_html/site/modules/ProcessMigrator/ProcessMigrator.module) 

So, it seems to have imported the categories and a load of tags, but only 5 posts and they have no content.

Any thoughts?

Edit: forgot to mention that it was with a Wordpress import

Share this post


Link to post
Share on other sites

Hey Joss,

Thanks for testing this out. Looks like you have come across an issue with a very recently added function to scan through RTE fields and automatically generate resized versions of images that have been embedded.

Actually I am wondering - is the WP still still online and working? It needs to be so that the module can download the images from the site and add them to each of the pages in PW.

Any chance you could PM me the WP xml file so I can test?

I am heading out for the rest of the day, but I will look into it in the morning for you - I am sure it will be an easy fix.

PS, Please make sure you are always using the latest version of Migrator and also MigratorWordpress (my fork - Nico's is not up to date)

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now


  • Recently Browsing   0 members

    No registered users viewing this page.

  • Similar Content

    • By BitPoet
      So I stumbled over the request to allow limiting templates to be used only once under every parent page in this thread
      and found that this would actually come in handy (also in a site I've built).
      The code can be found on github and soon in the module repo.
      After installation, you'll find a new checkbox "Only once per parent" in the family tab when editing a template.

    • By Robin S
      Password Generator
      Adds a password generator to InputfieldPassword.

       
      Usage
      Install the Password Generator module.
      Now any InputfieldPassword has a password generation feature. The settings for the generator are taken automatically from the settings* of the password field.
      *Settings not supported by the generator:
      Complexify: but generated passwords should still satisfy complexify settings in the recommended range. Banned words: but the generated passwords are random strings so actual words are unlikely to occur.  
      https://modules.processwire.com/modules/password-generator/
      https://github.com/Toutouwai/PasswordGenerator
    • By Robin S
      If you've ever needed to insert links to a large number of files within CKEditor you may have found that the standard PW link modal is a somewhat slow way to do it.
      This module provides a quicker way to insert links to files on the page being edited. You can insert a link to an individual file, or insert an unordered list of links to all files on the page with a single click.
      CKEditor Link Files
      Adds a menu to CKEditor to allow the quick insertion of links to files on the page being edited.

      Features
      Hover a menu item to see the "Description" of the corresponding file (if present). Click a menu item to insert a link to the corresponding file at the current cursor position. The filename is used as the link text. If you Alt-click a menu item the file description is used as the link text (with fallback to filename if no description entered). If text is currently selected in the editor then the selected text is used as the link text. Click "* Insert links to all files *" to insert an unordered list of links to all files on the page. Also works with the Alt-click option. Menu is built via AJAX so newly uploaded files are included in the menu without the page needing to be saved. However, descriptions are not available for newly uploaded files until the page is saved. Installation
      Install the CKEditor Link Files module.
      For any CKEditor field where you want the "Insert link to file" dropdown menu to appear in the CKEditor toolbar, visit the field settings and add "LinkFilesMenu" to the "CKEditor Toolbar" settings field.
       
      http://modules.processwire.com/modules/cke-link-files/
      https://github.com/Toutouwai/CkeLinkFiles
    • By matjazp
      A module for managing files and folders. Supports creating, opening (e.g. viewing, playing, editing), renaming, moving, copying, deleting and searching for files. You can also view and change (not supported on Windows) file and directory permissions. 
      https://github.com/matjazpotocnik/ProcessFileManager

      The author of FileManager component is (c) 2006 - 2018 Gerd Tentler, http://www.gerd-tentler.de/tools/filemanager/. I modified it to work with ProcessWire as a module. Please see license files on usage in commercial projects!
    • By Robin S
      An inputfield for displaying markup editable via CKEditor.
      The module is intended for use with the Form Builder module. Allows blocks of static text to be included within a form, which can be edited in the form settings using CKEditor.
      Usage
      Install the Markup CKEditor module.
      In the Form Builder module settings, add "MarkupCKEditor" to "Inputfield types to use with FormBuilder".
      In your form settings, add a new field of type "Markup CKEditor". Enter the text you want to show in this field using "Markup Text" on the "Details" tab.
      Screenshots


       
      http://modules.processwire.com/modules/inputfield-markup-ckeditor/
      https://github.com/Toutouwai/InputfieldMarkupCKEditor