Jump to content
adrian

Migrator

Recommended Posts

Run again and find out the id of the assets/files folder where the zip is stored. It should match the id of the Migrator Admin > Setup > Migrator page id, but some reason @NooseLadder is seeing it in one of his normal page id folders. Seems like it's a windows/xampp issue.

Just to clarify, the zip file (files.zip) was correctly created in the /1062/ folder but the error mssg pointed to /1022/ which belongs to another page.

  • Like 1

Share this post


Link to post
Share on other sites

.....................

@kongondo - thanks for offering. You are right, this thread is a behemoth - I will start a new one as soon as I am ready to release, but for now to test:

  1. Grab the module from: https://github.com/adrianbj/ProcessMigrator and install
  2. Make sure debug mode is on
  3. Run the module (Setup > Migrator)
  4. Choose Export
  5. Choose a Parent Page - to keep things simple, maybe something like the default PW's About page
  6. Export
  7. Check that the downloaded zip file contains a data.json file
  8. Let me know if you get any path related errors like @NooseLadder
  9. If you're feeling energetic, comment out line 335: //unlink($zipFilename);
  10. Run again and find out the id of the assets/files folder where the zip is stored. It should match the id of the Migrator Admin > Setup > Migrator page id, but some reason @NooseLadder is seeing it in one of his normal page id folders. Seems like it's a windows/xampp issue.

Thanks so much!

@Adrian,

Does what it says on the tin. No errors and the id of the assets/files folder where the zip is stored matches id of Migrator page..

  • Like 2

Share this post


Link to post
Share on other sites
Hi.

First off the work you guys have done with the Wordpress migration tool is amazing, unfortunalety I am not able to get it to work :(

I am running PW 2.4, I have tried to modules both in a WAMP environment and ISP provided environments, both in brand new PW installations and current ones.

I have loaded the blog.zip profile first, this works fine.

Then when I try to import the XML from wordpress (3.9.1) all I am getting is a message saying.

"Missing required ZIP or JSON source"

There is a possibility that I have missed something vital.

This are the basic steps that I take.

Install PW

Install module ProcessMigration

Install module MigratorWordpress

Migrating the blog.zip profile to /home/

Trying to migrate the wordpress.xml to /home/blog (root of blog recently created with blog.zip)

Any Ideas ?

Would really like to get this one working.

Thanks Ronnie

Share this post


Link to post
Share on other sites

@Ronnie,

Sorry about that - it was a simple fix - please try the latest version on Github and let me know how it goes for you. You might be still be affected by an outstanding multi-language issue but hopefully I can have the fixed soon.

@tobaco

Still have some issues with the language error and also trying to improve the Replace option - need to sort out recursive deletion of existing pages to work better, but I have fixed a bunch of issues around reporting/automatic installation and proper configuring of Inputfield types that are needed by the imported fields - for example, the icon select field now properly detects the need for installing InputfieldSelectMultipleTransfer and then uses it once installed. This should work for all input field types now.

Share this post


Link to post
Share on other sites

@Ronnie,

I should also mention that you do not need to import the blog.zip file. You can use Kongondo's Blog module (http://modules.processwire.com/modules/process-blog/) as the base, or just import your WP xml and manually create the templates to suit your needs.

If you decide to use the Blog module, here is a screenshot of the MigratorWordpress config settings that are required:

post-985-0-90394500-1404404097_thumb.png

Share this post


Link to post
Share on other sites

@tobaco - please try the latest version - I have made a lot of fixes and I now get a fully(?) working import of your json file!

Firstly, please run the export on your original site again first as I made a couple of fixes there. Then test on a clean PW install and use the REPLACE option so that your import will completely replace the existing pages from the default PW profile.

Please let me know if all the content appears to be populated as you expect - there are lots of fields with empty values in the json export, but I am not sure if that is correct or not, so I'll need your input on that.

There are still a couple of issues:

  1. Using the replace method a second time results in out of memory errors - I think it's the recursive deletion of the content from that repeater field
  2. I have temporarily disabled importing content for image fields inside repeaters as that is causing errors with your import. I am pretty sure images in repeaters worked fine in my previous testing, so I am not sure where that problem is yet.

Thanks again!

PS The latest version handles installation of all required language modules and gracefully warns of missing language packs and lets you install them as needed.

PPS - if you are doing lots of testing and need to be able to revert to a fresh install regularly, it pays to fire up PHPMyAdmin and create a clean copy of the DB after you have installed PW (or perhaps after you have also installed all your required modules) so that you can simply drop the actual DB and create a new copy from the backed up copy. You might be already aware of this approach, but it's a huge timesaver when testing things like this.

  • Like 1

Share this post


Link to post
Share on other sites

@Adrian

Thanks for the reply, unfortunately I am still experiencing the same issue.

I have used the latest versions from GitHub that you suggested.

When I turned on debug mode on my PW installation I see something that might be interesting.

I see some references to:
C:/Bitnami/processwire-2.4.0-3/apps/processwire/htdocs/site/assets/files/1043/migratorfiles

and

C:/Bitnami/processwire-2.4.0-3/apps/processwire/htdocs/site/assets/files/1043/migratorfiles\data.json

Attached file: 

post-1232-0-46162200-1404732759_thumb.jp

The only thing that is present in that folder is the WP xml file.

C:\Bitnami\processwire-2.4.0-3\apps\processwire\htdocs\site\assets\files\1043\neossida.xml

No migratorfiles folder / data.json file

Attached file:

post-1232-0-95093300-1404732759_thumb.jp

This file / folder is not present on my machine, I am not completely sure how this works but I quess that the plugin should transform the WP xml to a JSON file that is supposed to be used in the migration.

Maybe that would point you in some direction ? 

Thanks / Ronnie

post-1232-0-46162200-1404732759_thumb.jp

post-1232-0-95093300-1404732759_thumb.jp

Share this post


Link to post
Share on other sites

Sorry Ronnie,

I have tested more thoroughly this time and the latest version of Migrator should fix the issue.

Thanks again for letting me know how it goes.

Share this post


Link to post
Share on other sites

@Adrian.

Migrating WP sites like a boss now :)

Thanks for the fix, that seemed to do the trick, I have not tried it thoroughly yet, but I got past the previously issue and content is being created.

  • Like 4

Share this post


Link to post
Share on other sites

Great to hear :)

Now I just need to hear from @tobaco that everything is working for him with him and I'll be a happy camper - literally :) - I am heading camping for the rest of the week, but I'll follow up with any other issues here when I get back.

  • Like 1

Share this post


Link to post
Share on other sites

Hey Adrian,

I've got a merging of two PWs to deal with and I was wondering if this was possible with Migrator.

PW Site One essentially contains client information (business data - names, addresses, descriptions etc)

PW Site Two contains invoicing information - a site developed to track user billing and payment for the businesses in PW Site One. 

What I would like is to merge the two sites so the Site Two pages would become a branch of the Site One Tree, eg /invoicing/siteTwo

However, it looks like Migrator is developed to migrate dev to live etc, so presumably page ids would be the same etc. What happens if a data page from Site Two has the same id as a invoicing page from Site One for example. Is this magically 'handled' or would a overwrite occur?

Share this post


Link to post
Share on other sites

muzzer,

Your scenario is exactly what Migrator was designed to do. It does not migrate page IDs at all. Every reference to an id gets converted to a name/path on export and a new page id created on import to the new site.

I would still recommend testing the migration to a test PW install first until you have a full understanding of all the import settings and how they can affect the import. Also when you do it to the live site, please make sure to backup your database first. This module generally works as advertised, but there are some weird scenarios like what Macrura experienced with a page field that points to users that can cause problems. I will be fixing this shortly - I just need some more info from him.

Please let me know how it goes for you and if you think of any possible improvements in the way it works.

  • Like 2

Share this post


Link to post
Share on other sites

Super cool new feature just added - well at least I think it's cool :)

You can now automatically backup your entire site's database and templates directory during the import process. Backups are archived so you can go back to any previously created version whenever you want.

If something goes wrong during the import phase of the migration, or you simply imported the wrong json file, or you changed your mind, you can restore everything with the click of a button!

This database part of this functionality relies on Processwire's brand new WireDatabaseBackup class that Ryan committed today, so to use this functionality, go grab the latest dev version of PW and the latest version of this module.

Please let me know how it goes for you!

post-985-0-07817100-1408509206_thumb.png

post-985-0-78763500-1408509237_thumb.png

post-985-0-67320700-1408509271_thumb.png

  • Like 15

Share this post


Link to post
Share on other sites

Adrian, do you ever sleep? :P  This is awesome man! Quick question, maybe I missed it, maybe this (especially with the new additions) should now be a superadmin 's only toy? Just wondering....

Share this post


Link to post
Share on other sites

Adrian, do you ever sleep? :P  This is awesome man! Quick question, maybe I missed it, maybe this (especially with the new additions) should now be a superadmin 's only toy? Just wondering....

The backup/restore stuff was easy thanks to Ryan :)

I haven't given the module a permission setting so I think that limits it to superuser accounts. Are you seeing otherwise?

Something a little weird I just noticed - if you try to enter the url to a Process module, eg: 

http://pwtest.dev/admin/setup/migrator/

when logged in as a user without the appropriate permission, you get this error: TemplateFile: Unrecognized path - maybe that's better than a meaningful error saying you don't have permission or something?

Anyway, I think this is locked down sufficiently to superusers?

Share this post


Link to post
Share on other sites

Aah, didn't know permissions work like that. If so, it is locked down OK...

Share this post


Link to post
Share on other sites

@adrian How's further development going on ProcessMigrator?  I was playing around with it yesterday and it seems it's still quite buggy.  I tested it about 20 times on fresh installs and got a few errors which I can discuss with you.

I would like to feature it in a WordPress vs. ProcessWire screencast.  Your plugin solves a huge pain point.

Let me know if I can help.

  • Like 1

Share this post


Link to post
Share on other sites

Hi @jlahijani,

I haven't done much on it for a while now.

I did think that most bugs were sorted by now though. The biggest remaining issue (I thought) was page fields and how to handle them if their content pages are in a page tree that is not part of the exported parent. At the moment they are exported and recreated on the destination site as a child of home, which obviously may not be ideal, so I think I need to add some additional logic to recreate the entire tree above the page field parent (if it doesn't exist on the destination site), although I can see this being problematic too. Since you've been testing lots, I'd love any thoughts you might have on the best way to handle this.

Also, please send through all the details of the errors you got while testing - it will be a huge help I am sure.

As for Wordpress vs Processwire - are you wanting to feature Migrator's use as an importer of WP content (using Nico's plugin), or just its ability for migration of PW content from one site to another? If the former, then I think that functionality should be pretty robust.

Anyway, please let me know what you have uncovered - it would be great to get this to the next level!

Share this post


Link to post
Share on other sites

Good to hear.  I will put together some info about the bugs.

I'm planning on featuring the ProcessMigrator plugin as a general exporter/importer of content from one PW site to another.  For example, let's say you have a development and production version of a site, and a new set of content needs to be created (on dev), then eventually launched (on prod).  Rather than re-creating that content manually on production, I want to show how your plugin can achieve that.

WordPress has an export tool, however it's very limited and doesn't work that well for the situation I described.  There are some plugins to enhance it but I haven't had great success with those either.  There's one commercial plugin which I haven't used but seems to be the ultimate solution to do what I described: https://crowdfavorite.com/ramp/

ProcessMigrator, in my eyes, would be the analog to RAMP.

  • Like 1

Share this post


Link to post
Share on other sites

One thing that would be very helpful is if you can provide the exported JSON (or full zip package if that is relevant) that is causing issues when you import - this will make it much easier to debug.

Thanks again!

Share this post


Link to post
Share on other sites

I just want to drop a note here to publicly thank jlahijani and muzzer for lots of recent help getting several more bugs identified and fixed and at the same time mention that you should all grab the most recent version on Github (https://github.com/adrianbj/ProcessMigrator).

Thanks again guys!

  • Like 5

Share this post


Link to post
Share on other sites

WordPress has an export tool, however it's very limited and doesn't work that well for the situation I described.  There are some plugins to enhance it but I haven't had great success with those either.  There's one commercial plugin which I haven't used but seems to be the ultimate solution to do what I described: https://crowdfavorite.com/ramp/

 

ProcessMigrator, in my eyes, would be the analog to RAMP.

jlahijani - I finally had a look at the video for RAMP - it looks very cool, especially the automatic pushing from dev to production. Other than that, the big difference I noticed compared to Migrator is that currently Migrator can only export one page tree at a time and it has to be the entire tree of children below it. Sure you have the option to determine which pages get imported to the other site, but I think it would be a great enhancement to offer this selection during export as well. I am thinking the option to choose multiple pages and for each one (if it has children), determine whether to include the children or not. I think this approach would make it much easier to migrate a few newly developed pages, or several new sections, without the need to overwrite everything under an overarching  parent page. You don't actually have to do this with the existing Edit Imported Content option, but it is not as obvious or easy.

Anyway, it has given me some great ideas - thanks! 

  • Like 1

Share this post


Link to post
Share on other sites

Some more bug fixes - thanks again to muzzer! 

More interestingly though, during import you now have the option to automatically download and install any required missing fieldtypes/inputfields from the PW modules directory.

This is optional and I have noted that if you don't trust the source of the data you are importing, then you might want not want to enable this option.

It is off by default, in which case you'll receive notifications of what's missing so you can install manually before re-starting the import.

  • Like 5

Share this post


Link to post
Share on other sites

Hi, just thought I would have a go with this. I got this:

Error: Call to a member function size() on a non-object (line 2504 of /home/foodloversdiary/public_html/site/modules/ProcessMigrator/ProcessMigrator.module) 

So, it seems to have imported the categories and a load of tags, but only 5 posts and they have no content.

Any thoughts?

Edit: forgot to mention that it was with a Wordpress import

Share this post


Link to post
Share on other sites

Hey Joss,

Thanks for testing this out. Looks like you have come across an issue with a very recently added function to scan through RTE fields and automatically generate resized versions of images that have been embedded.

Actually I am wondering - is the WP still still online and working? It needs to be so that the module can download the images from the site and add them to each of the pages in PW.

Any chance you could PM me the WP xml file so I can test?

I am heading out for the rest of the day, but I will look into it in the morning for you - I am sure it will be an easy fix.

PS, Please make sure you are always using the latest version of Migrator and also MigratorWordpress (my fork - Nico's is not up to date)

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

  • Recently Browsing   0 members

    No registered users viewing this page.

  • Similar Content

    • By d'Hinnisdaël
      Happy new year, everybody 🥬
      I've been sitting on this Dashboard module I made for a client and finally came around to cleaning it up and releasing it to the wider public. This is how it looks.
      ProcessWire Dashboard

      If anyone is interested in trying this out, please go ahead! I'd love to get some feedback on it. If this proves useful and survives some real-world testing, I'll add this to the module directory.
      Download
      You can find the latest release on Github.
      Documentation
      Check out the documentation to get started. This is where you'll find information about included panel types and configuration options.
      Custom Panels
      My goal was to make it really simple to create custom panels. The easiest way to do that is to use the panel type template and have it render a file in your templates folder. This might be enough for 80% of all use cases. For anything more complex (FormBuilder submissions? Comments? Live chat?), you can add new panel types by creating modules that extend the DashboardPanel base class. Check out the documentation on custom panels or take a look at the HelloWorld panel to get started. I'm happy to merge any user-created modules into the main repo if they might be useful to more than a few people.
       Disclaimer
      This is a pre-release version. Please treat it as such — don't install it on production sites. Just making sure 🍇
      Roadmap
      These are the things I'm looking to implement myself at some point. The wishlist is a lot longer, but those are the 80/20 items that I probably won't regret spending time on.
      Improve documentation & add examples ⚙️ Panel types Google Analytics ⚙️ Add new page  🔥 Drafts 🔥 At a glance / Page counter 404s  Layout options Render multiple tabs per panel panel groups with heading and spacing between ✅ panel wrappers as grid item (e.g. stacked notices) ✅ Admin themes support AdminThemeReno and AdminThemeDefault ✅ Shortcuts panel add a table layout with icon, title & summary ✅ Chart panel add default styles for common chart types ✅ load chart data from JS file (currently passed as PHP array) Collection panel support image columns ✅ add buttons: view all & add new ✅
    • By Pip
      Hi everyone!
      I'm trying out the Login/Register module for my site. Noted that the module assigns the newly registered user to login-register role. 
      Once you modify the login-register role's permissions, particularly adding page-edit, the new member role will be set to guest. 
      Thing is I'd like to grant my new users the power to create their own pages. Any advice? 
      Thanks. 
    • By Gadgetto
      SnipWire - Snipcart integration for ProcessWire
      Snipcart is a powerful 3rd party, developer-first HTML/JavaScript shopping cart platform. SnipWire is the missing link between Snipcart and the content management framework ProcessWire.
      With SnipWire, you can quickly turn any ProcessWire site into a Snipcart online shop. The SnipWire plugin helps you to get your store up and running in no time. Detailed knowledge of the Snipcart system is not required.
      SnipWire is free and open source licensed under Mozilla Public License 2.0! A lot of work and effort has gone into development. It would be nice if you could donate an amount to support further development:

      Status update links (inside this thread) for SnipWire development
      2020-07-03 -- SnipWire 0.8.7 (beta) released! Fixes some small bugs and adds an indicator for TEST mode 2020-04-06 -- SnipWire 0.8.6 (beta) released! Adds support for Snipcart subscriptions and also fixes some problems 2020-03-21 -- SnipWire 0.8.5 (beta) released! Improves SnipWires webhooks interface and provides some other fixes and additions 2020-03-03 -- SnipWire 0.8.4 (beta) released! Improves compatibility for Windows based Systems. 2020-03-01 -- SnipWire 0.8.3 (beta) released! The installation and uninstallation process has been heavily revised. 2020-02-08 -- SnipWire 0.8.2 (beta) released! Added a feature to change the cart and catalogue currency by GET, POST or SESSION param 2020-02-03 -- SnipWire 0.8.1 (beta) released! All custom classes moved into their own namespaces. 2020-02-01 -- SnipWire is now available via ProcessWire's module directory! 2020-01-30 -- SnipWire 0.8.0 (beta) first public release! (module just submitted to the PW modules directory) 2020-01-28 -- added Custom Order Fields feature (first SnipWire release version is near!) 2020-01-21 -- Snipcart v3 - when will the new cart system be implemented? 2020-01-19 -- integrated taxes provider finished (+ very flexible shipping taxes handling) 2020-01-14 -- new date range picker, discount editor, order notifiactions, order statuses, and more ... 2019-11-15 -- orders filter, order details, download + resend invoices, refunds 2019-10-18 -- list filters, REST API improvements, new docs platform, and more ... 2019-08-08 -- dashboard interface, currency selector, managing Orders, Customers and Products, Added a WireTabs, refinded caching behavior 2019-06-15 -- taxes provider, shop templates update, multiCURL implementation, and more ... 2019-06-02 -- FieldtypeSnipWireTaxSelector 2019-05-25 -- SnipWire will be free and open source Plugin Key Features
      Fast and simple store setup Full integration of the Snipcart dashboard into the ProcessWire backend (no need to leave the ProcessWire admin area) Browse and manage orders, customers, discounts, abandoned carts, and more Multi currency support Custom order and cart fields Process refunds and send customer notifications from within the ProcessWire backend Process Abandoned Carts + sending messages to customers from within the ProcessWire backend Complete Snipcart webhooks integration (all events are hookable via ProcessWire hooks) Integrated taxes provider (which is more flexible then Snipcart own provider) Useful Links
      SnipWire in PW modules directory SnipWire Docs (please note that the documentation is a work in progress) SnipWire @GitHub (feature requests and suggestions for improvement are welcome - I also accept pull requests) Snipcart Website  

       
      ---- INITIAL POST FROM 2019-05-25 ----
       
    • By Sten
      Hello
      Till now I hacked something with the twig template but it works no more with new PW versions so I look forward to create a module. I am working on a site in multiple languages : French, English, Italian, German, Spanish, Portuguese, Hebrew, Russian. The new posts are entered in any language with a field for language. Till now, I got twig files to get the translations with constants defined for each part of the pages.
      So I'd like to create a module to include theses files added according to the url /fr/en/...
      Have you some observations to do before I begin about the direction to take ?
      Thank you
    • By ukyo
      Mystique Module for ProcessWire CMS/CMF
      Github repo : https://github.com/trk/Mystique
      Mystique module allow you to create dynamic fields and store dynamic fields data on database by using a config file.
      Requirements
      ProcessWire 3.0 or newer PHP 7.0 or newer FieldtypeMystique InputfieldMystique Installation
      Install the module from the modules directory:
      Via Composer:
      composer require trk/mystique Via git clone:
      cd your-processwire-project-folder/ cd site/modules/ git clone https://github.com/trk/Mystique.git Module in live reaction with your Mystique config file
      This mean if you remove a field from your config file, field will be removed from edit screen. As you see on youtube video.
      Using Mystique with your module or use different configs path, autoload need to be true for modules
      Default configs path is site/templates/configs/, and your config file name need to start with Mystique. and need to end with .php extension.
      Adding custom path not supporting anymore !
      // Add your custom path inside your module class`init` function, didn't tested outside public function init() { $path = __DIR__ . DIRECTORY_SEPARATOR . 'configs' . DIRECTORY_SEPARATOR; Mystique::add($path); } Mystique module will search site/modules/**/configs/Mystique.*.php and site/templates/Mystique.*.php paths for Mystique config files.
      All config files need to return a PHP ARRAY like examples.
      Usage almost same with ProcessWire Inputfield Api, only difference is set and showIf usage like on example.
      <?php namespace ProcessWire; /** * Resource : testing-mystique */ return [ 'title' => __('Testing Mystique'), 'fields' => [ 'text_field' => [ 'label' => __('You can use short named types'), 'description' => __('In file showIf working like example'), 'notes' => __('Also you can use $input->set() method'), 'type' => 'text', 'showIf' => [ 'another_text' => "=''" ], 'set' => [ 'showCount' => InputfieldText::showCountChars, 'maxlength' => 255 ], 'attr' => [ 'attr-foo' => 'bar', 'attr-bar' => 'foo' ] ], 'another_text' => [ 'label' => __('Another text field (default type is text)') ] ] ]; Example:
      site/templates/configs/Mystique.seo-fields.php <?php namespace ProcessWire; /** * Resource : seo-fields */ return [ 'title' => __('Seo fields'), 'fields' => [ 'window_title' => [ 'label' => __('Window title'), 'type' => Mystique::TEXT, // or InputfieldText 'useLanguages' => true, 'attr' => [ 'placeholder' => __('Enter a window title') ] ], 'navigation_title' => [ 'label' => __('Navigation title'), 'type' => Mystique::TEXT, // or InputfieldText 'useLanguages' => true, 'showIf' => [ 'window_title' => "!=''" ], 'attr' => [ 'placeholder' => __('Enter a navigation title') ] ], 'description' => [ 'label' => __('Description for search engines'), 'type' => Mystique::TEXTAREA, 'useLanguages' => true ], 'page_tpye' => [ 'label' => __('Type'), 'type' => Mystique::SELECT, 'options' => [ 'basic' => __('Basic page'), 'gallery' => __('Gallery'), 'blog' => __('Blog') ] ], 'show_on_nav' => [ 'label' => __('Display this page on navigation'), 'type' => Mystique::CHECKBOX ] ] ]; Searching data on Mystique field is limited. Because, Mystique saving data to database in json format. When you make search for Mystique field, operator not important. Operator will be changed with %= operator.
      Search example
      $navigationPages = pages()->find('my_mystique_field.show_on_nav=1'); $navigationPages = pages()->find('my_mystique_field.page_tpye=gallery');
×
×
  • Create New...