adrian

Module
Migrator

285 posts in this topic

Run again and find out the id of the assets/files folder where the zip is stored. It should match the id of the Migrator Admin > Setup > Migrator page id, but some reason @NooseLadder is seeing it in one of his normal page id folders. Seems like it's a windows/xampp issue.

Just to clarify, the zip file (files.zip) was correctly created in the /1062/ folder but the error mssg pointed to /1022/ which belongs to another page.

1 person likes this

Share this post


Link to post
Share on other sites

.....................

@kongondo - thanks for offering. You are right, this thread is a behemoth - I will start a new one as soon as I am ready to release, but for now to test:

  1. Grab the module from: https://github.com/adrianbj/ProcessMigrator and install
  2. Make sure debug mode is on
  3. Run the module (Setup > Migrator)
  4. Choose Export
  5. Choose a Parent Page - to keep things simple, maybe something like the default PW's About page
  6. Export
  7. Check that the downloaded zip file contains a data.json file
  8. Let me know if you get any path related errors like @NooseLadder
  9. If you're feeling energetic, comment out line 335: //unlink($zipFilename);
  10. Run again and find out the id of the assets/files folder where the zip is stored. It should match the id of the Migrator Admin > Setup > Migrator page id, but some reason @NooseLadder is seeing it in one of his normal page id folders. Seems like it's a windows/xampp issue.

Thanks so much!

@Adrian,

Does what it says on the tin. No errors and the id of the assets/files folder where the zip is stored matches id of Migrator page..

2 people like this

Share this post


Link to post
Share on other sites
Hi.

First off the work you guys have done with the Wordpress migration tool is amazing, unfortunalety I am not able to get it to work :(

I am running PW 2.4, I have tried to modules both in a WAMP environment and ISP provided environments, both in brand new PW installations and current ones.

I have loaded the blog.zip profile first, this works fine.

Then when I try to import the XML from wordpress (3.9.1) all I am getting is a message saying.

"Missing required ZIP or JSON source"

There is a possibility that I have missed something vital.

This are the basic steps that I take.

Install PW

Install module ProcessMigration

Install module MigratorWordpress

Migrating the blog.zip profile to /home/

Trying to migrate the wordpress.xml to /home/blog (root of blog recently created with blog.zip)

Any Ideas ?

Would really like to get this one working.

Thanks Ronnie

Share this post


Link to post
Share on other sites

@Ronnie,

Sorry about that - it was a simple fix - please try the latest version on Github and let me know how it goes for you. You might be still be affected by an outstanding multi-language issue but hopefully I can have the fixed soon.

@tobaco

Still have some issues with the language error and also trying to improve the Replace option - need to sort out recursive deletion of existing pages to work better, but I have fixed a bunch of issues around reporting/automatic installation and proper configuring of Inputfield types that are needed by the imported fields - for example, the icon select field now properly detects the need for installing InputfieldSelectMultipleTransfer and then uses it once installed. This should work for all input field types now.

Share this post


Link to post
Share on other sites

@Ronnie,

I should also mention that you do not need to import the blog.zip file. You can use Kongondo's Blog module (http://modules.processwire.com/modules/process-blog/) as the base, or just import your WP xml and manually create the templates to suit your needs.

If you decide to use the Blog module, here is a screenshot of the MigratorWordpress config settings that are required:

post-985-0-90394500-1404404097_thumb.png

Share this post


Link to post
Share on other sites

@tobaco - please try the latest version - I have made a lot of fixes and I now get a fully(?) working import of your json file!

Firstly, please run the export on your original site again first as I made a couple of fixes there. Then test on a clean PW install and use the REPLACE option so that your import will completely replace the existing pages from the default PW profile.

Please let me know if all the content appears to be populated as you expect - there are lots of fields with empty values in the json export, but I am not sure if that is correct or not, so I'll need your input on that.

There are still a couple of issues:

  1. Using the replace method a second time results in out of memory errors - I think it's the recursive deletion of the content from that repeater field
  2. I have temporarily disabled importing content for image fields inside repeaters as that is causing errors with your import. I am pretty sure images in repeaters worked fine in my previous testing, so I am not sure where that problem is yet.

Thanks again!

PS The latest version handles installation of all required language modules and gracefully warns of missing language packs and lets you install them as needed.

PPS - if you are doing lots of testing and need to be able to revert to a fresh install regularly, it pays to fire up PHPMyAdmin and create a clean copy of the DB after you have installed PW (or perhaps after you have also installed all your required modules) so that you can simply drop the actual DB and create a new copy from the backed up copy. You might be already aware of this approach, but it's a huge timesaver when testing things like this.

1 person likes this

Share this post


Link to post
Share on other sites

@Adrian

Thanks for the reply, unfortunately I am still experiencing the same issue.

I have used the latest versions from GitHub that you suggested.

When I turned on debug mode on my PW installation I see something that might be interesting.

I see some references to:
C:/Bitnami/processwire-2.4.0-3/apps/processwire/htdocs/site/assets/files/1043/migratorfiles

and

C:/Bitnami/processwire-2.4.0-3/apps/processwire/htdocs/site/assets/files/1043/migratorfiles\data.json

Attached file: 

post-1232-0-46162200-1404732759_thumb.jp

The only thing that is present in that folder is the WP xml file.

C:\Bitnami\processwire-2.4.0-3\apps\processwire\htdocs\site\assets\files\1043\neossida.xml

No migratorfiles folder / data.json file

Attached file:

post-1232-0-95093300-1404732759_thumb.jp

This file / folder is not present on my machine, I am not completely sure how this works but I quess that the plugin should transform the WP xml to a JSON file that is supposed to be used in the migration.

Maybe that would point you in some direction ? 

Thanks / Ronnie

post-1232-0-46162200-1404732759_thumb.jp

post-1232-0-95093300-1404732759_thumb.jp

Share this post


Link to post
Share on other sites

Sorry Ronnie,

I have tested more thoroughly this time and the latest version of Migrator should fix the issue.

Thanks again for letting me know how it goes.

Share this post


Link to post
Share on other sites

@Adrian.

Migrating WP sites like a boss now :)

Thanks for the fix, that seemed to do the trick, I have not tried it thoroughly yet, but I got past the previously issue and content is being created.

4 people like this

Share this post


Link to post
Share on other sites

Great to hear :)

Now I just need to hear from @tobaco that everything is working for him with him and I'll be a happy camper - literally :) - I am heading camping for the rest of the week, but I'll follow up with any other issues here when I get back.

1 person likes this

Share this post


Link to post
Share on other sites

Hey Adrian,

I've got a merging of two PWs to deal with and I was wondering if this was possible with Migrator.

PW Site One essentially contains client information (business data - names, addresses, descriptions etc)

PW Site Two contains invoicing information - a site developed to track user billing and payment for the businesses in PW Site One. 

What I would like is to merge the two sites so the Site Two pages would become a branch of the Site One Tree, eg /invoicing/siteTwo

However, it looks like Migrator is developed to migrate dev to live etc, so presumably page ids would be the same etc. What happens if a data page from Site Two has the same id as a invoicing page from Site One for example. Is this magically 'handled' or would a overwrite occur?

Share this post


Link to post
Share on other sites

muzzer,

Your scenario is exactly what Migrator was designed to do. It does not migrate page IDs at all. Every reference to an id gets converted to a name/path on export and a new page id created on import to the new site.

I would still recommend testing the migration to a test PW install first until you have a full understanding of all the import settings and how they can affect the import. Also when you do it to the live site, please make sure to backup your database first. This module generally works as advertised, but there are some weird scenarios like what Macrura experienced with a page field that points to users that can cause problems. I will be fixing this shortly - I just need some more info from him.

Please let me know how it goes for you and if you think of any possible improvements in the way it works.

2 people like this

Share this post


Link to post
Share on other sites

Super cool new feature just added - well at least I think it's cool :)

You can now automatically backup your entire site's database and templates directory during the import process. Backups are archived so you can go back to any previously created version whenever you want.

If something goes wrong during the import phase of the migration, or you simply imported the wrong json file, or you changed your mind, you can restore everything with the click of a button!

This database part of this functionality relies on Processwire's brand new WireDatabaseBackup class that Ryan committed today, so to use this functionality, go grab the latest dev version of PW and the latest version of this module.

Please let me know how it goes for you!

post-985-0-07817100-1408509206_thumb.png

post-985-0-78763500-1408509237_thumb.png

post-985-0-67320700-1408509271_thumb.png

15 people like this

Share this post


Link to post
Share on other sites

Adrian, do you ever sleep? :P  This is awesome man! Quick question, maybe I missed it, maybe this (especially with the new additions) should now be a superadmin 's only toy? Just wondering....

Share this post


Link to post
Share on other sites

Adrian, do you ever sleep? :P  This is awesome man! Quick question, maybe I missed it, maybe this (especially with the new additions) should now be a superadmin 's only toy? Just wondering....

The backup/restore stuff was easy thanks to Ryan :)

I haven't given the module a permission setting so I think that limits it to superuser accounts. Are you seeing otherwise?

Something a little weird I just noticed - if you try to enter the url to a Process module, eg: 

http://pwtest.dev/admin/setup/migrator/

when logged in as a user without the appropriate permission, you get this error: TemplateFile: Unrecognized path - maybe that's better than a meaningful error saying you don't have permission or something?

Anyway, I think this is locked down sufficiently to superusers?

Share this post


Link to post
Share on other sites

@adrian How's further development going on ProcessMigrator?  I was playing around with it yesterday and it seems it's still quite buggy.  I tested it about 20 times on fresh installs and got a few errors which I can discuss with you.

I would like to feature it in a WordPress vs. ProcessWire screencast.  Your plugin solves a huge pain point.

Let me know if I can help.

1 person likes this

Share this post


Link to post
Share on other sites

Hi @jlahijani,

I haven't done much on it for a while now.

I did think that most bugs were sorted by now though. The biggest remaining issue (I thought) was page fields and how to handle them if their content pages are in a page tree that is not part of the exported parent. At the moment they are exported and recreated on the destination site as a child of home, which obviously may not be ideal, so I think I need to add some additional logic to recreate the entire tree above the page field parent (if it doesn't exist on the destination site), although I can see this being problematic too. Since you've been testing lots, I'd love any thoughts you might have on the best way to handle this.

Also, please send through all the details of the errors you got while testing - it will be a huge help I am sure.

As for Wordpress vs Processwire - are you wanting to feature Migrator's use as an importer of WP content (using Nico's plugin), or just its ability for migration of PW content from one site to another? If the former, then I think that functionality should be pretty robust.

Anyway, please let me know what you have uncovered - it would be great to get this to the next level!

Share this post


Link to post
Share on other sites

Good to hear.  I will put together some info about the bugs.

I'm planning on featuring the ProcessMigrator plugin as a general exporter/importer of content from one PW site to another.  For example, let's say you have a development and production version of a site, and a new set of content needs to be created (on dev), then eventually launched (on prod).  Rather than re-creating that content manually on production, I want to show how your plugin can achieve that.

WordPress has an export tool, however it's very limited and doesn't work that well for the situation I described.  There are some plugins to enhance it but I haven't had great success with those either.  There's one commercial plugin which I haven't used but seems to be the ultimate solution to do what I described: https://crowdfavorite.com/ramp/

ProcessMigrator, in my eyes, would be the analog to RAMP.

1 person likes this

Share this post


Link to post
Share on other sites

One thing that would be very helpful is if you can provide the exported JSON (or full zip package if that is relevant) that is causing issues when you import - this will make it much easier to debug.

Thanks again!

Share this post


Link to post
Share on other sites

I just want to drop a note here to publicly thank jlahijani and muzzer for lots of recent help getting several more bugs identified and fixed and at the same time mention that you should all grab the most recent version on Github (https://github.com/adrianbj/ProcessMigrator).

Thanks again guys!

5 people like this

Share this post


Link to post
Share on other sites

WordPress has an export tool, however it's very limited and doesn't work that well for the situation I described.  There are some plugins to enhance it but I haven't had great success with those either.  There's one commercial plugin which I haven't used but seems to be the ultimate solution to do what I described: https://crowdfavorite.com/ramp/

 

ProcessMigrator, in my eyes, would be the analog to RAMP.

jlahijani - I finally had a look at the video for RAMP - it looks very cool, especially the automatic pushing from dev to production. Other than that, the big difference I noticed compared to Migrator is that currently Migrator can only export one page tree at a time and it has to be the entire tree of children below it. Sure you have the option to determine which pages get imported to the other site, but I think it would be a great enhancement to offer this selection during export as well. I am thinking the option to choose multiple pages and for each one (if it has children), determine whether to include the children or not. I think this approach would make it much easier to migrate a few newly developed pages, or several new sections, without the need to overwrite everything under an overarching  parent page. You don't actually have to do this with the existing Edit Imported Content option, but it is not as obvious or easy.

Anyway, it has given me some great ideas - thanks! 

1 person likes this

Share this post


Link to post
Share on other sites

Some more bug fixes - thanks again to muzzer! 

More interestingly though, during import you now have the option to automatically download and install any required missing fieldtypes/inputfields from the PW modules directory.

This is optional and I have noted that if you don't trust the source of the data you are importing, then you might want not want to enable this option.

It is off by default, in which case you'll receive notifications of what's missing so you can install manually before re-starting the import.

5 people like this

Share this post


Link to post
Share on other sites

Hi, just thought I would have a go with this. I got this:

Error: Call to a member function size() on a non-object (line 2504 of /home/foodloversdiary/public_html/site/modules/ProcessMigrator/ProcessMigrator.module) 

So, it seems to have imported the categories and a load of tags, but only 5 posts and they have no content.

Any thoughts?

Edit: forgot to mention that it was with a Wordpress import

Share this post


Link to post
Share on other sites

Hey Joss,

Thanks for testing this out. Looks like you have come across an issue with a very recently added function to scan through RTE fields and automatically generate resized versions of images that have been embedded.

Actually I am wondering - is the WP still still online and working? It needs to be so that the module can download the images from the site and add them to each of the pages in PW.

Any chance you could PM me the WP xml file so I can test?

I am heading out for the rest of the day, but I will look into it in the morning for you - I am sure it will be an easy fix.

PS, Please make sure you are always using the latest version of Migrator and also MigratorWordpress (my fork - Nico's is not up to date)

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!


Register a new account

Sign in

Already have an account? Sign in here.


Sign In Now

  • Recently Browsing   0 members

    No registered users viewing this page.

  • Similar Content

    • By ethfun
      I've created a Process Module, and its execute() function renders the HTML response to an AJAX request. The AJAX request is being made outside of the admin panel, and in all likelihood will be made by "guest" users.
      Within my ModuleName.info.json file, I have added these lines:
      "permission": "generate-thumbnails", "permissions": { "generate-thumbnails": "Generate Thumbnails" }, Then within the Admin panel, I gave guests access to run that module.
      Unless logged in, the response to the AJAX request is always the login form's HTML, rather than the HTML my execute() function creates.
      So two questions:
      Is it possible to give non-logged-in users the ability to run that function via the AJAX request, or Is there a better ProcessWire way to create HTML to use as an AJAX response? Thank you!
    • By Robin S
      I have had this module sitting in a 95% complete state for a while now and have finally made the push to get it out there. Thanks to @teppo for his Hanna Code Helper module which I referred to and borrowed from during development.
      http://modules.processwire.com/modules/hanna-code-dialog/
      https://github.com/Toutouwai/HannaCodeDialog
      HannaCodeDialog
      A module for ProcessWire CMS/CMF. Provides a number of enhancements for working with Hanna Code tags in CKEditor. The main enhancement is that Hanna tags in a CKEditor field may be double-clicked to edit their attributes using core ProcessWire inputfields in a modal dialog.
      Requires the Hanna Code module.
      Installation
      Install the HannaCodeDialog module using any of the normal methods.
      For any CKEditor field where you want the "Insert Hanna tag" dropdown menu to appear in the CKEditor toolbar, visit the field settings and add "HannaDropdown" to the "CKEditor Toolbar" settings field.
      Module configuration
      Visit the module configuration screen to set any of the following:
      Exclude prefix: Hanna tags named with this prefix will not appear in the CKEditor toolbar dropdown menu for Hanna tag insertion. Exclude Hanna tags: Hanna tags selected here will not appear in the CKEditor toolbar dropdown menu for Hanna tag insertion. Background colour of tag widgets: you can customise the background colour used for Hanna tags in CKEditor if you like. Dialog width: in pixels Dialog height: in pixels Features
      Insert tag from toolbar dropdown menu
      Place the cursor in the CKEditor window where you want to insert your Hanna tag, then select the tag from the "Insert Hanna tag" dropdown.

      Edit tag attributes in modal dialog
      Insert a tag using the dropdown or double-click an existing tag in the CKEditor window to edit the tag attributes in a modal dialog.

      Tags are widgets
      Hanna tags that have been inserted in a CKEditor window are "widgets" - they have a background colour for easy identification, are protected from accidental editing, and can be moved within the text by drag-and-drop.

      Options for tag attributes may be defined
      You can define options for a tag attribute so that editors must choose an option rather than type text. This is useful for when only certain strings are valid for an attribute and also has the benefit of avoiding typos.
      Add a new attribute for the Hanna tag, named the same as the existing attribute you want to add options for, followed by "__options". The options themselves are defined as a string, using a pipe character as a delimiter between options. Example for an existing attribute named "vegetables":
      vegetables__options=Spinach|Pumpkin|Celery|Tomato|Brussels Sprout|Potato You can define a default for an attribute as normal. Use a pipe delimiter if defining multiple options as the default, for example:
      vegetables=Tomato|Potato Dynamic options
      Besides defining static options as above, you can use one Hanna tag to dynamically generate options for another. For instance, you could create a Hanna tag that generates options based on images that have been uploaded to the page, or the titles of children of the page.
      Your Hanna tag that generates the options should echo a string of options delimited by pipe characters (i.e. the same format as a static options string).
      You will probably want to name the Hanna tag that generates the options so that it starts with an underscore (or whatever prefix you have configured as the "exclude" prefix in the module config), to avoid it appearing as an insertable tag in the HannaCodeDialog dropdown menu.
      Example for an existing attribute named "image":
      image__options=[[_images_on_page]] And the code for the _images_on_page tag:
      <?php $image_names = array(); $image_fields = $page->fields->find('type=FieldtypeImage')->explode('name'); foreach($image_fields as $image_field) { $image_names = array_unique( array_merge($image_names, $page->$image_field->explode('name') ) ); } echo implode('|', $image_names); Choice of inputfield for attribute
      For attributes which you have defined options for, you can choose the inputfield that is used to select from the options.
      Inputfields that support the selection of a single option are select (this is the default inputfield for attributes with options so it isn't necessary to specify it if you want it) and radios.
      Inputfields that support the selection of multiple options are selectmultiple, asmselect and checkboxes.
      You can also specify a checkbox inputfield - this is not for attributes with defined options but will limit an attribute to an integer value of 1 or 0.
      The names of the inputfield types are case-insensitive.
      Example for an existing attribute named "vegetables":
      vegetables__type=asmselect Descriptions and notes for inputfields
      You can add a description or notes to an attribute and these will be displayed in the dialog.
      Example for an existing attribute named "vegetables":
      vegetables__description=Please select vegetables for your soup. vegetables__notes=Pumpkin and celery is a delicious combination. Notes
      When creating or editing a Hanna tag you can view a basic cheatsheet outlining the HannaCodeDialog features relating to attributes below the "Attributes" config inputfield.
      Troubleshooting
      HannaCodeDialog includes and automatically loads the third-party CKEditor plugins Line Utilities and Widget. If you have added these plugins to your CKEditor field already for some purpose and experience problems with HannaCodeDialog try deactivating those plugins from the CKEditor field settings.
    • By gRegor
      Updated 2017-03-27:
      Version 1.1.3 released
      Updated 2016-04-11:
      Version 1.1.2 released

      Updated 2016-02-26:
      Officially in the module directory! http://modules.processwire.com/modules/webmention/

      Updated 2016-02-25:
      Version 1.1.0 is now released. It's been submitted to the module directory so should appear there soon. In the meantime, it's available on GitHub: https://github.com/gRegorLove/ProcessWire-Webmention. Please refer to the updated README there and let me know if you have any questions!
      ------------
      Original post:
       
      This is now out of date. I recommend reading the official README.
       
      I've been working on this one for a while. It's not 100%, but it is to the point I'm using it on my own site, so it's time for me to release it in beta. Once I finish up some of the features described below, I will submit it to the modules directory as a stable plugin.
      For now, you can install from Github. It works on PW2.5. I haven't tested on PW2.6, but it should work there.
      Feedback and questions are welcome. I'm in the IRC channel #processwire as well as #indiewebcamp if you have any questions about this module, webmention, or microformats.
      Thanks to Ryan for the Comments Fieldtype which helped me a lot in the handling of webmentions in the admin area.
      ProcessWire Webmention Module
      Webmention is a simple way to automatically notify any URL when you link to it on your site. From the receiver's perspective, it is a way to request notification when other sites link to it.
      Version 1.0.0 is a stable beta that covers webmention sending, receiving, parsing, and display. An easy admin interface for received webmentions is under development, as well as support for the Webmention Vouch extension.
      Features
      * Webmention endpoint discovery
      * Automatically send webmentions asynchronously * Automatically receive webmentions * Process webmentions to extract microformats   Requirements * php-mf2 and php-mf2-cleaner libraries; bundled with this package and may optionally be updated using Composer. * This module hooks into the LazyCron module.   Installation Github: https://github.com/gRegorLove/ProcessWire-Webmention

      Installing the core module named "Webmention" will automatically install the Fieldtype and Inputfield modules included in this package.   This module will attempt to add a template and page named "Webmention Endpoint" if the template does not exist already. The default location of this endpoint is http://example.com/webmention-endpoint   After installing the module, create a new field of type "Webmentions" and add it to the template(s) you want to be able to support webmentions. Sending Webmentions
      When creating or editing a page that has the Webmentions field, a checkbox "Send Webmentions" will appear at the bottom. Check this box and any URLs linked in the page body will be queued up for sending webmentions. Note: you should only check the "Send Webmentions" box if the page status is "published."   Receiving Webmentions This module enables receiving webmentions on any pages that have have "Webmentions" field, by adding the webmention endpoint as an HTTP Link header. If you would like to specify a custom webmention endpoint URL, you can do so in the admin area, Modules > Webmention.   Processing Webmentions (beta) Currently no webmentions are automatically processed. You will need to browse to the page in the backend, click "Edit," and scroll to the Webmentions field. There is a dropdown for "Visibility" and "Action" beside each webmention. Select "Process" to parse the webmention for microformats.   A better interface for viewing/processing all received webmentions in one place is under development.   Displaying Webmentions (beta) Within your template file, you can use `$page->Webmentions->render()` [where "Webmentions" is the name you used creating the field] to display a list of approved webmentions. As with the Comments Fieldtype, you can also generate your own output.   The display functionality is also under development.   Logs This module writes two logs: webmentions-sent and webmentions-received.   Vouch The Vouch anti-spam extension is still under development.   IndieWeb The IndieWeb movement is about owning your data. It encourages you to create and publish on your own site and optionally syndicate to third-party sites. Webmention is one of the core building blocks of this movement.   Learn more and get involved by visiting http://indiewebcamp.com.   Further Reading * http://indiewebcamp.com/webmention * http://indiewebcamp.com/comments-presentation * http://indiewebcamp.com/reply
    • By kixe
      This morning I pushed a module to github which extends the page api with moving and sorting capability.
      https://github.com/kixe/PageSortMove
      /** * CALLABLE * PageSortMove::execute($pageID, $newIndex = 0, $parentID = null); * $modules->get('PageSortMove')->execute($pageID, $newIndex = 0, $parentID = null); * * EXTENDED PAGE API * @property $page->first // under same parent * @property $page->last * * @method $page->first($parentID = null) * @method $page->last($parentID = null) * @method $page->sort($newIndex) * @method $page->move($newIndex, $parentID = null) */ Have a nice weekend.