Jump to content
adrian

Migrator

Recommended Posts

Great progress! I haven't tried the new features yet (maybe this is covered already) but it might be handy to have it spit out a list of any template files it moves. That would be to remind us to manually move over any include files or other resources those files depend on.

  • Like 1

Share this post


Link to post
Share on other sites

Great idea Steve - I'll do that for sure. 

Done and committed!

Edited by adrian

Share this post


Link to post
Share on other sites

Hi again everyone - I have spent quite a lot of time on this the last few days and it is now much more robust and feature rich.

In an effort to encourage some testing from you guys, I have attached an export of Ryan's blog profile that you can install into an existing PW site.

  1. Download the attached zip file
  2. Grab the latest version of Page Tree Migrator from Github (https://github.com/adrianbj/ProcessPageTreeMigrator) and install.
  3. Make sure your templates directory is writable by the server
  4. Run the module from the your Setup menu
  5. Choose "Import" and continue to Step 2
  6. Choose the Parent Page - choose "Home" which will install the blog as a first level child of your homepage.
  7. Make sure "Everything, including all data pages" is selected for Components to Import.
  8. From Data Source, choose "Zip File Upload", browse to the blog.zip file.
  9. Click "Upload and Create Content" and wait a few seconds.

That's it - you should have a fully functioning blog set up at: Home > Blog

It comes complete with the same test entries, images, gallery, tags, categories etc that Ryan's original one does, although I did add in one extra photo, embedded into an RTE and resized - the photo comes from Joss' Foundation 5 profile.

The one thing you'll notice is that the embedded Youtube link doesn't work, because the Video embedder module isn't installed. Page Tree Migrator will install core modules as needed, but obviously can't install other ones - yet :)

You'll notice that the blog is installed with its CSS and script files in a subdirectory of the templates folder to avoid an overwriting of existing files.

Please test and let me know what you think - Thanks!

blog.zip

  • Like 9

Share this post


Link to post
Share on other sites

I have attached an export of Ryan's blog profile that you can install into an existing PW site.

I had to re-read your last post! This is fantastic.  Thanks.

  • Like 1

Share this post


Link to post
Share on other sites

I've got two notices and a fatal error from your latest version on GitHub:

Notice: Trying to get property of non-object in D:\wamp\www\echofinance\site\modules\ProcessPageTreeMigrator.module on line 879
Notice: Trying to get property of non-object in D:\wamp\www\echofinance\site\modules\ProcessPageTreeMigrator.module on line 1049
Fatal error: Call to a member function getArray() on a non-object in D:\wamp\www\echofinance\site\modules\ProcessPageTreeMigrator.module on line 1052

Share this post


Link to post
Share on other sites

Hey Pete - sorry about that.

Sounds like you are exporting. The issue is that you haven't selected anything in the "Parent Page" or "Components to Export" option. My bad for forgetting to provide a friendly error instead of failing, but if you choose something in that step, you should be fine!

I'll fix it in the next version.

Let me know how you go :)

Share this post


Link to post
Share on other sites

Cutting edge php required? :)

Parse Error: syntax error, unexpected '[' (line 830 of /site/modules/ProcessPageTreeMigrator-master/ProcessPageTreeMigrator.module) 

Share this post


Link to post
Share on other sites

Oops - sorry about that Antti, and for the delayed response - was out enjoying some unseasonable sunshine!

I just pushed a quick fix that should work for older versions of php. I need to set up a 5.3 testing environment.

EDIT: Pete - the export process should now also provide friendly errors if a required option is not selected.

Edited by adrian

Share this post


Link to post
Share on other sites

Spectacular. I just moved a blog tree to another dev site which has saved me a good hour of setup time. Thank you!

  • Like 1

Share this post


Link to post
Share on other sites

Thanks Marty - glad to hear it is working for someone other than me - I was beginning to wonder :)

Also, I just set up php 5.3 testing environment and everything seems to be working there now too, so hopefully I won't be wasting Pete and Antti's time when they try again!

But please keep the bug reports and feature suggestions coming!

Share this post


Link to post
Share on other sites

If it helps my MAMP setup is running PHP 5.4.25

Actually there was a one things I noticed that didn't get packaged up in the zip I exported - and that was an include I used to show my blog categories. It might just be the way I've set it up but I'll put together something for you to look at later.

Share this post


Link to post
Share on other sites

Thanks for the report - I was developing it on 5.5 and have now tested on 5.3, so hopefully we are covered now :)

It makes sense that your include file was not added to the zip. Hopefully you noticed the message after the import to that effect - it mentions that you need to copy any .inc and other supporting files yourself. At the moment the migrator will only add template php files to the zip that are tied to the templates used in the pages that are exported. I have actually contemplated scanning through the template files looking for include, include_once, require, require_once and grabbing those files as well. I'll have a think about it and see if I think it makes sense to do that.

The blog profile zip I attached a few posts back was manually modified to contain all the associated inc, js, and css files, but everything else was direct from the automatic export process.

Share this post


Link to post
Share on other sites

I figured that might be the case as I hadn't logged that inc as a template at all - there was no real need to. I didn't see a message though - or maybe I missed it. I'll take a look next time I try it, which will be soon.

Share this post


Link to post
Share on other sites

Another pretty major update today. Here's the list of changes:

  1. Friendly error handling when modules/fieldtypes aren't installable on the destination site which also prevents any actions until you install the required modules.
  2. Support for migrating page data for all (hopefully) custom fieldtypes. I have tested with cropImage, MapMarker, Dimensions, and Phone. Let me know if you find one that doesn't work. Also, if you are playing around with MapMarker, be sure to make the changes shown in this PR: https://github.com/ryancramerdesign/FieldtypeMapMarker/pull/4/files - seems like it might only affect very recent devs of PW and hopefully Ryan will commit that change soon anyway.
  3. New Import Types: Append, Overwrite, Replace. Explained fully in the module, but the key new functionality is replace, which effectively allows you to edit the content in a section (Page Tree) of a site (say on your dev server), including changes to field settings, which fields are in the template (including repeater fields and their subfields), and the page data in the fields, and all these changes will be replicated on the destination site. This is very new, so let me know if you find something I have missed.

Remember, there is one happy customer already ....

Spectacular. I just moved a blog tree to another dev site which has saved me a good hour of setup time. Thank you!

But I am looking for more :)

If you want a quick demo, don't forget you can try installing Ryan's blog profile as a child of your Home using the zip attached to this post.

  • Like 1

Share this post


Link to post
Share on other sites

@adrian, I've been testing this a bit and getting this after export:

Notice: Trying to get property of non-object in /site/modules/ProcessPageTreeMigrator/ProcessPageTreeMigrator.module on line 1038

Warning: Invalid argument supplied for foreach() in /site/modules/ProcessPageTreeMigrator/ProcessPageTreeMigrator.module on line 1038

Not sure if this affects the import process, though. At least pages get imported properly, but there is something a bit strange there too; see attached screenshot. "Following template files" seems to suggest that something was (supposed to be) migrated there too, but now it looks like somethings missing below that headline?

Earlier this template had repeater field on it and I was seeing quite a bit more errors, so I removed the repeater field in case that was causing it somehow. Errors did actually go away, but only partly -- as you can see, some still remain.

I'm running 2.4 master branch on a local MAMP installation with PHP 5.5.10. You can see exported data here.

post-175-0-30741700-1394870451_thumb.png

  • Like 1

Share this post


Link to post
Share on other sites

Happy yet demanding :) I can't wait to test this update.

This Replace feature has me thinking (which I'll test soon). I know it's probably outside of the scope of what you're doing but how difficult would it be to periodically replace a page tree from one site to another? I'm thinking in terms of a site sharing a blog or news section. I know I could use Pages Web Service to show post lists but I'm thinking of shifting entire posts between sites regularly.

Share this post


Link to post
Share on other sites

Hey teppo,

Thank you for the testing. I can replicate that same export error. I hadn't come across it before, but looking at your export JSON, I noticed that the page field you included has a parent_id of "home". I never anticipated that. The module will actually find all page field parent/child trees and add them to the export, so if for example you had a pagefield that linked to a categories/category tree, then it would grab that info, export it and import it on the new site, even though it is not directly under the parent you chose when exporting.

I think the module actually handles this scenario quite well (despite the error which is an easy fix) - so if you do choose home, it won't try to export the entire home tree, but it will use the children of home on the new site, once you do the import, which is what happened for me when I imported your JSON. Does that make sense?

EDIT: Actually that 1038 error is because your page field doesn't have a "Template of Selectable Pages" set. I have always been testing with page fields with both "Parent of ..." and "Template of..." defined in the Input tab. I'll have to make sure there aren't any further complications here.

The message about importing template files should be hidden in this case, since it is clear you used the copy/paste approach. Obviously template files (*.php) can only be migrated with the "Save" output format. I'll take care of hiding that when it's not appropriate.

Now for the confusing one, the errors with repeater fields. I haven't had any issues with those in quite some time now so I am curious on the exact setup. Would you mind testing one again and send me the errors you're getting and also the JSON, and if they're not being included in the export, perhaps you could sketch out their structure.

One thing I am curious about on the repeater front - what option are you choosing for "Components to Export"? I have been doing almost all my recent testing with "Everything", so I am wondering if that might have something to do with it. I notice that none of your page fields have any content.

Thanks again, really appreciate this!

BTW, I have attached an export of a page tree that shows repeater and page fields working, along with images and RTE embedded images. Be sure to import "Everything" and choose "Replace". It should go into any parent, but I have been testing mostly with Home as the import's root is "About". It will replace your About (if you have one) and also install a US states page tree for the state selector field. There are a couple of repeaters at the bottom of the gallery child page.

files.zip

  • Like 1

Share this post


Link to post
Share on other sites

I've also been exporting "everything", though (like you said) the copy method obviously can't literally export everything.. :)

These are the errors I get in the same situation, with repeater field added. Ready items for this repeater is set to 3 and you can view the final JSON output here.

Notice: Trying to get property of non-object in /site/modules/ProcessPageTreeMigrator/ProcessPageTreeMigrator.module on line 1403

Warning: Invalid argument supplied for foreach() in /site/modules/ProcessPageTreeMigrator/ProcessPageTreeMigrator.module on line 1403

Notice: Trying to get property of non-object in /site/modules/ProcessPageTreeMigrator/ProcessPageTreeMigrator.module on line 1038

Warning: Invalid argument supplied for foreach() in /site/modules/ProcessPageTreeMigrator/ProcessPageTreeMigrator.module on line 1038

Warning: Invalid argument supplied for foreach() in /site/modules/ProcessPageTreeMigrator/ProcessPageTreeMigrator.module on line 1026

Notice: Trying to get property of non-object in /site/modules/ProcessPageTreeMigrator/ProcessPageTreeMigrator.module on line 1403

Warning: Invalid argument supplied for foreach() in /site/modules/ProcessPageTreeMigrator/ProcessPageTreeMigrator.module on line 1403

Notice: Trying to get property of non-object in /site/modules/ProcessPageTreeMigrator/ProcessPageTreeMigrator.module on line 1403

Warning: Invalid argument supplied for foreach() in /site/modules/ProcessPageTreeMigrator/ProcessPageTreeMigrator.module on line 1403

Notice: Trying to get property of non-object in /site/modules/ProcessPageTreeMigrator/ProcessPageTreeMigrator.module on line 1403

Warning: Invalid argument supplied for foreach() in /site/modules/ProcessPageTreeMigrator/ProcessPageTreeMigrator.module on line 1403

Testing your zip file didn't go exactly as planned, though that was probably user error; didn't select "replace", tried to "append" with parent set to "home" and as a result I got "Integrity constraint violation: 1062 Duplicate entry 'about-1' for key 'name_parent_id'".

Slightly weird thing here is that when trying this again with "replace", I got the same error, though this might be because something was broken earlier. Might have to setup another test site to see what's actually going on here -- if this works for you, there's most likely something wrong with my test site.

One specific thing I'm not really comfortable with, though I'm not entirely sure if that's possible to avoid either, is that even though the import fails things like fields and templates are still added and/or kept. This could potentially leave the site in a kind of a strange state, with some old content and some new. It'd be cool if failed import "cleaned up the mess", so to speak :)

Another thing that probably should be handled somehow are possible permission errors:

Warning: copy(/site/templates///basic-page.php): failed to open stream: Permission denied in /site/modules/ProcessPageTreeMigrator/ProcessPageTreeMigrator.module on line 449

Warning: copy(/site/templates///gallery.php): failed to open stream: Permission denied in /site/modules/ProcessPageTreeMigrator/ProcessPageTreeMigrator.module on line 449

The reason for these is obvious and (again) I should've known better, but then again, the module could also check for necessary permissions beforehand -- or perhaps just interrupt the import the moment this happens and notify user to make sure that permissions need to be tweaked?

.. and by the way, those template paths with three forward slashes look a bit strange, but that's far from fatal :)

PS. I actually enjoy testing this module and see a lot of potential in it. As a matter of fact it's already proven it's worth in some of our projects, saving me quite literally hours of work, on more than one occasion. If it sounds like I'm only seeing bad things here, that couldn't be further from the truth :)

  • Like 1

Share this post


Link to post
Share on other sites

Thanks again teppo - awesomely helpful!

So I think I have taken care of everything from your reports:

  1. If there are no template php files in the import, then the message is not displayed to avoid confusion.
  2. Write permissions are now checked on the destination templates directory (if there are template files to be imported), before anything happens and the user is alerted so they can fix and start again.
  3. Also fixed the extra forward slashes :)
  4. I have fixed the line 1403 errors for repeaters - that error only occurred when there were no entries for a repeater on a page which is why I hadn't seen it before. I am not getting the line 1026 error at all, so please let me know if that one still shows up for you.
  5. Fixed the line 1038 error - now the module will work without the page field's Template of Selectable Pages set. Along with this I have also put a limitation on automatically migrating content for page fields - it won't automatically export the entire page tree if a page field's Parent of Selectable Pages is Home (id#1, just in case the name was changed), and also only if the Parent of Selectable Pages is set - so, it won't potentially try to migrate pages from all over the page tree, which might happen if some is using Template, or even worse a Custom Selector or Custom PHP option.

Not sure why you are getting that integrity constraint violation. I know I have broken many a PW install while testing this module, so maybe if you could start with a fresh one and let me know, that would be great. By the way - I tested that files.zip with "Append" and it works as expected without errors, so I do think something is likely amiss at your end, but please let me know what you find.

Actually, I have managed to replicate something similar to this - it's amazing what you can sometimes miss due to the way you are doing something. Anyway, I am on it - it is related to the addition of the new REPLACE option, but I am having trouble tracking it down fully tonight, so it might have to wait another day or until I can find some more time, but I'll post again when that is fixed. In the meantime, so long as you delete the About tree from your install before importing, that zip should work!

Great thought on the issue of a failed import and the cleanup idea. Ideally I'd love for there to be no reason for a failed import, so maybe better checking for potential conflicts before anything happens is the easiest way. Undoing changes / cleanup might be tricky!

Really appreciate hearing that this has saved you lots of time - I would love this to be a "must have" PW module once some more kinks are ironed out.

Thanks again.

Edited by adrian

Share this post


Link to post
Share on other sites

Happy yet demanding :) I can't wait to test this update.

This Replace feature has me thinking (which I'll test soon). I know it's probably outside of the scope of what you're doing but how difficult would it be to periodically replace a page tree from one site to another? I'm thinking in terms of a site sharing a blog or news section. I know I could use Pages Web Service to show post lists but I'm thinking of shifting entire posts between sites regularly.

Hey Marty, anything for you :)

Do you mean automatically or manually on a regular basis? Manually will be no problem at all - that is what this is designed to do (once I get a for more of the REPLACE kinks sorted out). If you want automatic replacement, I am sure we could use a cron job to handle both the export and import - the key thing would be to save the zip on the server, instead of downloading it to local machine. Then get the zip from the destination server and trigger the import. All sounds do-able. That said, if it is just content we are talking about and no fields or templates would change, then I wonder if an import from a web service feed might not be a better approach - this could even bring across new images.

I'll sleep on it!

Share this post


Link to post
Share on other sites

Turns out I really broke some keys things when I added the import actions functionality (for append, overwrite, replace). I need sleep now and won't be able to get to this again for at least a day, maybe two. So, the most stable version right now is probably:

https://github.com/adrianbj/ProcessPageTreeMigrator/blob/019e006833ad970263533516aa7615b8c87af573/ProcessPageTreeMigrator.module

Of course this doesn't include the fixes for the PHP warnings that teppo found, nor the directory writing checks etc.

Anyway, I'll post again when the import actions additions are working properly.

Share this post


Link to post
Share on other sites

Ok, I have fixed another round of bugs, including one weird one getting an "Unknown column 'field_images.modified'" error when importing a page tree into a brand new install before ever having viewed a page on the site. Getting off track, but the issue was due to the need to trigger getDatabaseSchema on the images field which adds the modified and created DB fields. 

The things I've done is add optional support for migrating createdUser/modifiedUser and created/modified datetime for pages. I am using quiet save which is supposed to support createdUser, modifiedUser, and created, but it isn't supporting modifiedUser for me at the moment. I might resort to SQL, or I might decide it's actually best to just change the created versions and leave the modified to change to the user running the import and the current time.

On the user migration - if a user doesn't exist it will be created and assigned its roles from the source installation as defined in the JSON. Also, if the needed roles aren't available, they will be created too.

Let me know how it goes for you!

  • Like 1

Share this post


Link to post
Share on other sites

On a roll - after a chat with Joss, I have added a new option on export that allows you to select any additional required helper files such as .inc, .css, .js, etc. These selected files will be packaged in the zip for the migration.

This option is only displayed if you choose "save" because obviously not relevant to "copy" where it's only the JSON that can be exported.

Note that there is no need to select normal template files as these are still included in the export automatically.

  • Like 1

Share this post


Link to post
Share on other sites

Another pretty major update!

There is now an "Edit Imported Content" option during the import process that lets you determine exactly which fields and pages (from the JSON) get imported.

This should come in especially handy when importing the one of the Shared JSON packages. I have added a new one that is a good test for this, called "person" (maybe needs a more detailed name), that creates a person template with a complete range of contact/address fields. So, on import you can easily choose only the fields you want from the package.

I think this should be really handy for distributing standard templates, but with easy customization for each use case.

Share this post


Link to post
Share on other sites

Adrian, this module sounds more and more useful everytime. Next thing, you will be creating a "recording" mode that collects all changes that happen in PW during a certain period and makes it possible to export them as json to make late changes on published sites really easy! (did that sound like a suggestion?)

  • Like 2

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

  • Recently Browsing   0 members

    No registered users viewing this page.

  • Similar Content

    • By Macrura
      PrevNextTabs Module
      Github: https://github.com/outflux3/PrevNextTabs
      Processwire helper modules for adding page navigation within the editor.
      Overview
      This is a very simple module that adds Previous and Next links inline with the tabs on the page editor. Hovering over the tab shows the title of the previous or next page (using the admin's built in jqueryUI tooltips.)
      Usage
      This module is typically used during development where you or your editors need to traverse through pages for the purpose of proofing, flagging and/or commenting. Rather than returning to the page tree or lister, they can navigate with these links.
      Warnings
      If you are using PW version 2.6.1 or later, the system will prevent you from leaving the page if you have unsaved edits.
      For earlier versions, to avoid accidentally losing changes made to a page that might occur if a user accidentally clicks on one of these, make sure to have the Form Save Reminder module installed.
      http://modules.processwire.com/modules/prev-next-tabs/
    • By Gadgetto
      SnipWire - Snipcart integration for ProcessWire
      Snipcart is a powerful 3rd party, developer-first HTML/JavaScript shopping cart platform. SnipWire is the missing link between Snipcart and the content management framework ProcessWire.
      With SnipWire, you can quickly turn any ProcessWire site into a Snipcart online shop. The SnipWire plugin helps you to get your store up and running in no time. Detailed knowledge of the Snipcart system is not required.
      SnipWire is free and open source licensed under Mozilla Public License 2.0! A lot of work and effort has gone into development. It would be nice if you could donate an amount to support further development:

      Status update links (inside this thread) for SnipWire development
      2020-03-21 -- SnipWire 0.8.5 (beta) released! Improves SnipWires webhooks interface and provides some other fixes and additions 2020-03-03 -- SnipWire 0.8.4 (beta) released! Improves compatibility for Windows based Systems. 2020-03-01 -- SnipWire 0.8.3 (beta) released! The installation and uninstallation process has been heavily revised. 2020-02-08 -- SnipWire 0.8.2 (beta) released! Added a feature to change the cart and catalogue currency by GET, POST or SESSION param 2020-02-03 -- SnipWire 0.8.1 (beta) released! All custom classes moved into their own namespaces. 2020-02-01 -- SnipWire is now available via ProcessWire's module directory! 2020-01-30 -- SnipWire 0.8.0 (beta) first public release! (module just submitted to the PW modules directory) 2020-01-28 -- added Custom Order Fields feature (first SnipWire release version is near!) 2020-01-21 -- Snipcart v3 - when will the new cart system be implemented? 2020-01-19 -- integrated taxes provider finished (+ very flexible shipping taxes handling) 2020-01-14 -- new date range picker, discount editor, order notifiactions, order statuses, and more ... 2019-11-15 -- orders filter, order details, download + resend invoices, refunds 2019-10-18 -- list filters, REST API improvements, new docs platform, and more ... 2019-08-08 -- dashboard interface, currency selector, managing Orders, Customers and Products, Added a WireTabs, refinded caching behavior 2019-06-15 -- taxes provider, shop templates update, multiCURL implementation, and more ... 2019-06-02 -- FieldtypeSnipWireTaxSelector 2019-05-25 -- SnipWire will be free and open source Plugin Key Features
      Fast and simple store setup Full integration of the Snipcart dashboard into the ProcessWire backend (no need to leave the ProcessWire admin area) Browse and manage orders, customers, discounts, abandoned carts, and more Multi currency support Custom order and cart fields Process refunds and send customer notifications from within the ProcessWire backend Process Abandoned Carts + sending messages to customers from within the ProcessWire backend Complete Snipcart webhooks integration (all events are hookable via ProcessWire hooks) Integrated taxes provider (which is more flexible then Snipcart own provider) Useful Links
      SnipWire in PW modules directory SnipWire Docs (please note that the documentation is a work in progress) SnipWire @GitHub (feature requests and suggestions for improvement are welcome - I also accept pull requests) Snipcart Website  
      ---- INITIAL POST FROM 2019-05-25 ----
       
    • By horst
      Croppable Image 3
      for PW 3.0.20+
      Module Version 1.2.0
      Sponsored by http://dreikon.de/, many thanks Timo & Niko!
      You can get it in the modules directory!
      Please refer to the readme on github for instructions.
       
      - + - + - + - + - + - + - + - + - + - NEWS - 2020/03/19 - + - + - + - + - + - + - + - + - + - 
      There is a new Version in the pipe, that supports WebP too: 
       
      - + - + - + - + - + - + - + - + - + - NEWS - 2020/03/19 - + - + - + - + - + - + - + - + - + - 
       
       
      -------------------------------------------------------------------------
       
      Updating from prior versions:
       
      Updating from Croppable Image 3 with versions prior to 1.1.7, please do this as a one time step:
      In the PW Admin, go to side -> modules -> new, use "install via ClassName" and use CroppableImage3 for the Module Class Name. This will update your existing CroppableImage3 module sub directory, even if it is called a new install. After that, the module will be recogniced by the PW updater module, what makes it a lot easier on further updates.
      -------------------------------------------------------------------------
       
      For updating from the legacy Thumbnail / CropImage to CroppableImage3 read on here.
       
      -------------------------------------------------------------------------
       
    • By Robin S
      Inspired by a recent question.
      Image Crop Ratios
      Allows preset aspect ratios to be defined per image field for the ProcessWire image crop tool.
      The module adds a select dropdown to the crop tool. Choose an aspect ratio and the crop area will be fixed to that ratio.
      Screencast

      Installation
      Install the Image Crop Ratios module.
      Configuration
      Default aspect ratios for all image fields can be defined in the module config. Aspect ratios for specific image fields can be defined on the Input tab of the field settings. You can override the ratio settings in template context if needed. Insert a hyphen as the first item in the ratio settings unless you want to force a ratio to be applied to the crop tool. The hyphen represents a blank option that allows a free crop area to be drawn. Usage
      Click the "Crop" link on the details view of an image thumbnail. Click the "Crop" icon at the top of the editor window. Choose an option from the "Ratio" select dropdown.  
      https://github.com/Toutouwai/ImageCropRatios
      https://modules.processwire.com/modules/image-crop-ratios/
×
×
  • Create New...