Jump to content
adrian

Migrator

Recommended Posts

Ahh, that makes total sense - I am sure it is because all your fields become objects because of the two language versions eg:

"body": {
                    "data": "",
                    "data1029": ""
                },

rather than just simple strings.

Not sure how difficult this will be to fix - hopefully simple :)

  • Like 1

Share this post


Link to post
Share on other sites

Yep, that is. Probably some type sniffing and in case of object you can safely assume it will be multilang field. I have done much with those in API level, but I know soma is the man with those (like many others). Probably pretty simple, I think the number is language id, and without number it is default lang.

Share this post


Link to post
Share on other sites

Hi Adrian,

With your latest version I just get blank pages on my PW 2.3.0 install when I try to export anything. My server PHP version is 5.3.25. Perhaps I need a recent dev version?

There's something up with my install there I think. Tried it locally and it works perfectly. This will be a huge time saver Mr Jones. Thank you.

  • Like 1

Share this post


Link to post
Share on other sites

That's great news Marty :)

Unfortunately I haven't managed to make any time to work on this today and I am away for the next couple of days, but I hope next week to solve the multilanguage issue that Antti discovered - I have a plan and should be quite simple so long as the receiving PW installation has the needed languages installed - I should be able to issue a warning if it doesn't.

I also have some ideas on how to make migration of images and files work too and maybe even RTE embedded images. Not sure how important migration of this sort of content is though - maybe it is adding too much complexity. Do you have any thoughts on number 3 of the road map above: http://processwire.com/talk/topic/4420-page-list-migrator/?p=43582

BTW, I have spent many days in your part of the world when I was a keen climber - beautiful place!

Share this post


Link to post
Share on other sites

Awesome job Adrian! I can't wait to use this one. I'd be using it right now if I weren't holding a baby in one arm and typing with the other. :) But I did browse the code and thought it looked great. 

  • Like 1

Share this post


Link to post
Share on other sites

Thanks Ryan,

I hope I can make some time in the next couple of days to get a few more things sorted out with this. I am excited to get this release worthy!

  • Like 1

Share this post


Link to post
Share on other sites

Hey Antti,

I have just committed an update that I think fixes the multi-language issues. I know there are two different ways to do different language versions of a field, but from my understanding, this should work with both.

Would you mind checking on your end and see if the import is now working for you.

Thanks!

Share this post


Link to post
Share on other sites

Currently it gives me an error:

Parse error: syntax error, unexpected '[' in /.../site/modules/ProcessPageTreeMigrator/ProcessPageTreeMigrator.module on line 666 

And the code there is:

$newArray = [];

So it requires php 5.4. Just changed it to array() and now it works.

Importing full data still doesn't work, this time it seems to be repeaters:

FieldtypeRepeater: Created Repeater Page Parent: /processwire/repeaters/for-field-98/for-page-1067/

TemplateFile: Method RepeaterPageArray::setLanguageValue does not exist or is not callable in this context

For me export & import of templates and fields is the main selling point of this module (not the data).

  • Like 1

Share this post


Link to post
Share on other sites

Ah, sorry - been on 5.4 for so long I forgot about this.

Yes, repeaters aren't supported yet. For the moment I'll add that to the list of fieldtypes to ignore so at least there won't be an error.

Btw, the last update also now handles RTE links that have been processed by the PageLinkAbstractor module.

EDIT: Committed version with those small fixes. NB that the check to ignore repeaters is currently only on the export, so you would need to export again to check.

  • Like 3

Share this post


Link to post
Share on other sites

A small update - I have added support for import options so now you can decide whether you want to import just the fields and templates, or the entire page tree at the import stage, regardless of what is in the exported JSON file. Of course you can't import the page tree if it wasn't exported, but you can import just the fields and templates from a JSON file that contains the entire page tree.

I have also added support for importing page trees directly from the repo at: https://github.com/adrianbj/ProcessWirePageLists Just choose the "Shared JSON Packages" option when importing. 

I haven't tackled the repeater field etc issues yet, hopefully soon.

Will also be adding more shared packages soon and would love any contributions :)

See the attached screenshot showing the direct page tree import using Ryan's awesome new admin theme :)

post-985-0-22275500-1380836087_thumb.png

  • Like 5

Share this post


Link to post
Share on other sites

Hey adrian, this is great work, thanks a lot for the effort. I tested it with small chunks of datalists, selection values and so on, works like a charm.

On one specific page with FieldSetTabs I get this, after hitting the Export button in Step 2:

Error: Cannot break/continue 1 level (line 641 of /pwtesting/site/modules/ProcessPageTreeMigrator/ProcessPageTreeMigrator.module) 

This chunk seems to be responsible:

	if($type == 'fields'){

	    if($page->type instanceof FieldtypeFieldsetOpen) continue;

	    $data = array(

	        'name' => $page->name,
	        'label' => $page->label,
	        'description' => $page->description,
	        'template' => $template_name,
	        'flags' => $page->flags,
	        'type' => "{$page->type}",

	    );

	}

Also this might not be relevant yet, but on a page on which I use soma's new alpha module for ContentBlocks after exporting I get this:

Error: Call to a member function get() on a non-object (line 734 of /pwtesting/site/modules/ProcessPageTreeMigrator/ProcessPageTreeMigrator.module) 

I'm on PW 2.3.3.3

Perhaps it would be useful for users to see in the github readme what's already working and what not working yet. Perhaps a little roadmap as well.

Thanks again, this will be a huge timesaver and the more modular approach for migrating little logical/data chunks rather than whole profiles will come in very handy when developing locally and on server parallely.

Share this post


Link to post
Share on other sites

Thanks ozwim for the testing and bug reports. I'll see what I can do about getting these sorted out. I haven't tested with fieldsets at all yet. As for the error on line 734 - it is a language module related thing. I had seen that once before - should be easy enough to sort out.

Good idea for adding details to the github readme on what's working etc - I have a feeling it will be an increasing list for a while, before it starts getting smaller, but I am sure we'll get there :)

  • Like 1

Share this post


Link to post
Share on other sites

Ok, I have pushed another version to Github. v0.0.8 fixes the fieldsettab issue that Ozwim reported and introduces support for migrating repeater fields. It handles the creation of everything required on the source PW installation for the new repeater fields to work.

Next on the roadmap is full support for page fields - currently only the page field itself is migrated, but not the selectable pages fields and templates. Then it is on to migrating the page content for repeater and page fields

I would really appreciate any help testing out this new version. 

EDIT: Just updated the readme on github to detail what does and doesn't work, along with a rough roadmap - please let me know if your testing shows that I have any working/not working mistakes - I am sure there are some :)

  • Like 1

Share this post


Link to post
Share on other sites

Ok another update this morning - I have added support for full migration of page fields - it now automatically handles creation of the selectable pages fields, templates, and the pages themselves.

The one thing I haven't really figured out how to tackle is what parent to put these pages under. At the moment they end up under the same parent page that you choose when starting an import. This is probably not ideal, but otherwise it could result in the need to create a whole other lot of parent parents/templates/fields that you don't really need on the destination PW site. This way everything needed gets migrated but nothing more and you can easily move them into a new parent (like a tools parent) if needed. Anyone have any better ideas? You might need to run a test to see what I mean.

Again, this will probably need lots of testing so thanks in advance - there is a lot going on to make this all work!

Share this post


Link to post
Share on other sites

Another update - things are getting closer!

I have just added support for migration of the selected values of page fields when migrating "everything, including all data pages".

Only repeater field page data (the fields, templates etc are already working) to go, and then of course the inevitable bugs :)

EDIT:

Just found a bug in the page field data migration - hopefully solve this soon.

Repeater data is working now though - not on github yet, but I will push that too once I have the page field issue sorted out.

Edited by adrian
  • Like 1

Share this post


Link to post
Share on other sites
Getting stoked :)

The latest version on github now supports migration of repeater field page content and fixes the problems with migrating page field content.

That should take care of migrating the fields, templates, and page content for all the field types, not including page content for file and image fields as obviously these can't, or more accurately, shouldn't be included in a json file.

Multi-language fields should migrate perfectly so long as you have the required language support modules and languages installed on the destination PW installation. Let me know if you have any troubles here - I did see some weird errors during early testing - I hope they are all fixed now.

Please note: that most of my testing of late has been with the "Everything, including all data pages" option on export and import. I need to go back and test the other options soon, but I'll probably do that when I start implementing the series of checkboxes allowing you to select exactly which fields, templates, and pages to be exported and imported.

Also, I have been testing with the latest dev version of PW (downloaded today). Not sure if anything is critical, but if you are getting errors, please try upgrading PW first.

  • Like 3

Share this post


Link to post
Share on other sites

Sorry for yet another post, but I just fixed a bug that prevented certain fields being exported and assigned to templates in some situations. Not the most elegant solution (repetitive declaration in the JSON file), but it should work for the moment until I have time to rework slightly.

Grab the latest version from Github for testing :)

Share this post


Link to post
Share on other sites

Adrian, thanks for your work on this. I am super busy at the moment, but I have great plans for this (reusing common templates etc..). Will definitely give this good ride soon.

  • Like 1

Share this post


Link to post
Share on other sites

Just pushed a minor fix to prevent 403 forbidden errors when hitting the github page lists repository when starting the import process. Apparently they were forcing a user-agent header for several months now, but didn't seem to affect this module till recently. Anyway, should be fixed now.

Sometime early in the new year I will sort out all the php notices/warnings in this module and add some more features for finer detailed selection of what you want to export/import.

  • Like 1

Share this post


Link to post
Share on other sites

Just committed a fairly major update to support full migration of files and images, including rewriting of the src tag to match the ID of the new page for images embedded into RTE fields. Additionally, the module now also migrates the template .php files.

Files/images/template files and the json structure/data file are exported in a zip file which is then imported into the destination PW install.

So, you can now select one parent page during export and it will migrate the following content for all child pages of the selected parent:

  • All standard field types, including RTE, and decoding of links modified by the PageLinkAbstractor module and abstracting again on the destination PW install.
  • File/Image/CropImage fields including the actual files/images/thumbnails and all other variations
  • Repeater fields and all their required fields, templates, and all content, including files/images
  • Page fields (and the pages, templates, and fields that make up their selectable pages)
  • Multi-language versions of all field content
  • Templates (including Access, Family, URL and other settings) and the template .php files. It even grabs the appropriate file if you are using the "Alternate Template Filename" setting. NB the templates directory on the destination PW installation must be writable for these to be imported.
 
So, you could build sections of content on a local dev PW installation, export it, and then with a couple of clicks import everything into the live PW installation.
 
Some outstanding issues that I hope to get to shortly:
  • Need to support images inserted from a different page into an RTE field
  • Rewrite any references to page ids, eg $pages->get(xxxx) in template .php files so they will be converted to the correct id on the destination installation.
  • Need to look into the new core link abstractor that was added to PW 2.4 and see how to handle those links compared to the PageLinkAbstractor module.
  • I still want to add finer control for determining exactly what components will be exported and imported.
  • Generally needs lots more error checking for things like making sure required languages are installed on the destination PW install etc
  • Need to add checks so that existing template php files are not overwritten (or give the option to choose)
  • Might need to override PHP max_execution_time and other settings for larger exports and maybe chunk out zipping of all images to prevent memory issues on larger exports.

In its current state it is handling almost all my needs, but I would really like to have this robust enough for anyone to use, so if you have time please do some testing for me - THANKS!

At the moment, please only use it on two unimportant test PW installs!

Edited by adrian
  • Like 10

Share this post


Link to post
Share on other sites

Several of the outstanding issues and a couple of bug fixes have been taken care of this morning. Be sure to grab the latest from Github  :)

EDIT: Even more issues taken care of this afternoon. Updated in post #47 above.

Edited by adrian

Share this post


Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.


  • Recently Browsing   0 members

    No registered users viewing this page.

  • Similar Content

    • By Gadgetto
      SnipWire - Snipcart integration for ProcessWire
      Snipcart is a powerful 3rd party, developer-first HTML/JavaScript shopping cart platform. SnipWire is the missing link between Snipcart and the content management framework ProcessWire.
      With SnipWire, you can quickly turn any ProcessWire site into a Snipcart online shop. The SnipWire plugin helps you to get your store up and running in no time. Detailed knowledge of the Snipcart system is not required.
      SnipWire is free and open source licensed under Mozilla Public License 2.0! A lot of work and effort has gone into development. It would be nice if you could donate an amount to support further development:

      Status update links (inside this thread) for SnipWire development
      2020-01-21 -- Snipcart v3 - when will the new cart system be implemented? 2020-01-19 -- integrated taxes provider finished (+ very flexible shipping taxes handling) 2020-01-14 -- new date range picker, discount editor, order notifiactions, order statuses, and more ... 2019-11-15 -- orders filter, order details, download + resend invoices, refunds 2019-10-18 -- list filters, REST API improvements, new docs platform, and more ... 2019-08-08 -- dashboard interface, currency selector, managing Orders, Customers and Products, Added a WireTabs, refinded caching behavior 2019-06-15 -- taxes provider, shop templates update, multiCURL implementation, and more ... 2019-06-02 -- FieldtypeSnipWireTaxSelector 2019-05-25 -- SnipWire will be free and open source Plugin Key Features
      Fast and simple store setup Full integration of the Snipcart dashboard into the ProcessWire backend (no need to leave the ProcessWire admin area) Browse and manage orders, customers, discounts, abandoned carts, and more Process refunds and send customer notifications from within the ProcessWire backend Complete Snipcart webhooks integration (all events are hookable via ProcessWire hooks) Integrated taxes provider (which is more flexible then Snipcart own provider) Useful Links
      SnipWire in PW modules directory (alpha version only available via GitHub) SnipWire Docs (please note that the documentation is a work in progress) SnipWire @GitHub (feature requests and suggestions for improvement are welcome - I also accept pull requests) Snipcart Website  
      ---- INITIAL POST FROM 2019-05-25 ----
       
    • By d'Hinnisdaël
      Happy new year, everybody 🥬
      I've been sitting on this Dashboard module I made for a client and finally came around to cleaning it up and releasing it to the wider public. This is how it looks.
      ProcessWire Dashboard

      If anyone is interested in trying this out, please go ahead! I'd love to get some feedback on it. If this proves useful and survives some real-world testing, I'll add this to the module directory.
      Download
      You can find the latest release on Github.
      Documentation
      Check out the documentation to get started. This is where you'll find information about included panel types and configuration options.
      Custom Panels
      My goal was to make it really simple to create custom panels. The easiest way to do that is to use the panel type template and have it render a file in your templates folder. This might be enough for 80% of all use cases. For anything more complex (FormBuilder submissions? Comments? Live chat?), you can add new panel types by creating modules that extend the DashboardPanel base class. Check out the documentation on custom panels or take a look at the HelloWorld panel to get started. I'm happy to merge any user-created modules into the main repo if they might be useful to more than a few people.
       Disclaimer
      This is a pre-release version. Please treat it as such — don't install it on production sites. Just making sure 🍇
      Roadmap
      These are the things I'm looking to implement myself at some point. The wishlist is a lot longer, but those are the 80/20 items that I probably won't regret spending time on.
      Improve documentation & add examples ⚙️ Panel types Google Analytics ⚙️ Add new page  🔥 Drafts 🔥 At a glance / Page counter 404s  Layout options Render multiple tabs per panel panel groups with heading and spacing between ✅ panel wrappers as grid item (e.g. stacked notices) ✅ Admin themes support AdminThemeReno and AdminThemeDefault ✅ Shortcuts panel add a table layout with icon, title & summary ✅ Chart panel add default styles for common chart types ✅ load chart data from JS file (currently passed as PHP array) Collection panel support image columns ✅ add buttons: view all & add new ✅
    • By Robin S
      This module is inspired by and similar to the Template Stubs module. The author of that module has not been active in the PW community for several years now and parts of the code for that module didn't make sense to me, so I decided to create my own module. Auto Template Stubs has only been tested with PhpStorm because that is the IDE that I use.
      Auto Template Stubs
      Automatically creates stub files for templates when fields or fieldgroups are saved.
      Stub files are useful if you are using an IDE (e.g. PhpStorm) that provides code assistance - the stub files let the IDE know what fields exist in each template and what data type each field returns. Depending on your IDE's features you get benefits such as code completion for field names as you type, type inference, inspection, documentation, etc.
      Installation
      Install the Auto Template Stubs module.
      Configuration
      You can change the class name prefix setting in the module config if you like. It's good to use a class name prefix because it reduces the chance that the class name will clash with an existing class name.
      The directory path used to store the stub files is configurable.
      There is a checkbox to manually trigger the regeneration of all stub files if needed.
      Usage
      Add a line near the top of each of your template files to tell your IDE what stub class name to associate with the $page variable within the template file. For example, with the default class name prefix you would add the following line at the top of the home.php template file:
      /** @var tpl_home $page */ Now enjoy code completion, etc, in your IDE.

      Adding data types for non-core Fieldtype modules
      The module includes the data types returned by all the core Fieldtype modules. If you want to add data types returned by one or more non-core Fieldtype modules then you can hook the AutoTemplateStubs::getReturnTypes() method. For example, in /site/ready.php:
      // Add data types for some non-core Fieldtype modules $wire->addHookAfter('AutoTemplateStubs::getReturnTypes', function(HookEvent $event) { $extra_types = [ 'FieldtypeDecimal' => 'string', 'FieldtypeLeafletMapMarker' => 'LeafletMapMarker', 'FieldtypeRepeaterMatrix' => 'RepeaterMatrixPageArray', 'FieldtypeTable' => 'TableRows', ]; $event->return = $event->return + $extra_types; }); Credits
      Inspired by and much credit to the Template Stubs module by mindplay.dk.
       
      https://github.com/Toutouwai/AutoTemplateStubs
      https://modules.processwire.com/modules/auto-template-stubs/
    • By Mike Rockett
      Jumplinks for ProcessWire
      Release: 1.5.60
      Composer: rockett/jumplinks
      ⚠️ NOTICE: 1.5.60 is an important security patch-release for an XSS vulnerability discovered by @phlp. It's HIGHLY RECOMMENDED that all Jumplinks users update to the latest version as soon as possible.
      Jumplinks is an enhanced version of the original ProcessRedirects by Antti Peisa.
      The Process module manages your permanent and temporary redirects (we'll call these "jumplinks" from now on, unless in reference to redirects from another module), useful for when you're migrating over to ProcessWire from another system/platform. Each jumplink supports wildcards, shortening the time needed to create them.
      Unlike similar modules for other platforms, wildcards in Jumplinks are much easier to work with, as Regular Expressions are not fully exposed. Instead, parameters wrapped in curly braces are used - these are described in the documentation.
      Under Development: 2.0, to be powered by FastRoute
      As of version 1.5.0, Jumplinks requires at least ProcessWire 2.6.1 to run.
      View on GitLab
      Download via the Modules Directory
      Read the docs
      Features
      The most prominent features include:
      Basic jumplinks (from one fixed route to another) Parameter-based wildcards with "Smart" equivalents Mapping Collections (for converting ID-based routes to their named-equivalents without the need to create multiple jumplinks) Destination Selectors (for finding and redirecting to pages containing legacy location information) Timed Activation (activate and/or deactivate jumplinks at specific times) 404-Monitor (for creating jumplinks based on 404 hits) Additionally, the following features may come in handy:
      Stale jumplink management Legacy domain support for slow migrations An importer (from CSV or ProcessRedirects) Feedback & Feature Requests
      I’d love to know what you think of this module. Please provide some feedback on the module as a whole, or even regarding smaller things that make it whole. Also, please feel free to submit feature requests and their use-cases.
      Note: Features requested so far have been added to the to-do list, and will be added to 2.0, and not the current dev/master branches.
      Open Source

      Jumplinks is an open-source project, and is free to use. In fact, Jumplinks will always be open-source, and will always remain free to use. Forever. If you would like to support the development of Jumplinks, please consider making a small donation via PayPal.
      Enjoy! 🙂
    • By Robin S
      Add Image URLs
      Allows images/files to be added to Image/File fields by pasting URLs.

      Usage
      Install the Add Image URLs module.
      A "Paste URLs" button will be added to all image and file fields. Use the button to show a textarea where URLs may be pasted, one per line. Images/files are added when the page is saved.
       
      https://github.com/Toutouwai/AddImageUrls
      https://modules.processwire.com/modules/add-image-urls/
×
×
  • Create New...