Jump to content
adrian

Migrator

Recommended Posts

Ahh, that makes total sense - I am sure it is because all your fields become objects because of the two language versions eg:

"body": {
                    "data": "",
                    "data1029": ""
                },

rather than just simple strings.

Not sure how difficult this will be to fix - hopefully simple :)

  • Like 1

Share this post


Link to post
Share on other sites

Yep, that is. Probably some type sniffing and in case of object you can safely assume it will be multilang field. I have done much with those in API level, but I know soma is the man with those (like many others). Probably pretty simple, I think the number is language id, and without number it is default lang.

Share this post


Link to post
Share on other sites

Hi Adrian,

With your latest version I just get blank pages on my PW 2.3.0 install when I try to export anything. My server PHP version is 5.3.25. Perhaps I need a recent dev version?

There's something up with my install there I think. Tried it locally and it works perfectly. This will be a huge time saver Mr Jones. Thank you.

  • Like 1

Share this post


Link to post
Share on other sites

That's great news Marty :)

Unfortunately I haven't managed to make any time to work on this today and I am away for the next couple of days, but I hope next week to solve the multilanguage issue that Antti discovered - I have a plan and should be quite simple so long as the receiving PW installation has the needed languages installed - I should be able to issue a warning if it doesn't.

I also have some ideas on how to make migration of images and files work too and maybe even RTE embedded images. Not sure how important migration of this sort of content is though - maybe it is adding too much complexity. Do you have any thoughts on number 3 of the road map above: http://processwire.com/talk/topic/4420-page-list-migrator/?p=43582

BTW, I have spent many days in your part of the world when I was a keen climber - beautiful place!

Share this post


Link to post
Share on other sites

Awesome job Adrian! I can't wait to use this one. I'd be using it right now if I weren't holding a baby in one arm and typing with the other. :) But I did browse the code and thought it looked great. 

  • Like 1

Share this post


Link to post
Share on other sites

Thanks Ryan,

I hope I can make some time in the next couple of days to get a few more things sorted out with this. I am excited to get this release worthy!

  • Like 1

Share this post


Link to post
Share on other sites

Hey Antti,

I have just committed an update that I think fixes the multi-language issues. I know there are two different ways to do different language versions of a field, but from my understanding, this should work with both.

Would you mind checking on your end and see if the import is now working for you.

Thanks!

Share this post


Link to post
Share on other sites

Currently it gives me an error:

Parse error: syntax error, unexpected '[' in /.../site/modules/ProcessPageTreeMigrator/ProcessPageTreeMigrator.module on line 666 

And the code there is:

$newArray = [];

So it requires php 5.4. Just changed it to array() and now it works.

Importing full data still doesn't work, this time it seems to be repeaters:

FieldtypeRepeater: Created Repeater Page Parent: /processwire/repeaters/for-field-98/for-page-1067/

TemplateFile: Method RepeaterPageArray::setLanguageValue does not exist or is not callable in this context

For me export & import of templates and fields is the main selling point of this module (not the data).

  • Like 1

Share this post


Link to post
Share on other sites

Ah, sorry - been on 5.4 for so long I forgot about this.

Yes, repeaters aren't supported yet. For the moment I'll add that to the list of fieldtypes to ignore so at least there won't be an error.

Btw, the last update also now handles RTE links that have been processed by the PageLinkAbstractor module.

EDIT: Committed version with those small fixes. NB that the check to ignore repeaters is currently only on the export, so you would need to export again to check.

  • Like 3

Share this post


Link to post
Share on other sites

A small update - I have added support for import options so now you can decide whether you want to import just the fields and templates, or the entire page tree at the import stage, regardless of what is in the exported JSON file. Of course you can't import the page tree if it wasn't exported, but you can import just the fields and templates from a JSON file that contains the entire page tree.

I have also added support for importing page trees directly from the repo at: https://github.com/adrianbj/ProcessWirePageLists Just choose the "Shared JSON Packages" option when importing. 

I haven't tackled the repeater field etc issues yet, hopefully soon.

Will also be adding more shared packages soon and would love any contributions :)

See the attached screenshot showing the direct page tree import using Ryan's awesome new admin theme :)

post-985-0-22275500-1380836087_thumb.png

  • Like 5

Share this post


Link to post
Share on other sites

Hey adrian, this is great work, thanks a lot for the effort. I tested it with small chunks of datalists, selection values and so on, works like a charm.

On one specific page with FieldSetTabs I get this, after hitting the Export button in Step 2:

Error: Cannot break/continue 1 level (line 641 of /pwtesting/site/modules/ProcessPageTreeMigrator/ProcessPageTreeMigrator.module) 

This chunk seems to be responsible:

	if($type == 'fields'){

	    if($page->type instanceof FieldtypeFieldsetOpen) continue;

	    $data = array(

	        'name' => $page->name,
	        'label' => $page->label,
	        'description' => $page->description,
	        'template' => $template_name,
	        'flags' => $page->flags,
	        'type' => "{$page->type}",

	    );

	}

Also this might not be relevant yet, but on a page on which I use soma's new alpha module for ContentBlocks after exporting I get this:

Error: Call to a member function get() on a non-object (line 734 of /pwtesting/site/modules/ProcessPageTreeMigrator/ProcessPageTreeMigrator.module) 

I'm on PW 2.3.3.3

Perhaps it would be useful for users to see in the github readme what's already working and what not working yet. Perhaps a little roadmap as well.

Thanks again, this will be a huge timesaver and the more modular approach for migrating little logical/data chunks rather than whole profiles will come in very handy when developing locally and on server parallely.

Share this post


Link to post
Share on other sites

Thanks ozwim for the testing and bug reports. I'll see what I can do about getting these sorted out. I haven't tested with fieldsets at all yet. As for the error on line 734 - it is a language module related thing. I had seen that once before - should be easy enough to sort out.

Good idea for adding details to the github readme on what's working etc - I have a feeling it will be an increasing list for a while, before it starts getting smaller, but I am sure we'll get there :)

  • Like 1

Share this post


Link to post
Share on other sites

Ok, I have pushed another version to Github. v0.0.8 fixes the fieldsettab issue that Ozwim reported and introduces support for migrating repeater fields. It handles the creation of everything required on the source PW installation for the new repeater fields to work.

Next on the roadmap is full support for page fields - currently only the page field itself is migrated, but not the selectable pages fields and templates. Then it is on to migrating the page content for repeater and page fields

I would really appreciate any help testing out this new version. 

EDIT: Just updated the readme on github to detail what does and doesn't work, along with a rough roadmap - please let me know if your testing shows that I have any working/not working mistakes - I am sure there are some :)

  • Like 1

Share this post


Link to post
Share on other sites

Ok another update this morning - I have added support for full migration of page fields - it now automatically handles creation of the selectable pages fields, templates, and the pages themselves.

The one thing I haven't really figured out how to tackle is what parent to put these pages under. At the moment they end up under the same parent page that you choose when starting an import. This is probably not ideal, but otherwise it could result in the need to create a whole other lot of parent parents/templates/fields that you don't really need on the destination PW site. This way everything needed gets migrated but nothing more and you can easily move them into a new parent (like a tools parent) if needed. Anyone have any better ideas? You might need to run a test to see what I mean.

Again, this will probably need lots of testing so thanks in advance - there is a lot going on to make this all work!

Share this post


Link to post
Share on other sites

Another update - things are getting closer!

I have just added support for migration of the selected values of page fields when migrating "everything, including all data pages".

Only repeater field page data (the fields, templates etc are already working) to go, and then of course the inevitable bugs :)

EDIT:

Just found a bug in the page field data migration - hopefully solve this soon.

Repeater data is working now though - not on github yet, but I will push that too once I have the page field issue sorted out.

Edited by adrian
  • Like 1

Share this post


Link to post
Share on other sites
Getting stoked :)

The latest version on github now supports migration of repeater field page content and fixes the problems with migrating page field content.

That should take care of migrating the fields, templates, and page content for all the field types, not including page content for file and image fields as obviously these can't, or more accurately, shouldn't be included in a json file.

Multi-language fields should migrate perfectly so long as you have the required language support modules and languages installed on the destination PW installation. Let me know if you have any troubles here - I did see some weird errors during early testing - I hope they are all fixed now.

Please note: that most of my testing of late has been with the "Everything, including all data pages" option on export and import. I need to go back and test the other options soon, but I'll probably do that when I start implementing the series of checkboxes allowing you to select exactly which fields, templates, and pages to be exported and imported.

Also, I have been testing with the latest dev version of PW (downloaded today). Not sure if anything is critical, but if you are getting errors, please try upgrading PW first.

  • Like 3

Share this post


Link to post
Share on other sites

Sorry for yet another post, but I just fixed a bug that prevented certain fields being exported and assigned to templates in some situations. Not the most elegant solution (repetitive declaration in the JSON file), but it should work for the moment until I have time to rework slightly.

Grab the latest version from Github for testing :)

Share this post


Link to post
Share on other sites

Adrian, thanks for your work on this. I am super busy at the moment, but I have great plans for this (reusing common templates etc..). Will definitely give this good ride soon.

  • Like 1

Share this post


Link to post
Share on other sites

Just pushed a minor fix to prevent 403 forbidden errors when hitting the github page lists repository when starting the import process. Apparently they were forcing a user-agent header for several months now, but didn't seem to affect this module till recently. Anyway, should be fixed now.

Sometime early in the new year I will sort out all the php notices/warnings in this module and add some more features for finer detailed selection of what you want to export/import.

  • Like 1

Share this post


Link to post
Share on other sites

Just committed a fairly major update to support full migration of files and images, including rewriting of the src tag to match the ID of the new page for images embedded into RTE fields. Additionally, the module now also migrates the template .php files.

Files/images/template files and the json structure/data file are exported in a zip file which is then imported into the destination PW install.

So, you can now select one parent page during export and it will migrate the following content for all child pages of the selected parent:

  • All standard field types, including RTE, and decoding of links modified by the PageLinkAbstractor module and abstracting again on the destination PW install.
  • File/Image/CropImage fields including the actual files/images/thumbnails and all other variations
  • Repeater fields and all their required fields, templates, and all content, including files/images
  • Page fields (and the pages, templates, and fields that make up their selectable pages)
  • Multi-language versions of all field content
  • Templates (including Access, Family, URL and other settings) and the template .php files. It even grabs the appropriate file if you are using the "Alternate Template Filename" setting. NB the templates directory on the destination PW installation must be writable for these to be imported.
 
So, you could build sections of content on a local dev PW installation, export it, and then with a couple of clicks import everything into the live PW installation.
 
Some outstanding issues that I hope to get to shortly:
  • Need to support images inserted from a different page into an RTE field
  • Rewrite any references to page ids, eg $pages->get(xxxx) in template .php files so they will be converted to the correct id on the destination installation.
  • Need to look into the new core link abstractor that was added to PW 2.4 and see how to handle those links compared to the PageLinkAbstractor module.
  • I still want to add finer control for determining exactly what components will be exported and imported.
  • Generally needs lots more error checking for things like making sure required languages are installed on the destination PW install etc
  • Need to add checks so that existing template php files are not overwritten (or give the option to choose)
  • Might need to override PHP max_execution_time and other settings for larger exports and maybe chunk out zipping of all images to prevent memory issues on larger exports.

In its current state it is handling almost all my needs, but I would really like to have this robust enough for anyone to use, so if you have time please do some testing for me - THANKS!

At the moment, please only use it on two unimportant test PW installs!

Edited by adrian
  • Like 10

Share this post


Link to post
Share on other sites

Thanks Marty - I'd really appreciate that :)

Share this post


Link to post
Share on other sites

Several of the outstanding issues and a couple of bug fixes have been taken care of this morning. Be sure to grab the latest from Github  :)

EDIT: Even more issues taken care of this afternoon. Updated in post #47 above.

Edited by adrian

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

  • Recently Browsing   0 members

    No registered users viewing this page.

  • Similar Content

    • By bernhard
      --- Please use RockFinder3 ---
    • By MoritzLost
      Cacheable Placeholders
      This module allows you to have pieces of dynamic content inside cached output. This aims to solve the common problem of having a mostly cacheable site, but with pieces of dynamic output here and there.  Consider this simple example, where you want to output a custom greeting to the current user:
      <h1>Good morning, <?= ucfirst($user->name) ?></h1> This snippet means you can't use the template cache (at least for logged-in users), because each user has a different name. Even if 99% of your output is static, you can only cache the pieces that you know won't include this personal greeting. A more common example would be CSRF tokens for HTML forms - those need to be unique by definition, so you can't cache the form wholesale.
      This module solves this problem by introducing cacheable placeholders - small placeholder tokens that get replaced during every request. The replacement is done inside a Page::render hook so it runs during every request, even if the response is served from the template cache. So you can use something like this:
      <h1>Good morning, {{{greeting}}}</h1> Replacement tokens are defined with a callback function that produces the appropriate output and added to the module through a simple hook:
      // site/ready.php wire()->addHookAfter('CachePlaceholders::getTokens', function (HookEvent $e) { $tokens = $e->return; $tokens['greeting'] = [ 'callback' => function (array $tokenData) { return ucfirst(wire('user')->name); } ]; $e->return = $tokens; }); Tokens can also include parameters that are parsed and passed to the callback function. There are more fully annotated examples and step-by-step instructions in the README on Github!
      Features
      A simple and fast token parser that calls the appropriate callback and runs automatically. Tokens may include multiple named or positional parameters, as well as multi-value parameters. A manual mode that allows you to replace tokens in custom pieces of cached content (useful if you're using the $cache API). Some built-in tokens for common use-cases: CSRF-Tokens, replacing values from superglobals and producing random hexadecimal strings. The token format is completely customizable, all delimiters can be changed to avoid collisions with existing tag parsers or template languages. Links
      Github Repository & documentation Module directory (pending approval) If you are interested in learning more, the README is very extensive, with more usage examples, code samples and usage instructions!
    • By Craig
      I've been using Fathom Analytics for a while now and on a growing number of sites, so thought it was about time there was a PW module for it.
      WayFathomAnalytics
      WayFathomAnalytics is a group of modules which will allow you to view your Fathom Analytics dashboard in the PW admin panel and (optionally) automatically add and configure the tracking code on front-end pages.
      Links
      GitHub Readme & documentation Download Zip Modules directory Module settings screenshot What is Fathom Analytics?
      Fathom Analytics is a simple, privacy-focused website analytics tool for bloggers and businesses.

      Stop scrolling through pages of reports and collecting gobs of personal data about your visitors, both of which you probably don't need. Fathom is a simple and private website analytics platform that lets you focus on what's important: your business.
      Privacy focused Fast-loading dashboards, all data is on a single screen Easy to get what you need, no training required Unlimited email reports Private or public dashboard sharing Cookie notices not required (it doesn't use cookies or collect personal data) Displays: top content, top referrers, top goals and more
    • By daniels
      This is a lightweight alternative to other newsletter & newsletter-subscription modules.
      You can find the Module in the Modules directory and on Github
      It can subscribe, update, unsubscribe & delete a user in a list in Mailchimp with MailChimp API 3.0. It does not provide any forms or validation, so you can feel free to use your own. To protect your users, it does not save any user data in logs or sends them to an admin.
      This module fits your needs if you...
      ...use Mailchimp as your newsletter / email-automation tool ...want to let users subscribe to your newsletter on your website ...want to use your own form, validation and messages (with or without the wire forms) ...don't want any personal user data saved in any way in your ProcessWire environment (cf. EU data regulation terms) ...like to subscribe, update, unsubscribe or delete users to/from different lists ...like the Mailchimp UI for creating / sending / reviewing email campaigns *I have only tested it with PHP 7.x so far, so use on owners risk
      EDIT:
      Since 0.0.4, instructions and changelog can be found in the README only. You can find it here  🙂
      If you have questions or like to contribute, just post a reply or create an issue or pr on github, thanks!
    • By MoritzLost
      Sorry for the convoluted title. I have a problem with Process modules that define a custom page using the page key through getModuleInfo (as demonstrated in this excellent tutorial by @bernhard). Those pages are created automatically when the module is installed. The problem is that the title of the page only gets set in the current language. That's not a problem if the current language (language of the superuser who is installing the module) is the default language; if it isn't, the Process page is missing a title in the default language. This has the very awkward effect that a user using the backend in the default language (or any other language) will see an empty entry in the setup menu:

      This screenshot comes from my Cache Control module which includes a Process page. Now I realize the description sounds obscure, but for us it's a common setup: We a multiple bilingual sites where the default language is German and the second language is English. While the clients use the CMS in German, as a developer I prefer the English interface, so whenever I install a Process module I get this problem.
      As a module author, is there a way to handle this situation? I guess it would be possible to use post-installation hooks or create the pages manually, but I very much prefer the declarative approach. The page title is already translatable (through the __ function), but of course at the time of installation there is no translation, and as far as I'm aware it's not possible to ship translations with a module so they are used automatically. Could this situation be handled better in the core? I would prefer if the module installation process would always set the title of the Process page in the default language, instead of the language of the current user.
×
×
  • Create New...