Jump to content

Migrator


adrian

Recommended Posts

Ahh, that makes total sense - I am sure it is because all your fields become objects because of the two language versions eg:

"body": {
                    "data": "",
                    "data1029": ""
                },

rather than just simple strings.

Not sure how difficult this will be to fix - hopefully simple :)

  • Like 1
Link to comment
Share on other sites

Yep, that is. Probably some type sniffing and in case of object you can safely assume it will be multilang field. I have done much with those in API level, but I know soma is the man with those (like many others). Probably pretty simple, I think the number is language id, and without number it is default lang.

Link to comment
Share on other sites

Hi Adrian,

With your latest version I just get blank pages on my PW 2.3.0 install when I try to export anything. My server PHP version is 5.3.25. Perhaps I need a recent dev version?

There's something up with my install there I think. Tried it locally and it works perfectly. This will be a huge time saver Mr Jones. Thank you.

  • Like 1
Link to comment
Share on other sites

That's great news Marty :)

Unfortunately I haven't managed to make any time to work on this today and I am away for the next couple of days, but I hope next week to solve the multilanguage issue that Antti discovered - I have a plan and should be quite simple so long as the receiving PW installation has the needed languages installed - I should be able to issue a warning if it doesn't.

I also have some ideas on how to make migration of images and files work too and maybe even RTE embedded images. Not sure how important migration of this sort of content is though - maybe it is adding too much complexity. Do you have any thoughts on number 3 of the road map above: http://processwire.com/talk/topic/4420-page-list-migrator/?p=43582

BTW, I have spent many days in your part of the world when I was a keen climber - beautiful place!

Link to comment
Share on other sites

Hey Antti,

I have just committed an update that I think fixes the multi-language issues. I know there are two different ways to do different language versions of a field, but from my understanding, this should work with both.

Would you mind checking on your end and see if the import is now working for you.

Thanks!

Link to comment
Share on other sites

Currently it gives me an error:

Parse error: syntax error, unexpected '[' in /.../site/modules/ProcessPageTreeMigrator/ProcessPageTreeMigrator.module on line 666 

And the code there is:

$newArray = [];

So it requires php 5.4. Just changed it to array() and now it works.

Importing full data still doesn't work, this time it seems to be repeaters:

FieldtypeRepeater: Created Repeater Page Parent: /processwire/repeaters/for-field-98/for-page-1067/

TemplateFile: Method RepeaterPageArray::setLanguageValue does not exist or is not callable in this context

For me export & import of templates and fields is the main selling point of this module (not the data).

  • Like 1
Link to comment
Share on other sites

Ah, sorry - been on 5.4 for so long I forgot about this.

Yes, repeaters aren't supported yet. For the moment I'll add that to the list of fieldtypes to ignore so at least there won't be an error.

Btw, the last update also now handles RTE links that have been processed by the PageLinkAbstractor module.

EDIT: Committed version with those small fixes. NB that the check to ignore repeaters is currently only on the export, so you would need to export again to check.

  • Like 3
Link to comment
Share on other sites

  • 3 weeks later...

A small update - I have added support for import options so now you can decide whether you want to import just the fields and templates, or the entire page tree at the import stage, regardless of what is in the exported JSON file. Of course you can't import the page tree if it wasn't exported, but you can import just the fields and templates from a JSON file that contains the entire page tree.

I have also added support for importing page trees directly from the repo at: https://github.com/adrianbj/ProcessWirePageLists Just choose the "Shared JSON Packages" option when importing. 

I haven't tackled the repeater field etc issues yet, hopefully soon.

Will also be adding more shared packages soon and would love any contributions :)

See the attached screenshot showing the direct page tree import using Ryan's awesome new admin theme :)

post-985-0-22275500-1380836087_thumb.png

  • Like 5
Link to comment
Share on other sites

Hey adrian, this is great work, thanks a lot for the effort. I tested it with small chunks of datalists, selection values and so on, works like a charm.

On one specific page with FieldSetTabs I get this, after hitting the Export button in Step 2:

Error: Cannot break/continue 1 level (line 641 of /pwtesting/site/modules/ProcessPageTreeMigrator/ProcessPageTreeMigrator.module) 

This chunk seems to be responsible:

	if($type == 'fields'){

	    if($page->type instanceof FieldtypeFieldsetOpen) continue;

	    $data = array(

	        'name' => $page->name,
	        'label' => $page->label,
	        'description' => $page->description,
	        'template' => $template_name,
	        'flags' => $page->flags,
	        'type' => "{$page->type}",

	    );

	}

Also this might not be relevant yet, but on a page on which I use soma's new alpha module for ContentBlocks after exporting I get this:

Error: Call to a member function get() on a non-object (line 734 of /pwtesting/site/modules/ProcessPageTreeMigrator/ProcessPageTreeMigrator.module) 

I'm on PW 2.3.3.3

Perhaps it would be useful for users to see in the github readme what's already working and what not working yet. Perhaps a little roadmap as well.

Thanks again, this will be a huge timesaver and the more modular approach for migrating little logical/data chunks rather than whole profiles will come in very handy when developing locally and on server parallely.

Link to comment
Share on other sites

Thanks ozwim for the testing and bug reports. I'll see what I can do about getting these sorted out. I haven't tested with fieldsets at all yet. As for the error on line 734 - it is a language module related thing. I had seen that once before - should be easy enough to sort out.

Good idea for adding details to the github readme on what's working etc - I have a feeling it will be an increasing list for a while, before it starts getting smaller, but I am sure we'll get there :)

  • Like 1
Link to comment
Share on other sites

  • 3 weeks later...

Ok, I have pushed another version to Github. v0.0.8 fixes the fieldsettab issue that Ozwim reported and introduces support for migrating repeater fields. It handles the creation of everything required on the source PW installation for the new repeater fields to work.

Next on the roadmap is full support for page fields - currently only the page field itself is migrated, but not the selectable pages fields and templates. Then it is on to migrating the page content for repeater and page fields

I would really appreciate any help testing out this new version. 

EDIT: Just updated the readme on github to detail what does and doesn't work, along with a rough roadmap - please let me know if your testing shows that I have any working/not working mistakes - I am sure there are some :)

  • Like 1
Link to comment
Share on other sites

Ok another update this morning - I have added support for full migration of page fields - it now automatically handles creation of the selectable pages fields, templates, and the pages themselves.

The one thing I haven't really figured out how to tackle is what parent to put these pages under. At the moment they end up under the same parent page that you choose when starting an import. This is probably not ideal, but otherwise it could result in the need to create a whole other lot of parent parents/templates/fields that you don't really need on the destination PW site. This way everything needed gets migrated but nothing more and you can easily move them into a new parent (like a tools parent) if needed. Anyone have any better ideas? You might need to run a test to see what I mean.

Again, this will probably need lots of testing so thanks in advance - there is a lot going on to make this all work!

Link to comment
Share on other sites

Another update - things are getting closer!

I have just added support for migration of the selected values of page fields when migrating "everything, including all data pages".

Only repeater field page data (the fields, templates etc are already working) to go, and then of course the inevitable bugs :)

EDIT:

Just found a bug in the page field data migration - hopefully solve this soon.

Repeater data is working now though - not on github yet, but I will push that too once I have the page field issue sorted out.

Edited by adrian
  • Like 1
Link to comment
Share on other sites

Getting stoked :)

The latest version on github now supports migration of repeater field page content and fixes the problems with migrating page field content.

That should take care of migrating the fields, templates, and page content for all the field types, not including page content for file and image fields as obviously these can't, or more accurately, shouldn't be included in a json file.

Multi-language fields should migrate perfectly so long as you have the required language support modules and languages installed on the destination PW installation. Let me know if you have any troubles here - I did see some weird errors during early testing - I hope they are all fixed now.

Please note: that most of my testing of late has been with the "Everything, including all data pages" option on export and import. I need to go back and test the other options soon, but I'll probably do that when I start implementing the series of checkboxes allowing you to select exactly which fields, templates, and pages to be exported and imported.

Also, I have been testing with the latest dev version of PW (downloaded today). Not sure if anything is critical, but if you are getting errors, please try upgrading PW first.

  • Like 3
Link to comment
Share on other sites

Sorry for yet another post, but I just fixed a bug that prevented certain fields being exported and assigned to templates in some situations. Not the most elegant solution (repetitive declaration in the JSON file), but it should work for the moment until I have time to rework slightly.

Grab the latest version from Github for testing :)

Link to comment
Share on other sites

  • 1 month later...

Just pushed a minor fix to prevent 403 forbidden errors when hitting the github page lists repository when starting the import process. Apparently they were forcing a user-agent header for several months now, but didn't seem to affect this module till recently. Anyway, should be fixed now.

Sometime early in the new year I will sort out all the php notices/warnings in this module and add some more features for finer detailed selection of what you want to export/import.

  • Like 1
Link to comment
Share on other sites

  • 2 months later...

Just committed a fairly major update to support full migration of files and images, including rewriting of the src tag to match the ID of the new page for images embedded into RTE fields. Additionally, the module now also migrates the template .php files.

Files/images/template files and the json structure/data file are exported in a zip file which is then imported into the destination PW install.

So, you can now select one parent page during export and it will migrate the following content for all child pages of the selected parent:

  • All standard field types, including RTE, and decoding of links modified by the PageLinkAbstractor module and abstracting again on the destination PW install.
  • File/Image/CropImage fields including the actual files/images/thumbnails and all other variations
  • Repeater fields and all their required fields, templates, and all content, including files/images
  • Page fields (and the pages, templates, and fields that make up their selectable pages)
  • Multi-language versions of all field content
  • Templates (including Access, Family, URL and other settings) and the template .php files. It even grabs the appropriate file if you are using the "Alternate Template Filename" setting. NB the templates directory on the destination PW installation must be writable for these to be imported.
 
So, you could build sections of content on a local dev PW installation, export it, and then with a couple of clicks import everything into the live PW installation.
 
Some outstanding issues that I hope to get to shortly:
  • Need to support images inserted from a different page into an RTE field
  • Rewrite any references to page ids, eg $pages->get(xxxx) in template .php files so they will be converted to the correct id on the destination installation.
  • Need to look into the new core link abstractor that was added to PW 2.4 and see how to handle those links compared to the PageLinkAbstractor module.
  • I still want to add finer control for determining exactly what components will be exported and imported.
  • Generally needs lots more error checking for things like making sure required languages are installed on the destination PW install etc
  • Need to add checks so that existing template php files are not overwritten (or give the option to choose)
  • Might need to override PHP max_execution_time and other settings for larger exports and maybe chunk out zipping of all images to prevent memory issues on larger exports.

In its current state it is handling almost all my needs, but I would really like to have this robust enough for anyone to use, so if you have time please do some testing for me - THANKS!

At the moment, please only use it on two unimportant test PW installs!

Edited by adrian
  • Like 10
Link to comment
Share on other sites

Several of the outstanding issues and a couple of bug fixes have been taken care of this morning. Be sure to grab the latest from Github  :)

EDIT: Even more issues taken care of this afternoon. Updated in post #47 above.

Edited by adrian
Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
  • Recently Browsing   0 members

    No registered users viewing this page.

  • Similar Content

    • By LuisM
      Symprowire is a PHP MVC Framework based and built on Symfony using ProcessWire 3.x as DBAL and Service-Provider
      It acts as a Drop-In Replacement Module to handle the Request/Response outside the ProcessWire Admin. Even tough Symfony or any other mature MVC Framework could be intimidating at first, Symprowire tries to abstract Configuration and Symfony Internals away as much as possible to give you a quick start and lift the heavy work for you.
      The main Goal is to give an easy path to follow an MVC Approach during development with ProcessWire and open up the available eco-system.
      You can find the GitHub Repo and more Information here: https://github.com/Luis85/symprowire
      Documentation
      The Symprowire Wiki https://github.com/Luis85/symprowire/wiki How to create a simple Blog with Symprowire https://github.com/Luis85/symprowire/wiki/Symprowire-Blog-Tutorial Last Update
      16.07.2021 // RC 1 v0.6.0 centralized ProcessWire access trough out the Application by wrapping to a Service https://github.com/Luis85/symprowire/releases/tag/v0.6.0-rc-1 Requirements
      PHP ^7.4 Fresh ProcessWire ^3.0.181 with a Blank Profile Composer 2 (v1 should work, not recommended) The usual Symfony Requirements Features
      Twig Dependency Injection Monolog for Symprowire Support for .env YAML Configuration Symfony Console and Console Commands Symfony Webprofiler Full ProcessWire access inside your Controller and Services Webpack Encore support Caveats
      Symfony is no small Framework and will come with a price in terms of Memory Usage and added Overhead. To give you a taste I installed Tracy Debugger alongside to compare ProcessWire profiling with the included Symfony Webprofiler

      So in a fresh install Symprowire would atleast add another 2MB of Memory usage and around 40ms in response time, should be less in production due to the added overhead of the Webprofiler in dev env
       
    • By FireWire
      Hello community!

      I want to share a new module I've been working on that I think could be a big boost for multi-language ProcessWire sites.

      Some background, I was looking for a way for our company website to be efficiently translated as working with human translators was pretty laborious and a lack of updating content created a divergence between languages. I, and several other devs here, have talked about translation integrations and have recognized the power that DeepL has. DeepL is an AI deep learning powered service that delivers translation quality beyond any automated service available. After access to the API was opened up to the US, I built Fluency, a DeepL translation integration for ProcessWire.
      Fluency brings automated translation to every multi-language field in the admin, and also provides a translation tool allowing the user to translate their text to any language without it being inside a template's field. With Fluency you can:
      Translate any plain textarea or text input Translate any CKEditor content (yes, with markup) Translate page names for fully localized URLs on every page Translate your in-template translation function wrapped strings Translate modules Fluency is free, and now so is DeepL
      Since this module was first built DeepL has introduced free Developer accounts that allow anyone to start using Fluency at zero cost and beginning with the version 0.3.0 release Fluency now supports free DeepL accounts. As of June 2021 DeepL supports translation to 26 languages and continues to offer more!
      Installation and usage is completely plug and play. Whether you're building a new multi-language site, need to update a site to multi-language, or simply want to stop manually translating a site and make any language a one-click deal, it could not be easier to do it. Fluency works by having you match the languages configured in ProcessWIre to DeepL's. You can have your site translating to any or all of the languages DeepL translates to in minutes (quite literally).
      Let's break out the screenshots...
      When the default language tab is shown, a message is displayed to let users know that translation is available. Clicking on each tab shows a link that says "Translate from English". Clicking it shows an animated overlay with the word "Translating..." cycling through each language and a light gradient shift. Have a CKEditor field? All good. Fluency will translated it and use DeepL's ability to translate text within HTML tags. CKEditor fields can be translated as easily and accurately as text/textarea fields.

      Repeaters and AJAX created fields also have translation enabled thanks to a JavaScript MutationObserver that searches for multi-language fields and adds translation as they're inserted into the DOM. If there's a multi-language field on the page, it will have translation added.

      Same goes for image description fields. Multi-language SEO friendly images are good to go.

      Creating a new page from one of your templates? Translate your title, and also translate your page name for native language URLs. (Not available for Russian, Chinese, or Japanese languages due to URL limitations). These can be changed in the "Settings" tab for any page as well so whether you're translating new pages or existing pages, you control the URLs everywhere.

      Language configuration pages are no different. Translate the names of your languages and search for both Site Translation Files (including all of your modules)

      Translate all of the static text in your templates as well. Notice that the placeholders are retained. DeepL is pretty good at recognizing and keeping non-translatable strings like that. If it is changed, it's easy to fix manually.

      Fluency adds a "Translate" item to the CMS header. When clicked this opens up a modal with a full translation tool that lets the user translate any language to any language. No need to leave the admin if you need to translate content from a secondary language back to the default ProcessWire language. There is also a button to get the current API usage statistics. DeepL account owners can set billing limitations via character count to control costs. This may help larger sites or sites being retrofitted keep an eye on their usage. Fluency can be used by users having roles given the fluency-translate permission.

      It couldn't be easier to add Fluency to your new or existing website. Simply add your API key and you're shown what languages are currently available for translation from/to as provided by DeepL. This list and all configuration options are taken live from the API so when DeepL releases new languages you can add them to your site without any work. No module updates, just an easy configuration. Just match the language you configured in ProcessWire to the DeepL language you want it to be associated with and you're done. Fluency also allows you to create a list of words/phrases that will not be translated which can prevent items such as brands and company names from being translated when they shouldn't

       
      Limitations:
      No "translate page" - Translating multiple fields can be done by clicking multiple translation links on multiple fields at once but engineering a "one click page translate" is not feasible from a user experience standpoint. The time it takes to translate one field can be a second or two, but cumulatively that may take much longer (CKEditor fields are slower than plain text fields). There may be a workaround in the future but it isn't currently on the roadmap. No "translate site" - Same thing goes for translating an entire website at once. It would be great, but it would be a very intense process and take a very (very) long time. There may be a workaround in the future but it isn't on the roadmap. No current support for Inline CKEditor fields - Handling for CKEditor on-demand hasn't been implemented yet, this is planned for a future release though and can be done. I just forgot about it because I've never really used that feature personally.. Alpha release - This module is in alpha. Releases should be stable and usable, but there may be edge case issues. Test the module thoroughly and please report any bugs via a Github issue on the repository or respond here. Please note that the browser plugin for Grammarly conflicts with Fluency (as it does with many web applications). To address this issue it is recommended that you disable Grammarly when using Fluency, or open the admin to edit pages in a private window where Grammarly may not be loaded. This is an issue that may not have a resolution as creating a workaround may not be possible. If you have insight as to how this may be solved please visit the Github page and file a bugfix ticket.
      Requirements:
      ProcessWire  3.0+ UIKit Admin Theme That's Fluency in a nutshell. A core effort in this module is to create it so that there is nothing DeepL related hard-coded in that would require updating it when DeepL offers new languages. I would like this to be a future-friendly module that doesn't require developer work to keep it up-to-date.
      The Module Is Free
      This is my first real module and I want to give it back to the community as thanks. This is the best CMS I've worked with (thank you Ryan & contributors) and a great community (thank you dear reader).
      DeepL Developer Accounts
      In addition to paid Pro Developer accounts, DeepL now offers no-cost free accounts. Now all ProcessWire developers and users can use Fluency at no cost.
      Learn more about free and paid accounts by visiting the DeepL website. Sign up for a Developer account, get an API key, and start using Fluency today.
      Download & Feedback
      Download the latest version here
      https://github.com/SkyLundy/Fluency-Translation/archive/main.zip
      Github repository:
      https://github.com/SkyLundy/Fluency-Translation
      File issues and feature requests here (your feedback and testing is greatly appreciated):
      https://github.com/SkyLundy/Fluency-Translation/issues
       
      Thank you! ¡Gracias! Ich danke Ihnen! Merci! Obrigado! Grazie! Dank u wel! Dziękuję! Спасибо! ありがとうございます! 谢谢你!

    • By tcnet
      PageViewStatistic for ProcessWire is a module to log page visits of the CMS. The records including some basic information like IP-address, browser, operating system, requested page and originate page. Please note that this module doesn't claim to be the best or most accurate.
      Advantages
      One of the biggest advantage is that this module doesn't require any external service like Google Analytics or similar. You don't have to modify your templates either. There is also no Javascript or image required.
      Disadvantages
      There is only one disadvantage. This module doesn't record visits if the browser loads the page from its browser cache. To prevent the browser from loading the page from its cache, add the following meta tags to the header of your page:
      <meta http-equiv="Cache-Control" content="no-cache, no-store, must-revalidate" /> <meta http-equiv="Pragma" content="no-cache" /> <meta http-equiv="Expires" content="0" /> How to use
      The records can be accessed via the Setup-menu of the CMS backend. The first dropdown control changes the view mode.

      Detailed records
      View mode "Detailed records" shows all visits of the selected day individually with IP-address, browser, operating system, requested page and originate page. Click the update button to see new added records.

      Cached visitor records
      View modes other than "Detailed records" are cached visitor counts which will be collected on a daily basis from the detailed records. This procedure ensures a faster display even with a large number of data records. Another advantage is that the detailed records can be deleted while the cache remains. The cache can be updated manually or automatically in a specified time period. Multiple visits from the same IP address on the same day are counted as a single visitor.

      Upgrade from older versions
      Cached visitor counts is new in version 1.0.8. If you just upgraded from an older version you might expire a delay or even an error 500 if you display cached visitor counts. The reason for this is that the cache has to be created from the records. This can take longer if your database contains many records. Sometimes it might hit the maximally execution time. Don't worry about that and keep reloading the page until the cache is completely created.
      Special Feature
      PageViewStatistic for ProcessWire can record the time a visitor viewed the page. This feature is deactivated by default. To activate open the module configuration page and activate "Record view time". If activated you will find a new column "S." in the records which means the time of view in seconds. With every page request, a Javascript code is inserted directly after the <body> tag. Every time the visitor switches to another tab or closes the tab, this script reports the number of seconds the tab was visible. The initial page request is recorded only as a hyphen (-).

      New in version 1.1.0
      A new feature comes with version 1.1.0 which offers to record user names of loggedin visitors. Just activate "Record user names" and "Record loggedin user" in the module settings.
      Settings
      You can access the module settings by clicking the Configuration button at the bottom of the records page. The settings page is also available in the menu: Modules->Configure->ProcessPageViewStat.
      IP2Location
      This module uses the IP2Location database from: http://www.ip2location.com. This database is required to obtain the country from the IP address. IP2Location updates this database at the begin of every month. The settings of ProcessPageViewStat offers the ability to automatically download the database monthly. Please note, that automatically download will not work if your webspace doesn't allow allow_url_fopen.
      Dragscroll
      This module uses DragScroll. A JavaScript available from: http://github.com/asvd/dragscroll. Dragscroll adds the ability in view mode "Day" to drag the records horizontally with the mouse pointer.
      parseUserAgentStringClass
      This module uses the PHP class parseUserAgentStringClass available from: http://www.toms-world.org/blog/parseuseragentstring/. This class is required to filter out the browser type and operating system from the server request.
    • By clsource
      Inertia Adapter ProcessWire Module
      Hello! Long time no see.
      I created this module so you can use Inertia.js (https://inertiajs.com/) with ProcessWire.
      Description
      Inertia allows you to create fully client-side rendered, single-page apps, without much of the complexity that comes with modern SPAs. It does this by leveraging existing server-side frameworks.
      Inertia isn’t a framework, nor is it a replacement to your existing server-side or client-side frameworks. Rather, it’s designed to work with them. Think of Inertia as glue that connects the two. Inertia comes with three official client-side adapters (React, Vue, and Svelte).
      This is an adapter for ProcessWire. Inertia replaces PHP views altogether by returning JavaScript components from controller actions. Those components can be built with your frontend framework of choice.
      Links
      - https://github.com/joyofpw/inertia
      - https://github.com/joyofpw/inertia-svelte-mix-pw
      - https://inertiajs.com/
      Screenshots


       
    • By sms77io
      Hi all,
      we made a small module for sending SMS via Sms77.io. It supports sending to one and multiple users.
      You can download it from GitHub and follow the instructions on how to install it - it is quite easy. An API key is required for sending, get yours for free @ Sms77 and receive 0,50 €.
      Hope this helps somebody and we are open for improvement suggestions!
       
      Best regards
      André
×
×
  • Create New...