Jump to content
adrian

Migrator

Recommended Posts

GitHub: https://github.com/adrianbj/ProcessMigrator


 


This module has gone through lots of iterations with lots of new functionality each time. It is now a fully fledged content migration tool.


 


*** Please be sure to read the GitHub ReadMe to find out what it can do now as most of the posts in this thread are no longer correct regarding its functionality


 


 


Once it is release worthy, I'll create a fresh thread with all the details.


 


 


This modules allows export, sharing, and import of page lists via JSON files.


 


It takes care of replicating all the pages, as well as creating any templates and fields that are needed.


 


I have defined "Page Lists" as page trees (parent and children) that store selector values for a Page fieldtype. An example would be a list of countries that would be used to populate a countries drop-down select field. The fields might include: Country Name, 2-digit code, 3-digit code, number code.


 


I would like to suggest a place where we can post json files to be shared and updated - maybe a dedicated github repository?


Start of a repo of lists ready to import is now available: https://github.com/adrianbj/ProcessWirePageLists


 


It might handle migrating other simple pages trees as well, but it should not be considered a tool for migrating general pages as it does not handle associated files, nor does it handle fields which store arrays. Probably lots of other things it doesn't handle either :)


 


It now handles migrating all (I think) field types, including repeater fields, page fields, all Profields fields, multi-language versions of fields etc. The only omission is the actual uploaded files and images in file/image fields.


 


WARNING: This should be considered an Alpha module - please don't use this on a live site at the moment and be sure to back everything up before testing.


 


 


Would appreciate any feedback on the concept, the code, and the idea of a shared and community edited resource of these files.


 


Also, would love to hear what page lists would be good to share. Here are a few quick ideas:


  • States (separate files for each country)
  • Measurement units
  • Languages
  • Religions
  • Race
  • Academic subjects (chemistry, biology etc)
  • Publication types (book, journal article, newspaper article, newsletter, thesis etc)
  • Car makes and models

Anyone have a better idea for a name, or how to better describe "Page Lists"?


Edited by Nico Knoll
Added the "module" tag.
  • Like 18

Share this post


Link to post
Share on other sites

Very interesting!

Does it create the fields also when importing (those are defined in JSON, so that made me assume)? I would also think about JSON schema - should metadata and content data be in their own containers? Something like "meta" and "data"?

Share this post


Link to post
Share on other sites

Yep,

It creates all required templates and fields using the settings from the original version. It won't re-create or modify a field if it is already existing, but it will add an existing field of that name to the required template.

I did think about separating the metadata and content - went with the lazy option to start with until I get some feedback on what changes to functionality might be good.

At the moment, the first item is the parent, parent template and child template info. The second is the field info. The rest are the data.

  • Like 1

Share this post


Link to post
Share on other sites

So theoretically I could have one parent and one child page with all my fields setup, export it, and and then use this to import a field setup?

Share this post


Link to post
Share on other sites

Interesting thought Marty - I hadn't considered that at all, but I don't see why not :)

Of course it would mean all the fields would initially be added to the template of that child page, but I guess no big deal.

I wonder whether it would be worth taking it beyond its original goal and give it the ability to export all levels of children, grandchildren, great grandchildren etc, so an entire site of pages, templates and fields could be migrated. Obviously there are lots of issues to make this work, and is not why I built this. If I tried for that functionality, it would still fall short of the Profile Exporter, because it wouldn't support copying of template and other files, but it might still be a useful tool.

  • Like 1

Share this post


Link to post
Share on other sites

My mind works in mysterious ways :) Not every site I build is the same, there are always subtle differences. For things like blogs and galleries being able to export a basic page heirarchy (& fieldset) and recreate it on a new site build would be a huge, huge time saver.

  • Like 4

Share this post


Link to post
Share on other sites

:) I agree I think this could be cool. The profile exporter is great if you want to copy an entire setup, but it would be great for example to be able to "install" the pages/templates/fields for an image gallery from an existing site onto a new site with a couple of clicks at any point in the development of the site (unlike the site exporter where it is the entire site and you need to do it at the start).

Perhaps I could also add an option to just export/import the page/template/ field structure without the data!

I'll have to experiment with this in the next few days.

Thanks for the feedback

  • Like 3

Share this post


Link to post
Share on other sites

Looks cool, thanks for your work here. We got this discussion already to export and import template/fields and I think Ryan said we could build something with json or yaml. I'm not sure anyone else has started something but this could be a start.

Just briefly looking at the code it looks good. I'm not sure I understand why you have functions inside a class method? I'm sure it works but seems odd.

Also I'm not sure you are aware you can get template and fields with their settings with $field->getArray() $template->getArray() that will return an array with the properties. This might can be used to store them, though not sure it would be of help.

wire("templates")->get("basic-page")->getArray();

Would give something like:

Array
(
    [useRoles] => 1
    [editRoles] => Array
        (
            [0] => 1006
            [1] => 1675
        )

    [addRoles] => Array
        (
            [0] => 1006
            [1] => 1675
        )

    [createRoles] => Array
        (
            [0] => 1006
            [1] => 1675
        )

    [childrenTemplatesID] => 0
    [sortfield] => 
    [noChildren] => 
    [noParents] => 
    [childTemplates] => Array
        (
        )

    [parentTemplates] => Array
        (
        )

    [allowPageNum] => 1
    [allowChangeUser] => 0
    [redirectLogin] => 0
    [urlSegments] => 1
    [https] => 0
    [slashUrls] => 1
    [altFilename] => 
    [guestSearchable] => 0
    [pageClass] => 
    [pageLabelField] => 
    [noGlobal] => 0
    [noMove] => 0
    [noTrash] => 0
    [noSettings] => 0
    [noChangeTemplate] => 0
    [noUnpublish] => 0
    [nameContentTab] => 0
    [noCacheGetVars] => 
    [noCachePostVars] => 
    [useCacheForUsers] => 0
    [cacheExpire] => 0
    [cacheExpirePages] => Array
        (
        )

    [label] => 
    [tags] => 
    [roles] => Array
        (
            [0] => 37
            [1] => 1006
            [2] => 1675
        )

)

Also this is possible

wire("templates")->get("basic-page")->fieldgroup->getArray();
wire("fields")->get("body")->getArray();

Not sure how about template context settings. Anyway it would be a lot more work if you want to cover all settings.

  • Like 1

Share this post


Link to post
Share on other sites

Hey Soma,

Thanks for all the input. I have seen some discussion about export/import and I understand the challenges. I have also had a look through Antti's ProcessDataImport module. When I put this together I really wanted it to just handle simple page lists, so I wasn't planning on worrying about all the context settings and other issues that are a more challenging prospect.

Thanks for spotting the functions inside the method - this really was a quick and dirty initial effort designed to quickly copy some page lists to a new site I am working on. I'll make a note to reorganize those.

I remember seeing getArray in the cheatsheet some time ago, but didn't think about it for this use case at all. Could be very handy in making this module more capable - thank you! Seems like the only superfluous information are the roles settings - might be easiest to remove these from the array before json_encode'ing.

I guess I am trying to figure out how far to take this:

  1. It seems like it should be quite easy using getArray to grab most of the settings and iterate through them to set them on the new site. Are all of those settable via the API? - noGlobal etc?

    I think I should add support for all levels of children, grandchildren etc. 

    Plus an option to export/import structure (templates and fields and category pages) only - ie don't migrate the lowest level child pages - using a gallery example:

    Gallery
    --Nature
    ----Image 1
    ----Image 2
    --People
    --Kids

    It would migrate all the templates and fields for "Gallery" and all levels of nested pages, but it wouldn't migrate the Image pages themselves if "structure only" setting was checked.

     I think those enhancements would facilitate Marty's thoughts for migrating and setting up a full page hierarchy with templates and fields for a gallery etc.  
     
  2. Or perhaps it wouldn't be a big deal to also download any template files as well and include those in a zip along with the JSON file. The catch there might be permissions issues on the templates folder as this won't be writeable by default on most setups. Copying files from assets/files should be easy enough, but could result in a huge download package and also it might become interesting trying to reconnect these files to their appropriate new pages_id

I think for now at least, I'll aim for option 1. It seems like option 2 is something best tackled by Ryan as I am sure there are some big picture considerations that I'll miss at the moment. It would also become difficult deciding where to stop. For example I don't think a module like this should ever tackle migrating template access roles - I think it would be logical that these sorts of setting would be best redefined on the new site.  

Marty - can you think of anything I am overlooking in Option 1 for making this work for your idea?

Share this post


Link to post
Share on other sites

I have started a GitHub repo for Page Tree List JSON files: https://github.com/adrianbj/ProcessWirePageLists

I will add new ones as the need arises for me.

Still not sure if this is the best place for these. I am tempted to set up a simple site to house them that hooks into the Github API like the PW modules page.

Will see once I get an idea if people will actually use these or not. Might be more useful once I have taken care of some of the ideas in the above post.

Share this post


Link to post
Share on other sites

Thought I'd step in and show support for this - love the idea as a whole and Marty's hit upon an interesting use case there right away for easily setting up small sections of repetitive content that is a bit more modular perhaps than a site profile and useful when such things are outside the scope of a module themselves.

@Soma - I need to pay more attention to the cheatsheet as I'd not seen getArray() - would've been incredibly useful when I was debugging something the other day and print()'ing a page object... which causes a mess on the screen :D

  • Like 2

Share this post


Link to post
Share on other sites

Adrian I don't really have any more thoughts to add. A structure-only option sounds great. I can see this module being used in conjunction with the ImportPagesCSV and Wanze's Batcher a fair bit.

Cheers

Marty

Share this post


Link to post
Share on other sites

Thanks Marty,

I just added the structure only option and pushed to Github. At the moment it still creates the parent page, but not the child pages.  I am hoping to spend some time now looking into importing all levels of children which will be needed in some use cases, so I think I will need to change the structure behaviour to have three options - 1) all pages, 2) no pages at all, and 3) all levels of pages, except the very lowest level child pages so that categories and other levels are be created if wanted, without population of the actual content pages. Hope that makes sense.

I guess it will be an experiment to see what I think will be most useful / versatile for people.

  • Like 1

Share this post


Link to post
Share on other sites

Hey Marty - sorry for the confusion. You need to choose one of those options - obviously I need to come up with a better name. 

Save will spawn a download of the JSON file. Copy will output it on the screen so you can copy it.

I assume you have chosen a Parent page and structure option as well? I thought I had made them all required fields, but maybe not - will check.

I am getting close to having multiple levels of children etc working too - getting exciting :)

Share this post


Link to post
Share on other sites

Yeah I followed the steps. I chose a top level page with a few children then Structure only and Save. I tried all options actually and I only got an empty file.

Exciting is an understatement... :)

Share this post


Link to post
Share on other sites

Weird!

Would you mind trying to import that countries.json file for me and see if that works. If it does, could you then try to export that countries page tree and see if that works. 

I am running a fairly recent dev version, but I don't think I am using anything particularly new.

Sorry you are the guinea pig on this :)

Share this post


Link to post
Share on other sites

I imported the US States json file perfectly. I just can't seem to export. Perhaps it's my server setup. I'll give it a go locally at some point.

Share this post


Link to post
Share on other sites

Thanks Marty - strange the export is not working for you. Would definitely love to hear how it goes locally for you. Also could you give me a rundown of your server setup?

Do you have debug on? Any php errors?

I should be finished with the next round of functionality tomorrow - changing quite a few things, so we'll see if that helps as well. 

If not, hopefully someone else can confirm whether it is working for them or not and we can go from there.

Share this post


Link to post
Share on other sites

OK, I have a new version ready, with a new name: Page Tree Migrator

This new version has a new Github page: https://github.com/adrianbj/ProcessPageTreeMigrator

It can still handle simple "page list" imports and I have update the format of the countries.json and us-states.json files at: https://github.com/adrianbj/ProcessWirePageLists

WARNING: It should still be considered alpha at the moment, so please be careful !!

You will now need this new module to be able to import those files.

Whats' New

  1. Ability to import any page tree with infinite levels of children
  2. Templates are now created with all (I think?) settings from the original setup (including Family child and parent templates allowed).
  3. Three export options: 
        - Everything, including all data pages
        - Field, templates, and structural pages
        - Fields and templates only

    I am not really happy with the middle one yet (see Road Map for more details), but I think the other two are very useful.

Road Map

  1. Testing - It has been working great for me so far, but I am sure you guys will find some problems - I know it is far from perfect :)
  2. Code cleanup - at minimum I want to rename some variables etc to be more logical and add some detailed commenting. I am sure someone with more PW experience will see a mess in how I have achieved certain things. I know of a few things already that I want to redo, but for now I just want to get it out for testing :)
  3. Add more options for the "Field, templates, and structural pages" export option. At the moment this exports the templates, fields, and all pages, except for the final child level. My logic was geared towards something like a gallery with this structure:

    Gallery
    --Nature
    ----Image 1
    ----Image 2
    --People
    --Animals

    So, with this export option, you would only export the Gallery parent page, and the category pages (nature, people, animals). The image grandchildren won't be exported so you can use this to start with a clean gallery page tree.

    However, I think I should add more options that allow the user to select which levels of children to export and even which fields within these pages. That way you could choose to only export the title field for an entire set of pages, leaving the rest of the fields empty. Not sure of the most useful approach here. Any thoughts?
  4. Deal with any potential issues that occur when importing over an existing template if existing settings are different.
  5. What else do you guys want to see?
  • Like 2

Share this post


Link to post
Share on other sites

Just made a couple of quick updates on Github to v0.0.3 now to fix some path issues for the main parent in some situations. Hopefully fixed now, but we'll see :)

Off to sleep now - probably some more bug finding tomorrow :)

EDIT: Marty - let me know if this version works for you. 

Share this post


Link to post
Share on other sites

This looks even better. Import doesn't seem to work on my test site though, seems to some incompatibility with multilang features. This comes after using paste import:

Catchable fatal error: Object of class stdClass could not be converted to string in /srv/www/pw/frontendedit/wire/modules/LanguageSupport/FieldtypeTextLanguage.module on line 38

I am running PW 2.3.0 on that site.

Share this post


Link to post
Share on other sites

Thanks for finding that Antti, although I am having trouble reproducing it. Unfortunately I am pretty ignorant of the multilang features - typical English speaker I'm afraid :)

I just installed the Language Support (Core) module and ran a successful import without any errors.

I am running a fairly recent dev version so I wonder if that is the difference? Or maybe I actually need to install a language pack and set up an alternate language field to trigger the error. I'll keep investigating, but do you have any ideas?

BTW, I just fixed a problem with the country and state list files - my subconscious must have realized and woke me up (it's early here) - I should learn never to release code late at night :) This change won't be causing your error though.

Share this post


Link to post
Share on other sites

Tested it little bit more - it probably has something to do with the data structure I am trying to import:

{
    "pages": [
        {
            "name": "basicpage",
            "parent_name": "",
            "page_template": "basic-page",
            "status": 1,
            "sort": 7,
            "data": {
                "title": {
                    "data": "basicpage",
                    "data1029": "basicpage"
                },
                "headline": "",
                "summary": "",
                "body": {
                    "data": "",
                    "data1029": ""
                },
                "sidebar": "",
                "images": [],
                "related": [],
                "repeater": {
                    "data": "",
                    "count": 0,
                    "parent_id": 1052
                },
                "commentsSpecial": [
                    {
                        "id": 74,
                        "status": 1,
                        "data": "Hi there!",
                        "cite": "Antsa",
                        "email": "xxxx.xxxxx@xxxxxx.com",
                        "created": 1370721469,
                        "created_users_id": 41,
                        "ip": "91.156.11.190",
                        "user_agent": "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/27.0.1453.110 Safari/537.36",
                        "website": ""
                    }
                ],
                "thumbnail": []
            }
        },
        {
            "name": "templates",
            "parent_name": "basicpage/",
            "page_template": "basic-page",
            "status": 1,
            "sort": 0,
            "data": {
                "title": {
                    "data": "",
                    "data1029": ""
                },
                "headline": "",
                "summary": "",
                "body": {
                    "data": "",
                    "data1029": ""
                },
                "sidebar": "",
                "images": [],
                "related": [],
                "repeater": {
                    "data": "",
                    "count": 0,
                    "parent_id": 1059
                },
                "commentsSpecial": [],
                "thumbnail": []
            }
        }
    ],
    "templates": [
        {
            "template": "basic-page",
            "data": {
                "useRoles": 0,
                "childrenTemplatesID": 0,
                "sortfield": "",
                "noChildren": "",
                "noParents": "",
                "childTemplates": [],
                "parentTemplates": [],
                "allowPageNum": 0,
                "allowChangeUser": 0,
                "redirectLogin": 0,
                "urlSegments": 0,
                "https": 0,
                "slashUrls": 1,
                "altFilename": "",
                "guestSearchable": 0,
                "pageClass": "",
                "pageLabelField": "",
                "noGlobal": 0,
                "noMove": 0,
                "noTrash": 0,
                "noSettings": 0,
                "noChangeTemplate": 0,
                "noUnpublish": 0,
                "nameContentTab": 0,
                "noCacheGetVars": "",
                "noCachePostVars": "",
                "useCacheForUsers": 0,
                "cacheExpire": 0,
                "cacheExpirePages": [],
                "label": "",
                "tags": ""
            }
        }
    ],
    "fields": [
        {
            "name": "title",
            "label": "Title",
            "description": null,
            "template": "basic-page",
            "flags": 13,
            "type": "FieldtypePageTitleLanguage"
        },
        {
            "name": "headline",
            "label": "Headline",
            "description": "Use this instead of the Title if a longer headline is needed than what you want to appear in navigation.",
            "template": "basic-page",
            "flags": 0,
            "type": "FieldtypeText"
        },
        {
            "name": "summary",
            "label": "Summary",
            "description": null,
            "template": "basic-page",
            "flags": 1,
            "type": "FieldtypeTextarea"
        },
        {
            "name": "body",
            "label": "Body",
            "description": null,
            "template": "basic-page",
            "flags": 0,
            "type": "FieldtypeTextareaLanguage"
        },
        {
            "name": "sidebar",
            "label": "Sidebar",
            "description": null,
            "template": "basic-page",
            "flags": 0,
            "type": "FieldtypeTextarea"
        },
        {
            "name": "images",
            "label": "Images",
            "description": null,
            "template": "basic-page",
            "flags": 0,
            "type": "FieldtypeImage"
        },
        {
            "name": "related",
            "label": "Related pages",
            "description": null,
            "template": "basic-page",
            "flags": 0,
            "type": "FieldtypePage"
        },
        {
            "name": "repeater",
            "label": "Repeater",
            "description": null,
            "template": "basic-page",
            "flags": 0,
            "type": "FieldtypeRepeater"
        },
        {
            "name": "commentsSpecial",
            "label": "Comments",
            "description": null,
            "template": "basic-page",
            "flags": 0,
            "type": "FieldtypeComments"
        },
        {
            "name": "thumbnail",
            "label": "Image",
            "description": null,
            "template": "basic-page",
            "flags": 0,
            "type": "FieldtypeImage"
        }
    ]
}

If I export/import structure only, it works great!

Share this post


Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.


  • Recently Browsing   0 members

    No registered users viewing this page.

  • Similar Content

    • By gebeer
      I am happy to present my new fieldtype FieldtypeImageFromPage. It is made up of 2 modules:
      Fieldtype Image Reference From Another Page is a Fieldtype that stores a reference to a single image from another page. The image can be selected with the associated Inputfield.
      Inputfield Select Image From Page is an Inputfield to select a single image from images on a predefined page and it's children.
      And there also is a helper module that takes care of cleanup tasks.
      This module evolved out of a discussion about my other Module FieldtypeImagePicker.  It caters for use cases where a set of images is being reused multiple times across a site. With this fieldtype these images can be administered through a chosen page. All images uploaded to that page will be available in the inputfield.
      When to use ?
      Let editors choose an image from a set of images that is being used site-wide. Ideal for images that are being re-used across the site.
      Suited for images that are used on multiple pages throughout the site (e.g. icons).
      Other than the native ProcessWire images field, the images here are not stored per page. Only references to images on another page are stored. This has several advantages:
      one central place to organize images when images change, you only have to update them in one place. All references will be updated, too. (Provided the name of the image that has changed stays the same) Features
      Images can be manipulated like native ProcessWire images (resizing, cropping etc.) Image names are fully searchable through the API Accidental image deletion is prevented. When you want to delete an image from one of the pages that hold your site-wide images, the module searches all pages that use that image. If any page contains a reference to the image you are trying to delete, deletion will be prevented. You will get an error message to help you edit those pages and remove references there before you can finally delete the image. How to install and setup
      Download and install this module like any other modules in ProcessWire Create a page in the page tree that will hold your images. This page's template must have an images field Upload some images to the page you created in step 2 Create a new field. As type choose 'Image Reference From Another Page'. Save the field. In 'Details' Tab of the field choose the page you created in step 2 Click Save button Choose the images field name for the field that holds your images (on page template from step 2) Click Save button again Choose whether you want to include child pages of page from step 2 to supply images Add the field to any template You are now ready to use the field View of the inputfield on the page edit screen:

      View of the field settings

      The module can be installed from this github repo. Some more info in the README there, too.
      In my tests it was fairly stable. After receiving your valued feedback, I will eventually add it to the modules directory.
      My ideas for further improvement:
      - add ajax loading of thumbnails
      Happy to hear your feedback!
       
    • By gebeer
      Although the PW backend is really intuitive, ever so often my clients need some assistance. Be it they are not so tech savvy or they are not working in the backend often.
      For those cases it is nice to make some help videos available to editors. This is what this module does.
      ProcessHelpVideos Module
      A Process module to display help videos for the ProcessWire CMS. It can be used to make help videos (screencasts) available to content editors.
      This module adds a 'Help Videos" section to the ProcessWire backend. The help videos are accessible through an automatically created page in the Admin page tree. You can add your help videos as pages in the page tree. The module adds a hidden page to the page tree that acts as parent page for the help video pages. All necessary fields and templates will be installed automatically. If there are already a CKEditor field and/or a file field for mp4 files installed in the system, the module will use those. Otherwise it will create the necessary fields. Also the necessary templates for the parent help videos page and it's children are created on module install. The module installs a permission process-helpvideos. Every user role that should have access to the help video section, needs this permission. I use the help video approach on quite a few production sites. It is stable so far and well received by site owners/editors. Up until now I installed required fields, templates and pages manually and then added the module. Now I added all this logic to the install method of the module and it should be ready to share.
      The module and further description on how to use it is available on github: https://github.com/gebeer/ProcessHelpVideos
      If you like to give it a try, I am happy to receive your comments/suggestions here.
    • By Robin S
      A module created in response to the topic here:
      Page List Select Multiple Quickly
      Modifies PageListSelectMultiple to allow you to select multiple pages without the tree closing every time you select a page.
      The screencast says it all:

       
      https://github.com/Toutouwai/PageListSelectMultipleQuickly
      https://modules.processwire.com/modules/page-list-select-multiple-quickly/
    • By gebeer
      Hello all,
      sharing my new module FieldtypeImagePicker. It provides a configurable input field for choosing any type of image from a predefined folder.
      The need for it came up because a client had a custom SVG icon set and I wanted the editors to be able to choose an icon in the page editor.
      It can also be used to offer a choice of images that are used site-wide without having to upload them to individual pages.
      There are no image manipulation methods like with the native PW image field.
      Module and full description can be found on github https://github.com/gebeer/FieldtypeImagePicker
      Kudos to @Martijn Geerts. I used his module FieldTypeSelectFile as a base to build upon.
      Here's how the input field looks like in the page editor:

      Hope it can be of use to someone.
      If you like to give it a try, I'm happy to hear your comments or suggestions for improvement. Eventually this will go in the module directory soon, too.
    • By bernhard
      @Sergio asked about the pdf creation process in the showcase thread about my 360° feedback/survey tool and so I went ahead and set my little pdf helper module to public.
      Description from PW Weekly:
       
      Modules Directory: https://modules.processwire.com/modules/rock-pdf/
      Download & Docs: https://github.com/BernhardBaumrock/RockPDF
       
      You can combine it easily with RockReplacer: 
      See also a little showcase of the RockPdf module in this thread:
       
×
×
  • Create New...