Module Migrator

[adrian]

GitHub: https://github.com/adrianbj/ProcessMigrator

 

This module has gone through lots of iterations with lots of new functionality each time. It is now a fully fledged content migration tool.

 

*** Please be sure to read the GitHub ReadMe to find out what it can do now as most of the posts in this thread are no longer correct regarding its functionality

 

 

Once it is release worthy, I'll create a fresh thread with all the details.

 

 

This modules allows export, sharing, and import of page lists via JSON files.

 

It takes care of replicating all the pages, as well as creating any templates and fields that are needed.

 

I have defined "Page Lists" as page trees (parent and children) that store selector values for a Page fieldtype. An example would be a list of countries that would be used to populate a countries drop-down select field. The fields might include: Country Name, 2-digit code, 3-digit code, number code.

 

I would like to suggest a place where we can post json files to be shared and updated - maybe a dedicated github repository?

Start of a repo of lists ready to import is now available: https://github.com/adrianbj/ProcessWirePageLists

 

It might handle migrating other simple pages trees as well, but it should not be considered a tool for migrating general pages as it does not handle associated files, nor does it handle fields which store arrays. Probably lots of other things it doesn't handle either

 

It now handles migrating all (I think) field types, including repeater fields, page fields, all Profields fields, multi-language versions of fields etc. The only omission is the actual uploaded files and images in file/image fields.

 

WARNING: This should be considered an Alpha module - please don't use this on a live site at the moment and be sure to back everything up before testing.

 

 

Would appreciate any feedback on the concept, the code, and the idea of a shared and community edited resource of these files.

 

Also, would love to hear what page lists would be good to share. Here are a few quick ideas:

• States (separate files for each country)
  
• Measurement units
  
• Languages
  
• Religions
  
• Race
  
• Academic subjects (chemistry, biology etc)
  
• Publication types (book, journal article, newspaper article, newsletter, thesis etc)
  
• Car makes and models
  

Anyone have a better idea for a name, or how to better describe "Page Lists"?

Edited April 18, 2015 by Nico Knoll
Added the "module" tag.

[apeisa]

Very interesting!

Does it create the fields also when importing (those are defined in JSON, so that made me assume)? I would also think about JSON schema - should metadata and content data be in their own containers? Something like "meta" and "data"?

[adrian]

Yep,

It creates all required templates and fields using the settings from the original version. It won't re-create or modify a field if it is already existing, but it will add an existing field of that name to the required template.

I did think about separating the metadata and content - went with the lazy option to start with until I get some feedback on what changes to functionality might be good.

At the moment, the first item is the parent, parent template and child template info. The second is the field info. The rest are the data.

[Marty Walker]

So theoretically I could have one parent and one child page with all my fields setup, export it, and and then use this to import a field setup?

[adrian]

Interesting thought Marty - I hadn't considered that at all, but I don't see why not

Of course it would mean all the fields would initially be added to the template of that child page, but I guess no big deal.

I wonder whether it would be worth taking it beyond its original goal and give it the ability to export all levels of children, grandchildren, great grandchildren etc, so an entire site of pages, templates and fields could be migrated. Obviously there are lots of issues to make this work, and is not why I built this. If I tried for that functionality, it would still fall short of the Profile Exporter, because it wouldn't support copying of template and other files, but it might still be a useful tool.

[Marty Walker]

My mind works in mysterious ways Not every site I build is the same, there are always subtle differences. For things like blogs and galleries being able to export a basic page heirarchy (& fieldset) and recreate it on a new site build would be a huge, huge time saver.

[adrian]

I agree I think this could be cool. The profile exporter is great if you want to copy an entire setup, but it would be great for example to be able to "install" the pages/templates/fields for an image gallery from an existing site onto a new site with a couple of clicks at any point in the development of the site (unlike the site exporter where it is the entire site and you need to do it at the start).

Perhaps I could also add an option to just export/import the page/template/ field structure without the data!

I'll have to experiment with this in the next few days.

Thanks for the feedback

[Soma]

Looks cool, thanks for your work here. We got this discussion already to export and import template/fields and I think Ryan said we could build something with json or yaml. I'm not sure anyone else has started something but this could be a start.

Just briefly looking at the code it looks good. I'm not sure I understand why you have functions inside a class method? I'm sure it works but seems odd.

Also I'm not sure you are aware you can get template and fields with their settings with $field->getArray() $template->getArray() that will return an array with the properties. This might can be used to store them, though not sure it would be of help.

wire("templates")->get("basic-page")->getArray();

Would give something like:

Array
(
    [useRoles] => 1
    [editRoles] => Array
        (
            [0] => 1006
            [1] => 1675
        )

    [addRoles] => Array
        (
            [0] => 1006
            [1] => 1675
        )

    [createRoles] => Array
        (
            [0] => 1006
            [1] => 1675
        )

    [childrenTemplatesID] => 0
    [sortfield] => 
    [noChildren] => 
    [noParents] => 
    [childTemplates] => Array
        (
        )

    [parentTemplates] => Array
        (
        )

    [allowPageNum] => 1
    [allowChangeUser] => 0
    [redirectLogin] => 0
    [urlSegments] => 1
    [https] => 0
    [slashUrls] => 1
    [altFilename] => 
    [guestSearchable] => 0
    [pageClass] => 
    [pageLabelField] => 
    [noGlobal] => 0
    [noMove] => 0
    [noTrash] => 0
    [noSettings] => 0
    [noChangeTemplate] => 0
    [noUnpublish] => 0
    [nameContentTab] => 0
    [noCacheGetVars] => 
    [noCachePostVars] => 
    [useCacheForUsers] => 0
    [cacheExpire] => 0
    [cacheExpirePages] => Array
        (
        )

    [label] => 
    [tags] => 
    [roles] => Array
        (
            [0] => 37
            [1] => 1006
            [2] => 1675
        )

)

Also this is possible

wire("templates")->get("basic-page")->fieldgroup->getArray();
wire("fields")->get("body")->getArray();

Not sure how about template context settings. Anyway it would be a lot more work if you want to cover all settings.

[adrian]

Hey Soma,

Thanks for all the input. I have seen some discussion about export/import and I understand the challenges. I have also had a look through Antti's ProcessDataImport module. When I put this together I really wanted it to just handle simple page lists, so I wasn't planning on worrying about all the context settings and other issues that are a more challenging prospect.

Thanks for spotting the functions inside the method - this really was a quick and dirty initial effort designed to quickly copy some page lists to a new site I am working on. I'll make a note to reorganize those.

I remember seeing getArray in the cheatsheet some time ago, but didn't think about it for this use case at all. Could be very handy in making this module more capable - thank you! Seems like the only superfluous information are the roles settings - might be easiest to remove these from the array before json_encode'ing.

I guess I am trying to figure out how far to take this:

1. It seems like it should be quite easy using getArray to grab most of the settings and iterate through them to set them on the new site. Are all of those settable via the API? - noGlobal etc?
   
   I think I should add support for all levels of children, grandchildren etc. 
   
   Plus an option to export/import structure (templates and fields and category pages) only - ie don't migrate the lowest level child pages - using a gallery example:
   
   Gallery
   --Nature
   ----Image 1
   ----Image 2
   --People
   --Kids
   
   It would migrate all the templates and fields for "Gallery" and all levels of nested pages, but it wouldn't migrate the Image pages themselves if "structure only" setting was checked.
   
    I think those enhancements would facilitate Marty's thoughts for migrating and setting up a full page hierarchy with templates and fields for a gallery etc.  
    
2. Or perhaps it wouldn't be a big deal to also download any template files as well and include those in a zip along with the JSON file. The catch there might be permissions issues on the templates folder as this won't be writeable by default on most setups. Copying files from assets/files should be easy enough, but could result in a huge download package and also it might become interesting trying to reconnect these files to their appropriate new pages_id

I think for now at least, I'll aim for option 1. It seems like option 2 is something best tackled by Ryan as I am sure there are some big picture considerations that I'll miss at the moment. It would also become difficult deciding where to stop. For example I don't think a module like this should ever tackle migrating template access roles - I think it would be logical that these sorts of setting would be best redefined on the new site.  

Marty - can you think of anything I am overlooking in Option 1 for making this work for your idea?

[adrian]

I have started a GitHub repo for Page Tree List JSON files: https://github.com/adrianbj/ProcessWirePageLists

I will add new ones as the need arises for me.

Still not sure if this is the best place for these. I am tempted to set up a simple site to house them that hooks into the Github API like the PW modules page.

Will see once I get an idea if people will actually use these or not. Might be more useful once I have taken care of some of the ideas in the above post.

[Pete]

Thought I'd step in and show support for this - love the idea as a whole and Marty's hit upon an interesting use case there right away for easily setting up small sections of repetitive content that is a bit more modular perhaps than a site profile and useful when such things are outside the scope of a module themselves.

@Soma - I need to pay more attention to the cheatsheet as I'd not seen getArray() - would've been incredibly useful when I was debugging something the other day and print()'ing a page object... which causes a mess on the screen

[Marty Walker]

Adrian I don't really have any more thoughts to add. A structure-only option sounds great. I can see this module being used in conjunction with the ImportPagesCSV and Wanze's Batcher a fair bit.

Cheers

Marty

[adrian]

Thanks Marty,

I just added the structure only option and pushed to Github. At the moment it still creates the parent page, but not the child pages.  I am hoping to spend some time now looking into importing all levels of children which will be needed in some use cases, so I think I will need to change the structure behaviour to have three options - 1) all pages, 2) no pages at all, and 3) all levels of pages, except the very lowest level child pages so that categories and other levels are be created if wanted, without population of the actual content pages. Hope that makes sense.

I guess it will be an experiment to see what I think will be most useful / versatile for people.

[Marty Walker]

Perhaps it's my PW version (2.3) but all I get in the json.json file is [ ]. What's does Save &/or Copy do?

[adrian]

Hey Marty - sorry for the confusion. You need to choose one of those options - obviously I need to come up with a better name. 

Save will spawn a download of the JSON file. Copy will output it on the screen so you can copy it.

I assume you have chosen a Parent page and structure option as well? I thought I had made them all required fields, but maybe not - will check.

I am getting close to having multiple levels of children etc working too - getting exciting

[Marty Walker]

Yeah I followed the steps. I chose a top level page with a few children then Structure only and Save. I tried all options actually and I only got an empty file.

Exciting is an understatement...

[adrian]

Weird!

Would you mind trying to import that countries.json file for me and see if that works. If it does, could you then try to export that countries page tree and see if that works. 

I am running a fairly recent dev version, but I don't think I am using anything particularly new.

Sorry you are the guinea pig on this

[Marty Walker]

Happy to test this out. I'll let you know how I get on.

[Marty Walker]

I imported the US States json file perfectly. I just can't seem to export. Perhaps it's my server setup. I'll give it a go locally at some point.

[adrian]

Thanks Marty - strange the export is not working for you. Would definitely love to hear how it goes locally for you. Also could you give me a rundown of your server setup?

Do you have debug on? Any php errors?

I should be finished with the next round of functionality tomorrow - changing quite a few things, so we'll see if that helps as well. 

If not, hopefully someone else can confirm whether it is working for them or not and we can go from there.

[adrian]

OK, I have a new version ready, with a new name: Page Tree Migrator

This new version has a new Github page: https://github.com/adrianbj/ProcessPageTreeMigrator

It can still handle simple "page list" imports and I have update the format of the countries.json and us-states.json files at: https://github.com/adrianbj/ProcessWirePageLists

WARNING: It should still be considered alpha at the moment, so please be careful !!

You will now need this new module to be able to import those files.

Whats' New

1. Ability to import any page tree with infinite levels of children
2. Templates are now created with all (I think?) settings from the original setup (including Family child and parent templates allowed).
3. Three export options: 
       - Everything, including all data pages
       - Field, templates, and structural pages
       - Fields and templates only
   
   I am not really happy with the middle one yet (see Road Map for more details), but I think the other two are very useful.

Road Map

1. Testing - It has been working great for me so far, but I am sure you guys will find some problems - I know it is far from perfect
2. Code cleanup - at minimum I want to rename some variables etc to be more logical and add some detailed commenting. I am sure someone with more PW experience will see a mess in how I have achieved certain things. I know of a few things already that I want to redo, but for now I just want to get it out for testing
3. Add more options for the "Field, templates, and structural pages" export option. At the moment this exports the templates, fields, and all pages, except for the final child level. My logic was geared towards something like a gallery with this structure:
   
   
   Gallery
   --Nature
   ----Image 1
   ----Image 2
   --People
   --Animals
   
   So, with this export option, you would only export the Gallery parent page, and the category pages (nature, people, animals). The image grandchildren won't be exported so you can use this to start with a clean gallery page tree.
   
   However, I think I should add more options that allow the user to select which levels of children to export and even which fields within these pages. That way you could choose to only export the title field for an entire set of pages, leaving the rest of the fields empty. Not sure of the most useful approach here. Any thoughts?
4. Deal with any potential issues that occur when importing over an existing template if existing settings are different.
5. What else do you guys want to see?

[adrian]

Just made a couple of quick updates on Github to v0.0.3 now to fix some path issues for the main parent in some situations. Hopefully fixed now, but we'll see

Off to sleep now - probably some more bug finding tomorrow

EDIT: Marty - let me know if this version works for you. 

[apeisa]

This looks even better. Import doesn't seem to work on my test site though, seems to some incompatibility with multilang features. This comes after using paste import:

Catchable fatal error: Object of class stdClass could not be converted to string in /srv/www/pw/frontendedit/wire/modules/LanguageSupport/FieldtypeTextLanguage.module on line 38

I am running PW 2.3.0 on that site.

[adrian]

Thanks for finding that Antti, although I am having trouble reproducing it. Unfortunately I am pretty ignorant of the multilang features - typical English speaker I'm afraid

I just installed the Language Support (Core) module and ran a successful import without any errors.

I am running a fairly recent dev version so I wonder if that is the difference? Or maybe I actually need to install a language pack and set up an alternate language field to trigger the error. I'll keep investigating, but do you have any ideas?

BTW, I just fixed a problem with the country and state list files - my subconscious must have realized and woke me up (it's early here) - I should learn never to release code late at night This change won't be causing your error though.

[apeisa]

Tested it little bit more - it probably has something to do with the data structure I am trying to import:

{
    "pages": [
        {
            "name": "basicpage",
            "parent_name": "",
            "page_template": "basic-page",
            "status": 1,
            "sort": 7,
            "data": {
                "title": {
                    "data": "basicpage",
                    "data1029": "basicpage"
                },
                "headline": "",
                "summary": "",
                "body": {
                    "data": "",
                    "data1029": ""
                },
                "sidebar": "",
                "images": [],
                "related": [],
                "repeater": {
                    "data": "",
                    "count": 0,
                    "parent_id": 1052
                },
                "commentsSpecial": [
                    {
                        "id": 74,
                        "status": 1,
                        "data": "Hi there!",
                        "cite": "Antsa",
                        "email": "xxxx.xxxxx@xxxxxx.com",
                        "created": 1370721469,
                        "created_users_id": 41,
                        "ip": "91.156.11.190",
                        "user_agent": "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/27.0.1453.110 Safari/537.36",
                        "website": ""
                    }
                ],
                "thumbnail": []
            }
        },
        {
            "name": "templates",
            "parent_name": "basicpage/",
            "page_template": "basic-page",
            "status": 1,
            "sort": 0,
            "data": {
                "title": {
                    "data": "",
                    "data1029": ""
                },
                "headline": "",
                "summary": "",
                "body": {
                    "data": "",
                    "data1029": ""
                },
                "sidebar": "",
                "images": [],
                "related": [],
                "repeater": {
                    "data": "",
                    "count": 0,
                    "parent_id": 1059
                },
                "commentsSpecial": [],
                "thumbnail": []
            }
        }
    ],
    "templates": [
        {
            "template": "basic-page",
            "data": {
                "useRoles": 0,
                "childrenTemplatesID": 0,
                "sortfield": "",
                "noChildren": "",
                "noParents": "",
                "childTemplates": [],
                "parentTemplates": [],
                "allowPageNum": 0,
                "allowChangeUser": 0,
                "redirectLogin": 0,
                "urlSegments": 0,
                "https": 0,
                "slashUrls": 1,
                "altFilename": "",
                "guestSearchable": 0,
                "pageClass": "",
                "pageLabelField": "",
                "noGlobal": 0,
                "noMove": 0,
                "noTrash": 0,
                "noSettings": 0,
                "noChangeTemplate": 0,
                "noUnpublish": 0,
                "nameContentTab": 0,
                "noCacheGetVars": "",
                "noCachePostVars": "",
                "useCacheForUsers": 0,
                "cacheExpire": 0,
                "cacheExpirePages": [],
                "label": "",
                "tags": ""
            }
        }
    ],
    "fields": [
        {
            "name": "title",
            "label": "Title",
            "description": null,
            "template": "basic-page",
            "flags": 13,
            "type": "FieldtypePageTitleLanguage"
        },
        {
            "name": "headline",
            "label": "Headline",
            "description": "Use this instead of the Title if a longer headline is needed than what you want to appear in navigation.",
            "template": "basic-page",
            "flags": 0,
            "type": "FieldtypeText"
        },
        {
            "name": "summary",
            "label": "Summary",
            "description": null,
            "template": "basic-page",
            "flags": 1,
            "type": "FieldtypeTextarea"
        },
        {
            "name": "body",
            "label": "Body",
            "description": null,
            "template": "basic-page",
            "flags": 0,
            "type": "FieldtypeTextareaLanguage"
        },
        {
            "name": "sidebar",
            "label": "Sidebar",
            "description": null,
            "template": "basic-page",
            "flags": 0,
            "type": "FieldtypeTextarea"
        },
        {
            "name": "images",
            "label": "Images",
            "description": null,
            "template": "basic-page",
            "flags": 0,
            "type": "FieldtypeImage"
        },
        {
            "name": "related",
            "label": "Related pages",
            "description": null,
            "template": "basic-page",
            "flags": 0,
            "type": "FieldtypePage"
        },
        {
            "name": "repeater",
            "label": "Repeater",
            "description": null,
            "template": "basic-page",
            "flags": 0,
            "type": "FieldtypeRepeater"
        },
        {
            "name": "commentsSpecial",
            "label": "Comments",
            "description": null,
            "template": "basic-page",
            "flags": 0,
            "type": "FieldtypeComments"
        },
        {
            "name": "thumbnail",
            "label": "Image",
            "description": null,
            "template": "basic-page",
            "flags": 0,
            "type": "FieldtypeImage"
        }
    ]
}

If I export/import structure only, it works great!