adrian

Module
Migrator

289 posts in this topic

GitHub: https://github.com/adrianbj/ProcessMigrator


 


This module has gone through lots of iterations with lots of new functionality each time. It is now a fully fledged content migration tool.


 


*** Please be sure to read the GitHub ReadMe to find out what it can do now as most of the posts in this thread are no longer correct regarding its functionality


 


 


Once it is release worthy, I'll create a fresh thread with all the details.


 


 


This modules allows export, sharing, and import of page lists via JSON files.


 


It takes care of replicating all the pages, as well as creating any templates and fields that are needed.


 


I have defined "Page Lists" as page trees (parent and children) that store selector values for a Page fieldtype. An example would be a list of countries that would be used to populate a countries drop-down select field. The fields might include: Country Name, 2-digit code, 3-digit code, number code.


 


I would like to suggest a place where we can post json files to be shared and updated - maybe a dedicated github repository?


Start of a repo of lists ready to import is now available: https://github.com/adrianbj/ProcessWirePageLists


 


It might handle migrating other simple pages trees as well, but it should not be considered a tool for migrating general pages as it does not handle associated files, nor does it handle fields which store arrays. Probably lots of other things it doesn't handle either :)


 


It now handles migrating all (I think) field types, including repeater fields, page fields, all Profields fields, multi-language versions of fields etc. The only omission is the actual uploaded files and images in file/image fields.


 


WARNING: This should be considered an Alpha module - please don't use this on a live site at the moment and be sure to back everything up before testing.


 


 


Would appreciate any feedback on the concept, the code, and the idea of a shared and community edited resource of these files.


 


Also, would love to hear what page lists would be good to share. Here are a few quick ideas:


  • States (separate files for each country)
  • Measurement units
  • Languages
  • Religions
  • Race
  • Academic subjects (chemistry, biology etc)
  • Publication types (book, journal article, newspaper article, newsletter, thesis etc)
  • Car makes and models

Anyone have a better idea for a name, or how to better describe "Page Lists"?


Edited by Nico Knoll
Added the "module" tag.
18 people like this

Share this post


Link to post
Share on other sites

Very interesting!

Does it create the fields also when importing (those are defined in JSON, so that made me assume)? I would also think about JSON schema - should metadata and content data be in their own containers? Something like "meta" and "data"?

Share this post


Link to post
Share on other sites

Yep,

It creates all required templates and fields using the settings from the original version. It won't re-create or modify a field if it is already existing, but it will add an existing field of that name to the required template.

I did think about separating the metadata and content - went with the lazy option to start with until I get some feedback on what changes to functionality might be good.

At the moment, the first item is the parent, parent template and child template info. The second is the field info. The rest are the data.

1 person likes this

Share this post


Link to post
Share on other sites

So theoretically I could have one parent and one child page with all my fields setup, export it, and and then use this to import a field setup?

Share this post


Link to post
Share on other sites

Interesting thought Marty - I hadn't considered that at all, but I don't see why not :)

Of course it would mean all the fields would initially be added to the template of that child page, but I guess no big deal.

I wonder whether it would be worth taking it beyond its original goal and give it the ability to export all levels of children, grandchildren, great grandchildren etc, so an entire site of pages, templates and fields could be migrated. Obviously there are lots of issues to make this work, and is not why I built this. If I tried for that functionality, it would still fall short of the Profile Exporter, because it wouldn't support copying of template and other files, but it might still be a useful tool.

1 person likes this

Share this post


Link to post
Share on other sites

My mind works in mysterious ways :) Not every site I build is the same, there are always subtle differences. For things like blogs and galleries being able to export a basic page heirarchy (& fieldset) and recreate it on a new site build would be a huge, huge time saver.

3 people like this

Share this post


Link to post
Share on other sites

:) I agree I think this could be cool. The profile exporter is great if you want to copy an entire setup, but it would be great for example to be able to "install" the pages/templates/fields for an image gallery from an existing site onto a new site with a couple of clicks at any point in the development of the site (unlike the site exporter where it is the entire site and you need to do it at the start).

Perhaps I could also add an option to just export/import the page/template/ field structure without the data!

I'll have to experiment with this in the next few days.

Thanks for the feedback

3 people like this

Share this post


Link to post
Share on other sites

Looks cool, thanks for your work here. We got this discussion already to export and import template/fields and I think Ryan said we could build something with json or yaml. I'm not sure anyone else has started something but this could be a start.

Just briefly looking at the code it looks good. I'm not sure I understand why you have functions inside a class method? I'm sure it works but seems odd.

Also I'm not sure you are aware you can get template and fields with their settings with $field->getArray() $template->getArray() that will return an array with the properties. This might can be used to store them, though not sure it would be of help.

wire("templates")->get("basic-page")->getArray();

Would give something like:

Array
(
    [useRoles] => 1
    [editRoles] => Array
        (
            [0] => 1006
            [1] => 1675
        )

    [addRoles] => Array
        (
            [0] => 1006
            [1] => 1675
        )

    [createRoles] => Array
        (
            [0] => 1006
            [1] => 1675
        )

    [childrenTemplatesID] => 0
    [sortfield] => 
    [noChildren] => 
    [noParents] => 
    [childTemplates] => Array
        (
        )

    [parentTemplates] => Array
        (
        )

    [allowPageNum] => 1
    [allowChangeUser] => 0
    [redirectLogin] => 0
    [urlSegments] => 1
    [https] => 0
    [slashUrls] => 1
    [altFilename] => 
    [guestSearchable] => 0
    [pageClass] => 
    [pageLabelField] => 
    [noGlobal] => 0
    [noMove] => 0
    [noTrash] => 0
    [noSettings] => 0
    [noChangeTemplate] => 0
    [noUnpublish] => 0
    [nameContentTab] => 0
    [noCacheGetVars] => 
    [noCachePostVars] => 
    [useCacheForUsers] => 0
    [cacheExpire] => 0
    [cacheExpirePages] => Array
        (
        )

    [label] => 
    [tags] => 
    [roles] => Array
        (
            [0] => 37
            [1] => 1006
            [2] => 1675
        )

)

Also this is possible

wire("templates")->get("basic-page")->fieldgroup->getArray();
wire("fields")->get("body")->getArray();

Not sure how about template context settings. Anyway it would be a lot more work if you want to cover all settings.

1 person likes this

Share this post


Link to post
Share on other sites

Hey Soma,

Thanks for all the input. I have seen some discussion about export/import and I understand the challenges. I have also had a look through Antti's ProcessDataImport module. When I put this together I really wanted it to just handle simple page lists, so I wasn't planning on worrying about all the context settings and other issues that are a more challenging prospect.

Thanks for spotting the functions inside the method - this really was a quick and dirty initial effort designed to quickly copy some page lists to a new site I am working on. I'll make a note to reorganize those.

I remember seeing getArray in the cheatsheet some time ago, but didn't think about it for this use case at all. Could be very handy in making this module more capable - thank you! Seems like the only superfluous information are the roles settings - might be easiest to remove these from the array before json_encode'ing.

I guess I am trying to figure out how far to take this:

  1. It seems like it should be quite easy using getArray to grab most of the settings and iterate through them to set them on the new site. Are all of those settable via the API? - noGlobal etc?

    I think I should add support for all levels of children, grandchildren etc. 

    Plus an option to export/import structure (templates and fields and category pages) only - ie don't migrate the lowest level child pages - using a gallery example:

    Gallery
    --Nature
    ----Image 1
    ----Image 2
    --People
    --Kids

    It would migrate all the templates and fields for "Gallery" and all levels of nested pages, but it wouldn't migrate the Image pages themselves if "structure only" setting was checked.

     I think those enhancements would facilitate Marty's thoughts for migrating and setting up a full page hierarchy with templates and fields for a gallery etc.  
     
  2. Or perhaps it wouldn't be a big deal to also download any template files as well and include those in a zip along with the JSON file. The catch there might be permissions issues on the templates folder as this won't be writeable by default on most setups. Copying files from assets/files should be easy enough, but could result in a huge download package and also it might become interesting trying to reconnect these files to their appropriate new pages_id

I think for now at least, I'll aim for option 1. It seems like option 2 is something best tackled by Ryan as I am sure there are some big picture considerations that I'll miss at the moment. It would also become difficult deciding where to stop. For example I don't think a module like this should ever tackle migrating template access roles - I think it would be logical that these sorts of setting would be best redefined on the new site.  

Marty - can you think of anything I am overlooking in Option 1 for making this work for your idea?

Share this post


Link to post
Share on other sites

I have started a GitHub repo for Page Tree List JSON files: https://github.com/adrianbj/ProcessWirePageLists

I will add new ones as the need arises for me.

Still not sure if this is the best place for these. I am tempted to set up a simple site to house them that hooks into the Github API like the PW modules page.

Will see once I get an idea if people will actually use these or not. Might be more useful once I have taken care of some of the ideas in the above post.

Share this post


Link to post
Share on other sites

Thought I'd step in and show support for this - love the idea as a whole and Marty's hit upon an interesting use case there right away for easily setting up small sections of repetitive content that is a bit more modular perhaps than a site profile and useful when such things are outside the scope of a module themselves.

@Soma - I need to pay more attention to the cheatsheet as I'd not seen getArray() - would've been incredibly useful when I was debugging something the other day and print()'ing a page object... which causes a mess on the screen :D

2 people like this

Share this post


Link to post
Share on other sites

Adrian I don't really have any more thoughts to add. A structure-only option sounds great. I can see this module being used in conjunction with the ImportPagesCSV and Wanze's Batcher a fair bit.

Cheers

Marty

Share this post


Link to post
Share on other sites

Thanks Marty,

I just added the structure only option and pushed to Github. At the moment it still creates the parent page, but not the child pages.  I am hoping to spend some time now looking into importing all levels of children which will be needed in some use cases, so I think I will need to change the structure behaviour to have three options - 1) all pages, 2) no pages at all, and 3) all levels of pages, except the very lowest level child pages so that categories and other levels are be created if wanted, without population of the actual content pages. Hope that makes sense.

I guess it will be an experiment to see what I think will be most useful / versatile for people.

1 person likes this

Share this post


Link to post
Share on other sites

Hey Marty - sorry for the confusion. You need to choose one of those options - obviously I need to come up with a better name. 

Save will spawn a download of the JSON file. Copy will output it on the screen so you can copy it.

I assume you have chosen a Parent page and structure option as well? I thought I had made them all required fields, but maybe not - will check.

I am getting close to having multiple levels of children etc working too - getting exciting :)

Share this post


Link to post
Share on other sites

Yeah I followed the steps. I chose a top level page with a few children then Structure only and Save. I tried all options actually and I only got an empty file.

Exciting is an understatement... :)

Share this post


Link to post
Share on other sites

Weird!

Would you mind trying to import that countries.json file for me and see if that works. If it does, could you then try to export that countries page tree and see if that works. 

I am running a fairly recent dev version, but I don't think I am using anything particularly new.

Sorry you are the guinea pig on this :)

Share this post


Link to post
Share on other sites

I imported the US States json file perfectly. I just can't seem to export. Perhaps it's my server setup. I'll give it a go locally at some point.

Share this post


Link to post
Share on other sites

Thanks Marty - strange the export is not working for you. Would definitely love to hear how it goes locally for you. Also could you give me a rundown of your server setup?

Do you have debug on? Any php errors?

I should be finished with the next round of functionality tomorrow - changing quite a few things, so we'll see if that helps as well. 

If not, hopefully someone else can confirm whether it is working for them or not and we can go from there.

Share this post


Link to post
Share on other sites

OK, I have a new version ready, with a new name: Page Tree Migrator

This new version has a new Github page: https://github.com/adrianbj/ProcessPageTreeMigrator

It can still handle simple "page list" imports and I have update the format of the countries.json and us-states.json files at: https://github.com/adrianbj/ProcessWirePageLists

WARNING: It should still be considered alpha at the moment, so please be careful !!

You will now need this new module to be able to import those files.

Whats' New

  1. Ability to import any page tree with infinite levels of children
  2. Templates are now created with all (I think?) settings from the original setup (including Family child and parent templates allowed).
  3. Three export options: 
        - Everything, including all data pages
        - Field, templates, and structural pages
        - Fields and templates only

    I am not really happy with the middle one yet (see Road Map for more details), but I think the other two are very useful.

Road Map

  1. Testing - It has been working great for me so far, but I am sure you guys will find some problems - I know it is far from perfect :)
  2. Code cleanup - at minimum I want to rename some variables etc to be more logical and add some detailed commenting. I am sure someone with more PW experience will see a mess in how I have achieved certain things. I know of a few things already that I want to redo, but for now I just want to get it out for testing :)
  3. Add more options for the "Field, templates, and structural pages" export option. At the moment this exports the templates, fields, and all pages, except for the final child level. My logic was geared towards something like a gallery with this structure:

    Gallery
    --Nature
    ----Image 1
    ----Image 2
    --People
    --Animals

    So, with this export option, you would only export the Gallery parent page, and the category pages (nature, people, animals). The image grandchildren won't be exported so you can use this to start with a clean gallery page tree.

    However, I think I should add more options that allow the user to select which levels of children to export and even which fields within these pages. That way you could choose to only export the title field for an entire set of pages, leaving the rest of the fields empty. Not sure of the most useful approach here. Any thoughts?
  4. Deal with any potential issues that occur when importing over an existing template if existing settings are different.
  5. What else do you guys want to see?
2 people like this

Share this post


Link to post
Share on other sites

Just made a couple of quick updates on Github to v0.0.3 now to fix some path issues for the main parent in some situations. Hopefully fixed now, but we'll see :)

Off to sleep now - probably some more bug finding tomorrow :)

EDIT: Marty - let me know if this version works for you. 

Share this post


Link to post
Share on other sites

This looks even better. Import doesn't seem to work on my test site though, seems to some incompatibility with multilang features. This comes after using paste import:

Catchable fatal error: Object of class stdClass could not be converted to string in /srv/www/pw/frontendedit/wire/modules/LanguageSupport/FieldtypeTextLanguage.module on line 38

I am running PW 2.3.0 on that site.

Share this post


Link to post
Share on other sites

Thanks for finding that Antti, although I am having trouble reproducing it. Unfortunately I am pretty ignorant of the multilang features - typical English speaker I'm afraid :)

I just installed the Language Support (Core) module and ran a successful import without any errors.

I am running a fairly recent dev version so I wonder if that is the difference? Or maybe I actually need to install a language pack and set up an alternate language field to trigger the error. I'll keep investigating, but do you have any ideas?

BTW, I just fixed a problem with the country and state list files - my subconscious must have realized and woke me up (it's early here) - I should learn never to release code late at night :) This change won't be causing your error though.

Share this post


Link to post
Share on other sites

Tested it little bit more - it probably has something to do with the data structure I am trying to import:

{
    "pages": [
        {
            "name": "basicpage",
            "parent_name": "",
            "page_template": "basic-page",
            "status": 1,
            "sort": 7,
            "data": {
                "title": {
                    "data": "basicpage",
                    "data1029": "basicpage"
                },
                "headline": "",
                "summary": "",
                "body": {
                    "data": "",
                    "data1029": ""
                },
                "sidebar": "",
                "images": [],
                "related": [],
                "repeater": {
                    "data": "",
                    "count": 0,
                    "parent_id": 1052
                },
                "commentsSpecial": [
                    {
                        "id": 74,
                        "status": 1,
                        "data": "Hi there!",
                        "cite": "Antsa",
                        "email": "xxxx.xxxxx@xxxxxx.com",
                        "created": 1370721469,
                        "created_users_id": 41,
                        "ip": "91.156.11.190",
                        "user_agent": "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/27.0.1453.110 Safari/537.36",
                        "website": ""
                    }
                ],
                "thumbnail": []
            }
        },
        {
            "name": "templates",
            "parent_name": "basicpage/",
            "page_template": "basic-page",
            "status": 1,
            "sort": 0,
            "data": {
                "title": {
                    "data": "",
                    "data1029": ""
                },
                "headline": "",
                "summary": "",
                "body": {
                    "data": "",
                    "data1029": ""
                },
                "sidebar": "",
                "images": [],
                "related": [],
                "repeater": {
                    "data": "",
                    "count": 0,
                    "parent_id": 1059
                },
                "commentsSpecial": [],
                "thumbnail": []
            }
        }
    ],
    "templates": [
        {
            "template": "basic-page",
            "data": {
                "useRoles": 0,
                "childrenTemplatesID": 0,
                "sortfield": "",
                "noChildren": "",
                "noParents": "",
                "childTemplates": [],
                "parentTemplates": [],
                "allowPageNum": 0,
                "allowChangeUser": 0,
                "redirectLogin": 0,
                "urlSegments": 0,
                "https": 0,
                "slashUrls": 1,
                "altFilename": "",
                "guestSearchable": 0,
                "pageClass": "",
                "pageLabelField": "",
                "noGlobal": 0,
                "noMove": 0,
                "noTrash": 0,
                "noSettings": 0,
                "noChangeTemplate": 0,
                "noUnpublish": 0,
                "nameContentTab": 0,
                "noCacheGetVars": "",
                "noCachePostVars": "",
                "useCacheForUsers": 0,
                "cacheExpire": 0,
                "cacheExpirePages": [],
                "label": "",
                "tags": ""
            }
        }
    ],
    "fields": [
        {
            "name": "title",
            "label": "Title",
            "description": null,
            "template": "basic-page",
            "flags": 13,
            "type": "FieldtypePageTitleLanguage"
        },
        {
            "name": "headline",
            "label": "Headline",
            "description": "Use this instead of the Title if a longer headline is needed than what you want to appear in navigation.",
            "template": "basic-page",
            "flags": 0,
            "type": "FieldtypeText"
        },
        {
            "name": "summary",
            "label": "Summary",
            "description": null,
            "template": "basic-page",
            "flags": 1,
            "type": "FieldtypeTextarea"
        },
        {
            "name": "body",
            "label": "Body",
            "description": null,
            "template": "basic-page",
            "flags": 0,
            "type": "FieldtypeTextareaLanguage"
        },
        {
            "name": "sidebar",
            "label": "Sidebar",
            "description": null,
            "template": "basic-page",
            "flags": 0,
            "type": "FieldtypeTextarea"
        },
        {
            "name": "images",
            "label": "Images",
            "description": null,
            "template": "basic-page",
            "flags": 0,
            "type": "FieldtypeImage"
        },
        {
            "name": "related",
            "label": "Related pages",
            "description": null,
            "template": "basic-page",
            "flags": 0,
            "type": "FieldtypePage"
        },
        {
            "name": "repeater",
            "label": "Repeater",
            "description": null,
            "template": "basic-page",
            "flags": 0,
            "type": "FieldtypeRepeater"
        },
        {
            "name": "commentsSpecial",
            "label": "Comments",
            "description": null,
            "template": "basic-page",
            "flags": 0,
            "type": "FieldtypeComments"
        },
        {
            "name": "thumbnail",
            "label": "Image",
            "description": null,
            "template": "basic-page",
            "flags": 0,
            "type": "FieldtypeImage"
        }
    ]
}

If I export/import structure only, it works great!

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!


Register a new account

Sign in

Already have an account? Sign in here.


Sign In Now

  • Recently Browsing   0 members

    No registered users viewing this page.

  • Similar Content

    • By alexcapes
      Hi,
      I have an issue when using the SmartyPants Typographer text formatter.
      If the text has an HTML tag (such as italics) straight after the quotation mark, it reverses it. 
      So for example:

      This only seems to occur when an HTML tag is placed at the beginning. 
      Does anyone know how this could be corrected? 
    • By benbyf
      Done quite a bit of Googling and haven't quite worked out how one would create a field of type repeater and add fields to it via the API in a module. Anyone have any links or examples?
    • By benbyf
      HELLO!
      I've been working on a user messaging module which I am nearly ready to release as version 1.
      Currently I have the below functionality and I'm looking for feedback to what other things may be useful for people looking to add user to user messaging on their sites.
      compose message to one or more users known to PW js to enable tag based UI display of user names input on compose message form display all message threads with reply forms reply to message thread displaying each user name by each message in the thread display "unread" when a thread has new content that hasnt been seen by current user delete or unpublish message thread (configurable) display total message thread count display total unread threads (threads that have new replies that the current user has not seen) delete all message threads and associated data (not meant for the users to have access to) road map
      send email to user on new message to a thread they are included in
    • By louisstephens
      So I was working on a module and trying to include a library (so I could use it across the site) using $modules->get('myModule'); . However, When I go to include the library folder (modeled after the LibFlourish module created some time ago). I keep getting: 
      site/modules/MyModule/libraryName/ProcessWire\filetoinclude.php This comes with an error notifying that "failed to open stream: No such file or directory in ...".. Is ProcessWire supposed to be in the include url, or is there a way around this? Sorry, I am quite new to working with my own modules.
       
      I just noticed the sub forum regarding Development, if someone could move this to the correct area I would appreciate it.
    • By louisstephens
      I wasn't quite sure where to post this, as this is a question regarding the module (sorry if it is in the wrong place). I was wondering if anyone has used the Email to Page module and figured out a way to "move" incoming css to another field, or how to render the css so the message just renders utilizing the css. 
      Right now all of the message contents get dropped into a textarea field and display in a template as a jumbled mess. Perhaps I am missing a formatting option in the text area field to render the css and html.