Data Migration / Syncronization of Field & Template Definitions

[Jason Huck]

I've got a large number of new field definitions and associated data in a development site, all of which needs to be deployed to a site that's already live in production. I know I can export/import the field data via CSV, but I'd also like to avoid the laborious process of manually recreating all of the field definitions.

Looking at the database schema, it seems like I could do a structural sync (e.g., using a tool like Navicat) to add empty versions of all of the new per-field tables ("field_myfieldname"), then sync the data in the following tables:

fieldgroups

fieldgroups_fields

fields

fieldtype_options

It looks like that would recreate the structure. I wouldn't sync the actual field data that way, as it looks like an easy way to get ID mismatches. I can test this on a copy of the database, but thought I'd get opinions here. Seem like a reasonable approach, or bad idea (because...)?

Thanks,

Jason

[LostKobrakai]

I've switched to writing all changes affecting the db in migration files (simply using the pw api), which I can easily re-run on any other server. Took a lot of the pain away for me. It's otherwise not possible to automate db changes with servers, where content is independently created (e.g. overlapping id's). Additionally it's not limited to fields/templates either.

I'm not sure if your half way automatism will actually improve your workflow. You've still to update any added field data as well as any (even if rare) schema changes, that might have happened to tables. If you're sure you'll only be adding fields (without data) than it should work.

[szabesz]

@Jason Huck: Have you already checked out adrian's ProcessMigrator module?
https://github.com/adrianbj/ProcessMigrator
Personally I never used it, but adrian seems to be always online if you need help. Maybe he never sleeps

[Craig]

On the source site, go to Setup > Fields, and click the [Export] button at the bottom of the page. Choose your new fields, then click [Export]. This will give you a chunk of JSON-encoded text which represents your chosen fields. Copy the text, and paste it into the same area on the live site - Setup > Fields > [import].

Then do the same for templates if necessary, which will pick up the changes to add those new fields to the relevant templates if they already exist, or create the templates if they don't.

[SteveB]

Just the other day export/import for templates and files partly worked but many settings such as allowable template and parent for page fields were missed and a few fields were not assigned to all of the templates they belong on. I tried Migrator once (not recently) and it scrambled the page tree badly. To be fair, both of these episodes happened on migration attempts between different PW versions.

People bring up Migrator in topics like this but last time I looked I didn't see much detail on the forum about what's working and whether it's keeping up with changes in PW.

Template and field export/import from 2.7.3 to 3.0.8 saved me a lot of time but didn't fully resolve the interdependencies of fields and templates referencing each other.

[Craig]

I've found with the field/template import/export, it is sometimes necessary to do it twice. Once to create the entries (skipping the relationships/family settings), and the second time to apply the correct family settings.

[Jason Huck]

Thanks for the suggestions. I looked at the built-in field-level export/import (didn't even know that existed) as well as the Migrator module, but ultimately ended up syncing everything with Navicat. In this particular case there was very little data that differed on production that needed to be preserved, and most of that was self contained in its own tables (e.g., FormBuilder forms and entries), so it wasn't that difficult to isolate. I will keep these other options in mind for the future when circumstances might be different.

[horst]

Template and field export/import from 2.7.3 to 3.0.8 saved me a lot of time but didn't fully resolve the interdependencies of fields and templates referencing each other.

I've found with the field/template import/export, it is sometimes necessary to do it twice. Once to create the entries (skipping the relationships/family settings), and the second time to apply the correct family settings.

Exactly as @Craig said: for example, if you want import two templates that have a parent / children family relation, it cannot be imported in one go, as for both the other referenced family template isn't known in the system. But it works fine, if you first let out the family settings. Click / select YES for "Create this template?" for both templates and commit the form.

In the next screen, you simply click on ok and in the next step you paste in the data again and click peview. Now you will see both templates with the opened settings. One shows the new property "parentTemplates" and the other a new property "childTemplates". Lastly click commit changes and you are done.