Problem with templates loosing fields

[DpW-Mi]

I've run into a strange problem that is affecting one of our dev environments. The issue is that several of our templates, will on occasion, "loose" some of their assigned fields. When you go to view the page, things are broken because the fields are no longer there. Yet, the data is still there, so if the fields are re-added back to the templates, the pages render correctly and there is no data loss.

I haven't been able to track down what is causing this. As far as I know, this is the only environment that it's happening in.  Our main production environment is essentially the "live" version of that same dev site and it hasn't shown this behavior.  Right now, I've been trying to just isolate, what the cause might be.  I have a hook that tries to capture field differences in the number of fields before and the number after.  This has only really caught myself going in to "fix" the errant template.  It's happens occasionally, maybe once or twice a week. Sometimes it's one template affected, sometimes it's two.  Strangely, it really only seems to happen on our "content" templates, that handle the display of front-end content.  We have a number of "data" templates that we automatically import product data into, that don't show this behavior.  Those templates are auto-populated via the API and our own data endpoints, several times a day, so perhaps there isn't enough time for this behavior to happen.  It tends to happen pretty frequently in two fieldsets that we commonly arrange fields used in the output of pages on the site, but it also happens in parts of templates that have no fieldset.  Most of these fields are text fields, although there are few option selects and checkboxes and one image field that goes missing pretty regularly.  Doesn't seem to impact FormBuilder forms. We don't automatically add fields to our templates, and our set up isn't that exotic.  One thing is that our page IDs can get pretty high, since the import routine, adds/deletes pages regularly, but the import doesn't deal with templates, it just updates pages on the site.

I was speculating with another Dev on our team that maybe it has something to do with the versioning module, or maybe an errant repeater breaking something, but the more I dive into this, it seems like it might be something else.  It's possible it has something to do our DB setup, and maybe an encoding issue or a periodic write issue, but I'm still scratching my head. The hook I'm using right now to log template additions/deletions is: addHookAfter('Templates::save').

We're running ProcessWire 2.10 using PHP-FPM 7.4 with Apache handling permissions/redirects and nginx acting as a proxy. Servers are running Ubuntu LTS. DBs are all MySQL. We'll upgrade to the latest main branch of PW once we deploy our new servers, but my manager would like to get a handle on this template issue before we start to migrate the servers.

Any thoughts on what might be causing this?  I'm trying to log changes on the system when something related to templates happens.  I'm not sure if there is a handy way to try to log the mysql table for "templates" and if the id/name array associated with the template changes?  Or is there another part of ProcessWire's handling of templates themselves that I should look into? 

Thanks, Daniel

[elabx]

@adrian@adriana bit more of this odd behaviour where fields disappear from templates, also as misterious as our own (@adrian's and myself) experiences.

[DpW-Mi]

Hmm, well, at least it's good to know we're not the only ones seeing this.  It is strange though and no associated error or exception messages.  Seems to happen pretty silently.

[cwsoft]

3 hours ago, DpW-Mi said:

We're running ProcessWire 2.10 using PHP-FPM 7.4 with Apache handling permissions/redirects and nginx acting as a proxy. Servers are running Ubuntu LTS. DBs are all MySQL. We'll upgrade to the latest main branch of PW once we deploy our new servers, but my manager would like to get a handle on this template issue before we start to migrate the servers.

Can you create a 2nd instance of your dev env and update this one to latest PW master version? Maybe the issues you are facing right now will be disappear. 

[DpW-Mi]

18 hours ago, cwsoft said:

Can you create a 2nd instance of your dev env and update this one to latest PW master version? Maybe the issues you are facing right now will be disappear. 

It's a good idea to try that.  I don't have time this week, but I can probably try that next week.  I have been wondering if the latest master version of PW might help.