"flagged, corrupted" page referenced in page reference field disappears when navigating to that page

[johnstephens]

Here’s an unusual issue I’ve never seen before.

1. I have a page called "Settings" that uses a page reference field to allow an administrator to construct and reorder pages for the main navigation menu. That field references 6 separate pages.
2. The navigation menu template gets its PageArray() from the page reference field I just described, and prepends the site's homepage ('/'). That makes a total of 7 pages in the PageArray().
3. On just about every page of the site, I see all 7 pages in the navigation menu, no matter where I navigate.
4. But on one particular page, only 6 pages appear in the menu. Missing is the current page being viewed. I’ll call this the problem page.
5. If I change the template of the problem page to some other template, it appears in the navigation again, even at the problem page's URL.
6. But I have other pages that appear in the navigation that use the same template as the problem page, and they don’t disappear from the navigation when I navigate to their URLs.
7. I can’t find any difference in publication status or permissions between the problem page and the working pages that have the same template as the problem page. But maybe I’m not looking in the right place.

I’m totally baffled. Can anyone suggest steps I might take to troubleshoot this? Using TracyDebugger, I have already used bd() to output the navigation PageArray() on the problem page and other pages, and verified that the problem page really is absent from the array when viewing the problem page URL. But I don’t understand why or how to fix it.

[johnstephens]

AHA! I found a datum that sets the problem page apart.

When I look at the TracyDebugger bd() output for the problem page (ie. bd(pages()->get('/problem-page'))), I see it has a property called "status". When I output this on any other page, the "status" property has the value of "flagged". But the same bd() outputs a value of "flagged, corrupted" when I view this output on the problem page URL.

I still don't know how to fix it, but I'll post more here if I find anything. I still welcome any insight from those of you who recognize what may be going on!

Thanks in advance!

[johnstephens]

I was able to remove the "flagged" status by opening the page in ProcessWire and re-saving it. That didn't affect the "corrupted" status, though, and the page still does not show up in the navigation menu when viewing the problem page URL.

[wbmnfktr]

 Found these two links with more details about Page::statusCorrupted:

https://github.com/processwire/processwire-issues/issues/909

Even though I read both parts I'm not really sure what to do next.

I'd probably try saving that page again via API with output formatting off or maybe even just create a new page and move all content over to that new one.

 

[johnstephens]

21 minutes ago, wbmnfktr said:

I'd probably try saving that page again via API with output formatting off or maybe even just create a new page and move all content over to that new one.

Thank you, @wbmnfktr! I will read over the two links. As a test, I already created a new page and copied the content over to the new one by hand (not using the API), and the new one is having the same problem. I will check the links to see if I can make any sense of it.

[johnstephens]

Quote

What "formatted" means depends on the Fieldtype. But if you are observing corrupted status at all, then it means you might be treading into territory where corruption can occur (not necessarily will), like when you are using pages for both front-end output and manipulation at the same. I recommend using a page either for front-end output or for manipulation, but not both at the same time.

Since the github issue references saving the page, I guessed that this might be associated with using the PageFrontEdit module on this site. As a test, I uninstalled PageFrontEdit, and there was no effect. Still unclear on the diagnosis and steps to troubleshoot. I’ll keep banging around. Thank you again!

[wbmnfktr]

1 hour ago, johnstephens said:

the new one is having the same problem

Do you mind posting your menu-builder code and the things related to it. Maybe there is something off of some kind.

Another thought here... did the database crash in the past or something? I can't remember having that issue in any of my sites or projects so far.

[johnstephens]

2 hours ago, wbmnfktr said:

Do you mind posting your menu-builder code and the things related to it. Maybe there is something off of some kind.

Another thought here... did the database crash in the past or something? I can't remember having that issue in any of my sites or projects so far.

I’m not aware of the database ever crashing, but I don't know how I would detect that. I've never seen this in a ProcessWire project either.

Here is the how the page reference field is configured:

{
    "navigation_main": {
        "id": 130,
        "name": "navigation_main",
        "label": "Main navigation menu",
        "flags": 0,
        "type": "FieldtypePage",
        "description": "Select pages for main navigation menu",
        "derefAsPage": 0,
        "inputfield": "InputfieldAsmSelect",
        "parent_id": "/",
        "labelFieldName": "title",
        "collapsed": 0,
        "usePageEdit": 0,
        "findPagesSelector": "template=basic-page|template-a|template-b|template-c|template-d|template-e|template-f, has_parent!=7",
        "notes": "Pages in navigation must be direct children of the site's homepage.",
        "allowUnpub": "",
        "defaultValue": "",
        "template_id": "",
        "template_ids": "",
        "findPagesSelect": "",
        "labelFieldFormat": "",
        "addable": "",
        "showIf": "",
        "themeOffset": "",
        "themeBorder": "",
        "themeColor": "",
        "columnWidth": 100,
        "required": "",
        "requiredIf": ""
    }
}

And here is the code that generates the menu markup using that field's input:

$menu_main = $settings->navigation_main;

$menu_main->prepend(pages()->get('/'));

$menu_main_iterator = 0;

foreach ($menu_main as $item) {

  $menu_main_iterator++;

  $item_class = 'nav--item';

  $item_class .= $item->rootParent == $page->rootParent ? ' nav--active uk-active' : '';

  $item_class .= ' nav--' . $menu_main_iterator . 'th';

  echo "<li pw-append='nav-main' class='nav-main__item {$item_class}'><a class='nav-main__link' href='{$item->url}'>{$item->title}</a></li>";

}

I have another menu in the footer which lists the same page, but the PageArray() used by the footer menu is generated by a different API call which does not involve the navigation_menu field.

[wbmnfktr]

You could download a backup of your database and just look through it. Sometimes you'll something that's off. But I almost think that there must be another issue of some kind.

Ok... there is nothing in your code that seems wrong to me.

Still a weird issue. Especially with the new page you created. That's something I don't understand.

[johnstephens]

I narrowed down the issue again, but I’m not sure what it means. I’ll do some more testing and see if I can turn up something.

Here’s what I found: I duplicated the page again by creating a fresh page with the same template, just like before. The only content in the original page was a title and CKEditor field, so I copied the title first and tested it before copying the CKEditor field content.

It worked! The navigation link appeared as expected on other pages and on the page itself.

Then I copied the original content from the CKEditor field into the new page's CKEditor field and Saved again.

It broke! Clearing the CKEditor field restored the article to the favor of ProcessWire.

I glanced over the markup yesterday and didn’t spot anything egregious, but now I’ll take a closer look at it. I’m pretty sure the site owner wrote and formatted the content in a word processor and copied it into the CKEditor field, but I thought CKEditor would filter out anything that might cause problems.

I would love to hear any additional insight anyone has. How can HTML markup in a CKEditor field create a "corrupted" status?

[johnstephens]

It seems to be fixed. I don’t know why.

What I did was copy the Source markup from the CKEditor field into Vim and scour it. Since the content had a lot of link anchors, I added new lines around each one so that I could see them separately.

But then… I found nothing unusual. I added blockquote tags around one paragraph that was a quotation, and I switched a few em tags to cite since they were book titles, and I changed a few straight quotes (") and apostrophes (') in the markup to the curly versions (“, ”, and ’). I pasted the result back into CKEditor's Source panel and saved the page. And that fixed the problem.

Even though the problem is solved for now, it would be great to learn more about what went wrong, how it went wrong, and how we could prevent this again.

Thanks in advance for any guidance!

P.S. Is this the right sub-forum for this issue? I chose API because I thought that it was related to the interaction between my page reference field and the template markup, but that doesn't seem to be the case.

[wbmnfktr]

So... there is something in your CKEditor field that breaks parts of your site... weird.

I guess you still have the original content and therefore could try these tools:

https://www.soscisurvey.de/tools/view-chars.php

https://dostring.com/show-hidden-characters

They should show and tell which or what characters are hidden in plain sight.

As your client will continue to use his tools he knows this problem might occur again some day and maybe the sanitizer for textareas needs an upgrade of some kind.

 

Yet... good to see you got it fixed to some point. 

[johnstephens]

Thanks again, @wbmnfktr!

Hidden characters was one of my thoughts too. I see that all the hidden characters exposed by these two sites are also visible by default in my Vim configuration: spaces and linefeeds. Comparing both the original content and the version that works after my unsubstantial revisions using those two sites, I don't see any significant difference between the problem version and the working version. The problem version didn't have any suspicious hidden unicode characters or sequences that these tools exposed.

I'm still stumped, but at least now I know some new angles for troubleshooting that I didn't know before!

[wbmnfktr]

Off-topic but your VIM config sounds very interesting to me but maybe we should open another thread for that.

You said your client uses word processors and such... maybe... your client copied things into CKEditor that weren't filtered with the regular sanitizer features or maybe even punycode or things like that. Do you mind sharing the content from that field? Maybe some of us or maybe even only @ryan can look into it. It won't hurt anyone so far. (Just an idea... to be clear.)

At least we all can take something with us from this.