Front-end Editing adding extra line endings to my Markdown (ProcessWire 3.0.241)

[EyeDentify]

Hello There fellow PW users.

I am writing to get your advice on how to figure this problem out.

I have never used the Front-end Editing functions of PW before so i tought i try it out.

Using Ryans Front-end Page Editor module version 0.0.6.

And i am using option A (automatic editing) method described in the documentation together with:
 

<?PHP
/*
	In my setup 'main_markdown_content' is a textarea field with a 'Markdown/Parsedown' textformatter.
	and inputfieldtype 'Textarea' selected on details tab for the field.
	Also content type is 'Unknown/Text' is selected.

	I get a warning when editing this field:
	-"This field currently allows Markup/HTML in formatted output. If you do not intend to allow Markup/HTML, please select the “HTML Entity Encoder” 
	for “Text formatters”.

	But the field is storing Markdown and not HTML, as i understand it, the markdown gets rendered into HTML by the textformatter at runtime when
	outputting the field in my template ?

	
*/

/* using Option A (automatic editing) */
echo($page->main_markdown_content);
?>

So far so good, it works, the trouble comes when the field is saved when clicking "Save" button on the frontend.
For some reason new line endings are added to every line in my Markdown.

Lets say this is the markdown (sorry for the swedish):
 

# Sommar 2024
Sommar och sol, värme och regn om vart annat, för att inte tala om pollen.
/EyeDentify.

## Uppdateringar
- **2024-07-13:** Har uppdaterat Bulma.JS och önskar glad sommar.
- **2024-04-10:** Önskar en trevlig vår.
- **2023-12-24:** Önskar God Jul och Gott nytt år.
- **2023-01-11:** Det är ett nytt år och jag fortsätter jobba på siten. Har planer för att utveckla den. Mer om det senare.

And that after saving the frontend field ends up looking like this when i check it in the Admin editor:
 

# Sommar 2024

Sommar och sol, värme och regn om vart annat, för att inte tala om pollen.

/EyeDentify



## Uppdateringar

- **2024-07-13:** Har uppdaterat Bulma.JS och önskar glad sommar.

- **2024-04-10:** Önskar en trevlig vår.

- **2023-12-24:** Önskar God Jul och Gott nytt år.

- **2023-01-11:** Det är ett nytt år och jag fortsätter jobba på siten. Har planer för att utveckla den. Mer om det senare.

(data above is copied exactly from the field after a save and it adds extra spaces.)

Which is not what i want, the textformatter outputs the HTML as expected but it has more line endings and spaces between rows.

I have to edit out the extra spaces in the Admin backend editor to make it go back to my original Markdown.

I can´t figure out what is happening so i turned the editing of for now.

Would realy like to be able to use it.

You PW gurus have any ideas to what is going on ?
If you want any other info to help me, just say so and i will try to supply it.

Thanks again!
/EyeDentify

Edited August 14 by EyeDentify
Added PW version and module that i am running.

[BrendonKoz]

I haven't tried frontend editing yet, but since no one's yet replied, I have two possible thoughts:

1. You said there's a Markdown/Parsedown TextFormatter. Is there, by chance, a Newlines to XHTML Linebreaks TextFormatter enabled too? That might do it.
2. If you're manually typing this content in, just in case the interface is adding Windows' style line endings, try holding SHIFT when pressing ENTER/RETURN at the end of a line in order to remove the possibility of extra line-endings being included (ex: \r\n instead of just \n). If you're copying and pasting, similarly make sure the source text doesn't use those line-endings.

?

[EyeDentify]

10 hours ago, BrendonKoz said:

I haven't tried frontend editing yet, but since no one's yet replied, I have two possible thoughts:

1. You said there's a Markdown/Parsedown TextFormatter. Is there, by chance, a Newlines to XHTML Linebreaks TextFormatter enabled too? That might do it.
2. If you're manually typing this content in, just in case the interface is adding Windows' style line endings, try holding SHIFT when pressing ENTER/RETURN at the end of a line in order to remove the possibility of extra line-endings being included (ex: \r\n instead of just \n). If you're copying and pasting, similarly make sure the source text doesn't use those line-endings.

?

Hello @BrendonKoz i thank you for your reply.

You had the same thought i had about the textformatters so i took an extra look, and it´s only the "Markdown/Parsedown" formatter that is enabled.

Also i did some more experimenting with the frontend, and it seems something happens when i double click the part of the site i want to edit, in this case the Markdown and everytime the field is saved their is newlines added, even if i had not made any changes.

This is so strange.

But if i use your method to delete the \n and use shift-Enter for a \r\n and save the field no extra spaces are added.

I can´t be expected to go through every line and delete the \n and replace with \r\n every time i want to edit it.

Could it be some UTF-8 issue that i have failed to catch ?

Update 2024-08-16
This time when i went through all the lines and used BrendonKoz method the newlines was visible in the admin side textarea form, but not renderd on the frontend side.

Thanks in advance.

Edited August 16 by EyeDentify

[EyeDentify]

I did even some more testing after @BrendonKoz post gave me some ideas.

And i am pretty confident that the culprit is the "Markdown/Parsedown" textformatter.

Because i disabled the "Markdown/Parsedown" textformatter and had no formatter at all on the field.

Then when i edit it on the frontend and save, no extra lines or endings is added.

As soon i enabled the "Markdown/Parsedown" textformater the problem shows up again.

[EyeDentify]

I wanted to add some more information in case it plays a difference in figuring this out.

On the page/template that the Markdown field is used i also make use of the following JavaScript librarys:

- Fontawsome
- Google fonts

[wbmnfktr]

I'm using a similar setup with markdown and frontend-edit. Copied your snippet, added demo content, wrote text... no new lines added here.

I put some screenshots here to compare. If you want, drop me one of your .md files and I will test with that.

[wbmnfktr]

Ok, so after more playing around with the frontend-editing I got it to throw up - kind of.

I threw in a code comment:

```twig
<li>
  asdads
</li>
```

And I got a totally messed up textarea/markdown back. Everything in new lines/paragraphs. Totally unusable.

Went back to the admin, pasted in the old content. Everything was fine again.

Tried again and nothing happened. Besides the code block looked a bit off, went into edit-mode and saw this:

```twig

asdads

```

the <li>s were missing, cleaned that up, and got it to throw up again.

It takes some tries here but once in a while something goes wrong. Tried different HTML tags but those <li>s or code blocks seem to cause issues when added from the frontend. Tried the same in the admin and everything was fine.

[EyeDentify]

I thought i supply some screenshots of my settings on the "main_markdown_content" field that is using the "Markdown/Parsedown" textformatter.
Included a screenshot on the Front-End Page Editor settings as well for comparing.

First the fields Basics tab

Then the Details tab

Then the Markdown/Parsedown modules settings:

And finaly the Front-End Page Editor settings:

I find that if i involve the "HTML Entity Encoder" textformater trouble ensues. So i only use hte Markdown/Parsedown formatter alone.
That seems to work the best and lets me mix Markdown and HTML in the same textarea. I don´t use any of the advanced editors for the Textarea.
Thats just how i roll.

Hope this will give some more information to solve this mystery.

Edited August 16 by EyeDentify
Made the Sections headlines bold and more clear.

[wbmnfktr]

Two things are different in our setups.

1. HTML Entity Encoder is enabled
2. Safe Mode in Markdown/Parsedown is enabled

For both I can't tell why they are enabled. I guess it's the default setting. At least I can't remember changing that.

Another thing is that I usually don't mix Markdown and HTML in my textareas.
Just plain Markdown. This is important because the moment I try to fix the HTML in the code annotation that textarea starts become unstable. And only then. First it adds new lines to the code area, next it adds paragraphs everywhere - even in other list elements.

Adding only Markdown without any HTML doesn't change the behaviour. And that's what I tried first.

Something different but similar I noticed was that I can't set line-breaks/new lines from the frontend.
Usually in the backend I add two spaces at the end of the line, hit enter and start a new line.
That doesn't work in the frontend at all. It actually messes thinks up similar to your issue.

 

Just an idea that came up:
Could it be an issue with output formatting and that it is not working as expected in the frontend?

Edited August 16 by wbmnfktr
added note about idea

[EyeDentify]

43 minutes ago, wbmnfktr said:

Two things are different in our setups.

1. HTML Entity Encoder is enabled
2. Safe Mode in Markdown/Parsedown is enabled

For both I can't tell why they are enabled. I guess it's the default setting. At least I can't remember changing that.

Another thing is that I usually don't mix Markdown and HTML in my textareas.
Just plain Markdown. This is important because the moment I try to fix the HTML in the code annotation that textarea starts become unstable. And only then. First it adds new lines to the code area, next it adds paragraphs everywhere - even in other list elements.

Adding only Markdown without any HTML doesn't change the behaviour. And that's what I tried first.

Something different but similar I noticed was that I can't set line-breaks/new lines from the frontend.
Usually in the backend I add two spaces at the end of the line, hit enter and start a new line.
That doesn't work in the frontend at all. It actually messes thinks up similar to your issue.

 

Just an idea that came up:
Could it be an issue with output formatting and that it is not working as expected in the frontend?

 

My only issues with mixing HTML and Markdown came up when trying to edit with the Front-End textarea, never in the Admin side.
But i only use HTML mixed with Markdown when i forget the syntax. (getting old i guess).

Like you i have problems with the Front-end textarea adding linebreaks only if i change something, because if i leave it untouched the Front-end refuse to save because there is no need.

I have safemode turned of for some reason, can´t remember why right now.

I can manage without the Front-end stuff but it would be good to know what is causing this problem.

A question for you @wbmnfktr, could the order of your textformatters change your outcome to your problems ?
 

[wbmnfktr]

51 minutes ago, EyeDentify said:

could the order of your textformatters change your outcome to your problems ?

Changing the order results in chaos as the markdown will be shownas unrendered HTML - which is actually correct behaviour here and makes sense.

I just removed HTML Entity Encoder, disabled Safe Mode, and played more.

The result stays the same: at some point the textarea field or it's rendering becomes unstable and lots of <p> tags were added.

Then I looked into the dev tools and saw that being in frontend-edit mode it outputs actually HTML and not real markdown - like it does in the admin.

With CKEditor or TinyMCE fields it's fine and expected as those are WYSIWYG editor but here in Markdown it's not.

Tested further and found that it is the same with real plain text textarea fields.
It makes the rendered HTML editable and doesn't render the textarea as InputField like in the admin.

Frontend:

Backend:

So... not perfect for plain text/markdown fields.
I might be totally wrong here but it's either:

1. Output Formatting or
2. the fact frontend-editing doesn't load the real InputFields

It's curious that I only ran into this problem because I tried to run into it. I use this for quite some time now, 3 months at least, and never noticed an issue. Maybe I should look into each and every page now. ??

[EyeDentify]

19 minutes ago, wbmnfktr said:

Changing the order results in chaos as the markdown will be shownas unrendered HTML - which is actually correct behaviour here and makes sense.

I just removed HTML Entity Encoder, disabled Safe Mode, and played more.

The result stays the same: at some point the textarea field or it's rendering becomes unstable and lots of <p> tags were added.

Then I looked into the dev tools and saw that being in frontend-edit mode it outputs actually HTML and not real markdown - like it does in the admin.

With CKEditor or TinyMCE fields it's fine and expected as those are WYSIWYG editor but here in Markdown it's not.

Tested further and found that it is the same with real plain text textarea fields.
It makes the rendered HTML editable and doesn't render the textarea as InputField like in the admin.

Frontend:

Backend:

So... not perfect for plain text/markdown fields.
I might be totally wrong here but it's either:

1. Output Formatting or
2. the fact frontend-editing doesn't load the real InputFields

It's curious that I only ran into this problem because I tried to run into it. I use this for quite some time now, 3 months at least, and never noticed an issue. Maybe I should look into each and every page now. ??

I think your on to something about how the Front-End textarea is renderd, implemented or what have you, cause it´s the source of all my troubles anyway.
Never had the trouble in the backend funny enough.

Never thought of checking what happend in the DOM when the Front-End editor kicked in. Good find.

I seem to have stired up a hornets nest of trouble ? 

[BrendonKoz]

On 8/16/2024 at 3:25 PM, wbmnfktr said:

Frontend:

[...]

So... not perfect for plain text/markdown fields.
I might be totally wrong here but it's either:

1. Output Formatting or
2. the fact frontend-editing doesn't load the real InputFields

That appears to be a result of contenteditable being applied and used on an element; it creates newlines and <br> tags on each ENTER/RETURN press (or similar end-result with pasted content to render it similarly). FrontendEdit does take into consideration the difference between standard Textarea and Richtext fields...unfortunately, Markdown technically does validly accept HTML in its markup, so although it doesn't provide a WYSIWYG interface, I'm assuming it doesn't parse out and remove embedded HTML by design. I guess the problem here is that ... Actually, since the Markdown/Parsedown here is applied from a Textformatter and not a custom fieldtype, and the fieldtype is still a standard Textarea, I'm back to being confused. ? It might require a deeper dive into the code of that module than a cursory glance like I gave just now.

[EyeDentify]

26 minutes ago, BrendonKoz said:

That appears to be a result of contenteditable being applied and used on an element; it creates newlines and <br> tags on each ENTER/RETURN press (or similar end-result with pasted content to render it similarly). FrontendEdit does take into consideration the difference between standard Textarea and Richtext fields...unfortunately, Markdown technically does validly accept HTML in its markup, so although it doesn't provide a WYSIWYG interface, I'm assuming it doesn't parse out and remove embedded HTML by design. I guess the problem here is that ... Actually, since the Markdown/Parsedown here is applied from a Textformatter and not a custom fieldtype, and the fieldtype is still a standard Textarea, I'm back to being confused. ? It might require a deeper dive into the code of that module than a cursory glance like I gave just now.

Yes its like an episode of X-files... a real mystery.

But you make an interesting find with the contenteditable stuff.

I am not that advanced, but some day i might have a look at the modules code also, to se if i can figure out something.

[wbmnfktr]

I am still not sure it this is a weird glitch or bug or maybe even expected behaviour.

Until this very thread here I never really noticed any issues and didn't find any messed up articles so far, yet to be frank I rarely use that feature on a daily basis. I mostly enabled frontend-editing for some fields to clean up formatting, adding a sentence, fixing a typo, and similar tasks while browsing through the site.

On the other hand I'm in the process of migrating some client projects over to Markdown textareas as those clients feel more comfortable using plain-text than HTML editors - and they just can't really f**k up things that bad in there. So it could become a concern when I enable frontend-editing there. It's not planned, but could be the case.

@ryan , maybe you could give us more insights here? Is this expected behaviour or a weird glitch/bug?

 

[BrendonKoz]

I was curious, so setup a test environment. Blank profile. I created a `body` Textarea field and applied it to the `basic-page` template. I enabled the core's TextformatterMarkdownExtra module, and configured it as such:

- Parsedown Extra
- Safe mode: No (realistically this should be "yes", but setting it to yes caused HTML to be HTML encoded)

For the body (Type: Textarea) field:

- Text formatters: Markdown/Parsedown Extra (ONLY this; HTML Entity Encoder caused issues when combined with the Markdown formatter)
- Inputfield type: Textarea
- Content type: Markup/HTML
- Markup/HTML (Content type) [options]: None selected, though this is only due to it being a test; these should be fine to adjust

Choosing "Unknown/Text" for the Content Type while using Markdown seems to cause some issues when combined with Frontend Edit since the rendered text after being formatted is HTML, not text. I also had a strange scenario with Frontend Edit at one point where the text generated by Frontend Edit into the `body` field using contenteditable was "<div>Body text</div><div>More text</div>", but interestingly, ProcessWire only rendered "<div>Body text</dv>" into the HTML, though when I activated Frontend Edit to modify the field's value, the full text was displayed. I believe that may have to do with choosing "Unknown/Text" for the content type while using Frontend Edit due to how contenteditable works.

That being said, once I configured things as it is configured above, I was unable to reproduce any extra newlines being produced in the final markup, no matter how many times I edited and saved.

[wbmnfktr]

26 minutes ago, BrendonKoz said:

That being said, once I configured things as it is configured above, I was unable to reproduce any extra newlines being produced in the final markup, no matter how many times I edited and saved.

I could support that statement, yet I now have worse problems than before and makes it totally unusable.
I might missed something from your settings. Otherwise this is cursed.

Can you post screenshots of your settings, please.

[BrendonKoz]

If you're intermingling this field with actual HTML on purpose, it would make sense that there are problems. As I mentioned earlier, Markdown should accept HTML, but due to how this all works together, I can see scenarios where there would be issues. I just manually typed in <strong>Test this</strong> into the Frontend Editable textarea for the body field, and saved. The text was bolded. OK. I went back to edit it and the text was still bolded, but there were no HTML tags for me to modify to un-bolden it. CTRL+B didn't work because that's a browser control to open my bookmarks. It would have to be modified in the backend, or I'd have to combine Markdown with a richtext editor (which I have not tested, and there are other modules for that, too). Therefore, any broken HTML would just be broken HTML.

NOTE: I did not adjust any Input settings, so "Strip Tags" is not enabled.

I don't believe HTMLPurifier is processing this input as it's currently configured, which is its own problem... See below (the "Edit this page" should not be bolded, it's hardcoded text, part of the page template, not part of the body field.)

[BrendonKoz]

It might be best to disable Frontend Editing on fields that are going to be handled with Markdown (or possibly any fields that have Textformatters that swap in/out content dynamically). I'm thinking that because Frontend Edit is not aware of the additional processing that a textformatter will do, it is only presenting and saving the data as it sees it based on the Fieldtype, and (in the case of textarea) Content type therein.

@bernhard You have front-end editing in your Rockfrontend; does it use the core's frontend page editor, or did you use your own? I'm curious if you've run into any similar issues.

[wbmnfktr]

Ok, we have the same settings now. Besides mine is actually a Textarea (Multi-Language) field, but I don't think that's the issue here.

I uploaded a short clip of how the field and/or frontend-editing now behave. It worked way better before.

https://youtu.be/OOUKdEzH4Tc

After all I maybe really just disable it again. As mentioned before I rarely use it and I have a quick link to the edit screen anyways.

[EyeDentify]

1 hour ago, BrendonKoz said:

I was curious, so setup a test environment. Blank profile. I created a `body` Textarea field and applied it to the `basic-page` template. I enabled the core's TextformatterMarkdownExtra module, and configured it as such:

- Parsedown Extra
- Safe mode: No (realistically this should be "yes", but setting it to yes caused HTML to be HTML encoded)

For the body (Type: Textarea) field:

- Text formatters: Markdown/Parsedown Extra (ONLY this; HTML Entity Encoder caused issues when combined with the Markdown formatter)
- Inputfield type: Textarea
- Content type: Markup/HTML
- Markup/HTML (Content type) [options]: None selected, though this is only due to it being a test; these should be fine to adjust

Choosing "Unknown/Text" for the Content Type while using Markdown seems to cause some issues when combined with Frontend Edit since the rendered text after being formatted is HTML, not text. I also had a strange scenario with Frontend Edit at one point where the text generated by Frontend Edit into the `body` field using contenteditable was "<div>Body text</div><div>More text</div>", but interestingly, ProcessWire only rendered "<div>Body text</dv>" into the HTML, though when I activated Frontend Edit to modify the field's value, the full text was displayed. I believe that may have to do with choosing "Unknown/Text" for the content type while using Frontend Edit due to how contenteditable works.

That being said, once I configured things as it is configured above, I was unable to reproduce any extra newlines being produced in the final markup, no matter how many times I edited and saved.

Tried to changing the "Content Type" setting on the fields "Text formatters" tab to "Markup/HTML" and everything went crazy, so i quickly changed it back to "Unknown/Text". And all is well.

In the front end all the Markdown bunched up in a pile with no line endings or returns when i clicked saved.
And in the front end when i went to check on what showed up there, the Markup was unchanged except for some HTML tags had been added.
Very strange.

But i think for the time being i am going to let Markdown and Front-End editing stay away from each other until we figure out this mystery.

Edited August 20 by EyeDentify
Checked some spelling.