Using SmartyPants and HTML Entity Encoder Textformatters together

[Robin S]

Are the SmartyPants and HTML Entity Encoder textformatters compatible with each other?

If I apply Entity Encoder first and SmartyPants second then the quotes stay "dumb".

If I apply SmartyPants first and Entity Encoder second then I get code like

’

in my field output.

[cstevensjr]

I believe it depends on the particular textformatters that you are using. Some will work together and some won't.  ProcessWire's Wiki briefly talks about textformatters and their use.

Additional Link that explains Textformatters:

http://www.flamingruby.com/blog/taking-apart-a-simple-textformatter-module/

[Robin S]

Some will work together and some won't.

I can understand that, but in the case of SmartyPants and HTML Entity Encoder these are both essential formatters in my book. SmartyPants for proper quote mark typography and HTML Entity Encoder for valid HTML from plain text fields. I'm surprised they don't work together, which makes me think I must be missing something.

[cstevensjr]

From the first link I gave you:

You may also apply multiple text formatters at once. Simply select more than one, and drag them to the order you want them to process the text. This doesn't necessarily mean that you should add more than one though. For instance, if you added both "entity encoder" and "Markdown" you'd end up with double encoding, since Markdown already does it's own entity encoding (as do most lightweight markup languages).

An example of where you might want to apply two text formatters would be to combine Markdown with Smartypants. These two are designed to run together. According to the author, Smartypants is meant to run after Markdown, so we'd drag Smartypants to be below Markdown.

[Robin S]

Thanks, but I'm not clear on what you're getting at. My first post explains that using the textformatters in either order gives unsatisfactory results.

I believe the problem is that SmartyPants encodes the quote mark, dash and ellipsis punctuation as HTML entities (but no other characters, so it can't be used as a replacement for HTML Entity Encoder unfortunately), and then HTML Entity Encoder double-encodes the ampersand character in those entities.

Two possible solutions I can see:

1. Modify HTML Entity Encoder to allow the "double_encode" parameter to be used in htmlspecialchars, which avoids the double-encoding of existing HTML entities. The downside is that it wouldn't be possible to have literal text like "&" appear in the output. Funnily enough, this problem manifests in the PW wiki article you linked to, with phrases like

   Note how the ampersand is now entity encoded, being "&" rather than "&".

2. Modify SmartyPants so that it inserts UTF-8 characters rather than HTML entities. I tried this and it seems to be a good solution. If there are no major downsides to this maybe Ryan will consider adapting the core TextformatterSmartyPants module to allow this as an option.

[cstevensjr]

1. Markdown and Smartypants are meant to work together.  They are designed to work together.
2. Markdown does it own "entity encoding" so there is no need to use HTML Entity Encoder with Smartypants.
3. Use Markdown first then have Smartypants apply.

That's what I got from the information that I linked to.

[Robin S]

Ah, I get you now, and it works.

Though it does seem like using the Markdown parser on text that does not intentionally contain Markdown is risky - could create as many problems as it solves. I can imagine situations where an editor enters a plus, minus or asterisk at the start of a line and a list item inadvertently results.

I guess I could always add a strip tags textformatter at the end to remove any HTML tags added by the markdown textformatter, but that is getting kinda crazy.

[LostKobrakai]

Why not create a copy and remove the encoding functionality from smartypants? As long as you make sure everything is encoded afterwards there shouldn't be any concerns with that.

[Robin S]

Why not create a copy and remove the encoding functionality from smartypants? As long as you make sure everything is encoded afterwards there shouldn't be any concerns with that.

Thanks, yes, those were my thoughts also a couple of posts up. Seems to work well with UTF-8 characters instead of HTML entities. There's no need to entity encode the characters inserted by SmartyPants for modern websites using charset UTF-8.

I think this should be the default for TextformatterSmartypants, or at least an option, if this makes the module compatible with TextformatterEntities - using them together is pretty important for anyone who wants both valid HTML and correct punctuation from plain text fields. I'll try my hand at modifying TextformatterSmartypants to give the option of UTF-8 or HTML entities and submit a pull request.