Valid URL error in PW 2.7.2

[BFD Calendar]

I'm getting an 'Error found - please check that it is a valid URL' when I enter ""https://en.wikipedia.org/wiki/Sinéad_OConnor" automatically but then the "O'Connor" becomes "OConnor" and the link doesn't work.

Didn't happen in previous versions of PW.

Ps. Not that I'm such a big fan of Sinéad but I'm updating births in my BFD Calendar and hers is coming up in a few days.

[ryan]

Go to your URL field settings (Setup > Fields > your field) and check the box to "allow single/double quote characters in URLs". That should fix it. 

We started not allowing those characters in URLs by default because such URLs could be a security problem if someone forgets to enable the "HTML entity encoder" text formatter for their URL field, and happens to output the URL in an href attribute. 

[BFD Calendar]

That keeps the URL complete, but now the link is cut off at

https://en.wikipedia.org/wiki/Sinéad_O

because in my template I call for

echo "More info on <a href='{$page->bfd_url_wiki}' target='new'><b>Wikipedia</b></a>";

So there's a bit of a problem with double and single quotes.

[LostKobrakai]

Did you add the html entity encoder?

[BFD Calendar]

HTML Entity Encoder is on. Allow relative/local URLs is "yes". Allow internationalised domain names is "yes". Allow single/double quotes is "yes".

[BitPoet]

I just looked at this, as umlauts and other funny characters in URL paths are going to be on the table in one of my next work-related PW tasks next year, and I'm pretty sure that FieldtypeURL currently ignores all assigned textformatters. I've already filed a bug.

Also, HTML Entity Encoder is probably the wrong tool for that task. The way to go would IMHO be to rawurlencode all path components in the URL, encoding all non-reserved non-ascii characters to their percent encoding in line with RFC 3986.

[BitPoet]

Update: Ryan has already pushed a fix for the Textformatter issue to dev.

I've put together a small module that encodes relevant parts of a URL using parse_url and rawurlencode. If anybody wants to have a look, I'd be happy to get some feedback. It can be found on github.

[BFD Calendar]

BitPoet, good about the bug and module. But what I like about ProcessWire is that it keeps everything simple, and I don't want to use too many modules. Having URL fields should be a basic thing for which we don't need an extra module. Also, things that worked fine in all previous versions shouldn't move to a module all at once. I  have about 2000 Wikipedia URLs in as many different pages, see them go astray would be a little disaster.

Ps. Sinéad's birthday is today, but Nothing Compares to PW.

[BFD Calendar]

Can we expect a solution for this soon? Apparently PW 2.7.2 has changed all the URLs with an ' in a way that they don't work anymore. An old Peter O'Toole Wikipedia link doesn't work anymore, also the link I use to a Wikipedia image doesn't work anymore. http://www.birthfactdeathcalendar.net/people/otoole-peter/

I also hope the fix will repair all the URLs since finding every one with a ' would be a considerable amount of work.

[horst]

Sorry, but I really cannot follow what you are asking for.

As I could understand from reading here, there was a change in PW, related to security, to not support single and double quotes in URLs per default anymore, as it was done in the past. That's a good thing, I think.

And there is an option to (re)activate the old behaviour, for those, who need it.

Conclusion: new behaviour + old behaviour ? what more do you need ? or what is the part that I do not understand here ?

Also please refer to the according RFC 3986, where is described which characters are allowed to be present unescaped in an URL. Quotes and doublequotes are not in, or I may have overlooked it?

Also the example you have posted is wrong coded and cannot have worked before too:

echo "More info on <a href='{$page->bfd_url_wiki}' target='new'><b>Wikipedia</b></a>";

You cannot wrap single quotes around an URL if the URL itself contains (unescaped) single quotes!

Please, first fix your syntax and maybe, second, try to use more RFC conform URLs. (I know that browsers also work with malformed URLs, but since you now know that this is malformed, you shouldn't use it anymore).

[BFD Calendar]

Also the example you have posted is wrong coded and cannot have worked before too:

I'm sorry, but it worked fine from PW 2.0 to 2.6.18 at least for the past two years or so. Since 2.7.2 it doesn't work anymore. Or are you telling me that I have hundreds of links to pages and images that actually never worked? I do check my pages to see if they do what they are supposed to do.

[ryan]

BFD what's an example of the exact URL as you have it input in the admin page editor, and what's an example of the exact output you get in your source on the front end?

For your field settings, what textformatter plugins are being applied (details tab), if different from before? You mentioned the URLs were working before, when actually they really shouldn't have, so I'm wondering if they might be getting double encoded somewhere along the line. If some of your URLs are input encoded and some not it might be that you need to output the value a little differently on your front end to account for that. But I need to know what the exact inputs and outputs are first.

[BFD Calendar]

Ryan, I'll make an example on two sites that are running on different versions of PW with same input and output. I'll post it here when ready.

[BFD Calendar]

I put the site back to PW 2.6.18 and the links and images are working again.

http://www.birthfactdeathcalendar.net/people/otoole-peter/

Code for the photo:

// PHOTO

    if(count($page->bfd_people_picture)) {
        $images = $page->bfd_people_picture->getRandom(2);
        foreach($images as $image) {
        $thumbnail = $image->width(300);
        echo "<div id='images'><img class='photo' src='{$thumbnail->url}'></div>";
        }
    }
	elseif($page->bfd_image_from_url) {
    	$webimage = $page->bfd_image_from_url;
		echo "<div style='max-width:300px'><img src='{$webimage}' align='right' style='width:100%; height:100%' /></div>";
	}
	else {
		$homeimage = $homepage->images->getRandom()->width(300);
    	echo "<div id='images'><img class='photo' src='{$homeimage->url}'></div>no photo available";
	}

Input in the 'bfd_image_from_url' field is https://upload.wikimedia.org/wikipedia/commons/5/53/Peter_O%27Toole_-_1968.jpg

Code for the Wiki link:

    if ($page->bfd_url_wiki) {
	echo "More info on <a href='{$page->bfd_url_wiki}' target='new'><b>Wikipedia</b></a>";
	}; 

Input in the 'bfd_url_wiki' field is https://en.wikipedia.org/wiki/Peter_O'Toole

Both fields are URL fields. No text formatter for image and HTML Entity Encoder for Wiki link, Local/relative = yes, prepend root path = no for both.

I admit I'm by far not the best coder and of course I'm most willing to improve or change whatever is necessary to make things work. All I was saying is that this works fine in PW 2.6.18 and it doesn't work in 2.7.2

Thanks for help.

[BFD Calendar]

Looking further I notice a difference on how PW saves the links...

https://upload.wikimedia.org/wikipedia/commons/5/53/Peter_O'Toole_-_1968.jpg
becomes
https://upload.wikimedia.org/wikipedia/commons/5/53/Peter_O%27Toole_-_1968.jpg

while

https://en.wikipedia.org/wiki/Peter_O'Toole
just remains
https://en.wikipedia.org/wiki/Peter_O'Toole

But again, PW 2.7.2 does it differently...

[horst]

You can use escaped doublequotes in either of the versions in your code, than it should work in PW 2.7 too:

echo "More info on <a href=\"{$page->bfd_url_wiki}\" target='new'><b>Wikipedia</b></a>";

change href='...' to href=\"...\" in your template code, than it will work in every situation, regardles if you have escaped or unescaped single quotes in your URLs.

[ryan]

Testing here with PW 2.7.2 in a new URL field I setup called "href", I used the URLs you mentioned. For my field settings, I have "Allow single/double quotes" set to true, and for "Text formatters" I have "HTML Entity Encoder" selected. I tried with and without the "allow IDN" option, but it doesn't matter here since we're not testing with IDNs. Here are the results I got:

https://en.wikipedia.org/wiki/Peter_O'Toole

• Input to URL field in admin: https://en.wikipedia.org/wiki/Peter_O'Toole
• API code to output: echo "<a href='$page->href'>test</a>";
• Output in rendered page: <a href='https://en.wikipedia.org/wiki/Peter_O'Toole'>test</a> 
• Result: Clicking the link works, taking me to the page on Wikipedia

https://upload.wikimedia.org/wikipedia/commons/5/53/Peter_O'Toole_-_1968.jpg

• Input to URL field in admin: Input to URL field in admin: https://en.wikipedia.org/wiki/Peter_O'Toole
• API code to output: echo "<img src='$page->href' />";
• Output in rendered page: <img src='https://upload.wikimedia.org/wikipedia/commons/5/53/Peter_O'Toole_-_1968.jpg' />
• Result: Picture of Peter O'Toole

Basically, it's working exactly how it should. My best guess is that it wasn't working for you because you've got some other code in there somewhere HTML entity encoding the URLs, to make up for the bug prior to 2.7.2 where text formatters weren't being properly applied to URL fields. 

Another possibility, check if your $page possibly has output formatting turned off? It would need to be ON for this all to work. 

Looking further I notice a difference on how PW saves the links...

https://upload.wikimedia.org/wikipedia/commons/5/53/Peter_O'Toole_-_1968.jpg

becomes

https://upload.wikimedia.org/wikipedia/commons/5/53/Peter_O%27Toole_-_1968.jpg

 

 

That's interesting, so it looks like the URL is getting URL encoded, converting the apostrophe to a %27. I did try entering that too, but PW converted it back to an apostrophe and then entity encoded it, resulting in the same working output as in my second test above. If this is literally how your URLs are stored in the database, then my best guess is that when you are outputting the value, either the $page's output formatting is off for some reason, or the HTML entity encoder wasn't selected in your "text formatters" details tab, which would prevent a URL with an apostrophe from working, whether it was URL encoded or not. I would suggest upgrading again to 2.7.2 and double check all your settings, especially the one about allowing single/double quotes and the Text Formatters. If the URLs are still not working, then check to make sure in your template file that the page's output formatting is enabled, like this:

if($page->of()) {
  echo "<p>Output formatting IS enabled</p>";
} else {
  echo "<p>Output formatting IS NOT enabled</p>";
}

If it's not enabled, you'll want to track down where you are turning it off, typically with a $page->of(false); or $page->setOutputFormatting(false); 

 

Finally, check that you don't have something else entity encoding the URLs in your site. If you've got a line like this, you'll want to remove it:

$page->bfd_image_from_url = htmlentities($page->bfd_image_from_url); // avoid this

[BFD Calendar]

Thanks Horst and Ryan. My coding skills were probably the culprit in all this. Just weird that the problem suddenly appeared after upgrading. Maybe a coincidence, but I'm glad I learned something once again. Anyway, I'm up and running PW 2.7.2 again!