Debugging a server issue, text content is cut off on any ‘special’ character

[thijs]

Dear fellow PW-jedi’s,

I’ve run into quite a strange issue which I find hard to debug, I’m pretty sure the issue is not on the PW side of things but something server side. Anyway; I’ve moved a project to the production server, and when I save a page in the admin section, text is being cut off (not saved) where ever a ‘special’ character occurs, for instance accented characters or apostrophes. Has anyone experienced a similar problem before? Or any thoughts on how to debug this?

Always grateful to hear your thoughts!

T

[thijs]

Update; behaviour is the same with a fresh PW 2.2 install

Some more info:

When sending a post request on the production server, these are the response headers:

 

Cache-Control    no-store, no-cache, must-revalidate, post-check=0, pre-check=0

Connection    close

Content-Length    0

Content-Type    text/html; charset=ISO-8859-1

Date    Thu, 07 Mar 2013 14:22:29 GMT

Expires    Thu, 19 Nov 1981 08:52:00 GMT

Location    ./?id=1001

Pragma    no-cache

Server    Apache

 

 

 

These are the response headers on my development server

 

Cache-Control    no-store, no-cache, must-revalidate, post-check=0, pre-check=0

Connection    close

Content-Encoding    gzip

Content-Length    20

Content-Type    text/html

Date    Thu, 07 Mar 2013 14:22:25 GMT

Expires    Thu, 19 Nov 1981 08:52:00 GMT

Location    ./?id=1007

Pragma    no-cache

Server    Apache/2

Vary    Accept-Encoding,User-Agent

X-Powered-By    PHP/5.2.14

 

 

Could it be the charset=ISO-8859-1 on the production server?

[horst]

... 

Content-Type    text/html; charset=ISO-8859-1

 ...

Could it be the charset=ISO-8859-1 on the production server?

if you have used utf8 on local side I tend to nearly 100% to say: yes!

have a look here

maybe you should check: DB-charset, PHP-charset, HTML-charset

[thijs]

Thanks for your reply! Forcing the proper content type fixed the issue, I’ve included this in any page I render (also in the admin part):

header("Content-Type: text/html;charset=UTF-8");

[ryan]

Adding this to your .htaccess file might do the trick too:

AddDefaultCharset UTF-8

Make sure your document is also specifying a <meta charset>. 

[thijs]

Hi Ryan,

Thank you for your reply, amazing how no single thread remains unseen by you! For now I’ve added it to the .htaccess file as well,  I’ll be sure to remember this solution if I ever run into this issue again.

Thijs

[dev123]

This is my .htaccess

This is my csv which i used for import

When i just read csv using code:

and output it in browser it showd this:(special characters are shown with question mark in diamond)

when i added this line to my code : header('Content-Type: text/html; charset=iso-8859-1');

Webpage output:

So now i know that webpage is reading my special characters , but then when i import even using header line in processwire as in this code:

The import stops at special characters :

pls help!!So no matter how muchever i try i am unable to add text with special characters into processwire using api

Note: the same works when i jus copy the word with special character and add the page via admin

[horst]

In Short because on mobile: try to convert your csv file to UTF-8 and Set all other to UTF-8 too

[dev123]

horst..i even tried this now...skipping the csv thing entirely as in this example:

i.e. I just tried to create a page via api and gave it the title  "The smørrebrød"..still it doesnt get stored, the title after "The sm" gets truncated as it comes accross a special character.

And as before same thing works if done via admin.

[horst]

Ok, please check with wich charset your template file is stored!

(It needs to be UTF-8, what is _not_ the default on Win!)

And also you have two charset headers in your code example. The second one overrides the first!

EDIT:

please, if you want to show code examples, please can you post it as text, using the "<>" - code formatter button?

You can simply test if your template file is UTF-8 encoded: try your example with $p->title = utf8_encode("smorebrod");

If this works, than your file isn't utf-8 encoded.

Edited March 19, 2014 by horst

[dev123]

@horst ..you were bang on ! ..the problem was that my php code in templates wasn't by default getting encoded as utf-8 and to add to it, even my csv were not utf-8 encoded.

no headers required, i am able to add all special characters via csv import now..thanks a ton!

And yes, i'll be careful about adding code next time onwards !