WordPress.org

Ready to get started?Download WordPress

Forums

[resolved] Special characters converted to question marks on save (39 posts)

  1. Endolith
    Member
    Posted 6 years ago #

    If I create special characters using the "Custom characters" button they are erased and turned into question marks as soon as I press Save. Shouldn't WordPress be able to handle characters generated by itself??

  2. palamedes
    Member
    Posted 6 years ago #

    If you enter something in a different language that uses UTF-8 to display, such as Chinese (這是一個考驗) and you get nothing but ????? then you're not UTF-8 compliant.

    If you dont' see chinese in my post right now, then it's your browser. If you do see chinese then it might be your database.

  3. Endolith
    Member
    Posted 6 years ago #

    It's definitely not my browser. The special characters show up in the WordPress edit window but then turn into question marks when I press Save.

    Yes, I see Chinese characters. I'm in Ubuntu and my computer and browser can handle Unicode just fine.

  4. moshu
    Member
    Posted 6 years ago #

    Well, something in your environment is NOT handling utf-8 "just fine" - otherwise you wouldn't be here asking...

    Are you using the wysiwyg thingy or is it turned off?
    Is this a brand new WP install or was it an upgrade? If upgrade did it work before/ever?

  5. palamedes
    Member
    Posted 6 years ago #

    Is your database your local database?
    Do you have control over it or is it hosted for you?
    Do you know if IT can handle utf-8 characters?

    as Moshu said, somethings wrong as wordpress handles these things perfectly.

  6. Endolith
    Member
    Posted 6 years ago #

    Well, something in your environment is NOT handling utf-8 "just fine" - otherwise you wouldn't be here asking...

    Everything is just fine as far as my personal computer displaying Unicode. I can see Chinese, Greek, Math, Arabic, Hebrew, Cyrillic, etc. Ubuntu and Firefox are very good about this. The problem is with WordPress or the server that WordPress is running on.

    Are you using the wysiwyg thingy or is it turned off?

    Yes, I am generating these with the Insert custom character button with an Ω on it.

    Is this a brand new WP install or was it an upgrade? If upgrade did it work before/ever?

    This does not work on my standard install, which has been upgraded several times, and it also does not work on a fresh install that I just created.

    When I create custom characters using the tool in the WYSIWYG editor, they display fine, but when I press Save, they are converted into question marks.

    On a brand new fresh install, if I copy and paste these characters into the Post box:

    & " ¢ € £ ¥ © ® TM ‰ µ · • ... ′ ″ § ¶ ß
    ‹ › « » ‘ ' " " ‚ „ < > ≤ ≥ - - ¯ ‾ ¤ ¦
    ¨ ¡ ¿ ˆ ˜ ° - ± ÷ ⁄ × ¹ ² ³ ¼ ½ ¾ ƒ ∫ ∑
    ∞ √ ≈ ≠ ≡ ∏ ¬ ∩ ∂ ´ ¸ ª º † ‡ À Á Â Ã Ä
    Å Æ Ç È É Ê Ë Ì Í Î Ï Ð Ñ Ò Ó Ô Õ Ö Ø Œ
    Š Ù Ú Û Ü Ý Ÿ Þ à á â ã ä å æ ç è é ê ë
    ì í î ï ð ñ ò ó ô õ ö ø œ š ù ú û ü ý þ
    ÿ Α Β Γ Δ Ε Ζ Η Θ Ι Κ Λ Μ Ν Ξ Ο Π Ρ Σ Τ
    Υ Φ Χ Ψ Ω α β γ δ ε ζ η θ ι κ λ μ ν ξ ο
    π ρ ς σ τ υ φ χ ψ ω ← ↑ → ↓ ↔ ◊ ♣ ♥ ♦

    I get these after pressing Save:

    & " ¢ € £ ¥ © ® TM ‰ µ · • ... ? ? § ¶ ß
    ‹ › « » ‘ ' " " ‚ „ < > ? ? - - ¯ ? ¤ ¦
    ¨ ¡ ¿ ˆ ˜ ° - ± ÷ ? × ¹ ² ³ ¼ ½ ¾ ƒ ? ?
    ? ? ? ? ? ? ¬ ? ? ´ ¸ ª º † ‡ À Á Â Ã Ä
    Å Æ Ç È É Ê Ë Ì Í Î Ï Ð Ñ Ò Ó Ô Õ Ö Ø Œ
    Š Ù Ú Û Ü Ý Ÿ Þ à á â ã ä å æ ç è é ê ë
    ì í î ï ð ñ ò ó ô õ ö ø œ š ù ú û ü ý þ
    ÿ ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?
    ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?
    ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?

    Is your database your local database?

    I don't know what that means.

    Do you have control over it or is it hosted for you?

    WordPress was installed by Fantastico on cPanel. I have as much control over it as cPanel gives.

    Do you know if IT can handle utf-8 characters?

    Do I know if a database can handle UTF-8 characters? I don't know what you're asking.

  7. palamedes
    Member
    Posted 6 years ago #

    Is your host a windows machine or a linux machine? What exactly is it?
    What version is your mysql database?
    What version is your PHP?
    Do you have access to your mysql.ini file?

    These are very basic questions you'll need to be able to answer for us to help you.

    And I suggest you read up on UTF-8 to understand more about the problem:

    http://en.wikipedia.org/wiki/UTF-8

  8. Endolith
    Member
    Posted 6 years ago #

    I know about UTF-8 already. I don't know about mysql or PHP. I'll try to find this out.

  9. Endolith
    Member
    Posted 6 years ago #

    Apache version 1.3.37 (Unix)
    MySQL version 4.1.22-standard-log
    PHP version 4.4.6

  10. Endolith
    Member
    Posted 6 years ago #

    I can access phpMyAdmin, which shows my WordPress databases? I have three installations, including a brand new one, and it lists _wrdp1, _wrdp2, _wrdp3. Inside are "Tables" with "Collations", some of which are utf8_general_ci and some of which are latin1_swedish_ci. On the brand new installation, they are all latin1_swedish_ci.

  11. palamedes
    Member
    Posted 6 years ago #

    if you have myadmin for mysql that will help greatly.

    Go into the wordpress database for the one that you know you have entered multibyte characters and look at the wp_posts table.

    the post_content field will have the content of that post and should contain those characters.

    Are they ???? or actual characters?

    Or if you create a brand new post and save it as a draft, making sure its the only draft you can run the query;

    select post_content from wp_posts where post_status = 'draft';

    If the symbols in the database are ????? then it could be that the database isn't correctly set to utf8 or it could be that the multibyte string stuff for php isn't working.

    DO you have access to a phpinfo() page?

  12. palamedes
    Member
    Posted 6 years ago #

    If you do have access to a phpinfo page do a search for "mbstring".. "Multibyte Support" must be enabled.

    (Someone correct me if thats incorrect.. I assume you guys use mbstring for all the multibyte stuff as opposed to rolling your own.. )

  13. deuced
    Member
    Posted 6 years ago #

    some of which are utf8_general_ci and some of which are latin1_swedish_ci. On the brand new installation, they are all latin1_swedish_ci.

    I think the problem is collation. You should have all your DB utf8_general_ci. Otherwise your posts are saved wrong in the DB which then appearing unreadable in your blog.

    You 'll find more about converting DB characters sets HERE.

    PS: That might happened because you installed WP via Fantastico. So also check you config.php

  14. Endolith
    Member
    Posted 6 years ago #

    Go into the wordpress database for the one that you know you have entered multibyte characters and look at the wp_posts table.

    Are they ???? or actual characters?

    They are question marks in the phpMyAdmin display of wp_posts.

    DO you have access to a phpinfo() page?

    If you do have access to a phpinfo page do a search for "mbstring".. "Multibyte Support" must be enabled.

    '--enable-mbstring' '--enable-mbstr-enc-trans' '--enable-mbregex'

    mbstring
    Multibyte Support enabled
    Japanese support enabled
    Simplified chinese support enabled
    Traditional chinese support enabled
    Korean support enabled
    Russian support enabled
    Multibyte (japanese) regex support enabled

    mbstring.internal_encoding ISO-8859-1

    I think the problem is collation. You should have all your DB utf8_general_ci.

    They are apparently set as latin1_swedish_ci by default?

    PS: That might happened because you installed WP via Fantastico.

    Ah, ok.

    So also check you config.php

    What am I checking for?

  15. deuced
    Member
    Posted 6 years ago #

    In a default installation you should have these two lines in your config.php

    define('DB_CHARSET', 'utf8');
    define('DB_COLLATE', '');

    Your DataBase and all your tables should be utf8_general_ci and this is crucial for all of us using other than latin characters. Backup and then read the link i provided you.

    Shouldn't WordPress be able to handle characters generated by itself??

    WordPress gets and displays what's stored in your DataBase. If you see unreadable characters in your DataBase then these characters appear in your posts.

  16. palamedes
    Member
    Posted 6 years ago #

    Yep deuced is on to it.. Nice catch man.

  17. Endolith
    Member
    Posted 6 years ago #

    In a default installation you should have these two lines in your config.php

    define('DB_CHARSET', 'utf8');
    define('DB_COLLATE', '');

    I do, in wp-config.php

    define('DB_CHARSET', 'utf8');

    define('DB_COLLATE', '');

    Your DataBase and all your tables should be utf8_general_ci and this is crucial for all of us using other than latin characters. Backup and then read the link i provided you.

    They are not, though, and they are not configured this way by default. Why not? My brand new installation is all latin1_swedish_ci.

  18. Endolith
    Member
    Posted 6 years ago #

    I tried this plugin on my test installation:

    http://g30rg3x.com/utf8-database-converter/

    but any posts with special characters were destroyed. Are there any other solutions for this? Are there alternatives to WordPress that don't have problems like this?

  19. Endolith
    Member
    Posted 6 years ago #

    Oh wait. phpinfo has mbstring.internal_encoding ISO-8859-1, which is Latin1. Is that the fundamental problem?

  20. Endolith
    Member
    Posted 6 years ago #

    Alright screw it. I only have 12 posts, so I backed up the SQL database, backed up the actual content of the posts by saving them as HTML and copy-pasting them into Google Docs, went through and manually removed any non-ascii characters I could find in the hopes that they will convert without corruption, and now I'm going to use the plugin even though it's only for old versions of WordPress.

  21. Endolith
    Member
    Posted 6 years ago #

    .... and most of the posts survived. Some were truncated, but whatever, I made backups.

  22. Endolith
    Member
    Posted 6 years ago #

    Note: When backing up your website as HTML, don't click "HTML complete". Just "HTML", or it overwrites your file paths. :'(

  23. Deusdies
    Member
    Posted 6 years ago #

    Here's a solution for your problem:

    1) Open wp-config.php;
    2) Find:

    define('DB_CHARSET', 'utf8');
    define('DB_COLLATE', '');

    3) Change it to:

    /*define('DB_CHARSET', 'utf8');
    define('DB_COLLATE', '');*/

    Cheers ;-)

  24. theblaine.com
    Member
    Posted 6 years ago #

    I changed my wp-config file to what Deusdies suggested and it worked great!! Thanks.

  25. daveingland
    Member
    Posted 5 years ago #

    After upgrading an old version of WordPress and separately doing a new install for 2.5.1 I am having the same problem with foreign language characters being displayed as question marks and weird symbols. All is fine when typing out the post, but the problem comes when it is saved...it displays the weird symbols after that.

    On the upgrade from the older version of WordPress, that install displayed the foreign language (Korean) characters just fine. The new install never has.

    I found info in another board that advised to just delete:

      define('DB_CHARSET', 'utf8');
      define('DB_COLLATE', '');

    and when I did that, the Korean characters displayed perfectly. However, when I went to log in to administer the site I was locked out with error strings. The pages are visible, but I can't log in. Then, as advised in the above post to this forum I changed the wp-config.php to this:

      /*define('DB_CHARSET', 'utf8');
      define('DB_COLLATE', '');*/

    and now I can't even get the page to display even before I try to login. Now I get this error message:

      c_html/blog/wp-includes/pluggable.php on line 694

    Any ideas? The site that was upgraded to 2.5.1 is at: mylivingstone.com/blog.

    Thanks!

  26. daveingland
    Member
    Posted 5 years ago #

    Sorry, for some reason the full error message didn't post above. It is:

      Warning: Cannot modify header information - headers already sent by (output started at /home/mylivin/public_html/blog/wp-config.php:55) in /home/mylivin/public_html/blog/wp-includes/pluggable.php on line 694
  27. daveingland
    Member
    Posted 5 years ago #

    I just took a new wp-config.php file from WordPress and input the db info and the code from Deusdies and now it works fine. I guess somehow my wp-config.php file got corrupt.

  28. noach
    Member
    Posted 5 years ago #

    I've worked my way through all the fixes posted above with no success (I think I'm ok with the under-the-hood stuff but by choice would rather be driving ....).

    The situation is that in two separate new installs on two unrelated servers via Fantastico latest and cpanel latest displays only ????? when I post anything that my browser normally accepts as Hebrew utf-8. The situation is different on my wordpress.com blog: it works perfectly.

    Among the MySQL tweaks that I tried were:

    - changing the schema structure so that collation showed utf8_general_ci but the individual tables still showed latin1-swedish-ci

    - laboriously changing every char field (column) in every table to utf but the schema didn't show any difference afterwards

    -making sure that all my default encodings in phpSQLAdmin were utf8 I created a duplicate database and imported all the tables and data from the orginal, only to find that char fields had determinedly stuck to latin1-swedish-c1

    - having tried the edits to config.php, in exasperation I've returned here to keep the topic open.

    My best guess at this stage is that server caching got in the way of all these changes. I cleared my browser cache several times to make sure.

    Any advice? besides sticking only to the wordpress.com subdomain, which is not the most efficient solution at this end.

    Thanks

  29. deuced
    Member
    Posted 5 years ago #

    ...via Fantastico latest and cpanel...

    This is what I WOULD DO if i were you:

    - Install a clean WordPress via FTP (dont mess with one click things)
    - Make sure your config.php has this settings (which are the default):

    define('DB_CHARSET', 'utf8');
    define('DB_COLLATE', '');

    - After installation make sure all your tables are utf8 general_ci
    - Import what u got from WordPress.com or your backup

    It should work with no problems.

  30. noach
    Member
    Posted 5 years ago #

    Thanks - I'll give it a go.

Topic Closed

This topic has been closed to new replies.

About this Topic