phpmyadmin complaints

sneakyimp

so I think I have my phpMyAdmin installed properly, but it is complaining about a couple of things that I fear may have wider relevance in my wider MySQL universe:

complaint 1

The mbstring PHP extension was not found and you seem to be using a multibyte charset. Without the mbstring extension phpMyAdmin is unable to split strings correctly and it may result in unexpected results.

complaint 2

Cannot load mcrypt extension. Please check your PHP configuration.

Can anyone shed some light on this? What does it mean 'using a multibyte charset'? Is that a MySQL config or a phpmyadmin config?

sneakyimp

I think this is a PHP issue because I was able to install the two modules using the CentOS yum command:

yum install php-mbstring

that fixed the mbstring complaint.

then this fixed the mcrypt complaint:

yum install php-mcrypt

then of course the obligatory reboot:

service httpd restart

Horizon88

Looks to me like phpMyAdmin requires those two modules to run as recommended, and you just didn't have them installed, as you obviously noticed. 😃

sneakyimp

Yes but when it says 'you seem to be using a multibyte charset' what on earth is it referring to?? Is this a basic MySQL setting that I should think about changing? Does it mean the current table? Is it referring to mysql client, mysql server, or mysql connection?

and what the hecks are charsets good for anyway?

Horizon88

As I understand it, charsets allow you to use various characters from other languages. If you set it to Western IS-whatever, it'll use basic English characters, but Simple Chinese or whatever has support for chinese characters, instead of just showing random garbage.

As to what a multibyte charset is, I've never heard of it. 🙁

sneakyimp

In English, we have 26 letters (A-Z) and some punctuation. That means we can get away with using a byte for any given letter (which gives us 255 possibilities).

Chinese and Japanese don't have letters, they have some 20-40,000 'words' which constitute their language. For this reason, one needs two or more bytes to describe and distinguish all the possible characters for display.

What's confusing me is that I believe that the multi-byte character set support issue is one that relates not just to PHP and Apache and MySQL and phpMyAdmin but cuts right to the heart of your operating system. For instance, a simple text file might need to contain multibyte characters, in which case nano better be ready to interpret them. And what about the apache httpd.conf file? Can you have comments in chinese in there? Or will apache barf and fail to start?

I'm sure the multibyte character set issue manifests itself at many levels in your LAMP stack. Given that I've received a complaint from a layer in my stack, I'm hoping to understand if the complaint goes further or if I'm safe. Like what happens when my site becomes all the rage in Japan and someone tries to type Japanese comments in my forum? What happens then? Will my server crash? Will it output jibberish? Will my server handle it and output chinese chars which cause my visitors browsers to crash? Will a sentient virus results that takes over the entire NET?

troybtj

UTF-8 is a multibyte character set that gives more than 254 characters, some windows fonts have more characters than others. When in MS Win Character Map, it will show a 4 digit Hex code for the character instead of a 2 digit (255 chars).

This is something 'new-ish' in terminal emulation to give full line drawing in addition to character symbols for squared, cubed, half, quarter, etc, in addition to line drawing characters and the full alphabet. I had a nightmare with SecureCRT when it switched to needing the LANG environment variable to: LANG=en_US.UTF-8

I'm sure the multibyte character set issue manifests itself at many levels in your LAMP stack. Given that I've received a complaint from a layer in my stack, I'm hoping to understand if the complaint goes further or if I'm safe. Like what happens when my site becomes all the rage in Japan and someone tries to type Japanese comments in my forum? What happens then? Will my server crash? Will it output jibberish? Will my server handle it and output chinese chars which cause my visitors browsers to crash? Will a sentient virus results that takes over the entire NET?

No, they are stored as 2 or more bytes, and if you treat it as english UTF-8, then it will simply try to translate all the chinese into english, giving you box characters, and a lot of question marks, sort of what looking at a compressed file in a text editor would look like.

sneakyimp

troybtj wrote:
This is something 'new-ish' in terminal emulation to give full line drawing in addition to character symbols for squared, cubed, half, quarter, etc, in addition to line drawing characters and the full alphabet. I had a nightmare with SecureCRT when it switched to needing the LANG environment variable to: LANG=en_US.UTF-8

When you say 'newish in terminal emulation', are you talking about Linux? or SSH? Or windows?

troybtj wrote:
No, they are stored as 2 or more bytes, and if you treat it as english UTF-8, then it will simply try to translate all the chinese into english, giving you box characters, and a lot of question marks, sort of what looking at a compressed file in a text editor would look like.

I think you are mistaken--or at the very least incomplete. I created a file named 'foo.txt' using nano on a 32-bit CentOS linux box and the only thing i typed in it was the number 1. I saved the file and exited nano then listed the file:

[root@server2 html]# ls -l foo.*
-rw-r--r-- 1 root root 2 Oct 16 22:25 foo.txt

As you can see, that's two bytes - but one is for EOF. If you add 3 characters total (e.g., the file only contains the text '123' and nothing more) then the file is going to be 4 bytes.

[root@server2 html]# ls -l foo.*
-rw-r--r-- 1 root root 4 Oct 16 22:25 foo.txt

That's one byte per character and 1 byte for EOF. Or maybe that last byte is something else besides EOF but $file_size_in_bytes=($number_of_chars_in_file +1) for me.

Like I said, the multibyte charset issue exists at serveral levels and likely is handled by Linux, Apache, PHP, and MySQL in some way or other...or maybe not.

troybtj

I was referring to the windows.

Secure CRT 5 Acted very funny because the LANG environment (usually C) wouldn't work until I changed it to the UTF format. Now some curses apps don't like that string, expecting "C"...

Everything else in LAMP is handled by your LOCALE setting, which defines the character set, and if only 255 characters are needed, that is all that will be used. You can set the encoding/charset of each mysql table on creation.

--I should have stated they are stored as 1 or more bytes, but in Windows, it wants to default to two bytes per character in the character map (post Win2k), prior to that it gave the single byte code. Files are stored one byte per character unless using a multibyte character code, then it is stored at the required size.

--ETA: The "newish" term Emulation is xterm in textmode, the default for putty. GNU gettext is in the process of internationalization (I8N), both I8N libs and standard libs are available at this point.

Sorry for mixing apples and oranges (M$/*nix) and making this confusing.

Wiki Says it better: http://en.wikipedia.org/wiki/Variable-width_encoding

Here is an in depth, but outdated article on Unicode Transformation Format (UTF) of different byte sizes. (Base64 encoding is used in email attachments, for example, since the RFC says email must be 7 bits only)
http://czyborra.com/utf/

sneakyimp

alright then....i'm running windows xp here, a 32-bit version and I can easily see all the Japanese chars on http://ad.lolipop.jp/. So I know that windows, at least at the browser level, understands double-byte chars.

Just for kicks, I've pasted some of those japanese characters in this post here:
述べている何考えを有しない。 <-- translate this!

So I'm wondering now if the various linux machines I work on would be able to handle japanese....I tried pasting that in my ssh window and it replaced each char with a dot. I'm guessing that is puTTY that is causing the problem...or maybe it's my preferences? Or perhaps the Fedora linux box on the other side? How does one change the locale setting on a linux box to support multibyte char sets? Suppose I wanted to use UTF 8 instead of ASCII? Is that an install-time config? Is it in the kernel? Or can I just change it and PRESTO everything is suddenly multibye and happy about it? I tend to think it's not.

I'm also wondering if you can use Japanese chars in a filename. Does ext3 support it? What about NTFS? I tend to doubt it...but you never know.

And we haven't even really started in on the Apache/MySQL/PHP bits. This is really confusing. IMHO this should probably have its own thread entirely.

halojoy

sneakyimp wrote:
yum install php-mbstring
that fixed the mbstring complaint.

About the mbstring module.
There are two php.ini settings that several applications wants.
So I have set these both to pass

[mbstring]

; http input encoding.
mbstring.http_input = pass

; http output encoding. mb_output_handler must be
; registered as output buffer to function
mbstring.http_output = pass

;

troybtj

It (UTF-8) is supported by most all *nix vendors, and mostly supported by GNU (linux). It is just another setting in config files.

There are *nix distributions set for every european language (german, dutch, galaic, etc). That is where the whole termcap and terminfo variables come into play for what/how to display and take input.

On Web Pages, multibyte character sets have been universally supported, so they will display depending on your browser, many languages. There are several Cyrillic font packs for XWin to install if you are in a country that needs them.

If you install a system in English, it isn't a fast and easy task to convert it to russian, and some programs (less is an example) will display <0A><14> when it gets confused on hitting a UTF string, the number of programs that have this behavior are dwindling, however.

mysql, php, apache, linux kernel, and other "Major" items in the 2nd link above are now all UTF aware and handle them properly.

Microsoft NT and 2k treat all characters as UTF-16, 2 bytes/character. XP+ use UTF, where 2 bytes are only used as needed.

In mysql-manager (The XWin app), you have two areas to change both the encoding (UTF-x/other), and the character set (latin1, etc) on a database. If data is added that isn't in the charset for that database, it is "de-translated" into what usually looks like garbage, or a '?' for unknown. The rest of the major daemons and languages also support this, since german, english, russian, korean are very popular in the development of linux, they are supported, but the Operating system won't "Translate" properly, since translation means much much more than character mapping.