Encoding type of uploaded file

leatherback

Hi All,

Just a few days ago I ran into some encoding trouble with pages displayed from my website; If I -by ftp- upload a file encodes in say ISO8859-1, but my site is encoding UTF-8, the german special characters are not displayed correctly.

As I am building an international portal for the storage of hyperspectral signatures, I have many potential encodings. I have set my forms to allow_enc_type=UTF-8, which works for copy-past, or typed text.

How do I ensure proper encoding of files uploaded to the server? Each file will be read and inserted in separate sections of the database, so doing a sniffer on the file is not a big problem. However, I have no clue how to do this.

Anybody out there who has done this, or can advice me on how to do this?

Cheers,

Jelle. :o

stolzyboy

can you use the utf encode/decode functions??

http://us2.php.net/manual/en/function.utf8-decode.php //Converts a string with ISO-8859-1 characters encoded with UTF-8 to single-byte ISO-8859-1
http://us2.php.net/manual/en/function.utf8-encode.php //Encodes an ISO-8859-1 string to UTF-8

leatherback

Yes I could. Once I know which formatting the input file is in.

Let me specify: there are ways to convert one encoding to another. What I cannot figure out is how to find the encoding. My users will NOT be programmers, so asking them.. Will not result in useful answers I am afraid.

stolzyboy

i'm not sure if you've ran across this or not... and i admit i didn't fully read it, but it talks alot about encoding of files and PHP issues

http://www.phpwact.org/php/i18n/charsets?s=utf8

hth,
stolzyboy

leatherback

Well, I came across a lot of pages claiming PhP is not good at doing \it.. So I decided to put the challenge here, and see what the common ground is. For sure it can't be true that here is no good php way to detect character encoding!?

Roger_Ramjet

There is the mb detect encoding function. I think you would have to open the file and read some of it into the function to find out the encoding of the file.

The user notes have a lot to say on the issue so have a read.

leatherback

Heya Roger,

Over the weekend I will give those a try. However the seriously long thread of user comments makes me feel it will trip. Do you have experience using these?

Cheers,

Roger_Ramjet

Me? No. Best of luck with it.

stolzyboy

i read thru the MB as well before my previous post... however it didn't look too promising and i guess i figured you'd prolly ran across it already

leatherback

Hey Stolzy,

Yeah, It looks like I will have to use an external language to do this, which is a bit of insane, to be honest..

I am surprised however.. The web is an international environment, and you'd expect webserver to automatically figure out the encoding.

hm..

Roger_Ramjet

It looks like you are uip against it with this one Leatherback. I found a Sun Solaris utility auto_ef that does it, but it is only Solaris 10, can't find anything like it for linux. You can dl and use Solaris 10 for free so you could use it for your server platform. Don't know is it comes with the source, if it does then you could try porting if you don't want solaris.

Otherwise you can get Solaris 12 preview with the full code base and I assume that auto_ef is a part of it.

Have fun and let us know how you get on.

I'd have a go at porting it to Linux myself if I was not busy getting up to speed with VS 2005 express - need to make myself competitive in today's job market. Still might do when I need a break from that, be a good thing to put on my CV.

leatherback

Hey Roger,

Thx for the effort.

Problem is: in the end I want to makle the database library available for anyone to download. So I rather have everything as 'simple' as possible. Hit a button and go, is what I am after; Rather not having to install other software on the server (Let alone the techies (VPAC.org) on this project, who want to test-run all my stuff before it is life..). And no.. They did not have a solution either ;-)

I'll work something out. One day..

Roger_Ramjet

Well, what I thought you wanted to do was to find out what the incoming encoding is so that you could convert all to one standard for storage.

leatherback

Roger Ramjet wrote:
Well, what I thought you wanted to do was to find out what the incoming encoding is so that you could convert all to one standard for storage.

True, that is the primary aim. But things are never easy in life, are they ;-)

Seriously, the ideas are appreciated! It is just as I said.. The database will be running online (In fact, is life right now), and will need to be able to decode all kinds of encodings, which are unknown.

But once the database has verified, and is through beta testing, the package as a whole will be made available as well. And you know what it is like, packages which require all sorts of additional installs are just a pain..

MarkR

The main problem is that if it's just a plain text file, there's no reliable way of telling what encoding is in use.

FTP provides no metadata to indicate this, and operating systems usually don't store any on their filesystem either.

Some files contain a byte order mark, which will indicate if the file is UTF-8, or UTF-16, but otherwise, you're just going to have to guess.

One option would be to assume the file is in utf-8, and if it contains invalid UTF-8 sequences, then assume it's latin1.

But that does nothing to cater for other 8-bit character sets (e.g. latin2, cyrillic etc), or more exotic ones (big5, shift-jis)

Mark