Detecting Language a user puts in

[2deep]

I have been looking for something like this, but can only find detection of the users language. But thats not what I am looking for.

I tried to build it myself, but to be honest it's not very beautiful and not (not even nearly) a good solution.

My goal is: in a form, A user types into a tetarea and presses OK.

Now, how do you figure out, what language he has typed. (I need to check between THAI, ENGLISH, JAPANESE, CHINESE, FRENCH, GERMAN (at least, more would be better).

My first try was see if there is a word which is used often (eq. AND in English)
it works, but not full proof.

Who has an idea?

[2deep]

function AutoDetectLang($str) {
// try detecting English
$pos1 = strpos($str, " the ");
$pos2 = strpos($str, " and ");
$pos3 = strpos($str, "และ");
if ($pos1 > 0 and $pos2 > 0) {
return "EN";
} elseif ($pos3 > 0) {
return "TH";
} else {
return "UNABLE TO DETECT LANGUAGE";
}

// maybe should do a check on how many words vs characters (longest word in english is 30)

}

[Piranha]

I would say that what you try to do is impossible. Why? Because there are words that may be the same in more than one language. What language is it if the text consists only of those words? Then of course there may be quotations, how would you consider a text that is part english and part french? To solve the problem just make a radio button, then the user can easily select what language it is.

[NogDog]

I agree with Piranha that it's probably more accurate to have the user specify the language manually, but as an interesting exercise I came up with the following:

function guessLanguage($text)
{
   // words to look for by language:
   $words = array(
      'english' => array('a', 'an', 'the', 'and', 'or', 'I', 'you', 'he', 'she', 'they'),
      'french' => array('un', 'une', 'le', 'la', "l'", 'les', 'et', 'ou', 'je', 'tu', 'vous', 'il', 'elle', 'ils')
   );
   // convert to PCRE regexes:
   array_walk_recursive(
      $words, 
      create_function('&$v, $k', '$v = "/\\b$v\\b/i";')
   );
   foreach($words as $key => $array)
   {
      $count[$key] = 0;
      foreach($array as $regex)
      {
         $count[$key] += preg_match_all($regex, $text, $matches);
      }
   }
   arsort($count);
   return(array_shift(array_keys($count)));
}

[2deep]

First of all, thank you guys for thinking along.

I understand it is difficult and will never be 100% fullproof, but I would like to TRY to autodetect it, and if it can't detect it, the user should pick. Like a service to the customer.

Now the thing you tried NogDog is what I tried as well, the thing is, it works for english and french and german for the most part, but the problem is with the match it does not seem to tell the difference between THAI and Chinese.

Might have to do something with character encoding type.

I know it is a bit of a challenge, but if MS word can do it, we should be able to do it.

I will post later on today about my findings...

cheers guys

[NogDog]

To work with Unicode characters, you'll need to use PHP's Multibyte String functions.

[2deep]

<?
if ($POST['action'] == "") { ?>
<form method="post" action="etranslate.php">
<input type='hidden' name='action' value='1'>
<textarea id="elm1" name="elm1" rows="10" cols="90"></textarea>
<input type="submit" name="save" value="Submit" />
<input type="reset" name="reset" value="Reset" />
</form>
<?
} else {
echo "".AutoDetectLang($POST['elm1']);
}

function AutoDetectLang($str) {
if (preg_match('/[\&#3585;|\&#3586;|\&#3587;|\&#3588;|\&#3589;|\&#3590;|\&#3591;|\&#3592;|\&#3593;|\&#3594;|\&#3595;|\&#3596;|\&#3597;||@|&]/', $str)) {
echo "THAI was found.";
} else {
echo "THAI was not found.";
}
if (preg_match('/[\&#35830;|\&#27604;|\&#35199;|\&#36842;|\&#20234;|\&#33406;&#24343;|\&#21513;|\&#33406;&#23610;|\&#33406;|\&#26480;|\&#24320;|\&#33406;&#21202;|\&#33406;&#39532;||@|&]/', $str)) {
echo "CHINESE was found.";
} else {
echo "CHINESE was not found.";
}
}
?>

and I changed php.ini to:
[mbstring]
; language for internal character representation.
mbstring.language = Neutral

; internal/script encoding.
; Some encoding cannot work as internal encoding.
; (e.g. SJIS, BIG5, ISO-2022-*)
mbstring.internal_encoding = UTF-8

; http input encoding.
mbstring.http_input = auto

; http output encoding. mb_output_handler must be
; registered as output buffer to function
mbstring.http_output = UTF-8

; enable automatic encoding translation according to
; mbstring.internal_encoding setting. Input chars are
; converted to internal encoding by setting this to On.
; Note: Do not use automatic encoding translation for
; portable libs/applications.
mbstring.encoding_translation = On

; automatic encoding detection order.
; auto means
mbstring.detect_order = auto

; substitute_character used when character cannot be converted
; one from another
;mbstring.substitute_character = none;

; overload(replace) single byte functions by mbstring functions.
; mail(), ereg(), etc are overloaded by mb_send_mail(), mb_ereg(),
; etc. Possible values are 0,1,2,4 or combination of them.
; For example, 7 for overload everything.
; 0: No overload
; 1: Overload mail() function
; 2: Overload str() functions
; 4: Overload ereg() functions
mbstring.func_overload = 2

; enable strict encoding detection.
;mbstring.strict_encoding = Off

---

Because I was thinking, doing a check on foeign characters first and after that within the latin characters do a check on words (like yours and, a, an)

[Piranha]

2deep wrote:

I understand it is difficult and will never be 100% fullproof, but I would like to TRY to autodetect it, and if it can't detect it, the user should pick. Like a service to the customer.

I would do it the other way. Let the customer pick language at once, and if they don't pick a language you have to choose the language. Then of course it is possible that customers are always posting in the same language, then they only need to pick a standard language, not change every time they post a new text.

[2deep]

Well the actual thing I need to make is:

A user posts a file (.doc) then I read it out, (try autodetecting the language).

I would like to auto detect first, if not found then select.
Something like: we could not auto detect your file's language, or we found more then 1 language. please select your input language)

I think I should check on which alphabet is being used.
Latin
Thai
Chinese
Japanese

If in Latin, Can make more checks for "possible"-language

[Piranha]

Why didn't you start with that question? I don't know how .doc files is built, but I'm sure that there are information about the language in the file, if so then you don't have to invent something to chose the right language.

[2deep]

I know you can open the document, but for as far as I know the language property is not set in the document, it does it on the fly when opening the document.

can't find the property resource for ActiveX word.application as well.