carriage return is displayed with &iquest;

keneso

Hi,

in the mail I get from a form's textarea input the special chracters, and carriage returns are changed to ¿ (single for the special character, and double for the carriage return), making the inputted text hard to read.

Below is the processor's code.

I don't know php, so an actual solution would be very appreciated.

Thank you.

<?php
//MAIL HEADER INFORMATION
$EmailFrom = "site name";
$EmailTo = "email to send form to";
$Subject = "message from site";

// CLEANUP ANY NON-PRINTABLE OR UNWANTED CHARACTERS
function clean_text_string($string) { // FORCE IT TO ACCEPTABLE ESCAPABLE CHARACTERS ONLY
        $new    = trim(ereg_replace('[^\\' a-zA-Z0-9&!#$%()"+:?/@,_\\.\\-]', '&iquest;', stripslashes($string)));
        $new    = ereg_replace(' +', ' ', $new);
return ( $new );
}

// ITERATE OVER POST DATA
foreach ($_POST as $key => $value) {
   $clean_key   = strtolower(clean_text_string($key));
   if (substr($clean_key,0,1) == '_') continue; // SKIP FIELD NAMES THAT START WITH UNDERSCORE
   $clean_value = clean_text_string($value);
   if ($clean_value == '') continue; // SKIP EMPTY FIELDS
   $clean_post["$clean_key"] = $clean_value;
}

// NOW TEST FOR FIELDS THAT ARE REQUIRED
$required = array(); // ADD YOUR FIELDS AS NEEDED
$all_okay = TRUE;
foreach ($required as $key) {
   if ($clean_post["$key"] == '') {
      echo "<br/>$key is a required field\
";
      $all_okay = FALSE;
   }
}

// TEST FOR MISSING INPUT
if (!$all_okay) {
   $referer = $_SERVER['HTTP_REFERER'];
   echo "<br /><strong><a href=\\"$referer\\">Please click to go back, and fill the required fields!</a></strong>\
";
   die();
}

// PREPARE THE DATA
$name = Trim(stripslashes($_POST['name']));
$lastname = Trim(stripslashes($_POST['lastname']));
$email = Trim(stripslashes($_POST['email']));
$comment = Trim(stripslashes($_POST['comment']));

// PREPARE EMAIL BODY TEXT
$body = '';
$body .= "Dear " . $name . " " . $lastname . " thank you for your message. \n
we will contact you asap." . "\n \n"; // THIS IS TO HAVE PERSONALIZED MESSAGE
foreach ($clean_post as $key => $value) {
   $body .= $key . ': ' . $value . "\r\n";
}

$submitter = $_POST["email"];
$noreply = "no-reply@site.com";

// send email
$success = mail($EmailTo, $Subject, $body, "From: <$EmailFrom>");
		   mail($submitter, $Subject, $body, "From: <$noreply>");

// redirect to success page
if ($success){
header( "Location: path/to/thankyou.php" );
}

else
{print "There has been a technical problem, please resend, thank you."; }
?>

sid

Changing the line

        $new    = trim(ereg_replace('[^\\' a-zA-Z0-9&!#$%()"+:?/@,_\\.\\-]', '&iquest;', stripslashes($string)));

        $new    = trim(ereg_replace('[^\\' a-zA-Z0-9&!#$%()"+:?/@,_\\.\\-]', '', stripslashes($string)));

would probably be the quickest fix for someone without PHP experience.

keneso

Thank you.

I had thought about it, but wasn't sure if it could have been a security issue, I'll try it.

Ou of curiosity, what would it be for experienced people?

sneakyimp

keneso;11043589 wrote:
Ou of curiosity, what would it be for experienced people?

Well, among other things, an experienced coder would know that [man]ereg_replace[/man] and other ereg_* functions have been deprecated as of PHP 5.3.0. An experienced coder would also know what pattern matching is such that she would know what characters are being replaced with question marks and which characters are permitted to remain.

keneso

sid1 wrote:
Changing the line ... [cut]

Your suggestion removes the ¿ but it does not trigger the carriage return, now I get two words attached without space, space which before was filled by ¿
e.g.
user input in the form:

Hi my name is
keneso

what I see in my mail (before)

Hi my name is&iquest;keneso

what I see in my mail (after)

Hi my name iskeneso

sneakyimp;11043611 wrote:
Well, among other things, an experienced coder would know that [man]ereg_replace[/man] and other ereg_* functions have been deprecated as of PHP 5.3.0. An experienced coder would also know what pattern matching is such that she would know what characters are being replaced with question marks and which characters are permitted to remain.

Instead of telling me what an expert coder would know, perhaps would have been more intructive to tell me what solution would have been suggested to an expert coder; who BTW obviously wouldn't be that much of an expert if needed to ask something like what I asked.

If I were an expert I wouldn't have started the thread in the first place.
And BTW I knew about the ereg_replace, but as a NON expert don't know how to change to preg_replace, meaning not sure if it would be enough to just edit that single function (it's a function right?).

And I don't get a question mark but the whole ¿ written as it's written here, where it shoud be just a carriage return.

sneakyimp

Unless I'm mistaken, ¿ is the html code for an inverted question mark. If you put that sequence of characters in an HTML document, a browser will render it like this: ¿

The ereg_replace statement in that code basically says "take everything that is NOT one of these characters and replace it with ¿" -- or, after you've changed the code as sid suggested it'll change it to the empty string instead. You could try a space instead of the empty string.

As for a "solution", you haven't really specified what you are looking for. Additionally, I provided links to documentation that you might find helpful, but you appear to have no interest in it.

If -- and I can only guess this is what you want -- you want to keep the newline characters, you'll need to add a newline char to the string of chars in the first parameter supplied to your ereg statement.

NogDog

To really, fully answer your question as to what I would do (making the bold assumption that I qualify as an experienced PHP programmer), I would need to know the precise functional requirements as to why this is being done at all. (E.g., would any of this be necessary if everything from the HTML output to the database configuration/connection supported UTF-8?)

Weedpacket

With the same provisos as others have given (you know a lot more about your code than we do), using an inverted question mark as a replacement character for characters that cannot otherwise be represented is wrong; a proper replacement character should be used (encoded in HTML as either � or &#65533[noparse]😉[/noparse]. But, again, that depends on what the actual needs are (it shouldn't be used in domain names, for example, for security reasons).

keneso

Thank you all for your inputs, greatly appreciated.

sneakyimp1 wrote:
The ereg_replace statement in that code basically says "take everything that is NOT one of these characters and replace it with ¿" -- or, after you've changed the code as sid suggested it'll change it to the empty string instead.

Thank you for this clear explanation, I had kind of guessed that it had to do with replacing from the "replace" in ereg_replace, but didn't know what, and how, you puting it in plain English helped a lot.

sneakyimp wrote:
You could try a space instead of the empty string.

Done so already, thank for confirming.

sneakyimp wrote:
As for a "solution", you haven't really specified what you are looking for. Additionally, I provided links to documentation that you might find helpful, but you appear to have no interest in it.

My apologies, I thought stating that "carriage return", and "special characters" change to ¿ was clear, obviously I failed.
Believe me I read it, but it just goes beyond my capabilities, I can kind of understand it (though your clear English translation is more clear), but putting it into practice is something else, that is for a NON expert who is afraid of compromising security.

sneakyimp wrote:
If -- and I can only guess this is what you want -- you want to keep the newline characters, you'll need to add a newline char to the string of chars in the first parameter supplied to your ereg statement.

And you guess right, I would like to have the new line, and my special chracters display.

NogDog wrote:
To really, fully answer your question as to what I would do (making the bold assumption that I qualify as an experienced PHP programmer), I would need to know the precise functional requirements as to why this is being done at all. (E.g., would any of this be necessary if everything from the HTML output to the database configuration/connection supported UTF-8?)

Weedpacket wrote:
With the same provisos as others have given (you know a lot more about your code than we do), using an inverted question mark as a replacement character for characters that cannot otherwise be represented is wrong; a proper replacement character should be used (encoded in HTML as either � or &#65533😉. But, again, that depends on what the actual needs are (it shouldn't be used in domain names, for example, for security reasons).

As my last comment to sneakyimp I need to have the carriage return (new line), and special characters display in the email I get from my form.

First of all, I again apologize for not having been very clear about my need, secondly anyone here is an expert compared to my knowledge, thirdly I got this code from an existing form for a non-profit org that I am doing the work, and don't know who was the original coder.

The code was originally done to both send email, and to save into a database, and now I adapted it to only send the email.

It is a form for users of Italian, and English languages, the Italians have some accented vowels (i.e. à ò ì ù è) characters and when they fille the text area of the form, those characters are displayed as ¿ - now they are just empty spaces, as said earlier by removing the ¿ from ereg_replace; and obviously the carriage return.
While for the carriage return now with the emty space is readable, the special characters missing makes some text hard to interpret.

Basically my need is to add those special (accented) characters, and the new line to the ereg_replace [^\' a-zA-Z0-9&!#$%()"+:?/@,_\.\-], if I got right sneakyimp's suggestion.

Of course it would be greatly appreciated if I can change the deprecated ereg with preg_replace, I mean is it sufficient to just change ereg_replace with preg_replace, or by doing so I need to change the whole line with new string that I wouldn't know how to write?

Thank you again.

NogDog

The one difference you'll have to address with preg_replace() is that its regexes require beginning/ending delimiters (in part so that you can add optional modifiers after the closing delimiter). You can use most special characters as the delimiter, though "/" is sort of the standard. Whatever delimiter is used, you must then escape that character within the regex itself.

preg_replace('/[^a-z\/]/i', ' ', $string);

That would replace anything that is not a-z, A-Z (due to the case-insensitive "i" modifier), or "/".

keneso

Thank you all.

By googling, doing, and redoing, inspired by your suggestions I found this that works with ereg, I tried the same using preg, but it returns a partially blank mail, printing only the partse that do not have to do with the form's fields.

trim(ereg_replace('[^\' \p{L}a-zA-Z0-9&!#$%()"+:?/@,_\.\-\r]#u', ' ', stripslashes($string)));

I am happy with it, as like said it works, but if you can kindly help me with the preg to improve and have up to date code I would greatly appreciate.

sneakyimp

keneso;11043645 wrote:
My apologies, I thought stating that "carriage return", and "special characters" change to ¿ was clear, obviously I failed.

If you read your original post again, you'll realize that you didn't ask any questions or express what your desired outcome would be. Always remember to ask a question or at least clearly express what you want to happen.

keneso;11043645 wrote:
Believe me I read it, but it just goes beyond my capabilities, I can kind of understand it (though your clear English translation is more clear), but putting it into practice is something else, that is for a NON expert who is afraid of compromising security.

I think the documentation is pretty clear:

PHP Documentation wrote:
This function scans string for matches to pattern, then replaces the matched text with replacement.

keneso;11043645 wrote:
It is a form for users of Italian, and English languages, the Italians have some accented vowels (i.e. à ò ì ù è) characters and when they fille the text area of the form, those characters are displayed as ¿ - now they are just empty spaces, as said earlier by removing the ¿ from ereg_replace; and obviously the carriage return.
While for the carriage return now with the emty space is readable, the special characters missing makes some text hard to interpret.

Obviously, if you have to include a huge long list of all the possible accented characters in the world, this statement is going to get really complicated. This brings up the question the other guys were talking about: why are you doing this? I'm guessing that the original code took this fairly drastic measure to clean up the submitted data as some misguided attempt to approve security or to make sure the string could be emailed or stored in a database or something. What it in fact manages to accomplish is that the only characters permitted are those listed. Anything else (accented characters, other punctuation, newlines, Japanese text, Chinese text, arabic, etc.) will simply be removed.

A proper approach would not require that ereg_replace line, you would just need to be careful to make sure that your character encoding is correct, which involves making sure that your html page, your form, your database, etc. all have the same charset. You would also benefit from sending mail with PHPMailer or PEAR::Mail instead of the built-in PHP mail function. I believe those other tools should handle character encoding better, but am not 100% sure about it.

As for improving security, this drastic measure is probably pretty effective. One can't specify any html tags because the angle brackets are not permitted. Offhand, I don't even really know what the security threats are in an email message beyond specifying links or references to dangerous content. Perhaps someone else could chime in on this matter?

keneso

Thank you.

If you read your original post again, you'll realize that you didn't ask any questions or express what your desired outcome would be. Always remember to ask a question or at least clearly express what you want to happen.

I've already apologized for not having conveyed my needs, no need to reiterate.