Double quotes evaluate what is inside.
For example:
$cowsay = 'moo';
echo "Cows say $cowsay.";
would evaluate to "Cows say moo.".
If one used single-quotes:
$cowsay = 'moo';
echo 'Cows say $cowsay.';
would evaluate to "Cows say $cowsay.".
It's really that simple. For that reason (replacement/evaluation, or lack thereof), single quotes are marginally faster.
Another difference (due to replacement):
The only thing that needs escaping in a single-quoted string, is a single quote (\'), but anything else is ignored (including \n).