" is often slower, but easier to use. An example:
$var = "pie";
$string = "Bob like $var";
This then makes $string "Bob likes pie."
However, this:
$var = "pie";
$string 'Bob likes $var';
Does not equate to "Bob likes pie," but, rather, "Bob likes \$var."
Note that the $ is escaped, and there in lies the major difference.
Usually you will not notice any performance gain over using " and ', unless you put a \ or a $ inside. When it hits the $, it must read every possible letter for variable names, and stop when it reads one that doesn't work. For instance:
$var = "pie";
$string = "Bob likes $vars";
Will not work, since $vars is not defined. It does not bother to backwards check for $var, since that would slow it down even more.
So, while personal taste does come in, I prefer to just use ' in most cases, then concatenate variables:
$var = 'pie';
$string = 'Bob likes ' . $var;