Actually, the problem with
$query = "SELECT * FROM tablename ORDER BY '$ordering' ";
.. is that the result is
SELECT * FROM tablename ORDER BY 'dateSet DESC'
... and those single quote marks do not belong in the SQL. Just take them out and the problem will go away.
Jumping out of the string, and concatenating instead, as pierks suggests, is not necessary.
Read the section of the PHP manual that explains string substitution in double-quoted strings. It's very important and, judging from many postings here, is generally misunderstood.
http://us3.php.net/manual/en/language.types.string.php#language.types.string.parsing