insted of using != try using [man]strstr[/man] or [man]strpos[/man] because if they post from a page on your site like http://yoursite.com/page.php != will be true, even though they are coming from your site, its not an exact match.
if (strpos($_SERVER['HTTP_REFERER'], "yoursite.com") === FALSE) //bad referer
this will check if yourdomain.com is contained in the http referer. they can still use sockets and formulate their own http post requests and spoof the referer (if they are more than just kiddies copying your form). if it gets that bad, you may need to implement an image verification feature where they have to type the text from a randomly generated image.