Good idea on the referer, but not quite. If I were attmpt to exploit this, I would try with a referer of: "your domain here" and also try without a referer. One way or the other, this would bypass the referer check. That is easy enough to get around.
One thing you could try would be to set up htaccess directory protection in apache and give the users the user/password. Have a page with the user/pass combination, and right below have a link to the "htaccess protected directory" with the index file as the mailing file. Then again, if I were going to exploit this, I would have the user/pass and send it in with my request string.
You can spoof the session variable, referer, user-agent, etc pretty easily. In all honesty, I have never been able to think of a way to have a form mail without it bing exploitable. Even if you make people login, and let only those logged in use the form, it can be still bypassed.
This probably didn't help, but I tried. Food for thought. Please let me know what you decide to do. I would really like a good way to do this.