"since I can hit stop in my browser right after I submit it and the e-mail is still sent."
That's because you probably hit 'stop' right in the middle of the mail() function, and PHP cannot abort inside internal functions, only between lines of code, so PHP will abort right after mail() has finished sending the message.
The mail() function does a few mysterious things that make it slow down to a crawl.
Get your hands on an SMTP protocol routine and talk directly to the mailserver. Much faster.