Actually, there are two mail servers involved. The sending and the receiving. Even if one is fast, the other can still slow things down. You can open the email in your email client and view the full headers. This will show you how long the email got delayed at each mail server. (Read from the bottom to the top). Don't assume that your sending mail server is the slow down - it could be a problem at your receiving mail server.