If you want to utilize an e-mail system I'm pretty sure you're going to need to setup MX records for your domain and postfix if you wish to use it as a true e-mail address... i.e sending from joe@outside-domain.com to anyone@yourdomain.com.
If you're looking to simply do intrasite (your site only) messaging and require users to visit the site before hand and login, a PM system is fairly simple. It's just a matter of designing your database with the right tables. When someone sends a message to a user it goes into a messaging database with the following (or similar) information
+----+------------+--------------+----------+----------+
| ID | SENDER_ID | RECIPIENT_ID | SUBJECT | MESSAGE |
+----+------------+--------------+----------+----------+
Keep in mind you'll populate these fields through other queries you've done to retrieve this information anyway.
Next you need to generate a notification mechanism such as a pop-up alert when the user logs in; e-mail - again this would require postfix but not necessarily an MX record because you'd only be sending out; text message via an SMS gateway or hack workaround.
I did something similar for a web-based service system I was working on for a company that allowed for inter-office communication. Worked out nicely. May not have been the next big thing, but they were happy with it.