I've done stuff like that before with Java.
What happens is that a message is posted to a table with the receivers name against it. The receiver comes to the website and logs in (automatically, probably). Each page loads and the last thing it does is check the username against the message table. If there is a message then at the bottom of the page PHP writes
<?php <script language='javascript'> popPM();</script>
popPM is a javascript function which pops up a window calling, say, pm.php. That's simple javascript and really does very little.
pm.php then gets the username from the session or cookies, checks the table again and gets the list of messages, the oldest message, the newest or whatever your rule is and displays it.
The key is that the user has to be at your site (or an affiliated site that agrees to allow your popups - but that's another story).
