You'll need to use Javascript. Use your PHP code like you said and make it print out Javascript. Make sure it prints it out in a good place, anywhere should be Ok just as long as it's not in the middle of some other tag or something:
if ($messages != ""){
print "<script language="JavaScript">
<!--
var popup = window.open("http://www.yahoo.com/","mywindow",'location=1,status=1,scrollbars=1,resizable=1,width=320,height=200');
// -->
</script>";
}