In the insert.member.php I am checking if the user is exists in MySQL table.. If not, then I am INSERTING and then redirecting to another page but if exists I am displaying a popup dialog.
Assuming you've got a well-defined schema (i.e. you have UNIQUE/PRIMARY constraints on the appropriate column(s)), note that the first check is superfluous. It would be more efficient (and less code) to simply execute the INSERT statement and check if it fails due to duplicate data.
Regardless, this doesn't appear to be the source of your problem.
when the popup is shown, it will be shown in a blank page and the address on the browser address bar is insert_member.php
Right, because when you submit a form, your browser makes a new request (a POST one, in this case) in order to display a completely new document/page.
how can I let the popup message on the member.php?