I think what you are asking for is a persistant connection. I think ther is a mysql_pconnect() or something like that. But then too it is a good practice to:
Open connection
Preform Query
Close Connection
This can improve performance of the pages
If you are just using the mysql_connect then yes you will have to do this for every page...
Thanks
PHPdev