I've been writing a few scripts for a friend that scrape URLs and email addresses off of large websites (don't worry its not for spam). The most recent site I've been doing this for is so large that my scripts are taking a very long time to complete, and sometimes just never complete at all. The script's page in firefox just keeps trying to load forever.
This has also happened to me with a script that stores a large amount of data grabbed from some websites into a mySQL database. The script would complete and all the data would be in the DB, but it wouldn't stop in the browser window.
The code is pretty big so I'd rather not post it unless absolutely necessary. Does anyone have any ideas as to what would cause this to happen?
