[RESOLVED] Downloading Lots of Pages (100+)

[Duodecillion]

Im writing a script that has to download over 100 different web pages and then store the information in a database...problem is it stops downloading after a while (probably timed out or something)

The code below only echos up to 92 even though it should do it 100 times. Google is just an example, im downloading a different page every time. But this just shows the problem.

<?php

for ($i = 0; $i <= 100; $i++) {
		$strPage = file_get_contents("http://www.google.com");
		echo "$i<br>";
}

?>

I don't know any other way to do this other than redirecting the page to itself 100 times. Is that the only way?

thnx, yea im a noob. :quiet:

[Horizon88]

set_time_limit(0);

Edit: and redirecting to itself 100 times would only make the first page get called one hundred times, unless you were setting session variables or something. 😉

[halojoy]

Duodecillion wrote:

Im writing a script that has to download over 100 different web pages and then store the information in a database...problem is it stops downloading after a while (probably timed out or something)

The code below only echos up to 92 even though it should do it 100 times. Google is just an example, im downloading a different page every time. But this just shows the problem.

try with
[man]set_time_limit/man;
0 means unlimited max execution time, while PHP normally have 60 seconds setting, I think

I can not promise it will work for you. It may be some other problem.
But you can try and see what happens.

<?php

set_time_limit(0);

for ($i = 0; $i <= 100; $i++) {
		$strPage = file_get_contents("http://www.google.com");
		echo "$i<br>";
}

?>

[Horizon88]

halojoy wrote:

set_time_limit(0);
for ($i = 0; $i <= 100; $i++) {
		$strPage = file_get_contents("http://www.google.com");
		echo "$i<br>";
}

?>

That should actually be inside the for loop - as seen on the manual page:

When called, set_time_limit() restarts the timeout counter from zero. In other words, if the timeout is the default 30 seconds, and 25 seconds into script execution a call such as set_time_limit(20) is made, the script will run for a total of 45 seconds before timing out.

So to prevent timeouts you should put it inside the for loop.
I stand corrected, I think :

The maximum execution time, in seconds. If set to zero, no time limit is imposed.

You could probably leave it outside the for loop as well like halojoy said.

[Duodecillion]

That seemed to work. Thanks 🙂

[Horizon88]

No problem - now, if there's an issue with the site you're ripping, and it causes a timeout that lasts longer than 30 seconds, your script will still fail - if you want more robustness you could try hooking a system call using something like wget to rip these website, but then it's no longer a solely PHP solution.

Also, don't forget to mark the thread Resolved under the Thread Tools menu, if you would be so kind.

Also, welcome to PHPBuilder!

[Duodecillion]

Marked.

I have another question. Is there a way to have a server run a php page at schedule times? Like If I wanted to do this daily without having to go to my site and running the page. I highly doubt there is but just wanted to ask

[bradgrafelman]

Duodecillion wrote:

I highly doubt there is but just wanted to ask

Oh? Why so doubtful? Search around for "cronjob" or "crontab" or "cron".

[Duodecillion]

k i'll look into it, thanks again