Using curl to retrieve 100+ PDF reports

Spames

I have a script set up that uses curl to retrieve PDF reports from another web server. The problem is that it takes ~10 seconds for the remote server to generate a single PDF. I have about 120 reports to retrieve, so doing some simple math.... 10*120 / 60 = 20 minutes of execute time for the PHP script.

Is it a bad idea to increase the execute time of a PHP script all the way up to 20 or say 30 minutes?

My instincts tell me I need a way to break these down into groups of 5 or so at a time, and run small sets every 10 minutes until the set is finished. But ... to automate that task, I haven't figured out a way to do this without having 20 separate PHP scripts for each little set, and 20 separate cron jobs to run each of these, starting at 2AM, and going up 10 minutes on the other scripts.

I guess the other thing I'm worried about is crashing the remote server, since It's part of another branch of the organization.

Any advice?

Spames

Apparently, General help isn't the place to be asking questions 😃

Weedpacket

Don't make such posts (especially so quickly); it's considered rude. When you posted I was curled up in bed sharing a good movie. I guess other people just happen to have their own lives to lead.

You could go the cronjob approach; you wouldn't need to have twenty separate scripts. PHP can take command line arguments, so you can just run "php fetchpdf.php 1", "php fetchpdf.php 2", or however you want to indicate which batch to request.

If you know it takes about ten seconds to request one pdf, reset the timeout to something like twenty seconds before requesting each one; in principle that will keep up for as long as it takes to get everything (assuming no single one of them take more than twenty seconds).

MarkR

You should definitely run this offline rather than in a web-driven PHP script.

If you're going to do a task which is expected to take longer than the user may want to wait, and/or must not be interrupted, you should do it in an offline script.

This could be invoked from cron (You can test it by running it from the command line).

Because it can take as long as it wants, you could sleep for some time between requests - to reduce the bandwidth used over a given time period.

But as long as you're not hitting it with more than one request at once, I can't see why you can't just do all 120 of them sequentially.

If you're trying to use this as a method to synchronise two directories, there are other ways of doing this that don't involve PHP - and they're probably better (hint: rsync, wget etc)

Mark

Plasma

You can limit a script's execution individually via set_time_limit(), instead of doing it for every php script.

There should be no problem doing it this way either.