How many iterations is too many

karlovac

How many iterations would you consider to be too many?

One critical area of my app takes XML data from a flash movie, compares it with data in a database, and makes additions or deletions based on that.

Typically it may go through 228 * 136 = 31008 iterations. (There's two nested for loops), so it has to go through at least that many. I would have thought that for a computer 30K iterations is nothing.

I'm getting Fatal error: Maximum execution time of 30 seconds exceeded after 190 of the outer 228 iterations (i.e. 83%).

-Antun

Mordecai

Open php.ini, and edit this line:
max_execution_time = 30 ; Maximum execution time of each script, in seconds

That is, if you want to up the max execution time. It'll run, but it'll be slow.

jpmoriarty

you cant really make sweeping statements like "I would have thought that for a computer 30K iterations is nothing.": it's completely dependent on what it's doing in those iterations. If you're doing an addition or something, that's fine, but if in each iteration you were querying a database and performing quite a complicated query, then each iteration can take up to a second, so 30k iterations could take over 8 hours.

The problem is probably the speed at which it can take the data from the flash movie - bare in mind if it cant take over 30 seconds, then you've got to be performing each iteration in 1/1000th of a second, and that's going to be heavily influenced by how long it takes to ask a question of the flash file and get a response from it. Let alone the querying of the database to see the comparison information.

And all that is presuming that your site is hosted on it's own server (which may well be the case) but if your site is hosted by an isp it's highly likely you'll be sharing resources with other web sites.

karlovac

Thanks. I know about the max_execution_time variable. I don't know if I can up it because it's being hosted by a hosting company and I don't know what their policies are.

-Antun

Originally posted by Mordecai
Open php.ini, and edit this line:
max_execution_time = 30 ; Maximum execution time of each script, in seconds

That is, if you want to up the max execution time. It'll run, but it'll be slow.

karlovac

Originally posted by jpmoriarty
you cant really make sweeping statements like "I would have thought that for a computer 30K iterations is nothing.": it's completely dependent on what it's doing in those iterations.

You're absolutely right. I should have been clearer. All I'm doing in the iterations is testing for equality, and setting flag variables.

I've managed to optimize it a little by recursively breaking out of the script when a match is found (don't know why I wasn't doing that before).

The problem is probably the speed at which it can take the data from the flash movie

But I'm posting from the SWF to my PHP script. Does PHP start counting from when the request is made or from when its finished posting?

-Antun

plaggypig

the answer to your question is infinite.

can't you improve the logic somehow? that many iterations seems a bit daft to me, especially if it's open to the public - if it was some kind of admin feature for a select group of people then i would be a little more sympathetic.

p.s. somewhat off topic but i've noticed the max execution time doesn't seem to take into consideration the time for file() to download over a http stream.

andy.

karlovac

Hey Andy

Actually this is an admin feature - only one person ever uses it. When I initially built the app, it was more practical to compare beginning and end scenarios on the server, and determine what had to be done.

-Antun

Originally posted by plaggypig
if it was some kind of admin feature for a select group of people then i would be a little more sympathetic.

plaggypig

okay, it's just when you said "flash" i immediately assumed it was some sort of elaborate script for saving highscores for some wacky game or other 😉

have you really optimised your loop as much as possible? this is all i can really suggest. if it's a for loop then make sure your conditional test is as slick as possible - this is something a lot of people overlook, but it can make the world of difference.

andy.

karlovac

Hey Andy

How would I optimize the conditional test? There's a number of assignments in there for clarity, e.g.

$newSlotId = $newSlots[$j][0];

and then

if ( ($slotId ==  $newSlotId) && ($volunteerId == $newVolunteerI

Do the assignments affect speed?

Thanks,

Antun

Originally posted by plaggypig
have you really optimised your loop as much as possible? this is all i can really suggest. if it's a for loop then make sure your conditional test is as slick as possible - this is something a lot of people overlook, but it can make the world of difference.

plaggypig

Well, it would be better to do:

if ($slotId ==  $newSlots[$j][0] && $volunteerId == $newVolunteers[$j][0])

That's not what I meant by conditional test, though. Paste me the entire loop and I'll have a look at it for you.

Andy.

karlovac

Here's the code:-

for ($i=0; $i<count($existingAssignments); $i++) {
    $assignmentId = $existingAssignments[$i]["id"];
    $volunteerId = $existingAssignments[$i]["volunteerId"];
    $slotId = $existingAssignments[$i]["slotId"];
    $isThere = 0;
    for ($j=0; $j<count($newSlots); $j++) {
        $newSlotId = $newSlots[$j][0];
        for ($k=0; $k<count($newSlots[$j][8]); $k++) {
            $newVolunteerId = $newSlots[$j][8][$k][1];
            if ( ($slotId ==  $newSlotId) && ($volunteerId == $newVolunteerI
d) ) {
                // the slot still exists
                $isThere = 1;
                // AK: I've added these breaks in to reduce total
                // iterations.
                break;
            }
        }
        if ( $isThere ) { break; }
    }
    if (!$isThere) {
        // delete the assignment : build query string
        if ($count > 0) $qs .= "OR ";
        $qs .= "(a_id = " . $assignmentId . ") ";
        $count++;
    }
}

I don't really like the idea of not assigning cryptic statements to logically-named variables (although I've done it myself in this bit: $k<count($newSlots[$j][8])) - $newSlots[$j][8] should really have been reassigned and named above. If assignment in PHP is that expensive, then I'll change it.

-Antun

plaggypig

okay, i'm not going to try and understand the logical aspect of what you're trying to accomplish as i'm not able to visualise your data structures. however, there are some things you can do to speed up your code:

1) for ($i=0; $i<count($existingAssignments); $i++) {

count() is going to be called on every iteration of your loop(s), which would be painfully slow. you should do:

$count = count($array);
for ($i=0; $i<$count; $i++) {

2) i've no idea if this would be any more efficient, but there's no need to have certain evaluations, like you have, nested inside parentheses:

if ( ($a == $b) || ($c == $d) ) {

... this works, but so will:

if ($a == $b || $c == $d) {

3) again, i'm not sure how much faster this would be, but, what about referencing to your $existingAssignments array by number rather than associatively?

4) as for variable assignment, i don't really know how expensive it is - you should try benchmarking it, as well as these other suggestions, and see what works best.

andy.

karlovac

count() is going to be called on every iteration of your loops, which would be painfully slow. you should do:

$count = count($array);
for ($i=0; $i<$count; $i++) {

This sounds really promising. I'm going to give it a go.

-Antun

epimeth

I've been up all night and its 6:20am, so I'm kinda fuzzy... but $...ID seems to be leaping out of the screen and hitting me over the head when I look at your code... pulling only existingAssignments with the ID's you need from the database... that should reduce the load significantly... unless I misunderstood what you're doing, which very well may be....