[RESOLVED] Memory and loops

Kudose

'ello all.

I've got a loop here which seems to not free the memory it uses.

while($totalJobs < $total){
          echo 'Job Loop Current: ' . number_format(memory_get_usage(), 0, '.', ',') . " bytes\n";
          // be nice and sleep for a few seconds
          sleep(rand(1,5));
          $params['start'] += $params['limit'];
          $query = buildQuery($params);
          $target = $base.$query;
          $data = parseXML($target, $_db, $errors);
          $totalJobs += $params['limit'];
          $query = $target = $data = NULL;
        }

That isn't the initial loop ... it had unset and parseXML being assigned to nothing.

With each loop, I can see the memory increase another 200k, until eventually the machine start swapping and cries.

I've read a little bit about a new memory patch in 5.3, but I am curious to know if I should hassle with upgrading (currently on PHP 5.2.0-8+etch13) or if my script is foobar.

TIA!

bpat1434

I'm going to take a guess and say it has something to do with external functions you call, and possibly the fact that $params['start'] might be getting a bit large. I've never really tracked how much memory I really use ( i guess I should ). The only time I've run into issues is with infinite loops and running out of memory to allocate 😉 That's always fun to try and debug.....

Weedpacket

As far as upgrading goes, the PHP changelog, or the bugtracker, mentions a few memory leak fixes (mainly centring around reflection) and overhauls (particularly in 5.2.1) between your version and the current stable. I wouldn't upgrade to an alpha version (which is where 5.3 is at right now) unless you're specifically planning to test out the new features.

As far as the leak goes: I don't see $params['start'] growing large - it looks like it's an int, and an int is four bytes, unless it gets too large, in which case it's eight bytes. I do see mention of a $_db, there and maybe that's doing something. It may also be interesting to see the behaviour if you request real usage from memory_get_usage(), rather than just allocated usage: if that doesn't go creeping upwards it would be an especially clear sign that something inside is wonky and it's not a problem on your part.

[ Duplicate section cut out by bpat1434 ]

Heh, I got through that and still managed to forget to mention that Xdebug has memory profiling 🙂

Kudose

Here is a bit more testing results based on Weedpacket's feedback. $params is an array of parameters (as you probably guessed) and 'start' and 'limit' are type int.

$_db is my DB class, running memory stats on the loop that uses it to insert 200 records (in parseXML) yields no memory increase.

Here is the core of the script that is yielding the stats below:

echo 'Working with '.$abbr." @ ".date('H:i:s')."\n";
      echo 'Job Loop Current: ' . number_format(memory_get_usage(true), 0, '.', ',') . " bytes\n";
      // build query
      $params['l'] = $abbr;
      $query = buildQuery($params);

  // target site
  $target = $base.$query;

  // fire
  $total = (int)parseXML($target, $_db, $errors);
  $totalJobs += $params['limit'];
  $hits[$abbr] = $total;

  // calc total number of pages
  $numPages = ceil($total / $params['limit']);

  // once $totalJobs >= $total, continue
  if($numPages > 1){
    while($totalJobs <= $total){
      echo 'Inner Current: ' . number_format(memory_get_usage(true), 0, '.', ',') . " bytes\n";
      $params['start'] += $params['limit'];
      $query = buildQuery($params);
      $target = $base.$query;
      $data = parseXML($target, $_db, $errors);
      $totalJobs += $params['limit'];
      $query = $target = $data = NULL;
    }
  }
  $query = $target = $data = $total = $numPages = $totalJobs = NULL;
  // reset params
  $params['start'] = 1;

Results:

Working with AL @ 09:16:08
Job Loop Current: 262,144 bytes
Inner Current: 262,144 bytes
Inner Current: 524,288 bytes
Inner Current: 524,288 bytes
Inner Current: 524,288 bytes
Inner Current: 786,432 bytes
Inner Current: 786,432 bytes
Inner Current: 786,432 bytes
Inner Current: 1,048,576 bytes
Inner Current: 1,048,576 bytes
Inner Current: 1,048,576 bytes
Inner Current: 1,310,720 bytes
Inner Current: 1,310,720 bytes
Inner Current: 1,310,720 bytes
Inner Current: 1,572,864 bytes
Inner Current: 1,572,864 bytes
Inner Current: 1,572,864 bytes
Inner Current: 1,835,008 bytes
Inner Current: 1,835,008 bytes
Inner Current: 1,835,008 bytes
Inner Current: 2,097,152 bytes
Inner Current: 2,621,440 bytes
Inner Current: 2,621,440 bytes
Inner Current: 2,621,440 bytes
Inner Current: 2,621,440 bytes
Inner Current: 2,621,440 bytes
Inner Current: 2,621,440 bytes
Inner Current: 2,621,440 bytes
Inner Current: 2,883,584 bytes
Inner Current: 2,883,584 bytes
Inner Current: 2,883,584 bytes
Inner Current: 2,883,584 bytes
Inner Current: 3,145,728 bytes
Inner Current: 3,145,728 bytes
Inner Current: 3,145,728 bytes
Inner Current: 3,407,872 bytes
Inner Current: 3,407,872 bytes
Inner Current: 3,407,872 bytes
Inner Current: 3,407,872 bytes
Inner Current: 3,670,016 bytes
Inner Current: 3,670,016 bytes
Inner Current: 4,718,592 bytes
Inner Current: 4,718,592 bytes
Inner Current: 4,718,592 bytes
Inner Current: 4,718,592 bytes
Inner Current: 4,718,592 bytes
Inner Current: 4,718,592 bytes
Inner Current: 4,718,592 bytes
Inner Current: 4,718,592 bytes
Inner Current: 4,718,592 bytes
Inner Current: 4,718,592 bytes
Inner Current: 4,980,736 bytes
Inner Current: 4,980,736 bytes
Inner Current: 4,980,736 bytes
Inner Current: 5,242,880 bytes
Inner Current: 5,242,880 bytes
Inner Current: 5,242,880 bytes
Inner Current: 5,242,880 bytes
Inner Current: 5,505,024 bytes
Inner Current: 5,505,024 bytes
Inner Current: 5,505,024 bytes
Inner Current: 5,767,168 bytes
Inner Current: 5,767,168 bytes
Inner Current: 5,767,168 bytes
Inner Current: 5,767,168 bytes
Inner Current: 6,029,312 bytes
Inner Current: 6,029,312 bytes
Inner Current: 6,029,312 bytes
Inner Current: 6,291,456 bytes
Inner Current: 6,291,456 bytes
Inner Current: 6,291,456 bytes
Inner Current: 6,291,456 bytes
Inner Current: 6,553,600 bytes
Inner Current: 6,553,600 bytes
Inner Current: 6,553,600 bytes
Inner Current: 6,815,744 bytes
Inner Current: 6,815,744 bytes
Inner Current: 6,815,744 bytes
Inner Current: 6,815,744 bytes
Inner Current: 7,077,888 bytes
Inner Current: 7,077,888 bytes
Inner Current: 7,077,888 bytes
Inner Current: 8,912,896 bytes
Inner Current: 8,912,896 bytes
Inner Current: 8,912,896 bytes
Inner Current: 8,912,896 bytes
Inner Current: 8,912,896 bytes
Inner Current: 8,912,896 bytes
Inner Current: 8,912,896 bytes
Inner Current: 8,912,896 bytes
Inner Current: 8,912,896 bytes
Inner Current: 8,912,896 bytes
Inner Current: 8,912,896 bytes
Inner Current: 8,912,896 bytes
Inner Current: 8,912,896 bytes
Inner Current: 8,912,896 bytes
Inner Current: 9,175,040 bytes
Inner Current: 9,175,040 bytes
Inner Current: 9,175,040 bytes
Inner Current: 9,175,040 bytes
Inner Current: 9,437,184 bytes
^C

A bit erratic if you ask me (other than the consistent increase of 262,144 bytes).

Kudose

This issue was isolated to the XP machine the code was being executed on and there is no memory leak running the same script on another XP machine.