problem with writing to a large file

mrbaseball34

I'm creating a feed file for Google base in XML. This file can get quite large,
depending on the number of products I have. It is anywhere from 20MB to 35MB+.

Opening the file like this:

  $fp = fopen($filename, 'xb');

Now, I'm using code like this to add text to the file after each product node is built $items contains the XML source for each product in the table In this case, I'm limiting the number of products to 13,000, but I have over 24,000 in my table:

  $this->_addToFeed($fp, $items);

I'm also clearing out $items at each iteration through the products table.

  private function _addToFeed($fp, $str) {
    fwrite($fp, $str, strlen($str));
  } // private function _addToFeed($str)

Now I keep getting, a "The connection was reset" error and then it goes to a dialog that says "Apache HTTP server has encountered a problem and needs to close."

I do close the file after the last node is written but it never gets to that spot in my code.

I'm running WAMPServer 2.0 with PHP v5.3.1, mySQL v5.1.36 and Apache Server v2.2.11. What could be causing this in my Apache setup?

bradgrafelman

Does the page take a long time to load?

Do you have log_errors set to On and error_reporting set to E_ALL? If so, are there any PHP error messages logged?

EDIT: Also, what does the Apache error log say for the request(s) that fail?

mrbaseball34

For some reason, I'm getting a memory limit error and Apache is restarting:

php_error_log:
[04-Oct-2010 15:33:43] PHP Fatal error:  Allowed memory size of 134217728 bytes exhausted (tried to allocate 50 bytes)

apache_error_log:
[Mon Oct 04 15:33:43 2010] [notice] Parent: child process exited with status 4294967295 -- Restarting.

Since I'm writing the data out to the file, why would I get the memory limit error? I'm increasing to 256MB to see if it helps.

bradgrafelman

mrbaseball34 wrote:
why would I get the memory limit error?

Eh... it's impossible to say with the information we have. Can you post the full code?

You might also try doing some crude profiling by outputting [man]memory_get_usage/man in various points throughout the script to see which line(s) cause large spike(s) in usage.

mrbaseball34

Do you know of any code that will allow me to create multiple files around 20MB in the loop?
Mine doesn't seem to be working correctly. I will post more code later.

mrbaseball34

Brad, I went through the entire flow and I am using the exact same flow when writing the text (Tab, Pipe or Tilde delimited) output as when I'm writing XML output. I originally was using SimpleXML but dropped that idea and now am just using straight text, XML formatted, of course.

I can't figure out why it does what it does. You can see the entire flow in this file:
http://www.mechanictoolsonline.com/eddie/code.txt
Note that this is part of a module for PrestaShop e-commerce solution and there is a lot of code in there that you may not understand.

mrbaseball34

Yo, Brad, did you get a chance to look at the code?
I still can't figure that one out.