Cakephp running out of memory

sarahk

Hi,

On a cakePHP intranet I've written I have a problem with a really large csv file.

I've had to break it into "pages" and run multiple queries to build up the csv file slowly and that's worked for queries up to 1000 records.

After that I run out of juice and the page fails. No errors, nothing. It's fast so it's not a timeout. I'm passing &$data to help preserve memory, but not sure what the next step should be.

Any tuning experts around?

thanks

Sarah

Weedpacket

I don't have any experience with cakePHP; but keeping that caveat in mind it would occur to me to see if the dbms you're using could be coerced into importing the CSV data directly: once PHP has initiated the operation it's out of the loop.

sarahk

Hi Weedpacket, good to see you're still here 😉

The CSV is output...

Cake is great but it does get a bit memory hungry. I can't quite work out my problem here. Runs ok for one query with 1200 lines and falls over for another after 500.

These aren't cake queries, but they are run through the cake engine and have outer joins. Could be the nulls are tripping things up...

Weedpacket

Sorry, managed to miss that "build up" bit. I'd probably be irritating and suggest that the dbms might do a CSV export, but.....

[unhelpful waffle elided]

If there are any spots where you know you've finished with a big wodge of data, you could try immediately unsetting them. You mention paging the results out a thousand lines a time; could you reduce that to one line, appending to a temporary file?

I am assuming this is a case of PHP's exhausting its memory but it usually triggers a fatal error in that case. It's possible that it's crashing completely, but in that case I'd expect the server to send back a 500 response. (Incidentally, in checking that memory limit message, I tried a test script here; the memory limit is set at 128MB, but PHP didn't quit until it had claimed 500MB and wanted another gig!)

I'm going to have a nosey around Cake to see what it's up to; any suggestions of anything I ought to look at first?

It could also be something up on the database side; can't think what but perhaps you could try emulating your code and running the queries against the database manually.

This sounds like the sort of problem where I'd need to be sitting at the machine to diagnose. Get a feel for it. Bit tricky to do, here 🙂

sarahk

Cake takes the data and puts it all into arrays which can be incredibly helpful - once you get the hang of it.

I am working on a new project where the 1:M thing will be extreme. In this case I'm building the relationships upwards but not down (they are two way) to preserve the juice. I also have some lookup tables which I won't be linking in at all but calling using $this->requestAction()

Now, in my csv I could get the dbms to do it (safe mode allowing) but I've built all the smarts surrounding the permissions per column into a custom csv script and would be loathe to start over. I may need to bite that bullet.

As for getting started - take a look at the todo tutorials. They're very good at introducing the concepts.

I also use Joomla, WordPress and custom code. Cake is awesome for those projects that don't fit a standard CMS structure.