Script dies without error or warning

LSB

Hi,

I'm doing some parsing of poker hand history files and made a system that is working well but with one major exception.
When the files are larger than somewhere around 100k the script will die before it reaches the end of the file.
If I use firefox it starts outputting and it takes about 4 seconds and then I get a "connection reset" page.
In IE it takes a couple of seconds and I get a "page not available".

The script reads a local file as input and I read hand by hand using file_get_contents($filename, FALSE, NULL, $offset, $end)
That input is then processed using a state machine which goes through an array with states containing different preg statements to extract the information from the input. Each hand is in the region of 30 lines.
It stores the data in an array and when the hand is finished parsing it saves it into an MySQL database and I clear the array and start processing the next hand.

I tried the script locally using a default xampp install on XP as well on my linux host with same results.

When I use Charles debugging proxy the page does not get replaced with the connection reset page and I can follow the output of the script all the way until it quits.
The scrip uses about 5Mb before it dies according to memory_get_usage, and upping memory limit or execution times have no effect.

I have tried with the following php.ini setting with no effect on how long the script will run:
pcre.backtrack_limit = 1000000
pcre.recursion_limit = 1000000
ignore_user_abort = On
max_execution_time = 120
max_input_time = -1
memory_limit = -1

pcre.recursion_limit sometimes crash apache at that extreme setting, but when it doesn't the script stops at the same place.

I have error reporting set to E_ALL but receive nothing in neither php or apache error logs.

It seems like there is some resource limitation since if I remove a few lines in the start of the file it will parse a few lines more, so the problem is not relating to a certain part of the code or the input.

Any ideas about if there is any resource settings that could fix the problem?

I'm not so keen on rewriting too much of the code, but if I at least had a clue what to look for in my code to find the culprit, but without knowing what resource is exhausted and no error messages to help me I really don't know where to start.

Any ideas greatly appreciated!

chasiv

Not sure if this will help or not (I'm still kinda new to PHP - about a year), but I used to receive a lot of errors for exceeding the 30 second limit on scripts. So when executing large processes (like importing Excel tables into database tables) I use

set_time_limit(0)

and then after execution, I reset it to 30 with

set_time_limit(30)

Not sure this will resolve your problem with reading the file, but it can't hurt to try it.

LSB

I think that's the same as setting max_execution_time in php.ini
And the script is only executing a few seconds, so it doesn't seem to be the issue.
Anyway i tried it but result is the same 🙁

Also I noticed that when i don't output the data but only the id of each hand the script will only run about half as long.
If it was a resource issue like I have previously suspected I guess one would expect the opposite.

Dev_M_nX

Hi, How much memory is php using ?
I've had this exact problem before, and when I increased the memory_limit in my php.ini file and restarted apache, it worked...

LSB

I disabled the memory limit, and according to memory_get_usage and memory_get_peak_usage it's not using more than about 5Mb

Dev_M_nX

Just checking, you set the memory_limit to -1 for an unlimited memory usage... right ?

Also, is it your home server or an outside server ?
Either way, how much memory do you have to play with ?

Dev_M_nX

I'm going to need help with the regulars on this one... I surfed around for a bit... and found this .... http://bugs.php.net/bug.php?id=40800

shows how if there is a high enough throughput, then the system (using a 32bit integer) wraps around as 4GB-1 sized integer....

The only catch I found was that the error in that post doesn't explain the lack of error in this post ?

Hmmm... 😕

LSB

Yeah...that should be unlimited, but I have tried set values like 100M as well.

At home I run xampp on XP with 1Gb RAM.
I have the same problem on my linux webhost.

LSB

Unless the reporting of memory_get_usage is very unreliable I have problem seeing how it could be problem with PHP running out of memory.

I'm surprised though that I posted the problem in a few places and so far I have not had anyone suggest possible issues that could result the script dying without any error or warning whatsoever.

Dev_M_nX

Well... I think this goes beyond me...

Don't be discouraged though, the guys here are good, and I may not be new to the PHP world, but its not an everyday deal with me, so I can only help with what I've experienced.

Hope one of the excellent PHP blokes here can help you, they've got me out of the stickies before and have done so for much more for others... keep searching the solution is here or out there somewhere... 🙂

LSB

Thanks for the encouraging words.
I wish there was something error message so I could search for answers....but "php script dies" or "php script connection reset" doesn't seem quite specific enough, and I still haven't got anything much more specific to go on 🙁

NogDog

I only have some general suggestions.

Try to make sure you fclose() your file connections as soon as you are done reading them.

You might try adding the "S" modifier to your preg regular expressions, if you think they qualify per this comment from the manual:

When a pattern is going to be used several times, it is worth spending more time analyzing it in order to speed up the time taken for matching. If this modifier is set, then this extra analysis is performed. At present, studying a pattern is useful only for non-anchored patterns that do not have a single fixed starting character.

If you are doing a lot of database queries to insert or update records, you might consider putting lists of values into an array or string to be used in one INSERT or REPLACE query.

LSB

I'm using file_get_contents() and since it doesn't return a handle I guess fclose() is not applicable then?
I have been trying with fopen() and fread() as well as using cURL, but it makes no difference in how long the script runs.

The "S" modifier seems like it would be useful since the same expressions is run many times. I tested with it and the script didn't get any further though, but I will test if it will run a bit faster at least and the suggestion is much appreciated.

I disabled the database features to try to narrow down the problem, but that has no effect.

NogDog

What version of PHP are you using?

This bug claims to have been fixed in 4.3.9 and 5.0.2.

LSB

On my local machine it's 5.2.4...on the host I'm not sure without looking, but it's >5

Memory usage is not abnormal at all with the script, at least not what I can see with memory_get_usage() and memory_get_peak_usage() nor by checking in the task manager.

NogDog

Does it stop in the same place regardless of which machine it runs on? If it's different, then it would seem to be some system resource it's eating up, but if it dies in the identical place it would seem to be something peculiar with the script and PHP themselves. (I know that doesn't help a lot, but at least it might help us think what to concentrate on.)

About all I can think of to do at this point would be something creative with [man]declare[/man] and [man]register_tick_function/man to see if you can log things like memory usage and anything else of interest every couple of ticks or so, to see if anything interesting pops up.

LSB

Had another look at the issue today and so far I only noted that the parsing of large files fail on my webhost, but have been only been trying to find the issue on my local machine. I wanted to avoid messing about with the files on the webhost since they should be accessible to the client for testing.

But following your advice I did run some tests on the webhost and it turns out that the issue there was not the same as on my local machine.
The script runs a bit longer but then died with a memory warning. Upping the limit with ini_set made it run through fine though 🙂
Now I feel really stupid wasting all this time on it 🙁

Still a mystery what goes on with my local machine, but I don't really have to care about that as long as the server that will run the script in the end doesn't suffer the same problem.

Many thanks for the help!