exex() too slow?

unknownsheep

Hello, I'm running linux debian and I'm trying to parse large files with loads of data, to sort out the most of the junk I dont want I'm using the linux function grep.

The php function looks like this:
exec('cat filename | grep "K;"',$array);

Basically I pipeline the file to the function grep that filters out all lines that dont include the characters K;. Even with the filtering there is about 5000 lines of data left that needs to be filled in that $array.

But im getting this error when im running it: grep: writing output: Broken pipe. I googled some and apparently this means that grep is unable to pipe the data any further and since php is next im guessing that's the problem.

Of course I'm not a 100% sure of this but since it doesnt work im looking for help or atleast someone that can point me in the right direction.

Thank you.

bradgrafelman

Why not just use PHP for this?

$array = array();
$fp = fopen('filename', 'r');

while(!feof($fp)) {
	$data = fgets($fp);

if(strpos($data, 'K;') !== FALSE)
	$array[] = $data;
}

unknownsheep

Well, I was but grep is many times faster and because of the great amount of data php will hog up a lot of resources for a considerably amount of time if I use that method.

NogDog

Not sure it would make a difference, but can't you just call grep without the cat? E.g.:

grep "K;" filename

unknownsheep

Yes, that works too but it doesn't make any difference.

bradgrafelman

unknownsheep wrote:
because of the great amount of data php will hog up a lot of resources for a considerably amount of time

Actually, PHP will only use as much memory as it needs to in order to hold the array of applicable data. In other words, it'll use the same amount of memory as if you had used grep to process only the lines with "K;".

unknownsheep

bradgrafelman;10878695 wrote:
Actually, PHP will only use as much memory as it needs to in order to hold the array of applicable data. In other words, it'll use the same amount of memory as if you had used grep to process only the lines with "K;".

Its possible but grep still does it many times faster.

NogDog

Might be interesting to try it with [man]preg_grep/man and see how it compares.

$results = preg_grep('/K;/', file($filename));

unknownsheep

NogDog;10878761 wrote:
Might be interesting to try it with [man]preg_grep/man and see how it compares.
$results = preg_grep('/K;/', file($filename));

That will use a lot of memory indeed, im using ini_set("memory_limit","500M"); and still gets Fatal error: Allowed memory size of 524288000 bytes exhaustet.

NogDog

Hmmmm...well, if you can't figure out why you get the pipe error, all I can think of is trying to put the grep into a shell sript, then have the PHP script call that shell script. 🙁

PS: Have you tried it with shell_exec() instead of exec()?