grep log parsing

ntburchf

hello all!

Trying to parse though about 50 log files that range in size from 500k to 18mg.

I can get it to read and display the search term I am looking for, but it only displays 1 instance of each term per file, per line.
eg- I search for computed, and it returns this

1. 00000001.log : [06.03.2004 02:07:23] Player GUID Computed 961e36c44dcfc7291e21abd(-) (slot #7) 212.129.139.xxx:1190 -=DoW=-_SniperTARGET 
2. 00000002.log : [06.03.2004 11:52:02] Player GUID Computed efdd86dad1a2d8496530fcec7af2(-) (slot #2) 217.93.86.xx:64975 }D.K.K{Trainbow} 
3. 00000004.log : [06.03.2004 15:08:30] Player GUID Computed 0ec19f3dc6e70830332989437ea(-) (slot #3) 68.41.229.xxx:2671 -=DoW=-_romeo-23

it should return about 100+ hits of computed per log file.
can not figure out how to get it to display all instances in each file.

<?
$title = "Search";
?>
    <P>
        <FORM ACTION="<? echo "$PHP_SELF"; ?>" METHOD="POST">
        <INPUT TYPE="text" NAME="searchstr" value="<? echo "$searchstr"; ?>"
               SIZE="20" MAXLENGTH="30">
        <INPUT TYPE="submit" VALUE="Search!">
        </FORM>
    </P>

    <?
    if ( ! empty($searchstr) ) {
        // empty() is used to check if we've any search string
        // if we do, call grep and display the results.
        echo "<HR>\n";
        // call grep with case-insensitive search mode on all files
        $cmdstr = "grep -i $searchstr *";
        $fp = popen( $cmdstr, "r" ); //open the output of command as a  pipe
        $myresult = array(); // to hold my search results

        while( !feof( $fp )) {
       $buffer = fgetss ($fp, 4096);  // may have done this wrong 

        list($fname, $fline) = split(":",$buffer, 2);

        // I think this is the part that's giving me only 1 hit per file
           if ( !defined($myresult[$fname]))
            $myresult[$fname] = $fline;
            }

    // we have results in a hash. lets walk through it and print it
              if ( count($myresult) ){
                echo "<OL>\n";
                while(list($fname,$fline) = each($myresult))
                echo "<LI>
               <A HREF=\"$fname\">$fname</A> : $fline </LI>\n";
                echo "</OL>\n";
            } else {
                // no hits
                echo "Sorry. Search on <B>$searchstr</B>
                      returned no results.<BR>\n";
            }
            pclose($fp);
        }
   ?>

Thanks a ton in Advance
TB..

ntburchf

Any clues as to what i change here so that it shows all instances of each search


// I think this is the part that's giving me only 1 hit per file 
               if ( !defined($myresult[$fname])) 
                $myresult[$fname] = $fline; 
                } 

    // we have results in a hash. lets walk through it and print it 
              if ( count($myresult) ){ 
                echo "<OL>\n"; 
                while(list($fname,$fline) = each($myresult)) 
                echo "<LI> 
               <A HREF=\"$fname\">$fname</A> : $fline </LI>\n"; 
                echo "</OL>\n"; 
            } else {

drew010

try changing the beginning of the while loop to this:

$results = array();
while( !feof( $fp )) {
    $results[] = trim(fgets($fp)); //read 1 line at a time
}

foreach($results as $one) {
    echo "$one<br />\n";
}

see if you can do anything with that. im not sure its the solution but it seems like it could work.

ntburchf

Getting close 🙂
That now displays all instances of search term in one log, but ignores all the others logs

I think I have these arrays are hosed

<?
        if ( ! empty($searchstr) ) {
            // empty() is used to check if we've any search string
            // if we do, call grep and display the results.
            echo "<HR>\n";
            // call grep with case-insensitive search mode on all files
            $cmdstr = "grep -i $searchstr *";
            $fp = popen( $cmdstr, "r" ); //open the output of command as a  pipe

$myresults = array(); 

   while( !feof( $fp )) { 
$myresults[] = trim(fgets($fp)); //read 1 line at a time 
} 

foreach($myresults as $one) { 
    echo "$one<br />\n"; 

list($fname, $fline) = split(":",$buffer, 2); 

if ( !defined($myresults[$one])) 
                $myresults[$one] = $fline; 
                } 

    // we have results in a hash. lets walk through it and print it
              if ( count($myresults) ){
                echo "<OL>\n";
                while(list($one,$fline) = each($myresults))
                echo "<LI>
               <A HREF=\"$one\">$one</A> : $fname </LI>\n";
                echo "</OL>\n";
            } else {
                // no hits
                echo "Sorry. Search on <B>$searchstr</B>
                      returned no results.<BR>\n";
            }
            pclose($fp);
        }
    ?>

drew010

if its only showing all the results from one file, make sure php is sending back all the data, that way we can see how to capture it. for debugging, just try

$cmdstr = "grep -i $searchstr *";
$fp = popen( $cmdstr, "r" );

echo "<pre>\n";
while( !feof( $fp )) { 
    echo fgets($fp);
}
echo "</pre>\n";

that should show the full output of the grep command in the format php receives it.

ntburchf

Feel like an idiot..
I moved all the log files but 1, so that I could just work with that one file...feels foolish now..

I'll try you additions and let ya know..

THANKS

EDIT
WoW!!
Thanks, it now works, not like I wanted it to, but it does search every log now. And Fast!!
20 megs of logs in approx 2.5 secs

had to remove this

//if ( !defined($myresults[$one]))
// $myresults[$one] = $fline;

due to it was doubling up everything. would display the search finds eg 15000 hits, then at the bottom of the page it would list again all 15000 hits, in href format