Processing file char by char

[bubblenut]

This is more a conceptual problem rather than a specific PHP problem. I have to be honest, I'm not actually writing the program in PHP, but I could be.

Right, here's the deal.

Reading in lines form a log file. If you're interested they're holding logs generated by iptables.

I am reading the file in char by char and need a way of determining whether the line I'm currently on is one that I want to looking at, this is fine for the date time part I can just say "do the first 16 lines match the regex "/^{[A-Z][a-z]{2}\s[0-9]\s[0-9]{2}:[0-9]{2}:[0-9]{2}\s""} but but all lines should match that and if they don't there's somthing going wrong in the loggin system.
The next parameter would be the machine name but this is not a known size so we cannot restrict by size. I guess we could skip along to another stage once the date info has been matched but this seems like unecesarry work. Here is a rough idea of what I'm doing so far.

define('NEWLINE',"\n");
$line="";
$skip_line=false;
$file = new FileReader('log_file');
while($file->current_pos < $file->FILE_SIZE) {
  $c = $file->getChar();
  if($c == NEWLINE) {
    $line = "";
    $skip_line=false;
    continue;
  } else if ($skip_line==true) {
    continue;
  } else {
    $line+=$c;
    if(strlen($line) == 16 && !preg_match('/^[A-Z][a-z]{2}\s{1,2}[0-9]{1,2}\s[0-9]{2}:[0-9]{2}:[0-9]{2}$/',$line)) {
      $skip_line=true;
    } 
  }
}

That's as far as I've got and I'm a little stuck.

Thanks in advance
Bubble

[AstroTeg]

I'm not sure what your motive is here.

Character by character reading is good when you're after a particular character, you're counting the occurrence of a particular character, or you're streaming data in/out. Looking for character combinations, such as a date and then something else gets messy very quickly with character-by-character reading. It requires you to keep a buffer, store everything in the buffer/stack, then keep checking the buffer/stack for a match, then figure out when the clear the buffer/stack.

I think you're on the right track with using regular expressions. Can you post a sample of the iptable's log file you're working with? (I don't have access to one at the moment and google didn't find any messy examples). Also, what data are you wishing to dig out? Maybe there's another approach to the problem.

[bubblenut]

Can't really give out any of the logs I'm afraid as they're from the firewall of one of the routers on the production server subnet and I don't know what's sensitive and what's not.
The reason I'm going char by char comes down to the using another language, I'm using java with the nio package because the onous was on speed. These logs are pretty huge and this program (when finished) is supposed to process all the lines, figure out which ones are important and then pull out the relavent data. It is then supposed to compile the data according to how it was called. For example the user may just want to know the total number of stopped tcp packets and icmp packets, or maybe the number of stopped packets which came from within the production server subnet. Don't really know.

I've got around it (a little sloppily could still use some pointers) like this. I can't be bothered to translate it into php so here's the java for that little bit.

import java.io.*;
import java.nio.*;
import java.nio.channels.*;
import java.util.regex.*;
import java.lang.*;

public class IPHunter {
    static final char NEWLINE = '\n';
    static final int CHUNK = 4*1024;
    static final Pattern CHECKLINE = Pattern.compile("^[A-Z][a-z]{2}\\s{1,2}[0-9]{1,2}\\s[0-9]{2}:[0-9]{2}:[0-9]{2}");
    static final Pattern CHECKDATE = Pattern.compile("^(A|D|F|J|M|N|O|S)(a|d|f|j|m|n|o|s){2}\\s{1,2}[0-9]{1,2}\\s[0-9]{2}(:[0-9]{2}){2}\\s$");
    static final Pattern CHECKGOOD = Pattern.compile("kernel: FW ALERT : deny_log$");

public static void main( String args[] ) {
    //Argument Handling Object
    Arguments arguments = new Arguments(args);
    //File reading variables
    FileChannel log_fc = new FileInputStream(arguments.INPUT_LOG_FILE).getChannel(); //the file channel
    long log_size = log_fc.size(); //the total size of the file
    long running_total = 0; //the running total of how much of the file has been read (chunk by chunk)
    long chunk_size = 0; //the suze of the current chunk (will always be CHUNK except for on the last iteration where it will be log_size-running_total)
    MappedByteBuffer log_bb; //the mapped byte buffer used to pull the data out of the file

    String line=""; //A string to hold each line as it's being built
    boolean skip_line=false; //if we find that the line is not wanted flick this to ditch the rest

    //the confirmation vars
    String check_line = "";
    String CHECK_AGAINST = "FW ALERT";
    boolean good_line = false;
    int section_start = 0;
    int section_end = 0;
    int check_iterator = 0;

    //Check that the file actually has anything in it
    if( log_size == 0 ) {
        System.out.println("The file was empty :[ ");
        System.exit(-1);
    }

    //Keep running the loop for as long as there is more to read
    while( running_total < log_size ) {
        //read in chunk
        if( running_total+CHUNK > log_size ) {
            log_bb = log_fc.map(FileChannel.MapMode.READ_ONLY, running_total, log_size);
            chunk_size = log_size - running_total;
            running_total = log_size;
        } else {
            log_bb = log_fc.map(FileChannel.MapMode.READ_ONLY, running_total, running_total+CHUNK);
            chunk_size = CHUNK;
            running_total += CHUNK;
        }

        //loop through the chunk
        for( int i = 0; i < chunk_size; ++i ) {
            //get each line char by char
            char c = Character.toLowerCase( (char) log_bb.get() );
            if( c == NEWLINE ) {
                //reset everything
                if(good_line==true) {
                    //process line;
                    System.out.println(line);
                }
                line = "";
                skip_line=false;
                section_start=0;
                continue;
            } else if( skip_line == true ) {
                //skip on if we've decided we don't like the line; it's a fikkle friendship ;]
                continue;
            } else if(good_line==true) {
                line += Character.toString(c);
            } else {
                //add to the string
                line += Character.toString(c);
                if(good_line!=true) {
                    //we're at that magic number
                    if( line.length() == 16 ) {
                        Matcher match_tmp = CHECKDATE.matcher(line);
                        if(!match_tmp.matches()) {
                            skip_line=true;
                        }
                    } else if( section_start == 0 && line.length() > 16 && c.equals(new Character(' '))) {
                        section_end=section_start=line.length();
                    } else if( section_start != 0 && line.length()-section_start == 28) {
                        Matcher match_tmp = CHECKGOOD.matcher(line);
                        if(!match_tmp.matches()) {
                            skip_line=true;
                        } else {
                            good_line=true;
                        }
                    }
                }
            }
        }
    }
}
}

class Arguments {
    public String INPUT_LOG_FILE;

public Arguments( String args[] ) throws IllegalArgumentException {
    if( args.length == 0 ) {
        System.out.println("Error in Arguments");
//          throw IllegalArgumentException("Arguments array is empty");
        }
        for( int i=0; i<args.length ; ++i ) {
            if(args[i].indexOf("f") == 1) {
                INPUT_LOG_FILE = args[i].substring(2);
            }
        }
    }

}

Cheers
Bubble