[RESOLVED] log file manipulation

sgray

I have a log file in the form of:

<edit> see attached file for layout </edit>

I'm wanting to build an array which contains the rule number, as well as the first instance of the first UID that follows this rule. In this example, i'd like to write 'rule-7' to array $a, and the first uid {4223C20D-4173-4956-8D70-06AB1AF1B778} to array b$. I figure I then can combine the arrays to form $relationship[rule7] = {4223C20D-4173-4956-8D70-06AB1AF1B778}. It's important I only get the first UID as there are 5-6 other UID's before another rule number appears. I'd be so very greatful if anyone could lend me a hand with either the code or commands to do this.

Thanks,

Stew

bradgrafelman

If you can get that log data into a PHP variable, you can use this:

$pattern = '/\((rule-[0-9]+).*\{((?:[[:alnum:]]+-){4}[[:alnum:]]+)\}/Us';
preg_match_all($pattern, $log_data, $matches);

to get an array $matches like this:

Array
(
	...

[1] => Array
    (
        [0] => rule-7
        [1] => rule-858
    )

[2] => Array
    (
        [0] => 4223C20D-4173-4956-8D70-06AB1AF1B778
        [1] => 6660085D-4173-4956-8D70-06ADBB61B778
    )

)

Note that I removed the [0] index because that's just going to contain garbage we're not really interested in. Some things you should note about my example regexp:

'rule' is case-sensitive (throw an 'i' at the end of the pattern to change this)
The GUID must be surrounded by braces {} and consist only of alphanumeric characters (case doesn't matter) and hyphens.
The GUID parts aren't fixed in size, only in number (i.e. it just looks for 5 sets of alphanumeric text joined with hypens, it doesn't check to see that the first block contains 8 characters, the middle three contain 4 characters, etc. etc.

sgray

Thanks for the response, I'd wondered if this has been put in the too hard basket 😛 I'm pretty envious of your regex skills there. With the way the data will be arranged in the $matches array, is there going to be any way to "resolve" rules to UID's? I have a php script which runs through some firewall log files and strips out all of the UID's and counts the number of times it shows in the logs. Ie UID XXXXXX shows up ten times (suggesting that rule has been hit 10 times). The log file refers to UID's rather than rule numbers and there is no correlation between the two.

Another file which I'll be parsing with your code shows both the UID and rule number. I now need my first script to be able to look at the $matches array and calculate that UID = ruleX. With this I'll eventually have the result I'm after - "rule 5 has been hit 5 times". It's a way of improving firewall efficiency by placing the most used rules at the top of a security policy, and by being able to remove rules which are no longer being used.

Ive been using test log files that have about 200 entries in them, I'm hoping my code will work quickly on the real deal - log files containing upwards of 500,000 lines!

Thanks heaps for your help.

bradgrafelman

So you want to search using the UID? In that case, I think we should simplify our $matches array into a more simpler array, like so:

// after the preg_match_all() bit...
$matches = array_combine($matches[1], $matches[2]);

Now it's just a matter of using [man]array_search/man to retrieve the rule with the given GUID.

Couple of things to note about my use of [man]array_combine/man:

This function requires PHP5. As with almost all other functions that require PHP5, someone has contributed a "workaround" for those running PHP4 here.
I'm assuming that rule names/id's/whatever are unique (i.e. no rule can have more than one GUID tied to it).

EDIT: By the way... I'm curious to see how this approach withstands heavier loads, so once you scale it up to the 500k lines, let us know how it performs?

sgray

the $data source for preg_match_all can only be a string right? I've been loading the file using file($logfile) which loads it into an array. For text manipulation is there a better way of opening the files? does opening the file as a string strip formatting such as tabs and carriage returns?

also, how is that array output generated. print_r($matches) or print_r($matches[0]) etc dont output what I'm after, or what you've shown (might be for the reasons outlined above). I've attempted to construct some dodgey for loops and things with preg_match_all but maybe it's not working as it should.

thanks

bradgrafelman

sgray wrote:
For text manipulation is there a better way of opening the files? does opening the file as a string strip formatting such as tabs and carriage returns?

The function [man]file_get_contents/man will do what you need here. And no, loading the entire file into a string doesn't modify any tab characters or anything like that.

sgray wrote:
also, how is that array output generated. print_r($matches) or print_r($matches[0]) etc dont output what I'm after, or what you've shown (might be for the reasons outlined above).

Well, if you didn't load the contents of the file into a string, then yeah, that'd be your problem. Try fixing that first. To get my sample output, I just did a print_r($matches) and omitted the first index returned (for brevity, as stated above).

sgray

sweet, almost there. I have the script resolving UID to rule numbers now. Thanks heaps 🙂 The last part is generating the html tables. I have the following array defined as $raw;

[4] => Array
    (
        [0] => 777
        [1] => 592
        [2] => 592
        [3] => 777
    )

[5] => Array
    (
        [0] => 94008E69-9822-4031-8DB8-75FA70F72F50
        [1] => 983269E8-9D56-40A9-A524-B4DF88F62CB9
        [2] => 983269E8-9D56-40A9-A524-B4DF88F62CB9
        [3] => 94008E69-9822-4031-8DB8-75FA70F72F50

my PHP table code looks like the following:

<?php
echo "<table border=\"1\" cellspacing=0><tr><td>source</td><td>dest</td><td>proto</td><td>port</td><td>policy</td><td>hits</td><tr>";
$num = count($raw[1]);
while ($num > -1) {
  echo "<tr><td>" .$raw[1][$num]. "</td></td><td>" .$raw[2][0]. "</td><td>" .$raw[3][0]. "</td><td>" .$raw[4][0]. "</td><td>" .$resolution[1]. "</td><td>hits</td><tr>";
  $num = num -1;
}
php?>

When using something like .$raw[2][0]., it works fine. When using the dynamic equivalent .$raw[2][$num]. it doesnt show anything. I figure it's translating the $num part, however I'm not quite sure why. What have I done wrong here?

bradgrafelman

Instead of emulating a [man]for[/man] loop, why not use one:

for($i=count($raw[1]); $i>-1; $i--) {
  echo "<tr><td>" .$raw[1][$num]. "</td></td><td>" .$raw[2][$num]. "</td><td>" .$raw[3][$num]. "</td><td>" .$raw[4][$num]. "</td><td>" .$resolution[1]. "</td><td>hits</td><tr>";
}

If that doesn't work... you'll have to explain more about your array structure (or show us a print_r() on it).

Also... when posting PHP code, please use the board's [PHP][/PHP] bbcode tags - they make the code much, much easier to read (as well as aiding the diagnosis of syntax errors). You should edit your post now and add these bbcode tags to help others in reading/analyzing your code.

sgray

Thanks a lot for all your help with this one bradgrafelman. I got it working yesterday and ran it through some of the larger log files. It parses them at a rate of 350,000 lines every 10seconds. I had to bump the max memory and script execution times but it works beautifully now. Cheers 🙂

bradgrafelman

Glad you got it working. I've been a bit busy as of late, and hadn't had time to respond (though I did leave the e-mail marked unread in my inbox, hoping it would spur me to return... which it finally did! :p).

Anyway... don't forget to mark this thread resolved.