Log file php script use.

DeathfireD

Hello, I happen to come accross this nice php script/snippet used for displaying the amount of bandwidth your web server is using by reading your log files. Now the only issue im having with it is that it uses the log files to determin how much bandwidth iv been using but the log files are set to be deleted every 2 days to make room for the new ones. Now the php script reads from 2 log files, the current day log file and the day before it. Seeing as how my web server software uses log file names EXYYMMDD.LOG (YY=year; MM=month; DD=day)I dont know how I would have the script work without editing it everyday. I dont know php that good so I cant add in anything to update it on its own. So i will post the script here and I hope someone can help me with this issue/problem.

The log file names take the form EXYYMMDD.LOG (YY=year; MM=month; DD=day) and are stored in the same directory as the the web server software executable.

<?

# CONFIGURE THIS:
$todaylog = "/home/ivan/logs/access_log"; # today's logfile
$yesterdaylog = "/home/ivan/logs/access_log.1"; # yesterday's rotated log

# REMOVE OR COMMENT THIS LINE:
print "edit the configuration section inside this file."; exit;

error_reporting(1);

$fd = fopen("$todaylog","r"); 

while ($y = fgets($fd,1024)) { $linecount++; }

$y = NULL;

fclose($fd);

$fd = fopen("$todaylog","r"); 

while ($x = fgets($fd,1024)) { 


if($count == 0) { 
# get the first timestamp
list( , , , $date) = explode(" ", $x);
$date = str_replace("[", "", $date);
list($day, $month, $year) = explode("/", $date, 3);
list($year, $hour, $minute, $second) = explode(":", $year, 4);
$formdate = "$day $month $year $hour:$minute:$second";
$starttime = strtotime("$formdate");
}

if($count == ($linecount - 1)) { 
# get the last timestamp
list( , , , $date) = explode(" ", $x);
$edate = str_replace("[", "", $date);
list($day, $month, $year) = explode("/", $edate, 3);
list($year, $hour, $minute, $second) = explode(":", $year, 4);
$eformdate = "$day $month $year $hour:$minute:$second";
$endtime = strtotime("$eformdate");
}

list( , , , , , , , , , $size) = explode(" ", $x, 10);

if($size != "-") { $total = ($total + $size); }
$count++;
}

fclose($fd); 



$fd = fopen("$yesterdaylog","r"); 

while ($x = fgets($fd,1024)) { 

list( , , , , , , , , , $size) = explode(" ", $x, 10);
if($size != "-") { $ytotal = ($ytotal + $size); }
}


fclose($fd);

$change = ($total / $ytotal);
$change = (($change * 100) - 100);

$logduration = ($endtime - $starttime);
$totalsecs = 86400; # total seconds in a day

$timeofdaycovered = ($totalsecs / $logduration);
$actualchange = ($change * $timeofdaycovered);

$change = round($change, 1);
$actualchange = round($actualchange, 1);

$percentofday = (100 / $timeofdaycovered);
$percentofday = round($percentofday, 1);

if($change > 0) { $change = "+$change"; }
if($actualchange > 0) { $actualchange = "+$actualchange"; }

$ytotal = ($ytotal / 1024);
$total = ($total / 1024);

$total = round($total, 1);
$ytotal = round($ytotal, 1);

print "Yesterday's bandwidth: <b>$ytotal</b> KBytes<br>\n";
print "Today's bandwidth so far: <b>$total</b> KBytes<br>\n";
print "Change in bandwidth: <b>${change}%</b><br>\n";
print "Time covered in today's log: <b>$logduration</b> seconds, (<b>${percentofday}%</b> of day) <br>\n";
print "Estimated change in bandwidth (by midnight): <b>${actualchange}%</b>";

?>

Installer

This should work:

$log_path = 'path/to/log/';
$todaylog = $log_path . date('Eymd.\\L\\O\\G');
$yesterdaylog = $log_path . date('Eymd.\\L\\O\\G', strtotime('yesterday'));

DeathfireD

Originally posted by Installer
This should work:

$log_path = 'path/to/log/';
$todaylog = $log_path . date('Eymd.\\L\\O\\G');
$yesterdaylog = $log_path . date('Eymd.\\L\\O\\G', strtotime('yesterday'));

[/B]

hmmm what about the EX at the start of the log file. see for example todays log file looks like this ex050427.log The ex is added infront of all the log files.

Installer

Sorry, I missed the "X", but a look at the manual page for date() would show you how to easily fix it; like this:

$todaylog = $log_path . date('EXymd.\\L\\O\\G'); 
$yesterdaylog = $log_path . date('EXymd.\\L\\O\\G', strtotime('yesterday'));

(Assuming everything's in uppercase, like in your first post.)

Edit: If you copy and paste the code, be sure there's one backslash in front of each of the letters of "LOG".

DeathfireD

Thanks I was looking at date() and edited the snippet like 20 times. Had no luck so I came here. Thanks I wasnt 100% on how to do it but I was on the right track lol.

Installer

Glad to help. 🙂

DeathfireD

Hmmm ok well I went to test it all out. I made a new folder I placed 2 of my log files in it then I placed the edited php file in it and tryed viewing it. At first all I got was

Yesterday's bandwidth: 0 KBytes
Today's bandwidth so far: 0 KBytes
Change in bandwidth: -100%
Time covered in today's log: 0 seconds, 0% of day
Estimated change in bandwidth (by midnight): 0%

Then I tryed editing some things and now I get a white page. Bolth times I tryed it it didnt work.

Heres a link to the folder where all the files are + the current source.

The php page

The Source

The log files are in the folder Im gonna edit them so its less of a security risk for me :p

Please go and see if you can solve this weid problem.

Installer

Replace the error reporting line with these:

error_reporting(E_ALL);
ini_set('display_errors', '1');

I think you'll then find that your files aren't being opened because you don't have "/" at the end of your log path. So the script is trying to open files named "calcbandEX040428.LOG" and "calcbandEX040427.LOG", which of course don't exist anywhere.

DeathfireD

you know the funny thing is I tryed that and it still didnt work. The only diffrence was it took very log to load then after a few mins it eathor displayed a white page or a php memory error poped up on my server. I have a feeling my log files are to big or somthing.

Each one ranges from 6MBs to 30MBs Depending on traffic for that day.

Ill keep fooling with the paths. Thanks for your help, if you happen to come accross anything that seems to be wrong can post back to me, thanks.

Edited
Tested it with smaller log files and it kinda worked. Numbers where displayed. But anything bigger then 3MBs seems to just time out the script and display a white page. 🙁 Dont know why...any ideas on how to solve this? Click on the links and you can see for your self.

Installer

Yeah, the first link apparently timed out for me (no display).

The only speed-up suggestion I have right now is to replace the first "fclose()" and the second "fopen()" (right after $y = null) with "fseek($fd, 0);", but that probably won't make a lot of difference.

Can you post one or two sample lines from one of the logs? I could then construct a couple bogus files myself to test code on. You can change anything in them that might be sensitive, as long as the format is the same.

DeathfireD

K here. I took one of the logs and deleted alot out of it so theres only about 20 lines. I also replaced the IPs with *'s.

Heres the log

Just delete the here in the start of the log name and your good to go.

Installer

It looks like your problem is with max execution time, since the script works. I did some rewriting, and was able to cut the time by almost two-thirds, so give this a try:

$log_path = 'http://ordway.ath.cx:5121/TDA/calcband/';
$todaylog = $log_path . date('exymd.\L\O\G');
$yesterdaylog = $log_path . date('exymd.\L\O\G', strtotime('yesterday'));

$fd = fopen($todaylog, 'r'); 
$linecount = 0;
while (fgets($fd, 1024)) { 
    $linecount++; 
}
fseek($fd, 0);
$count = -1;
$total = 0;
while ($line = fgets($fd,1024)) { 
    if (++$count == 0) { 
        $starttime = time_mark($line);
    }
    if ($count == ($linecount - 1)) { 
        $endtime = time_mark($line);
    }
    $size_pos = strrpos($line, ' ') + 1;
    $total += substr($line, $size_pos);
}
fclose($fd); 

$fd = fopen($yesterdaylog, 'r'); 
$ytotal = 0;
while ($line = fgets($fd, 1024)) { 
    $size_pos = strrpos($line, ' ') + 1;
    $ytotal += substr($line, $size_pos);
}
fclose($fd);

$change = ($total / $ytotal * 100) - 100;
$logduration = $endtime - $starttime;
$timeofdaycovered = 86400 / $logduration;
$actualchange = round($change * $timeofdaycovered, 1);
$change = round($change, 1);
$percentofday = round(100 / $timeofdaycovered, 1);
if ($change > 0) { 
    $change = '+' . $change; 
}
if ($actualchange > 0) { 
    $actualchange = '+' . $actualchange; 
}
$total = round($total / 1024, 1);
$ytotal = round($ytotal / 1024, 1);

print "Yesterday's bandwidth: <b>" . $ytotal . "</b> KBytes<br>\n
       Today's bandwidth so far: <b>" . $total . "</b> KBytes<br>\n
       Change in bandwidth: <b>" . $change . "%</b><br>\n
       Time covered in today's log: <b>" . $logduration . "</b> seconds, (<b>" . $percentofday . "%</b> of day) <br>\n
       Estimated change in bandwidth (by midnight): <b>" . $actualchange . "%</b>";

function time_mark($str)
{
    $date_pos = strpos($str, '[') + 1;
    $date = substr($str, $date_pos, 11);
    $date = str_replace('/', '-', $date);
    $time = substr($str, $date_pos + 12, 8);
    return strtotime($date . ' ' . $time);
}

The major gain was from the new code in the first while loop; I used a different method and put it in a function.

hth

DeathfireD

Ya looks good just tryed it though and only displayed O's So that means its not reading the log files. Next I went and tryed editing the LOG to L\O\G still nothing. So ...ya its not reading the log files 🙁

New link

Source 2

Installer

If it's not reading the files, then those error reporting lines I posted a while back would have caused messages to be displayed. You did add them, didn't you?

I'll look at it some more in a couple hours.

DeathfireD

I didnt add them. I just coped and pasted what you posted in a new php file and also pasted the source in a txt file so you can view it. View the post I made before you.

Edited
Ok added it and theres a list of errors.

Edited 2
Ok Fixed the LOG thing but now I get

Warning: fseek(): stream does not support seeking in C:\folder\TDA\calcband\testtest.php on line 16

Fatal error: Maximum execution time of 30 seconds exceeded in C:\folder\TDA\calcband\testtest.php on line 33

You can view this by clicking the above link. The source has also been updated.

Installer

The fseek works on mine, must be a version difference. So we'll go back to the fopen.

Let's put in a bunch more error checking. Replace the entire first part of the script with this:

error_reporting(E_ALL); 
ini_set('display_errors', '1');

$log_path = 'http://ordway.ath.cx:5121/TDA/calcband/';
$todaylog = $log_path . date('exymd.\L\O\G');
$yesterdaylog = $log_path . date('exymd.\L\O\G', strtotime('yesterday'));

echo '$todaylog = ' . $todaylog . '<br />';
echo '$yesterdaylog = ' . $yesterdaylog . '<br />';
if (!file_exists($todaylog)) {
    exit('File ' . $todaylog . ' does not exist.');
}
if (!file_exists($todaylog)) {
    exit('File ' . $todaylog . ' does not exist.');
}

$fd = fopen($todaylog, 'rb');

if (!$fd) {
    exit('File ' . $todaylog . ' not opened');
}

$linecount = 0;
while (fgets($fd, 1024)) { 
    $linecount++; 
}
fclose($fd);

echo '$linecount = ' . $linecount . '<br />';
if (!$linecount) {
    exit('No records found in ' . $todaylog);
}

$fd = fopen($todaylog, 'rb');

if (!$fd) {
    exit('File ' . $todaylog . 'not opened');
}

$count = 0;
$total = 0;
while ($line = fgets($fd,1024)) { 
    if (++$count == 1) {

    echo $line . '<br />';

    $starttime = time_mark($line);
}
if ($count == $linecount) { 

    echo $line . '<br />';

    $endtime = time_mark($line);
}
$size_pos = strrpos($line, ' ') + 1;
$total += substr($line, $size_pos);
}
fclose($fd);

echo '$count = ' . $count  . '<br />';
echo '$starttime = ' . $starttime . '<br />';
echo '$endtime = ' . $endtime . '<br />'; 

$fd = fopen($yesterdaylog, 'rb');

if (!$fd) {
    exit('File ' . $yesterdaylog . ' not opened');
}

$ytotal = 0;
while ($line = fgets($fd, 1024)) { 
    $size_pos = strrpos($line, ' ') + 1;
    $ytotal += substr($line, $size_pos);
}
fclose($fd);

echo '$ytotal = ' . $ytotal . '<br />';

(Leave the "$change = ..." and all after it as is.) See what you get now.

DeathfireD

What php version are you using? Current? Im using php 4.3.9

View errors

As you can see its not reading the log files......there in there but it just dosn't want to read them.

also updated source Source

Installer

Remove the "file_exists()" lines, and try again.

DeathfireD

K I removed the lines but now Im getting the

"Fatal error: Maximum execution time of 30 seconds exceeded in C:\Dans\TDA\calcband\testtest.php on line 67"

Same link for script and source.

Installer

I'm seeing results; do you have it working?

At any rate, this revision should be faster.

error_reporting(E_ALL); 
ini_set('display_errors', '1');

$host = 'ordway.ath.cx';
$port = '5121';
$path = '/TDA/calcband/';
$todaylog = date('exymd.\L\O\G');
$yesterdaylog = date('exymd.\L\O\G', strtotime('yesterday'));

if (!ini_get('allow_url_fopen')) {
    exit('URL fopen not allowed');
}

$contents = file('http://' . $host . ':' . $port . $path . $todaylog);
$linecount = count($contents);
$count = 0;
$total = 0;
foreach ($contents as $line) {
    if (++$count == 1) {

    echo $line . '<br />';

    $starttime = time_mark($line);
}
if ($count == $linecount) { 

    echo $line . '<br />';

    $endtime = time_mark($line);
}
$size_pos = strrpos($line, ' ') + 1;
$total += substr($line, $size_pos);
}

$contents = file('http://' . $host . ':' . $port . $path . $yesterdaylog);
$ytotal = 0;
foreach ($contents as $line) {
    $size_pos = strrpos($line, ' ') + 1;
    $ytotal += substr($line, $size_pos);
}

echo '$ytotal = ' . $ytotal . '<br />';

$change = ($total / $ytotal * 100) - 100;
$logduration = $endtime - $starttime;
$timeofdaycovered = 86400 / $logduration;
$actualchange = round($change * $timeofdaycovered, 1);
$change = round($change, 1);
$percentofday = round(100 / $timeofdaycovered, 1);
if ($change > 0) { 
    $change = '+' . $change; 
}
if ($actualchange > 0) { 
    $actualchange = '+' . $actualchange; 
}
$total = round($total / 1024, 1);
$ytotal = round($ytotal / 1024, 1);

print "Yesterday's bandwidth: <b>" . $ytotal . "</b> KBytes<br>\n
       Today's bandwidth so far: <b>" . $total . "</b> KBytes<br>\n
       Change in bandwidth: <b>" . $change . "%</b><br>\n
       Time covered in today's log: <b>" . $logduration . "</b> seconds, (<b>" . $percentofday . "%</b> of day) <br>\n
       Estimated change in bandwidth (by midnight): <b>" . $actualchange . "%</b>";

function time_mark($str)
{
    $date_pos = strpos($str, '[') + 1;
    $date = substr($str, $date_pos, 11);
    $date = str_replace('/', '-', $date);
    $time = substr($str, $date_pos + 12, 8);
    return strtotime($date . ' ' . $time);
}

Edit: Parse error corrected.