Log class -- how to support many processes writing same log file?
Results 1 to 7 of 7

Thread: Log class -- how to support many processes writing same log file?

  1. #1
    Senior Member
    Join Date
    Apr 2003
    Location
    Silver Lake
    Posts
    4,874

    Log class -- how to support many processes writing same log file?

    I've been using this Log class to conveniently write to a log such that the log file will not grow without bound. The class takes care of rotating the log file and keeping a few backups so that the file never gets too big. I was considering using it in a multiprocessing application -- i.e., there are many processes potentially trying to write the same exact log file. If that log file happens to be in need of rotation, there is the potential for exceptions to be thrown if two processes are trying to write and/or rotate the log at the same time. The odds are pretty long but it could happen.

    Can anyone suggest how I might improve this log class such that many processes can simultaneously write to the same log file without the possibility of processes trying to write a file that is being rotated or rotate a file that is being written?

    PHP Code:
    <?php
    /**
     * A class to simplify log management. Easy to instantiate a logging object
     * which will take care to rotate logs when the current log becomes large
     * so we don't have to worry about logs growing without bound and using
     * up all the hard drive space.
     */

    class Log {
        
    /**
         * The maximum filesize (in bytes) permitted for a log.
         * I have set it here to 100MB
         * @var int
         */
        
    const MAX_LOG_SIZE 100000000;
        
    /**
         * The number of old copies of the file we keep
         * @var int
         */
        
    const BACKUP_FILE_COUNT 5;

        
    /**
         * The path and filename of the log file
         * E.g., /path/to/file.log
         * @var string
         */
        
    private $full_log_file_path NULL;
        
    /**
         * Path to the directory where these log files are getting saved
         * @var string
         */
        
    private $log_dir NULL;
        
    /**
         * The basename of the log file path.
         * E.g., file.log
         * @var string
         */
        
    private $basename NULL;
        
        
    /**
         * constructor, duh. creates an instance of this class
         * you can happily use for logging. This routine parses the provided log path
         * into parts in order to manage log rotation, etc.
         * @param string $full_log_file_path The path and filename of the main log file to be written
         */
        
    public function __construct($full_log_file_path) {
            
    $this->full_log_file_path $full_log_file_path;
            
    $path_parts pathinfo($full_log_file_path);
            
            
    $this->log_dir $path_parts["dirname"];
            
    $this->basename $path_parts["basename"];
            
            
    // if no path was specified, assume CWD:
            
    if (strlen($this->log_dir) == 0){
                
    $this->log_dir ".";
            }
            
            
    // validation / error checking
            
    if (!$this->basename || strlen($this->basename) == 0){
                throw new 
    Exception("Unable to create Log object.  Basename is not valid");
            }
            
            
    clearstatcache();
            if (!
    is_writable($this->log_dir)){
                throw new 
    Exception("Log constructor failed. Log directory (" $this->log_dir ") is not writable");
            }
        }
        
        
        
        
    /**
         * This function rotates the logs, deleting
         * one if we have reached our limit.  It's called
         * when our current log file has met or exceeded
         * self::MAX_LOG_SIZE
         */
        
    function rotate_logs() {
            
    //delete the last log if it exists
            
    $last_log $this->get_log_filename(self::BACKUP_FILE_COUNT);
            if (
    file_exists($last_log)) {
                if (!
    unlink($last_log)) {
                    throw new 
    Exception("Log::rotate_logs failed.  Unable to delete last log (" $last_log ")");
                }
            }

            
    // rotate the other backup logs
            
    for($i=self::BACKUP_FILE_COUNT$i>1$i--) {
                
    $old_log $this->get_log_filename($i 1);
                
    $new_log $this->get_log_filename($i);
                if (
    file_exists($old_log)) {
                    
    rename($old_log$new_log);
                }
            }
        
            
    // rename the main log file to .1
            
    $current_log $this->get_log_filename();
            
    $new_log $this->get_log_filename(1);
            if (
    file_exists($current_log)) {
                
    rename($current_log$new_log);
            }
        }

        
    /**
         * Returns a complete path to either the main log file
         * or, if optional argument $num is supplied, returns 
         * the full path to one of the numbered backup files
         * (e.g., /path/to/file.log.1)
         * @param int $num
         */
        
    function get_log_filename($num=NULL) {
            if (
    is_null($num)) {
                return 
    $this->full_log_file_path;
            } else {
                return 
    $this->full_log_file_path "." $num;
            }
        } 
    // get_log_filename()

        /**
          * This routine writes a message to the map log, taking care to
          * rotate it first if the resulting file would exceed the threshold size
          */
        
    function write($msg) {
            
    clearstatcache();
            
    $msg_length strlen($msg);
            if (
    $msg_length self::MAX_LOG_SIZE) {
                throw new 
    Exception("Unable to write log message to " $this->full_log_file_path " because the message length (" $msg_length ") exceeds the maximum log size (" self::MAX_LOG_SIZE ")");
            }
            if (
    file_exists($this->full_log_file_path)) {
                if (!
    is_writable($this->full_log_file_path)) {
                    throw new 
    Exception("Unable to write log file. " $this->full_log_file_path " is not writable");
                }
                
    $pending_file_size filesize($this->full_log_file_path) + $msg_length;
                if (
    $pending_file_size self::MAX_LOG_SIZE){
                    
    $this->rotate_logs();
                }
                
            } elseif (!
    is_writable($this->log_dir)) {
                    throw new 
    Exception("Unable to create log file because directory " $this->log " is not writable");
            }
            
    file_put_contents($this->full_log_file_path"[" date("Y-m-d H:i:s") . "] - " $msg "\n"FILE_APPEND);
        } 
    // write()
        
    // class Log
    IMPORTANT: STOP using the mysql extension. Use mysqli or pdo instead.
    World War One happened 100 years ago. Visit Old Grey Horror for the agony and irony.

  2. #2
    Senior Member traq's Avatar
    Join Date
    Jun 2011
    Location
    so.Cal
    Posts
    949
    Maybe have a file in the log directory that you could use as a flag?

    PHP Code:
    // . . .

    private $flag;

    function 
    __construct(){
    // . . .
        // get file handle for the log_access_flag
        // 'c' creates file if not exists (should only need happen once in all eternity)
        
    $this->flag fopen$this->log_dir.'/log_access_flag','c' );
        if( ! 
    $this->flag ){
            
    thrown new Exception"Log constructor failed. Unable to open/create file: {$this->log_dir}/log_access_flag" );
        }

    // . . .
    }

    function 
    __destruct(){
        
    // explicitly release any locks (required since PHP 5.3.2)
        
    $this->log_lockLOCK_UN );
    }

    function 
    rotate_logs(){
        try{
            
    // put a write lock on the flag file so other processes won't try to write at the same time
            
    $this->log_lockLOCK_EX );
    // . . .

    // . . .
            // revert to read lock when you're done
            
    $this->log_lockLOCK_SH );
        }
        catch( 
    Exception $e ){
            throw new 
    Exception"Log::rotate_logs() failed: ".$e->getMessage() );
        }
    }

    function 
    write$msg ){
        try{
            
    // put a write lock on the flag file so other processes won't try to write at the same time
            
    $this->log_lockLOCK_EX );
    // . . .

    // . . .
            // revert to read lock when you're done
            
    $this->log_lockLOCK_SH );
        }
        catch( 
    Exception $e ){
            throw new 
    Exception"Log::write() failed: ".$e->getMessage() );
        }
    }

    /**
     * checks|gets log on flag file (creates file if it doesn't exist).
     *
     * @param int $mode    LOCK_SH (read lock; "registers" this process as using the logs), or
     *                     LOCK_EX (write lock; for writing or rotating logs), or
     *                     LOCK_UN (unlock; when you're done so other processes may do as they will)
     * @return bool        true if lock obtained; false otherwise
     */
    function log_lock$mode=LOCK_SH ){
        
    // checks file handle
        
    if( ! is_resource$this->flag ) ){
            
    // request lock on flag
            
    if( flock$this->flag,$mode ) ){
                return 
    true;
            }
        }
        
    // if we're here, everything went wrong
        
    throw new Exception"Unable to lock log access file using mode:$mode." );

    This is just a suggestion, off the top of my head. Completely untested.

    Disadvantage is that, if any given process hangs, it might prevent all others from accessing the logs too.
    Last edited by traq; 06-06-2013 at 11:03 PM.

  3. #3
    Senior Member
    Join Date
    Apr 2003
    Location
    Silver Lake
    Posts
    4,874
    Thanks, traq, for your thoughtful post. I'm pretty sure that structure that you've proposed is going to be *more* reliable than my code, but I wonder if it is *completely* reliable? My fear is that any process that is about to write to the log (or rotate the log files) might be interrupted and de-scheduled in between the lock check and the point at which the process actually attempts to write/rotate the log file. The problem with testing it is that the likelihood of failure is really remote -- even remoter if we can reduce the number of instructions between the lock check and the file-writing. I.e., it would be really hard to get the error condition to arise -- which is a good thing in that problems are infrequent but a bad thing in that it's hard to produce the error condition.

    I'm starting to think I might use MySQL to log these messages to a table. It's my understanding that MySQL is suitable for use in these multi-threaded situations.
    IMPORTANT: STOP using the mysql extension. Use mysqli or pdo instead.
    World War One happened 100 years ago. Visit Old Grey Horror for the agony and irony.

  4. #4
    Senior Member traq's Avatar
    Join Date
    Jun 2011
    Location
    so.Cal
    Posts
    949
    I honestly don't know. It's an untested idea. However, IF all of your processes are using this same class to write, they should all respect the file locks. By having both write() and log_rotate() get an exclusive lock, do their stuff, and then release it, it should create a queue of sorts among all the processes that are using it. I'd have to read up more on how "advisory file locking" works in PHP, but I would imagine it would work across processes (wouldn't be too useful if it didn't).

    On the other hand, yes, DB would be a good solution too. I'm going with SQLite as the default for the logging class I'm writing.

    edit

    BTW, just noticed a typo in my code above: in the log_lock() method, the check should read if( is_resource(...) ) - NOT if( ! is_resource(...) ).
    Last edited by traq; 06-11-2013 at 03:32 PM.

  5. #5
    Senior Member
    Join Date
    Apr 2003
    Location
    Silver Lake
    Posts
    4,874
    Thanks again for your thoughtful input. I ended up creating a log that writes to a DB table and I'll just rely on MySQL to deal with the concurrency. It seems to be working.
    IMPORTANT: STOP using the mysql extension. Use mysqli or pdo instead.
    World War One happened 100 years ago. Visit Old Grey Horror for the agony and irony.

  6. #6
    High Energy Magic Dept. NogDog's Avatar
    Join Date
    Aug 2006
    Location
    Ankh-Morpork
    Posts
    13,941
    If nothing else, it might help log searches.
    Please give us a simple answer, so that we don't have to think, because if we think, we might find answers that don't fit the way we want the world to be." ~ from Nation, by Terry Pratchett

    "But the main reason that any programmer learning any new language thinks the new language is SO much better than the old one is because hes a better programmer now!" ~ http://www.oreillynet.com/ruby/blog/...ck_to_p_1.html


    eBookworm.us

  7. #7
    Senior Member
    Join Date
    Apr 2003
    Location
    Silver Lake
    Posts
    4,874
    Quote Originally Posted by NogDog View Post
    If nothing else, it might help log searches.
    Agreed. I spent a significant amount of time contemplating what kind of structure to give my log table and ultimately just made it really simple.

    On the other hand, databases don't have a tail -f command where you can just watch the log filling up. I expect I'll concoct some kind of AJAX log-monitoring script.
    IMPORTANT: STOP using the mysql extension. Use mysqli or pdo instead.
    World War One happened 100 years ago. Visit Old Grey Horror for the agony and irony.

Thread Information

Users Browsing this Thread

There are currently 1 users browsing this thread. (0 members and 1 guests)

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •