Suggestion for parsing through massive images directory?

[kilbey1]

Hi folks,

I'm looking for coding suggestions. My attack plan to get images information from a directory into a MySQL 5 DB, which start out as many compressed files. Once these files are unzipped into a directory, my method has been to recurse through the images directory - which has thousands of images - and load the file data into an array for processing and loading into the DB (id, path, extension, etc.).

However, I have reached the max array limit, so this will no longer work. I am looking for other solutions.

One idea I had is rather than recurse through all the folders in the images directory at once, to instead put each sub-folder into the array, load the images into the DB, than empty the array, adding the next sub-folder into the array, etc.

I don't know XML, but I was curious as to the benefits of putting all the data information into an XML file and using this to load the DB. I still see myself running up against array limits.

Thanks,
Eve

[mpb001]

XML would be incredibly fat due to it's verbosity and be slow to parse. The ideal way to do it is recursively, using objects to manage the sub arrays. If you were talking between things where serialisation is not compatible( like a java web service ) XML is needed as a communication method.

Flickr now offer serialised php arrays instead of xml if you want. Talking to the system internally via xml is usually referred to as Enterprise 😉

Recursion is a bit hard to do without testing but the following should give you an idea of a possible path to take. Implementation is a matter of taste.

<?php
class DirContents
{
    private $sub_directories    = array();
    private $files              = array();
    private $current_directory  = array();
    private $is_valid           = false;

public function __construct()
{
}

/**
 * @param string $folder : Full path to folder
 */
public function ParseDirectory( $current_directory )
{
    $this->current_directory = $current_directory; 

    /**
    * psueodocode
    * 1. check where folder exists and has files/folders etc, set $is_valid = true 
    * 2. set files
    * 3. set sub dirs - ignoring . and .. ( they are special )
    */    
}   

public function IsValid()
{
    return $this->is_valid;   
}


public function GetCurrentDirectory()
{
    return $this->current_directory;  
}


public function GetSubDirectories()
{
    return $this->sub_directories;   
}

public function GetFiles()
{
    return $this->files;   
}
}



class DBFilePathDetailsMigrator
{
    public function __construct()
    {
    }


function Migrate( DirContents $contents )
{
    if( $contents->IsValid() )
    {
        $current_dir = $contents->GetCurrentDirectory();     
        $files       = $contents->GetFiles();  
        $sub_dirs    = $contents->GetSubDirectories();

        $this->DealWithFiles( $current_dir , $files ); 
        $this->DealWithSubDirectories( $current_dir , $sub_dirs );
    }
}


private function DealWithFiles( $current_dir, $files )
{
    if( count( $files ) == 0 )
    {
       return false;
    }   

    foreach( $files AS $file )
    {
        //do what you need to do with database or whatever   
    }
}


private function DealWithSubDirectories( $current_dir, $sub_dirs )
{
    if( count( $sub_dirs)  == 0  )
    {
       return false;
    }

    foreach( $sub_dirs  AS $sub_dir )
    {
        $full_path = $current_dir . $sub_dir;

        $contents  = new DirContents();   
        $contents->ParseDirectory(  $full_path );
        $this->Migrate( $contents );
    }
}

}
?>

In either case memory limits and time limits will have to be watched. Were you ingoring . and .. as directories? Bad things happen if you don't.

[kilbey1]

Yeah, I'm using a recursive directory class:
http://lixlpixel.org/recursive_function/php/recursive_directory_scan/

But, my directory - filled with the image folders that get extracted from the many zip files - is really large. Getting everything into 1 array reaches PHP's memory limits.

So I wondering if my approach would be to rather than passing /parent_folder/ and everything below into the class, do something along these lines:
1. CD into the parent directory
2. get the sub-directory names into an array
3. using a foreach or similar, pass each subdirectory name into the class- which would, in effect, run the class many times, but would not reach memory limits. It may mean altering the classs, as I haven't looked at its limitations in a while.

I'm wondering if this logic is sound, or if there is another approach I should look into?

Thanks,
Eve