[RESOLVED] How to keep XML database synchronised with Ziparchive

jeri

Hi there,

I would like some help in understanding the following:
I am developing an internet site based on an .XML database. This database is regularly updated to my server from a third party in the form of a ziparchive. This zip archive containing the actual database test.xml and images.jpg.

At present what I'd thought to do is this: Each time a visitor is looking at the site, my code extracts all files from the archive to a directory (overwriting files from the last time the files where extracted). And then I use simplexml to show the site using the files in this directory.

And now my questions: Is my working method correct?, I mean is it the proper way to extract the contents of the archive each time my site is visited? or is there a way to read and use the contents of a zipped archive directly without extracting the files to a folder each time. In that case can someone help me on my way with the code to use? (look at the code below so far)

Because if my method is correct, then it leads me to the following questions: If the database is updated by the third party (a new ziparchive is ftp'd to my server with data and files added and data and files deleted) and it is extracted; all new files will be added and old ones overwritten, no problem there, but files deleted from the archive will not be deleted from the folder, resulting in using up diskspace on the server uselessly. So In that case can any one help me with some code how to "synchronise" the folder containing the files and the zipped archive.

Here is the (simple but working) code I am using up to now:

// extract archive to folder
    $zip = new ZipArchive;
    $zip->open('./base/Test.zip');
    $zip->extractTo('./base/');
    $zip->close();


// the simple xml part   

// specify url of xml file
$url = "base/Test.xml";
// get xml file contents
$xml = simplexml_load_file($url)or die ("Unable to load XML file!");

// Example loop through
foreach($xml->image as $image)
{
	// Some code
}
// loop ends

I hope I was clear and someone can help me?

etully

You should not unzip the files every time your site is visited by a new user. This is terribly slow and will cause massive CPU use. You should only unzip the files once, right when they are received from the 3rd party.

As for your syncing problem, the first question you are going to have to answer is whether or not you are forced to use this "XML Database" in a zip file thing. If you get to choose another method for file delivery, you'll probably want to design something better - look into rsync. If you are stuck with this zip file thing, then why not simply erase every single file on your site first and then unzip the zip archive? If you don't want to do that (which is fine, I don't blame you), then do this: When you receive a zip file, extract it and use simpleXML to find the filename of every existing file and record them to a table in your MySQL database. Then, the next time you get a zip archive, compare it to what already exists in MySQL. Any new files get added to MySQL. Any files that exist in MySQL but not the new zip file get removed from MySQL and deleted from the filesystem.

jeri

etully, Thank you very much for your time and comprehensible answer. I am working for a customer using an established software who's only possibility is this XML database zip thing so I am forced to work with that. Each time unzipping is terribly slow was exactly my worry, and reason for my quastion but I didn't seem to find an answer on line how other programmers solve this.

The goal of course is that all is done automatically and I do not know when I have received a new Zip file, also as far as I know a PHP script will only execute upon someone opening the page (visiting the site).

Simply erasing the whole folder and unzipping the new contents is the easiest solution so I will go with that. I will then have to find a solution to, detect when a new archive was uploaded on my server so that it can be extracted. (I thought my provider has some possibility to automatically execute scripts on regular intervals, in that case: problem solved, I'll just write a small script detecting new zip file, deleting the folder contents and extracting the zip on day's interval's or so)

If that is not possible One other way maybe to do this with is probably, whenever my page is requested loading the site based upon the existing (old) database first, and the script will then look if there is a new zip file, if so extract its contents and refresh the page afterwards, in that way for a visitor the page loads quickly.

sorry for my long story :o but any other comments are highly appreciated!

etully

jeri, The repeated execution is the easy part. And you don't have to settle for merely once a day. Cron is the tool you're looking for. (Windows servers have an equivalent which I can't help you with). Cron is so wildly flexible that you can run the script at any interval you can imagine. Every day, every minute, or 20 minutes after the top of the hour every hour from 9-5 on Mon-Fri.

Step 1. Ask your ISP what text editor is the default when you edit crontab
(It will be pico, vi, emacs, or Jove).
Step 2. Google for how to save and exit that text editor
Step 3. Learn to SSH into your server.
Step 4. type crontab -e
Step 5. type:
0,30 wget http://www.yourdomain.com/process_incoming_xml_zip.php
Step 6. Save and exit the text editor

Google for crontab to find out what the "0,30 " means and how to change it to arrive at any execution interval you desire.

Your process_incoming_xml_zip.php will need:
if (file_exists("/path/to/wherever/xml/file/arrives/filename.xml.zip"))

Don't run it every minute because unzipping the files might take a long time and you don't want the script to start running while the script is already running from a previous cron invocation a minute earlier. A cleaner way to handle this would be: When you detect the file, rename it immediately and then begin processing it. That way, if cron runs your processing script again, it won't see the file and start processing it.

After your script finds the file, deletes all the files and unzips the archive, then you should use:
unlink("/path/to/wherever/xml/file/arrives/filename.xml.zip");

That will delete the file so that you are ready for the next arrival.

Don't worry about running your script once a minute. The server is designed to handle hundreds of hits per second - one file running once a minute puts zero load on your server - so small it's not measurable.

jeri

Thank you etully, yes perfect this all will do the trick, in fact I have just found out that my ISP has already standard a very easy to configure tool to execute a script at whatever regular interval I want. I'll look into cron also, interesting information, did not know this yet. The rest is now a piece if cake as I will not have problems writing this small script. Thank you again etully for all your worthy comments and advise.

Regards