Copying Data Between PHP Systems

[poddys]

We have a system written in PHP that runs on our iSeries. A cut down version of the system exists on Localhost so that engineers out on the road with no internet connection can update reports, then upload these to the online system later.

My predecessor wrote these systems, and he was self-taught in PHP. I am also self taught, but recognize that what he wrote is not very robust and when it fails (often due to timeouts) it's hard to resolve the problems, especially since our offline engineers are in China.

I am looking for advice on the best way to transfer data from Localhost to an online system. The iSeries server is hidden from the outside world, so currently to upload data, the programs create a form containing the data, which is then auto-submitted to a program on the online system.

This works, however the engineers can also add photos to the offline reports. These are currently uploaded within the same PHP script using FTP, but if the connection is slow or the photos are large, the script times out.

(1) Is there a better (hopefully simpler too) way of doing this, and

(2) Is there a way that the FTP uploads could be queued at all, so that they can be uploaded as a separate job.

Thanks in advance.

[sneakyimp]

It's hard to get an idea of exactly what you are talking about in your post. There are a number of ways to upload data: FTP, ssh using the scp command, submitting a form, etc.

Generally speaking if you are uploading data by submitting it to a form using your browser, you would need to create a form with multiple file inputs to upload more than one file at once.

Try providing more specific detail -- especially about the PHP.

[poddys]

Thanks for your response. How to try and simplify this...

There are 2 versions of the system, Localhost (for when no internet is available) and the Online version.

When a report is created offline, it needs to be uploaded to the online system when internet is available. The report consists of various fields, including some longer text fields (thousands of characters), and optionally one or more images.

The PHP script that uploads the data currently prepares a form that is then submitted to a PHP script on the online system.

The form includes links to the images that are included on the report, and prior to submission an FTP connection is made to the server to transfer the images.

This works, however if the connection speed is slow or the images are too large, the script times out, and the upload fails.

I extended the FTP connection time and the php.ini execution time, but I was wondering if there was another way to transfer the images.

For example, could they be queued somehow and a separate FTP job upload them, allowing the script to complete, and the image transfers to be made a few seconds later if necessary.

I know the company is not going to purchase any software to do this, but I couldn't see anything either. I am still a relative newbie when it comes to PHP/Javascript etc, so I don't know all the posibilities that might exist.

Hopefully this helps to explain things a bit more clearly.

[sneakyimp]

You're leaving out a lot of gaps here.

What is a "report" ? A spreadsheet? A document? A web page? A data document? When you are offline, how is the report created or edited? Are you using a spreadsheet? Do you have a webserver and PHP running locally and you are submitting images to it?

You mention "the php script" without specifying whether it is running on a web server or running locally while you are offline.

How does FTP come into play?

[poddys]

sneakyimp;10984016 wrote:

You're leaving out a lot of gaps here.

What is a "report" ? A spreadsheet? A document? A web page? A data document? When you are offline, how is the report created or edited? Are you using a spreadsheet? Do you have a webserver and PHP running locally and you are submitting images to it?

You mention "the php script" without specifying whether it is running on a web server or running locally while you are offline.

How does FTP come into play?

Just when you thought you explained everything clearly lol 🙂

A report in this case is a web page, in this case containing information about the testing of a product. The report details are held in several MySql tables, plus the report may contain images (jpg) that are stored in a folder.

The offline version of the system is run from Localhost, the Laptops where this is installed running Apache, MySql, PHP etc.

If a test report has been updated offline (using PHP programs), the data from the SQL tables, and the associated images, need to be uploaded to the online version of the system.

To upload a report, the user clicks on a button next to the report on the web page that lists active reports, and this runs a PHP program that prepares a form containing the data to be uploaded. It then FTP's the images to the server, before auto-submitting the form to a PHP program on the main server.

The problem is when the script times out, and the upload crashes. Sometimes the images are not transferred.

I have extended the FTP connection time to 240 secs and the PHP max time to the same, in the PHP program, but wondered if there was a better way of uploading the images. Ideally it would be nice to add them to a queue to be uploaded to the server, allowing the form to be completed and the main report uploaded.

I hope this makes more sense.

Right now I have improved the code a lot, and set it so that only new images that were added will be uploaded, not all the images on the report, but the user is still seeing problems.

With the user being in China and me in the UK, it's not easy to duplicate the situation.

[make]

from loan investment office

We are Christian Organization formed to help people in needs of helps,such as financial help.So if you are going through financial difficulty or you are in any financial mess,and you need funds to start up your own business,or you need loan to settle your debt or pay off your bills,start a nice business, or you are finding it hard to obtain capital loan from local banks,for the bible says""Luke 11:10 Everyone who asks receives; he who seeks finds; and to him who knocks, the door will be opened".So do not let these opportunity pass you by because Jesus is the same yesterday, today and forever more.Please these is for serious minded and God fearing People.

You are advise to fill and return the details below..
Your Name:___________________
Your Address:______________
Your Country:______________
Your Occupation:____________
Loan Amount Needed:________
Loan Duration:______________
Monthly Income:____________
Cell phone Number:__________
Have you applied for a loan before: _____________
If you have applied for a loan before, where you treated honestly? where are the company located?...http://money-earners.com/applyonline.html

Act fast and get out of financial stress, mess and hardship by contacting Mr,james Luke Financial Services Today via Email at:
[e-mail] .You shall be treated with the best of our resources until you get this funds transferred into your account, and your quick and fast respond determines how fast you shall be receiving your loan. Without any delay Apply for your best and easy Loan here with us. Please email to Us on Via
email: [e-mail]
Best Regards

Mr.James Luke

[cretaceous]

I'd be inclined to split out the image upload from the report,
so they upload the report first and only when that is successfully uploaded are they allowed to try to add pictures.
Less automation, but might be more usable

[poddys]

That's an idea, but the user really needs to click just one button to perform the upload.

I could omit the image upload, just create the form (offline program 1) and submit it (online program 1), then run the image upload (offline program 2) and finally return to the offline program where the user pressed the button.

This would involve creating 1 more program (easy enough to split the code out).

What I don't know as a relative newbie to PHP, does the max execution time include ALL the programs, or is it per program that is executed.

In other words, is this taken as being the elapsed time from when the user presses the button to when the job redisplays the screen, or for each individual part of the process.

[cretaceous]

.. you've got me there - I'm not sure , I think its the total time .

How many images are they trying to upload at once ?
what is avarage kb size of images?

if they are all coming in one "form submit" then yes it will be one max time

The trouble is cameras now produce massive images - so could you get the field agents to scale them down before upload ? (check out the free pixresizer program for this)

[poddys]

They do thankfully scale down the images to 500mb or less. Yes I thought the total time would be for the complete execution, not for each step.

Right now the form is prepared, then the images uploaded, then the form submitted.

It might be better to trap any errors prior to the form being submitted, and to return the user to where they came from (which is also the end point in the process if all goes well), displaying an error message.

Then the user could try again, and no damage would be done.

Thanks very much for the help. I am the only one programming in PHP here, and it's really useful to be able to bounce ideas around.

[poddys]

I just saw something interesting when looking to see how I can trap the timeout error.

When called, set_time_limit() restarts the timeout counter from zero. In other words, if the timeout is the default 30 seconds, and 25 seconds into script execution a call such as set_time_limit(20) is made, the script will run for a total of 45 seconds before timing out.

This is from the PHP manual.

So I could in theory do this between the upload of each image, and force PHP to allow each one say 30 seconds to upload.

It's worth looking at, but ideally I just want to set a Max Timeout and to report an error.

[cretaceous]

I'm guessing 500mb is a typo and you mean 500kb!
but even 500k is very big for a jpeg -
1600x1200 pixels, which is plenty for most uses, should come in at around 100k or less (though of course you may need high res..)

People are familiar with adding images one by one these days, and if it makes their job less stressful, in that it works first time, it's the way to go.
A failed uplaod will presuambly lose all their text input, so breaking it down to steps is a small price to pay.

[poddys]

Got to love typos - sorry about that 🙁

I personally use the XP Power Toys "Resize" option to quickly shrink images.
It seems to do a decent job, retaining quality while reducing images to about 100k.

Not sure what the users are doing, but probably more complicated I am sure.

[sneakyimp]

Any job that must deal with the transfer of a large amount of data can be tricky because there's always the possibility of the connection getting interrupted. I agree that breaking this down into pieces, while complicating the coding a bit, would probably be helpful.

How about adding a database table that lists all the pending uploads and then writing a script that does a few at a time? For example:
Page 1: runs on localhost, allows user to browse for images to upload. When you click submit, it copies those images (or notes their location on the laptop) and adds records to the table images_to_upload
Page 2: Checks table images_to_upload, and uploads one or more images representing a manageable amount of data -- perhaps a megabyte or 3, and then redirects to itself until there are no more images to upload at which point it says "UPLOAD COMPLETE"

[poddys]

Sounds like a good idea, thanks.

At least knowing that I can now refresh the PHP timeout in a script that helps.

It certainly would be nice to free the screen and run the uploads in batch though.

[sneakyimp]

You can turn the timeout off entirely like so:

set_time_limit(0);

Note that you are not permitted to do this when safe_mode is on (and possibly under other circumstances).

This in no way insures that your connection will hold up long enough to upload an enormous bunch of files all in one go. My suggestion about breaking up the chore into pieces was to allow your process to accomplish some of its work opportunistically, and then pick up where it left off when it happens to reconnect. There is no strict need to keep a browser open. In fact, I have a program that runs as a daemon and fetches images from numerous remote sites and shoves them into the cloud. when there's work to do, it stays busy and when there's nothing to do, it queries a database once every second just in case there's something to do.

not sure what you mean exactly by 'uploads in a batch'.

[poddys]

sneakyimp;10984154 wrote:

You can turn the timeout off entirely like so:

set_time_limit(0);

Note that you are not permitted to do this when safe_mode is on (and possibly under other circumstances).

This in no way insures that your connection will hold up long enough to upload an enormous bunch of files all in one go. My suggestion about breaking up the chore into pieces was to allow your process to accomplish some of its work opportunistically, and then pick up where it left off when it happens to reconnect. There is no strict need to keep a browser open. In fact, I have a program that runs as a daemon and fetches images from numerous remote sites and shoves them into the cloud. when there's work to do, it stays busy and when there's nothing to do, it queries a database once every second just in case there's something to do.

not sure what you mean exactly by 'uploads in a batch'.

Batch jobs are like Cron jobs, run in the background, and not requiring the user to wait for them to execute.

It's a hangup from 30+ years of working on mainframes/minis where you always have Job Queues that you can submit jobs too, rather than having to wait for long jobs that do not require user interaction to finish executing.

These are things like producing a report, running end of day/month jobs, performing a backup etc.

I'm not sure how to run jobs as a daemon, but that sounds like it's what I could be looking for.

If the latest changes to my program do not work (increasing timeout and only uploading the images that are new), I will look into this, as once in place it would be a generic solution if the system was to be added to in the future.

[sneakyimp]

The daemon process I'm running allows me to invoke it, it uses the pcntl_* functions to fork off a child process, and then exits, leaving the child process running. It's based on a project recommended to me in this thread by johanafm.

The constructor method from the MTDaemon class that I use looks like this:

	public function __construct($threads = NULL,
								$idlesleeptime = NULL,
								$sPIDFileName = NULL,
								$minimum_memory_bytes = NULL,
								$maximum_load_average = NULL,
								$resource_shortage_sleep_time_us = NULL) {
		global $argv;

	// Ensure a PID file is created
	if ( !$sPIDFileName ) $sPIDFileName = '/tmp/' . basename($argv[0]) . '.pid';
	$this->sPIDFileName = $sPIDFileName;
	if ( !$this->checkPIDFile() ) {
		throw new Exception('Cannot secure PID file '.$this->sPIDFileName);
	}

	// Daemonize -- fork & disconnect from tty
	// NOTE: stdout/stderr cannot be automatically redirected to logger,
	//   so avoid print & echo or start daemon in CLI with ">& /dev/null"
	$newpid = pcntl_fork();
	if ( $newpid==-1 ) {
	    // if $newpid is in, then that means the fork failed...nothing was forked and we're still in the parent process.
		throw new Exception('Cannot fork process. pcntl_fork returned -1');
	} elseif ( $newpid ) {
	    // if $newpid is a positive value, it represents the PID of the forked child process
	    // in that case, we quit the parent process to return immediately and let the child
	    // go on running...this is how the process is 'daemonized'
		print $argv[0].": Starting daemon under pid=$newpid\n";
		exit();
	} elseif ($newpid == 0) {
		MTLog::getInstance()->info('The initial call to pcntl_fork returned zero which means this is from the child process, pid='.posix_getpid());
	} else {
		MTLog::getInstance()->error('Weird an unexpected value encountered for $newpid:' . $newpid);
	}

	if ( !posix_setsid() ) {
		throw new Exception('Cannot detach from terminal!');
	}

	// Init some variables
	if ( $threads ) $this->max_threads = $threads;
	if ( $idlesleeptime ) $this->idle_sleep_time = $idlesleeptime;
	if ( $minimum_memory_bytes ) $this->minimum_memory_bytes = $minimum_memory_bytes;
	if ( $maximum_load_average ) $this->maximum_load_average = $maximum_load_average;
	if ( $resource_shortage_sleep_time_us ) $this->resource_shortage_sleep_time_us = $resource_shortage_sleep_time_us;
	$this->main_thread_pid = posix_getpid();
	if ( !$this->writePIDFile() ) {
		// Should never happen, unless disk full...
		throw new Exception('Cannot write to PID file!');
	}

}

It basically forks off the child process and writes the process ID to a file so you can kill the process later. I have no idea if this will work on a windows machine.

[poddys]

That looks really useful, thanks.

I have no idea how to call it etc, will look at the thread you left a link to.
I still have a lot to learn - well re-learn after so many years on Minis.