My script is stopping after a thousand iteration or so ?

habbardone

I have my script running, but it is stopping after running for
a few hours.

The script is comparing text files one at a time.
It does NOT use mysql at all.

The script compares 2500 files to each other.
So that is 2500 x 2500 combinations = 6.25 million combinations

For some reason it is stopping.

The script is running on a remote server.

The script writes a log file as it goes,
The last time I ran the script (yesterday) the last entry
was 13 vs 1623 , so it completed (12 x 2500) + 1623 = 31,623

The day before it stopped in a different place ...
6 vs 2109, so it completed (5 x 2500) + 2109 = 14,609

I checked the apache error log and there are no reported errors,

The script just stops.

Why would this happen ?

I start the script from my home laptop and the script
runs on the server and the server reports no errors.

Maybe it is because my laptop shuts down after
being idle for a while ... but once the script is running
on the server would it care if the browser that started it closes ?

I could adjust my laptop settings to keep it running all night I guess :rolleyes:

Any thoughts ?

Shall I post the script ?

Thanks

Weedpacket

I could adjust my laptop settings to keep it running all night I guess

If you don't want your laptop turning off then telling it to not turn off sounds like a reasonable option. Especially if it had previously been told to turn off.

but once the script is running
on the server would it care if the browser that started it closes

See [man]connection-handling[/man].

habbardone

Thanks for the hints !

I did both! Stopped the laptop from closing and
added the ignore_user_abort(true).

So it is continuing to run now.

But it is taking a long time.
I have 5000 files to check against each other,
so 25 million comparisons to do. It looks like it will take
a week on my server !

I think I need to convert it to run on my home pc maybe
as a C or C++ application ?

Or something else ?

Derokorian

The thing that I notice is you say x2500 no matter what file its on. But if you have a list of four files:

FileA
FileB
FileC
FileD

Then once you've check A against the other 3, you never need to check it again. So in this example, there is 9 combinations to check, not 4x4=16. Because you check A against B, C, and D, then you check B against C and D (because you've already compared against A in the first iteration), then finally you check C against D as your last check.

Weedpacket

As Derokorian said; I was assuming for some reason that the comparison was non-symmetric, where the results of comparing file A to file B are independent of the results of comparing file B to file A. If that's not the case then instead of 5000×5000=25,000,000 comparisons you only have 5000×(5000+1)/2=12,502,500 comparisons to make. (Or, according to use the earlier numbers, 3,126,250 comparisons instead of 6,250,000).

habbardone

Yes, you are probably both correct, all I was emphasising
is that there is a lot of processing and that it is taking days.
If it is three days rather then seven it is still too long !

I am looking at converting the script into c++ so I can
avoid using my server.

This thread was very helpful though as it prompted me to look at and
use the ignore_user_abort(true) function and changed the settings on my laptop.

So ... thanks again.

habbardone

Yes, you are probably both correct, all I was emphasising
is that there is a lot of processing and that it is taking days.
If it is three days rather then seven it is still too long !

I am looking at converting the script into c++ so I can
avoid using my server.

This thread was very helpful though as it prompted me to look at and
use the ignore_user_abort(true) function and changed the settings on my laptop.

So ... thanks again.

habbardone

Hello ...

Having thought about your comments overnight ...

I have changed the inner loop to ...

  // Now select the second article for the comparison

  for ($art2 = $art1+1; $art2 <= $art_cnt; $art2++) {

I think this will ensure they still all get compared without re-checking
the files already checked.

Does this apply your logic correctly ?

Many thanks again for pointing out the duplication. 🙂