Multi-fopen; delaying script; file r/w; daemons

[Anon]

Well I figured I could either split this up into multiple seperate posts, or just make it one big post.

To avoid extra emailings for subscribers, I chose the latter option 🙂

So feel free to skip a question if you have no idea. 1 answer or idea is better than none 🙂

Now, the questions in nicely subtitled format:

       Multi-threading fopen

Here's my problem.

Upon execution my script grabs a page from a remote internet site, parses the data, then writes it to a file (it can send to database, but this was just easier at the moment, so it can be changed). It then grabs another page, and does the same with that.

The problem is, the target site is slower than the server I use. ALOT slower.

It takes, on average, 6 seconds per query to get the information. My script must query over 1200 times to complete 1/4 of it's work.

The site changes the information every 30 minutes, so I need to grab the data in under 30 minutes.

Is there any way to use multiple fopen()s in the same script, at the same time, to accomplish what I want to do? (the target server seems to restrict used bandwidth per query, as hundreds of people use the site at the same time without extended delays)

I "could" just split the job into multiple parts and then have each script execute it's own small block of data, and have them all go at the same time.

That would work, but it is definately not wanted. It feels like a hackjob of a work around, and I'm not getting paid to do this, so I might as well learn something new in the process 🙂

     Time-delay in a script

For various reasons, I often come upon a problem that could just be solved by waiting a given amount of time before continuing.

A script keeps looking to see if a given change has occurred every 10 seconds, and stops looking after 60 seconds (to avoid endless loops), for instance.

Is there absolutely ANY way to delay a script for a given amount of time , short of forcing a script to use some sort of insanely complicated mathematic equation or some sort of fopen to a slow/non-existant source (those slowing the process down by however long it takes the server to decide the connection failed)?

I've considered weird loops that do very odd things to try and delay the execution, dhtml refreshes on zombie-slave terminals, timed redirections to form submission urls, and all sorts of twisted techniques.

My hope is there is some form of PHP function to handle this that I'm simply not aware of. Or for that matter, a function in any language. It would seriously make my life easier...and happier 😉

crosses fingers

Begin writing/reading at a given point

If I had a file with an alphabetical only word (A-Z), followed by numbers (data for the word that comes before it), then followed by another word, is it possible to start fread at a given word, and/or stop it at a given point (let's say, the next word)?

Or do I just need to fread all the data into a variable and just do a regex on it?

On a similar though more important note, fwrite. How can I begin writing at a given point in a file?

For instance if I wanted to write a string into a text file after the word "hello", without deleting anything (just insert), how would I do that?

I can't find any mention of it in the fwrite documentation, and the question has been bothering me for ahwile. I get the feeling the answer is staring me right in the face (read: in the manual), but it's evaded me as of yet (the answer, not the manual).

The only idea I have had however, is to use some sort of regex to get to a certain point in the file, and then use some sort of ereg_replace to add the data.

In other words, I don't have much of a clue how to do it.

And on a last note:

           Daemons

Is there a way to activate a function at a given time of day, or every so many units of time?

Other than the UNIX program that does execution upon timing (ctab,tabtime, something like that...can't recall atm), that is. Prefereably a solution within PHP, but I'm doubting it.

Just one of my pipe-dreams, which I get the feeling will get resigned to a "maybe someday" hope...or some brand of useful app (one never knows).

end;

Heh, well that's all the questions I've had (that I can remember) since even hearing about PHP, that I haven't already had answered.

Any help on any of these questions would be much appreciated.

And thanks in advance! (though I'm sure I'll end up saying thanks multiple-more times after helping)

[Anon]

Multi-threading fopen

That would require a multi-threaded PHP which doesn't exist yet
as far as I know (PHP can't even fork() for that matter, which
actually makes sense at least in its Apache-module version.)
You could probably exec() something that would do the gathering
for you, that could be a multiprocess or multithreaded as you
care to make it. Just create a unique string to use as a file
prefix or temp directory that you pass to it, so your waiting
PHP script can tell which result files belong to it and which
don't.

Time-delay in a script

sleep() will do this; just don't get too carried away with your
polling and everything will be fine. sleep() itself doesn't incur
cpu-busy overhead, of course, but every time you wake up
and go check results it does.

Begin writing/reading at a given point

I'll start by hitting my macro key that says "It sounds like you
really need a database here." However, in certain limited
situations you might get a file to work to do what you need,
if (1) you can design it to have fixed-length "records", and (2)
you can devise a means of searching it. (If you already know the
position in the file, you can use fseek() to change the current
position in the file such that the next fwrite() will overwrite it--
assuming, that is, that you opened the file in "r+" mode.)
Just reading the entire file into memory and then writing it back
imposes a huge bottleneck, because only one script at a time
can be doing this--i.e. you'll have to lock the whole file the entire
time.

Daemons ... timed execution

cron is definitely the way to go here. You just use any of the many
available HTTP-getting tools (wget, curl, or even the little perl script
I posted here a while back), write a 2-line shell script to call it,
and tell cron about it by entering in the crontab. QED.

[Anon]

Crontab, that's it. Heh, I thought I had heard of it before 🙂

Thanks for all the answers.

And now for the question I forgot to ask.

Instant browser-feedback

Generally, my query program initializes and prints out the basic html header code and a "Query started" headline.

It then enters a loop, performs the query, prints the number it completed (00001), then goes back to the start of the loop.

The problem is, if after 10 minutes (which the script still isn't done yet) I check the source code of the html page, it has yet to print any "completed" strings to the browser. It hasn't even gotten the begining html code yet.

In other words, I have absolutely no way of knowing how close to finished the script is.

The only time I've gotten it to print was after about 40 minutes, where it informed me it had finished 50 iterations.

The file where it was writing however, showed that it had completed 1100.

So, 2 part question.

1: How can I send informtion to the browser and then have the browser display that information (even with a reasonably long delay) as the browser continues to get updates.

2: In my loop I do not use fclose() as I would then have to stick fopen() in the loop to re-open my results file. I imagine that would add a considerable amount of I/O which isn't needed.

The odd thing is that once I moved the fclose() and fopen() outside the loop, when I view the results file during the running of the script, it seems empty. But I'm 90% sure that it's already used fwrite() in the loop multiple times. But it seems the results don't show up immediately.

Is this just an odd feature where I have to use fclose() before all the information will be printed in the file?

[Anon]

Here's what I've found to work in some cases:

<?php
echo "<html><body>Processing your order <br></body></html>\n";
for ($i = 0; $i < 10; $i++ )
  {
  sleep(2);
  echo "<html><body>     $i <br></body></html>\n";
  flush();
  }

echo "<html><body>Done processing.<br></body></html>\n";
flush();
sleep(2);
?>
<html>
<head>
<meta http-equiv="Refresh" content="2; URL=/">
</head>
<body><a href="/">click here if you aren't forwarded
automatically.</a></body>
</html>

[Anon]

Oops, things got messed up there in the translation.

The echo in the body of the loop shouldn't be wrapped to the next line.
Also, the concluding comments were dropped. Sending each partial
piece you want displayed in its own <html> </html> segment
and flushing() to force the data out to the client. This appears to
work better with NS than with IE; writing larger chunks by appending
a bunch of trailing spaces, which might help to fill up some local
buffer somewhere, helps the process to work better with IE.