php robot

[Anon]

Hi,
I would like to design a search engine robot in PHP. Is this possible? I would like to go to various sites, and read the url content and index for searching. I can easily write a script to do my work, but i am stuck on its deployment. How do I run the sript so that it runs continously and extracts the info for two or three days ? How do i avoid the script timeout ? How should i design it so that system resources are conserved ?
One way i know is to have two scripts, each one calling the other one at the end. this way, two scripts would continously run!

What is Cron, can I use this to accomplish my job ? Can i make a cgi executable with php and run it from the command prompt?
Please help.

[Anon]

Cron is a unix thing, allowing you to schedule programs/scripts to be executed at surtain times/dates.

In your app you could use it to fire up a robot script every 5 minutes and have that script gather information about one url for example.
Don't make the first script call the next, because if the first script fails for some reason, it won't get to call the second script and your app stops.

You can avoid a script-timeout by setting the max_execution_time to a higher value.

Be sure to include something in your code that forces the script to stop when something goes wrong/seems to go wrong, or your machine could fill-up with crashed scripts.

In order to run PHP scripts from the commandline you must re-compile PHP into CGI mode. This makes PHP work juswt like perl:

php script.php

The CGI version can exist alongside the apache module version of PHP.

[Anon]

Alternatively, you can make a crontab entry to use lynx to hit the page via http request, which i use on a system i am building for timed events which can be scheduled to run at certain times of the day.