BOTS and URL

wolf81

HI people, this is a two question post. I was wondering if any one knows where I could start looking into PHP bots and spyders. Any articles and info will be greatly appreciated.

The second question is, can PHP go through a web site and retreive all the links in a web site. For example. If there is a index.php page and the index.php page has 4 link. 1. contact.php, 2. home.php, 3. stuff.php, and 4. morestuff.php. the php script will go through the index.php page identify the 4 links, then open the links and look for more links within these pages.
Is there a built in function such as 'fread' or something like that for this.

Thanks jn Advance.

Anon

hey, PHP can definately act like a bot. I've done one myself. This site has some useful info:

http://www.robotstxt.org

To get every link in a page. Just create a regular expression that picks out links. You then open each link and get every link within that page. etc etc. I know this regex works:

(eregi("(href = ['\"]?)([^'\" >]*)(['\" >])",$nuline, $text)

Hope that helps!!

Anon

Cool thanks.. its hard to find some good info concerning PHP bots. But again thanks for the help.

Armando P.