Spider or crawler PHP-code

Anon

I have searched the net (including PHPbuilder) extensively for some ready-to-run-code or even sample code for a spider or crawler, which can crawl a page and follow links. I found nothing useful.

I have also started coding a crawler myself, but got stuck on several issues. So now I'm here:

Does anybody know of sample PHP-code for the kernel in a spider system, which traverses a site and follows links? I just need to be able to follow local links as I'm only crawling my own site.

Martin

Anon

Hi Martin,

I don't have a ready to roll solution, but maybe Snoopy is something. It's a great PHP class which can fetch a page, including redirects, frames etc. It also features a function to extract href's from the fetched data. I guess it's quite easy to build a loop around this class which crawls through your site.

It can be found at Freshmeat (http://freshmeat.net/projects/snoopy/?highlight=snoopy)

Wim

Anon

Hi Martin,

have a look at Dreyer Media (http://www.dreyermedia.com) there is a project going on called PhpSpider. It is not finished yet, but there is a working demo for a spider.

Cheers
Barnie

Anon

Hi Martin,

have a look at Dreyer Media (http://www.dreyermedia.com) there is a project going on called PhpSpider. It is not finished yet, but there is a working demo for a spider.

Cheers
Barnie

Sorry for second post, replied to wrong message :-)