Looking for a server-sided web crawler

[pfeiffel]

Good evening,

for a couple of months now I am looking for a web spider. However, I couldn't find what I was looking for.

Basically, it should be similar to Xenu Link Sleuth, however, it should be programmed in a server-sided language such as PHP.

At a minimum I am looking for these features:

• crawling a whole list of websites and checking the http header status of all external links
• exporting the result (URL+http header code) in a csv or writing it in a mysql database
• multi-threading

If anyone of you has an idea where I could look or which existing (PHP) script would be easy to tweak I am more than grateful!

Thanks a lot.

Mark

[dagon]

if you can't find one you have two options, write one or pay some one to write one.

[pfeiffel]

I will do this. But I am pretty sure that something similar is existing.