I'm just wondering out of curiosity, how are spider/crawler scripts made? Just the basic setup and stuff is all i'm wondering (and also how do you get it to "follow" links?) I already searched a couple of times on google and found some stuff, but some of it just didn't have the info i was looking for.

    You may want to read This thread for some posts of mine and others on this topic.

    Moreover, try doing a forum search for my previous posts mentioning the word "Spider", I've posted about it several times (Spoiler: I did write a web spider in PHP)

    Mark

      I'll try that out, thanks.

        Write a Reply...