well what is the limit of crawler if after using some crawling function we give some site link which how much pages or how much data we can get and did crawler has any limit of crawling the site ?
infact going to make very simple search engine that can cache keywords title & description of site automatic when people give their Link of site
please give me some guaidness and details on this
realcoder,
I am unsure of what exactly you are asking -- however PHP is NOT a good platform for developing a web crawler.
There are many reasons for this, but here are a few:
PHP needs to be compiled into p-code for processing by an interpreter (such as PHP-CLI) .. this is less efficient than compiling an application in a language such as C/C++.
PHP generally has a script timeout, thus your crawler won't be able to run too quickly.
If I do not understand your question, please let me know.