Spiders

[bigdaddysheikh]

Hello,

This is my first time posting on this message board. I like this board because it has solved lots of my problems. I want to make a search engine. In order for that i need to learn how to make a spider that will go to the url that are submitted and check the meta tags adn store it in the database. The thing is I don't know anything about spiders or how they work or how i initiate them? Can some one please guide me to a newbie tutorial on spiders or any other info that would be helpful.

Thank you for your time.

[ucbones]

I just implemented a very small, very simple search engine on my site. It does require you, however to add each URL you want to search into a MySQL database, although it shouldn't be hard to get this to be automatic if you're lazy, or it's a big site. My script essentially does two things.

First, we define the content of the page. In all probability, your site contains a lot of headers and footers, which you probably don't want to include, so it takes only the content of the site. It grabs the title of the page, and inserts the contents (after strip_tags, and strtolower) into a DB, along with the header. You repopulate the search engine using a separate script, and with my relatively small site (it will soon get a lot bigger...), takes about 1 second to build the search engine (that's about 0.05sec per page). It then retrieves data from the database via MATCH... AGAINST (look for FULL TEXT SEARCH in the MySQL manual) and sorts them by relevancy. The only annoyance is that the relevancy it comes up with is very low, so I increase it a little... Wish I could figure out a better way of doing that!

If you want to take a look I can e-mail the files...

ucbones

[edit]try http://www.onlamp.com/lpt/a/2753/ my engine is built on this code...[/edit]