help / advice

[ulikechicken]

Hi.

I am building a website similar to http://www.newsnow.co.uk for the purpose of this post i suggest you look at the site as i don't really know the terminology to describe it.

However I am a newbie and struggling a little to understand how to do this. my thoughts are that I could use a spider to search for the keywords i determine.

however i am confused over how to use spiders.

would i use a table of the websites i want to search for articles and then get the spider to return the address to the database
so that i can output the link on my website ?

if this isn't right then please give me whatever help you can.

has anyone seen any tutorials or posts or anything that can help me ?

thank you

paul

[AstroTeg]

I'd recommend starting off small. Work on creating a spider (or script) that can go out to a page of your choosing and have it determine if the article on that page is one you want or not. If it is, develop the code to have it save the URL. If its not, develop the code to mark that page as read (assuming you don't plan to visit that url anytime in the near future - might include a timestamp of when you last visited the url).

Once you you've got this working, then its a matter of expanding your bot a bit to crawl a list of urls (ones that you manually provide). Once this is working, you might go back to the bot and have it look for new possible urls and save that to your url list to crawl.

"Divide and conquer"