PHP Spider to a DB

[Anon]

Basically, I'm looking to write an engine that uses a spider to grab the data from another site and pull it into a database in a one time shot. For example...an engine that goes to amazon.com and grabs every book title, author, and ISBN and stores it into my database. This script would only run once. If anyone has any ideas for this please let me know.

[kid_c]

OK, basically I can give you a few guidelines on how you'd do this..

1 - make sure set_time_limit(0) =)

2 - fopen a main page, and grab any relevant links that you'll need ot get to the other pages (using ereg_replaces, splits,whatever you wish)

3 - make your script follow those links deeper, grabbing what you require, then stealing the 'next page' link

really its hard to say, because this totally depends on the site you wish to grab the data from.....but hopefully that will start you off.....

just remember, writing this script can be quite time-consuming, as sometimes pages change or methods are in place to block your IP if you access too many pages in a certain time period.

Good Luck,

--
Keith Bussey
[e-mail]
Programmer - WISOL.com
[phone] ext. 225

[jordan]

Wow. You, sir, are a masochist.