Remote site map?

[zdislaw]

Does anyone have a good starting point, existing script even, for spidering a (very large) remote site and listing the URLs - and maybe <title> tags - of all pages that the site contains.

I'd like to end up with some sort of delimited file that I can bring in to either a database or spreadsheet.

I'm not looking for a handout, I'll be happy to code it myself. I'm just hoping for a nudge in the right direction if there is one.

Thanks!
Tim

[bubblenut]

If you're running on a *nix box it's probably going to be easiest to do it with wget, if not, well, we'll need to get a little funky.
Dance the recursion with me baby...

It's been a long day

[zdislaw]

I'll be running it on a nice lil linux box, but the site I want to index is (god help me) IIS.

[Weedpacket]

You can run any program you want on your computer - doesn't matter what their site is running on.

[bubblenut]

sounds like wget's your man. It has quite a lot of options so it may take a while to read through enough of the manual to find exactly the ones you need but believe me, it'll be a lot quicker than writing your own 🙂
Good Luck and let us know how you got on.
Bubble