I'm trying to write my own search engine at the mo.
try reading the files in as the url
e.g. file('http://www.yoursite.com', 'r')
this should ask for the page the way you want and then parse it for links and follow them. Just like a proper search engine!
I think this will work I haven't tied it, it's just an idea.
bo!