file() requests in remote server logs

[Anon]

Hi -

Does anyone know how file($remoteURL) appears in the remote URLs server logs? Is it as a direct request, or as a referrer from the local URL that executed the file() function?

The reason I need to know is that I have a lot of code that an administer executes to check remote site details with those in the database (eg prices of items) to keep the database synchronised. But I don't want the URLs of the code/executing page showing up in server logs. They will attract attention since the remote sites will be hit thousands of times during execution of the code.

If the requesting URL will show up, how do I cloak it to eg
http://somesite.com/cgi/robots.cgi
to show the remote admin that its simply an automated code of somesort and not user hits?

Hope this makes sense!

[warmwind]

This sounds like a job for CURL!

http://www.php.net/manual/en/ref.curl.php

It allows you to set your user agent, referring url, and then grab the file:

$ch = curl_init("http://target.site.com");
curl_setopt($ch,CURLOPT_USERAGENT,"Sammy Search Bot");
curl_setopt($ch,CURLOPT_REFERER,"some URL");

curl_setopt($ch, CURLOPT_RETURNTRANSFER,1);
$result=curl_exec ($ch);
curl_close ($ch);

If you're only grabbing html, you should be fine .. they can reverse lookup your IP, but by then it will be TOO LATE!! HAHAHAHAHA!