Some websites cannot be obtained with file_get_contents

teguh123

<?
function usecurl($url)
{
$ch = curl_init();
curl_setopt ($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt ($ch, CURLOPT_URL, $url );
//curl_setopt ($ch, CURLOPT_USERAGENT, "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.1) Gecko/20061204 Firefox/1.6");

curl_setopt ($ch, CURLOPT_USERAGENT, "Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.8.1) Gecko/20061010 Firefox/2.0");
curl_setopt ($ch, CURLOPT_HTTP_VERSION, CURL_HTTP_VERSION_1_1);

$output = curl_exec ( $ch );
curl_close($ch);

    return $output;   
}


echo strlen(file_get_contents ("http://google.com")) . "<br>";// works
echo strlen(file_get_contents ("http://en.wikipedia.org/wiki/Special:Random")) . "<br>"; //doesn't work

echo strlen(usecurl ("http://google.com")) . "<br>";// works
echo strlen(usecurl ("http://en.wikipedia.org/wiki/Special:Random")) . "<br>"; //doesn't work

echo (usecurl ("http://google.com")) . "<br>";

All right, where do I went wrong here? What curl_setopt should I put in there so I can insert the content of http://en.wikipedia.org/wiki/Special:Random into a string variable.

Notice that if I add

php_value user_agent "Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.8.1) Gecko/20061010 Firefox/2.0"

<IfModule mod_security.c>
SecFilterScanPost
</IfModule>

echo strlen(file_get_contents ("http://en.wikipedia.org/wiki/Special:Random")) . "<br>";

will produce a non zero value indicating that it works. However, I want to do it with curl. That's because adding that doesn't work in hostgator.com

MarkR

Generally speaking, if a web site doesn't want you to do automated requests, it may block them.

You shouldn't try to work around this limitation, because it's both impolite and liable to failure in the future (when they IP block you for breaking their TOS).

Just don't do it.

You can set the user agent with a header which you can specify in the stream context, as well as with ini setting user_agent - see the http wrapper for more details.

You should have sufficient debugging to see what the error is. Enable all error reporting and logging.

Mark