[RESOLVED] How to Get HTTP Status Code of Web Pages

dougmcc1

How can I use PHP to get the status code of webpages? All I want is to submit a list of URLS and then output the URL and status code of each.

kitchin

Do you have CURL compiled in your PHP? Maybe there is an easier way, but this will give you the HTTP header block...

function get_http_header($url) {
  $ch = curl_init(); // create a new CURL resource
  curl_setopt($ch, CURLOPT_URL, $url);
  curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
  curl_setopt($ch, CURLOPT_NOBODY, true);
  curl_setopt($ch, CURLOPT_HEADER, true);
  $st= curl_exec($ch);
  curl_close($ch);
  return $st;
?>

Weedpacket

You could also use [man]fsockopen[/man], see what the first line of the response says. Use a HEAD request instead of GET for efficiency.

MarkR

You can use fopen on the URL to get the contents of it, then use stream_get_meta_data to retrieve the HTTP status code out of the stream metadata.

The only snag is, if the HTTP status code is an error (for example, 401, 403, 404, 500 etc), fopen will fail and throw an error. You can get the code by parsing this error, but that's all I'm afraid.

So some careful error handling will be necessary.

Mark

kitchin

For PHP4 in Windows, (installer version), you can enable CURL by editing php.ini in your Windows directory. Search for the line with php_curl.dll and remove the semicolon in front of it.

bradgrafelman

Weedpacket's suggestion is the way I would go. It's efficient, doesn't require any extensions such as cURL, and gets the job done.

All you'd need to do is open a socket, send a request such as 'HEAD /images/test.jpg HTTP/1.0' and then parse the first line returned.

Example data returned from a HEAD request:

HTTP/1.1 404 Not Found
Date: Sat, 25 Feb 2006 16:40:10 GMT
Server: Apache/1.3.31 (Win32) PHP/4.3.7
Connection: close
Content-Type: text/html; charset=iso-8859-1

MarkR

Do not write your own HTTP implementation. PHP has a perfectly good one, built-in, no extensions required.

Use fopen with a URL, it will work. If you want to do a head request, this is possible by setting the stream context options - see the http wrapper docs for more info.

Mark

bradgrafelman

Using [man]fopen[/man] will result in a warning if the URL is not found (404), or any other HTTP error occurs (500, 401, 403, etc.).
It will also use a GET request, forcing the server to load the entire page. Waste of resources.

That's why opening a socket and sending a HEAD request is the best way to go here.

EDIT: Make sure the user isn't trying to point the script back at itself. This can cause an endless loop I believe.

EDIT2: I do agree on thse use of cURL over any fopen, fsockopen, etc. function, however, in the event that you need to retrieve the full contents of the remote website, as it performs much more reliably and quickly than the file functions.

MarkR

bradgrafelman wrote:
1. Using [man]fopen[/man] will result in a warning if the URL is not found (404), or any other HTTP error occurs (500, 401, 403, etc.).

This error can be trapped and behaviour set up accordingly.

2. It will also use a GET request, forcing the server to load the entire page. Waste of resources.

Only if you don't tell it to use a HEAD by setting the method parameter in the stream context.

EDIT: Make sure the user isn't trying to point the script back at itself. This can cause an endless loop I believe.

Not if you require a POST method to do this.

EDIT2: I do agree on thse use of cURL over any fopen, fsockopen, etc. function, however, in the event that you need to retrieve the full contents of the remote website, as it performs much more reliably and quickly than the file functions.

Well, I don't. fopen() is the right way to go.

Mark

bradgrafelman

Note: Context support was added with PHP 5.0.0.

And if people don't want to upgrade to PHP5?

visualAd

dougmcc1 wrote:
How can I use PHP to get the status code of webpages? All I want is to submit a list of URLS and then output the URL and status code of each.

I have created a script which I used to poll a server to check that it was not down. Although, PHP has a stream timeout, it does not appear to have a connection timeout. Some operating systems will respond with a connection refused when trying to connect to an unbound port.

However, some routers and some os'es will simply ignore the request and let the client on the otherside timeout. As PHP does not appear to provide a timeout for connections; short of using the max_execution time (which generates a fatal error when exceeeded), you could find that your uses are waiting a long time when they execute your program, if any of the sites happen to be down.

MarkR

I've been working with HTTP lately, and I've found that although PHP's HTTP implementation is alright for simple stuff, it's not really good enough for something cleverer.

Specifically, there's no way in the stock HTTP implementation to set the timeout separately at different stages of the request - such as connect, waiting for headers or waiting for the content etc.

If the server accepts the connection but hangs indefinitely after doing so, PHP doesn't appear to have a timeout at this stage at all. This is obviously not helpful for an automated web checker, spider, robot etc.

I've written my own HTTP implementation which fixes these bugs, but it's not yet complete.

I recommend either using curl (if that gives you enough control), or trying to live with the shortcomings of the php one in the interim.

Mark

bokehman

MarkR wrote:
Well, I don't. fopen() is the right way to go.

If you believe downloading a complete file is the right way to go just to check it exists, you are in the wrong line of work. What if the file was 100 megabytes?

MarkR wrote:
, there's no way in the stock HTTP implementation to set the timeout separately at different stages of the request - such as connect, waiting for headers or waiting for the content etc.

If the server accepts the connection but hangs indefinitely after doing so, PHP doesn't appear to have a timeout at this stage at all. This is obviously not helpful for an automated web checker, spider, robot etc.

stream_set_timeout(); can be altered before any step.

Here's a function that will get the status code without downloading the contents:

function http_file_status($url, $followRedirects = true)
{
   $url_parsed = parse_url($url);
   if (!@$url_parsed['scheme']) $url_parsed = parse_url('http://'.$url);
   extract($url_parsed);
   if(!@$port) $port = 80;
   if(!@$path) $path = '/';
   if(@$query) $path .= '?'.$query;
   $out = "HEAD $path HTTP/1.1\r\n";
   $out .= "Host: $host\r\n";
   $out .= "Connection: Close\r\n\r\n";
   if(!$fp = @fsockopen($host, $port, $es, $en, 5)){
	   return false;
   }
   fwrite($fp, $out);
   while (!feof($fp)) {
	   $s = fgets($fp, 128);
	   if(($followRedirects) && (preg_match('/^Location:/i', $s) != false)){
		   fclose($fp);
	       return http_file_status(trim(preg_replace("/Location:/i", "", $s)));
       }
       if(preg_match('@HTTP[/]1[.][01x][\s]{1,}([1-5][01][0-9])[\s].*$@', $s, $matches))
            {
	            $status = $matches[1];
            }
   }
   fclose($fp);
   if(!empty($status))
   {
	   return $status;
   }
   return false;
}

visualAd

bokehman wrote:
If you believe downloading a complete file is the right way to go just to check it exists, you are in the wrong line of work. What if the file was 100 megabytes?

stream_set_timeout(); can be altered before any step.

Yes, but stream_set_timeout() is only for an already established connection. They need a function like connect_timeout(). I solved the problem by executing the script as a separate process and setting the timout in the script which started it.

bokehman

visualAd wrote:
Yes, but stream_set_timeout() is only for an already established connection. They need a function like connect_timeout(). I solved the problem by executing the script as a separate process and setting the timout in the script which started it.

stream_set_timeout() works on any established stream whereas setting a time limit with fsockopen works when establishing a connection.

MarkR

bokehman wrote:
If you believe downloading a complete file is the right way to go just to check it exists, you are in the wrong line of work. What if the file was 100 megabytes?

fopen can be used to make HEAD requests (create a stream context).

Moreover, even if you use a GET, you don't have to read the entire stream to establish that the link is valid.

And my point about timeouts still stands:

I want to be able control:
1. The TCP connect timeout
2. The timeout waiting for the beginning of the response
3. The timeout waiting for the end of the headers

independently.

There is no way of controlling even two of them independently with fopen. stream_set_timeout only works on an already-established stream, and when you call fopen(), the stream has not yet been created.

The problem I have is that fopen() seems to* wait indefinitely for the end of the headers after a tcp connection is established (it will timeout during connection). This means that some web servers (which are broken in some way) will hang my app.

Mark

I haven't verified this definitively in tests, it's just an observation. I could be wrong.

bokehman

MarkR wrote:
fopen can be used to make HEAD requests (create a stream context).

Moreover, even if you use a GET, you don't have to read the entire stream to establish that the link is valid.

And my point about timeouts still stands:

I want to be able control:
1. The TCP connect timeout
2. The timeout waiting for the beginning of the response
3. The timeout waiting for the end of the headers

independently.

There is no way of controlling even two of them independently with fopen. stream_set_timeout only works on an already-established stream, and when you call fopen(), the stream has not yet been created.

The problem I have is that fopen() seems to* wait indefinitely for the end of the headers after a tcp connection is established (it will timeout during connection). This means that some web servers (which are broken in some way) will hang my app.

Mark

I haven't verified this definitively in tests, it's just an observation. I could be wrong.

You really don't know what you are talking about. When fopen is used with an http wrapper the whole file is downloaded to local before the script continues with any other operation. Any subsequent operation (feof(), fgets(), etc.) is performed on the local instance and not directly on the stream, since the stream has already been downloaded and closed.

By the way stream_set_timeout() is used in conjunction with fsockopen() and not fopen(). With fsockopen() timeout is one of the arguments so can be set before trying to establish a connection.

dougmcc1

It looks like there was a lot of back and forth in this thread on the best approach and I tried variations of all of them using the PHP Manual and such but can't get anything to work, let alone determine the most effective solution.

We do go through a proxy which complicates things, I should've mentioned that at the beginning. Plus, it's hard to follow the variables and what's going on in the code. Can anyone provide the code for getting the status code of "http://www.domain.com"? If not, please explain in more detail how to write it as I'm finding it very hard to follow the examples and PHP functions.

Thanks so much for the feedback so far.

dougmcc1

How can I connect through a proxy using the coding examples provided in this thread?

dougmcc1

Nevermind, I was able to get the answer from another forum.