data mining

diodave

This code returns inconsistent results. It's intended to go to the url indicated, gather the source code, and display it. You can see it in action at http://www.dio.org/catholictimes/cnsstory.php

If the story comes up incomplete, hit refresh a couple times. Eventually it displays the whole story. My thoughts are that it should either work or it shouldn't, not half and half.

function cnsstory(){
	$date = date('Ymd');
	$suffix = ($date-1).'.htm';
	$theurl= 'http://www.catholicnews.com/data/stories/'.$suffix;

while(!($fp = fopen($theurl, 'r')))
{
	$date -= 1;
	$suffix = ($date-1).'.htm';
$theurl= 'http://www.catholicnews.com/data/stories/'.$suffix;
}

$fp = fopen($theurl, 'r');
$contents = fread($fp, 1000000);
echo $contents;
fclose($fp);
}

thanks

Weedpacket

Originally posted by diodave
thanks

You're welcome.

One problem I have with your code is this mindless "$contents = fread($fp, 1000000);" bit. First, it means reserving almost a megabyte of memory to store the retrieved file (no matter how small the file might be), second it assumes that the entire file is going to be delivered in one piece (which, over http, isn't necessarily the case).

It basically ignores the warnings and examples on the [man]fread[/man] manual page:

Warning
When reading from network streams or pipes, such as those returned when reading remote files or from popen() and proc_open(), reading will stop after a packet is available. This means that you should collect the data together in chunks as shown in the example below.
$handle = fopen("http://www.example.com/", "rb");
$contents = "";
do {
   $data = fread($handle, 8192);
   if (strlen($data) == 0) {
       break;
   }
   $contents .= $data;
} while (true);
fclose($handle);

diodave

thanks