Parsing HTTP

mrtgrady

I'm trying to parse a file request to a Web page using fopen to grab the http feed.

Here's the code I'm using for testing:

<?
$file="http://www.phpbuilder.com/";
$file_array=file($file);

if ( is_readable($file)):
print "readable ";
$fr=fopen($file,"r");
$contents=fread($fr,filesize($file));
fclose($fr);
else :
if (file_exists ($file)):
print $file;
print " ";
print "Can see file over HTTP but not get file!";
else:
print $file;
print " ";
print "Can't even see file.";
endif;
endif;

$findStart = "<TITLE";
$findEnd = "/TITLE>";
$startScrape = strpos($contents, $findStart);
$endScrape = strpos($contents, $findEnd);
print $startScrape;
print " ";
print $endScrape;
print " ";
$lengthString = $endScrape - $startScrape;
print "$lengthString";
print " ";
$ourString = substr($contents, $startScrape, $lengthString);

print "$ourString";

phpbuidler's URL is there for an example and eventually it will be replaced by a URL I am entitled to access.

OK, the way I see it this should work as fopen can take input from URLs, but it doesn't. Can anyone explain why this code does not work and how I could approach this any other way in PHP?

Many thanks

Mark G

MrAlaska

I am too wired to figure check out your code, but here is a functional snippet that will parse the title out of Yahoo's webpage.

<?php
$file = fopen ("http://www.yahoo.com/", "r");
while (!feof ($file)) {
    $line = fgets ($file, 1024);
    if (eregi ("<title>(.*)</title>", $line, $out)) {
        $title = $out[1];
        break;
    }
}
fclose($file);
echo $title;
?>

Works on my system, anyway. I grabbed it pretty much as is off the PHP site. (I think)

Weedpacket

What is $file_array being used for?
From the manual entry for is_readable():

This function will not work on remote files; the file to be examined must be accessible via the server's filesystem.

mrtgrady

OK, two things -

I never said all the code was functional. $file_array clearly does nothing and I'm aware of the is_readable() function only accessing local files as this script was originally used for parsing local files. That's not the bit that is giving me the problem, it's why I'm getting nothing into the $file variable when I use fopen which DOES access files remotely. I thought this would be obviously what I was after, but it seems not. My apologise for the confusion.
RTFM replies are not helpful. I have 3 PHP manuals and they all say the $file variable should hold the output of the HTTP request. Please don't bother replying to my request if you're going to criticise and tell me to read the manual. The point of these forums is that I HAVE read the manual and I can't see how to solve the problem.

Now, can someone please help me solve this issue?

Thanks
Mark G

Weedpacket

Originally posted by mrtgrady
OK, two things -

I never said all the code was functional. $file_array clearly does nothing and I'm aware of the is_readable() function only accessing local files as this script was originally used for parsing local files. That's not the bit that is giving me the problem, it's why I'm getting nothing into the $file variable when I use fopen which DOES access files remotely. I thought this would be obviously what I was after, but it seems not. My apologise for the confusion.
[/b]

Thank you for clearing that up. Since you never explained that I was supposed to ignore the is_readable() statement and my psychic abilities are on the fritz, I presumed that the is_readable() statement and the if{} block it controls (which contains the fopen() statement) were part of the script you were having trouble with. Of course, taking them out means that there are no fopen() statements left.

2. RTFM replies are not helpful. I have 3 PHP manuals and they all say the $file variable should hold the output of the HTTP request. Please don't bother replying to my request if you're going to criticise and tell me to read the manual. The point of these forums is that I HAVE read the manual and I can't see how to solve the problem.

Then you'll be surprised to learn that I've met people on this forum who weren't aware that a manual exists, and have been getting all their code out of third-party books, friends, and spoon-fed to them on these forums.

Hence the sig:

mrtgrady

I'm not surprised to find out that people freeload, but that's not what I'm discussing or trying to do. I'm looking for an explanation as to why fopen of an HTTP source is not being fed into the variable $file.

Can I please have an explanation I can understand and avoid a flame war?

Mark G

MrAlaska

Originally posted by mrtgrady
... I'm aware of the is_readable() function only accessing local files...

Originally posted by Weedpacket
...the is_readable() statement and the if{} block it controls (which contains the fopen() statement) ... means that there are no fopen() statements left...

Makes sense so far...

Originally posted by mrtgrady
Now, can someone please help me solve this issue?

That didn't work? Maybe you should post the code you are actually trying.

mrtgrady

I've just created this script just to test whether anything is being read from the URL:

<?
$file="http://www.phpbuilder.com/";
$fr=fopen($file,"r");
$contents=fread($fr,filesize($file));
fclose($fr);

print "$fr";
print " ";
print filesize($file);
print " ";
print "$file";
print " ";
print "$contents";

and the output I get is:

Resource id #1

http://www.phpbuilder.com/

This means either I misunderstand what can be done with fopen and fread or something else is going on. Any ideas?

Thanks
Mark G

MrAlaska

Are you suppressing errors? This is what I got with that code:

Warning: stat failed for http://www.phpbuilder.com/ (errno=2 - No such file or directory) in c:\program files\easyphp\www\temp.php on line 18
Resource id #2

(line 18 contains '$contents=fread($fr,filesize($file));'

Warning: stat failed for http://www.phpbuilder.com/ (errno=2 - No such file or directory) in c:\program files\easyphp\www\temp.php on line 23

(line 23 contains 'print filesize($file);')

I am not used to the approach you are taking, but it appears filesize is not supported on remote files. This is from the manual:

"This function will not work on remote files; the file to be examined must be accessible via the server's filesystem"
http://www.php.net/manual/en/function.filesize.php

Entering the filesize explicitly as an integer worked on my system.

mrtgrady

After a few hours reading and searchiing I've got to this code:

<?
$fcontents = file ('http://www.php.net/');
while (list ($line_num, $line) = each ($fcontents)) {
// echo "Line $line_num:; ", htmlspecialchars ($line), " \n";
}

$fcontents = implode ('', file ('http://www.php.net/'));
// echo " ", htmlspecialchars ($fcontents), " ";

$contents= $fcontents;

$findStart = "<title>";
$findEnd = "</title>";
$startScrape = strpos($contents, $findStart);
$endScrape = strpos($contents, $findEnd);
print $startScrape;
print " ";
print $endScrape;
print " ";
$lengthString = $endScrape - $startScrape;
print "$lengthString";
print " ";
$ourString = substr($fcontents, $startScrape, $lengthString);
print "=> $ourString";
?>

FYI trying to get the content between the title tags is arbitrary as later I will be grabbing data from between two custom tags to make the pages I'm trying to buiild. www.php.net is also only used for testing purposes.

OK, it seems to work, but only to a point. Whilst it detects the poistions of the tags that delineate the beginning and end of the block it still doesn't seem able to put the content I'm after into a string that can be printed.

Any ideas on this one?

Thanks
Mark G

Weedpacket

For something like this (pulling out the content between <title> tags) I often use the preg_match() function:

if(preg_match('/<title[^>]*>((?:(?!</\s*title\s*>).)*)</\s*title\s*>/i', $contents,$match);

With the content being in $match[1].

Chances are the whitespace padding sequences (\s) could be dropped, and the [^>] sequence is only there because of habit (skipping attributes), so you could probably get away with

if(preg_match('/<title>((?:(?!</title>).)*)</title>/i', $contents,$match);

But that's how I might do it. In your case, you don't see the contents of $ourString because you failed to take the length of the <title> tag into account (you should be starting from $startScrape+strlen("<title>"), in other words). Not doing that means $ourString starts with "<title>" - and, well, it gets rendered as a title.

mrtgrady

Thanks for explaining why I don't see anything - I should have really figured that out, but then again I've been staring at this screen for hours.

I did say that the title tags were being used arbitrarily - it is not my intention to only grab page titles, but content as well or instead of titles in most cases.

I have written the following code, which now seems to do what I need:

$fcontents = implode ('', file ('http://sourceforge.net/projects/gpu/'));
// print htmlspecialchars ($fcontents);

$findStart = "";
$findEnd = "";

$startScrape = strpos($fcontents, $findStart);

$endScrape = strpos($fcontents, $findEnd);

$lengthString = $endScrape - $startScrape;

$ourString = substr($fcontents, $startScrape, $lengthString);
$newString = eregi_replace("<a href=", "\0 mypage.php?", $ourString);

print $newString;
?>

With this I can target specific custom tags as my start and end points and then rewrites links to pass the existing links into mypage.php as querystrings.

I've not written mypage.php yet but this script does solve the problem I had in the first place.

Thanks for the help and suggestions.

Mark G

DigitalExpl0it

http://www.daemonprojects.com/fun/getsource.php