how to download dbase-driven web pages?

Anon

Does anyone know of a way to download web pages that are served up by databases?

i.e. what I need is something similar to:

wget http://www.somesite.com/index.php?var1=value1&var2=value2

I need this functionality because I have a script that will then scan the downloaded page for the relevent bits and cross-reference those bits with the corresponding entries in my database, reporting if there is a difference. It's pretty much essential to me, or else my database is going to become unsynchronised rather quickly!

I've searched all over the place, but I think I may be a bit out of luck...

Anon

doncha just hate it when that happens -
I've been searching for this answer for many hours, and been perusing it in my lil head for many a day...
and just when I gave in and posted the question...Bamm, I found the answer!

I hadn't bothered to consult the curl manual...

curl -O http://www.somesite.com/index.php -d var1=value1 -d var2=value2

will download the corresponding page.

If anyone has a more useful way of scanning remote web pages for given parameters and returning what it finds, I'd much appreciate it

(my database is basically of comparison shopping site listing all the prices of a load of different items from different companies - if one company changes their price, my dbase is unsynced, thus I am trying to make an app that will check my dbase with the corresponding company pages and check for price discrepencies. With curl, I may just be able to do it)

Anon

I don't get it; what does curl do in this case that wget can't also do?

Or, for that matter, PHP's own ability to open and retrieve remote files?

Anon

so for me:
wget http://www.phpbuilder.com/forum/read.php3?num=1&id=132175&thread=132171

on the command line flat out doesn't work.

in a php script:
fopen("http://www.phpbuilder.com/forum/read.php3?num=1&id=132175&thread=132171", "w+");

sits around, presumably attempting to get the file, but fails to write anything,

whereas

curl -O http://www.phpbuilder.com/forum/read.php3 -d num=1 -d id=132175 -d thread=132171

successfully downloads a file named read.php3 containing the html for your reply.

If I'm missing a beat on fopen, then I'd be much happier to use that!

Anon

In your first example, wget won't work from the command line because the ampersand is a special character in Unix shells. It means "fork another process and do everything before me in the background." That breaks the URL, as you can imagine. You have two choices: Escape the ampersands with backslashes, or quote the argument.

On your second example, fopen() has two problems. One is that you're opening a remote file in write/append mode, and not in read mode, which is an error. The other is that you're not doing anything with the returned file handle. The fopen() call is designed to be used in concert with fread() and fclose(). If you just want to grab the file, don't use fopen(), just use file():

<?
$foo =
implode('',file("http://www.phpbuilder.com/forum/read.php3?num=1&id=132175&thre
echo $foo; // example
?>

That saves a step, and avoids the problem of "oops, I don't have write permissions in this directory."

tasistro

curl has:

support for ssl
goes through proxies

Andy, you're on the right track.

Saludos
Gerardo

Anon

cheers folks -

never said I was any good at this stuff
;-)

I've just been playing getting libcurl built in, so now I have the best of both worlds. Still in the linear range of that ole learning curve, with no end in site!