Collecting Mutiple Files Behind Basic IIS Athentication Script Problem

[ccsnet]

Hi, this is my first post on here so go easy 😉

I got the following script from the main php site but been a noob I'm not doing too well. Ideally I want to log in to the site which sits behind basic IIS athentication ( user ID & PW ) and either grab all the files I need in one go.... around 20 and copy to a specific dir on my own site and grab each one and pull the data in to SQL. I started by just trying to get access and copy the files in the first place. Eventually I want to get all the data in to a MySQL DB which I will be using a C# application with.

So far I have not been able to even get one file transfered over and now I am gettin server 500 problems. Because of this and a lack of debug skills ( and tbh noobness ) I'm not sure where its even going wrong.

Can any one help me ?

<?php

/* Simple PHP Script to login on a Basic Authentication page. */
/* Access Configuration */

define ('x401_host', 'EDITED');
define ('x401_port', '80');
define ('x401_user', 'EDITED');
define ('x401_pass', 'EDITED);

/* Function */
function get401Page($file) 
{
   $out  = "GET $file HTTP/1.1\r\n";
   $out .= "Host: ".x401_host."t\r\n";
   $out .= "Connection: Close\r\n";
   $out .= "Authorization: Basic ".base64_encode(x401_user.":".x401_pass)."\r\n";
   $out .= "\r\n";

   if (!$conex = @fsockopen(x401_host, x401_port, $errno, $errstr, 10)) return 0;
   fwrite($conex, $out);
   $data = '';
   while (!feof($conex));
   {
       $data .= fgets($conex, 512);
   }
   fclose($conex);
   return $data;
}

/* Code */
if ($source = get401Page('/absolute/path/file.php?get=value'))
	{ 
	echo $source;
	} 
	else;
	{
  	echo "I can't connect!";
	}

?>

I have also tried to use curl but again this is going wrong

<?php
// http://eugeneciurana.com/~zhenya01/books/webprog/pcook/ch17_07.htm
// http://www.sda-asia.com/php_information/psecom,id,4,articles,550.html

$local = "/England/UnplannedEvent/metadata.xml";
$remote = "http://EDITED/England/UnplannedEvent/metadata.xml";
$id = "EDITED"; 
$password = "EDITED"; 
$loginURL = "EDITED.com";
$fh = fopen($local, 'w') or die($php_errormsg);
$agent = "Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.4) Gecko/20030624 Netscape/7.1 (ax)"; 
$POSTFIELDS = 'UserName='.$id.'&hiddenVar1=hiddenVar1Value&UserPass='.$password; 

$ch = curl_init(); 
curl_setopt($ch, CURLAUTH_NTLM);
curl_setopt($ch, CURLOPT_USERPWD,"$id:$password");
curl_setopt($ch, curlOPT_URL,$loginURL);
curl_setopt($ch, curlOPT_SSL_VERIFYHOST,2); 
curl_setopt($ch, curlOPT_USERAGENT, $agent); 
curl_setopt($ch, curlOPT_RETURNTRANSFER,1); 
curl_setopt($ch, curlOPT_FOLLOWLOCATION,1); 
curl_setopt($ch, curlOPT_SSL_VERIFYPEER,FALSE); // this line makes it work under https 

$result=curl_exec ($ch);

// SET FILE TO DOWNLOAD
// $local is the location to store file on local machine
curl_setopt($ch, CURLOPT_URL, '$remote' CURLOPT_FILE, $fh);

// EXECUTE REQUEST (FILE DOWNLOAD)
$content = curl_exec ($ch);

curl_close ($ch); 

echo("Results: <br>".$result);

?> 

Many Thanks In Advance

Terran

[sneakyimp]

what's wrong with the curl approach? are you getting errors?

[ccsnet]

sneakyimp;10894427 wrote:

what's wrong with the curl approach? are you getting errors?

Yep - the line

curl_setopt($ch, CURLOPT_URL, '$remote' CURLOPT_FILE, $fh);

Is producing a Parse error: syntax error, unexpected T_STRING error...... I've tried a number of approaches but my head is filled with c#, php, sql, html and xml and I just cant see the wood for the trees at the moment nore can I actually see which bit its not happy with....

Curl is probably the best way and I know I can get it in to SQL after that its just getting past this point of getting access.

One thing I have learnt reading the data providers document is that they support HTML 1.1 GET and POST to login..... from what I have read curl is the right thing to use - may be I am wrong.

The URL is h**p://datex2.tistrafficinfo.com/

Any thoughts ?

Terran

[sneakyimp]

There seem to be several problems with that line. [man]curl_setopt[/man] only takes 3 parameters. Looks to me like you are maybe missing a concatenation operator or something in that line. Also, '$remote' results in exactly that string -- the word '$remote' with dollar sign and all. the variable $remote will not be interpreted as a var inside single quotes. You have to get double quotes to get that behavior.

[ccsnet]

sneakyimp;10894449 wrote:

There seem to be several problems with that line. [man]curl_setopt[/man] only takes 3 parameters. Looks to me like you are maybe missing a concatenation operator or something in that line. Also, '$remote' results in exactly that string -- the word '$remote' with dollar sign and all. the variable $remote will not be interpreted as a var inside single quotes. You have to get double quotes to get that behavior.

Tweeked it to look like this....

curl_setopt($ch, CURLOPT_URL, "'$remote'", CURLOPT_FILE, $fh);

The error went away but it does not seem to work.... I need to see if I have access to my logs from my ISP

Terran

[ccsnet]

Yep that looks like that part is fixed but I seem to be getting this....

"30 Nov 2008 15:08:39","/homepages/21/d201264549/htdocs/trafficsite/datexii/datexiidle.php:
45","(Warning) fopen(http://www.website.com/England/UnplannedEvent/metadata.xml) [<a href='function.fopen'>function.fopen</a>]: failed to open stream: HTTP wrapper does not support writeable connections."

So I am guessing I need to look at editing this line....

$fh = fopen($local, 'w') or die($php_errormsg);

Terran

[sneakyimp]

You need to read my post again. I don't think this line is valid:

curl_setopt($ch, CURLOPT_URL, "'$remote'", CURLOPT_FILE, $fh);

You are providing 5 parameters and curl_setopt only takes 3 as far as I know.

Also, "'$remote'" is just plain ridiculous. Why on earth are you quoting a string with both single quotes and double quotes?

[ccsnet]

sneakyimp;10894483 wrote:

You need to read my post again. I don't think this line is valid:

curl_setopt($ch, CURLOPT_URL, "'$remote'", CURLOPT_FILE, $fh);

You are providing 5 parameters and curl_setopt only takes 3 as far as I know.

Also, "'$remote'" is just plain ridiculous. Why on earth are you quoting a string with both single quotes and double quotes?

Ahhhhhhh now I follow.... I had //ed out some of the above lines....

Re the "'$remote'", from the example I have seen it had 'website' as the example... if $remote = "website.com" at the moment I thought I needed to ensure the 's where in....

I also had the thought today that I'm not sure if the file I am trying to write too has write permisions althoough I have assumed it has.

Let me go have another play... once again thanks for your help.

Terran

[sneakyimp]

You probably don't need any quotes around $remote at all if this is what you are trying to do:

curl_setopt($ch, CURLOPT_URL, $remote);

[ccsnet]

$fh = fopen($local, 'w') or die($php_errormsg);

// SET FILE TO DOWNLOAD
$ch = curl_init();
curl_setopt($ch, CURLAUTH_NTLM); 
curl_setopt($ch, CURLOPT_USERPWD,"$id:$password");
curl_setopt($ch, CURLOPT_URL, "'$remote'");
curl_setopt($ch, CURLOPT_FILE, $fh);

// EXECUTE REQUEST (FILE DOWNLOAD)
$content = curl_exec ($ch);

curl_close ($ch);

curl_setopt($ch, CURLAUTH_NTLM); 2 Param
curl_setopt($ch, CURLOPT_USERPWD,"$id:$password"); 2 Param
curl_setopt($ch, CURLOPT_URL, "'$remote'"); 2 Param
curl_setopt($ch, CURLOPT_FILE, $fh); 3 Param

Sneaky - I spent some time checking what you said and looking at some other examples to make sure I was not getting this too wrong now.

I undertsand you where saying that I had too many parms eg 5 on the line you highligted which when looking I assumed means on each setopt line ( based on other examples I have seen ) before running the curl exec.

To that end I cut my code down to the above.... I still got the same error.

Strangley.... I remove all the lines with setop in them and I still get the same error......

Re the "' around remote - I know what your saying how ever again I have seen a few more examples with 's around the path and if I remote them I get a T String error.

Another problem I found when using the first version of my script was that it wanted the physical local location of where to save eg /homepages/21/d201264549/htdocs/trafficsite/datexii/England/UnplannedEvent/metadata.xml for me.

Terran

[sneakyimp]

what is the error?

[ccsnet]

sneakyimp;10894558 wrote:

what is the error?

01/12/2008 16:38 "/homepages/21/d201264549/htdocs/trafficsite/datexii/datexiidle3.php:
56" (Warning) Wrong parameter count for curl_setopt()

Terran

[sneakyimp]

Once again, curl_setopt requires 3 parameters. Obviously, on line 56, you have supplied either more or less than 3.

[ccsnet]

<?php

$local = "/homepages/21/*******/htdocs/trafficsite/datexii/England/UnplannedEvent/metadata.xml";
$remote = "http************/England/UnplannedEvent/metadata.xml";
$id = "*****"; 
$password = "******"; 
$loginURL = "*******";
$fh = fopen($local, 'w') or die($php_errormsg);

echo "Ready To Start ----> ";

// SET FILE TO DOWNLOAD
$ch = curl_init();
curl_setopt($ch, CURLAUTH_NTLM);
curl_setopt($ch, CURLOPT_USERPWD,"$id:$password");
curl_setopt($ch, CURLOPT_URL, $remote);
curl_setopt($ch, curlOPT_RETURNTRANSFER,TRUE);
// curl_setopt($ch, CURLOPT_FILE, $fh);

echo "Curl opts Done ----> ";

echo "$ch ----> ";
echo "$fh ----> ";

// EXECUTE REQUEST (FILE DOWNLOAD)
$content = curl_exec ($ch);

echo "Curl Exec Done: ----> ";
echo "$content";

curl_close ($ch);
?>

Right - big thanks the above seems to be working now as I am getting a 1 as an out put which having read up seems to indicate no data is been collected. I'm guessing this is due to the authentication not working some how so I'll have a look at what is needed tommrrow as its late now here.

I know it uses 1.1 html standards and I have seen you can force curl to use diffrent methods including this so I'll have a look tommorrow unless you have any off the cuff thoughts to send me in the right direction at all ?

Once again thanks for your help.....

Terran

[sneakyimp]

You still have some problems. Only 2 params here, you need one more:

curl_setopt($ch, CURLAUTH_NTLM);

typo here...php may be case-sensitive, depending on the version:

curl_setopt($ch, curlOPT_RETURNTRANSFER,TRUE);

that could be why you are getting 1 instead of the contents

If you hope to get anywhere as a programmer, you are going to have to pay more attention.