help with CURL (doing two things at once)

[marcnyc]

Hello, I need some help with CURL.
What I would like to achieve is to enter a restricted area of my site through POST authentication and then go to a second URL in order to update something in my database everytime this script I am writing is called.
Basically there is an administrator/index.php page that has the log in form and then there is a administrator/index2.php page that has the options. By clicking on one of those options the database is updated. What I have been trying to do for hours now is to log in to index.php (which normally would direct me to index2.php (that's what the index.php page is coded to do) and after that to go to the URL of index2.php that performs the update.

Here is what I have so far:

        // Administrator area LOGIN
        $url2 = 'http://www.chaindlk.com/content/administrator/index.php';
        $urlstring2 = 'usrname=MY-USERNAME-HERE&pass=MY-PASSWORD-HERE&submit=Login';
        $ch2 = curl_init();
        curl_setopt($ch2, CURLOPT_FRESH_CONNECT, 1);
        curl_setopt($ch2, CURLOPT_VERBOSE, 1);
        curl_setopt($ch2, CURLOPT_URL,$url2);
        curl_setopt($ch2, CURLOPT_CONNECTTIMEOUT, 9);
        curl_setopt($ch2, CURLOPT_HEADER, 0);
        curl_setopt($ch2, CURLOPT_POST, 1);
        curl_setopt($ch2, CURLOPT_POSTFIELDS, $urlstring2);
        curl_setopt($ch2, CURLOPT_FOLLOWLOCATION, 0);
        curl_setopt($ch2, CURLOPT_MAXREDIRS, 0);
        curl_setopt($ch2, CURLOPT_REFERER, 'http://www.chaindlk.com/content/administrator/index.php');
        curl_setopt($ch2, CURLOPT_RETURNTRANSFER, 1);
        $data2 = curl_exec($ch2);
        curl_error($ch2);
        curl_close($ch2);

if ( substr_count(strtolower($data2), strtolower("<script>document.location.href='index2.php';</script>")) >= 1 ) {
//echo "<script>alert('so far so good');</script>";
        $url3 = 'http://www.chaindlk.com/content/administrator/index2.php?option=com_comprofiler&task=syncUsers';
        $urlstring3 = 'option=com_comprofiler&task=syncUsers';
        $ch3 = curl_init();
        curl_setopt($ch3, CURLOPT_VERBOSE, 1);
        curl_setopt($ch3, CURLOPT_URL,$url3);
        curl_setopt($ch3, CURLOPT_HEADER, 0);
        //curl_setopt($ch3, CURLOPT_HTTPGET, 1);
        curl_setopt($ch3, CURLOPT_POST, 1);
        curl_setopt($ch3, CURLOPT_POSTFIELDS, $urlstring3);
        curl_setopt($ch3, CURLOPT_FOLLOWLOCATION, 0);
        curl_setopt($ch3, CURLOPT_REFERER, 'http://www.chaindlk.com/content/administrator/index2.php?option=com_comprofiler&task=tools');
        curl_setopt($ch3, CURLOPT_RETURNTRANSFER, 1);
        $data3 = curl_exec($ch3);
        curl_error($ch3);
        curl_close($ch3);
}

Well if I uncomment the line

echo "<script>alert('so far so good');</script>

for debugging I actually see that message which means I have successfully logged in. Once logged in though I can't seem to go to the second url because the system considers me logged out for some reason...

I have also tried moving the curl_close($ch2); after the if statement in case (I wasn't sure) the curl $ch3 has to be executed within the $ch2 (before the $ch2 is closed), but that didn't work either...

Does anybody have any ideas?
I'd appreciate any input...

Are the sessions screwing with me?
All I'd really like to know (generally speaking) is if I can, within a CURL operation execute a second CURL operation, or, to put it in different words, if once I have accesses a page through CURL I can access a secong page through CURL maintaining the authentication going...

Thanks a lot...

[marcnyc]

Just for debugging purposes (in case you want to look at it), here is the code of administrator/index.php and administrator/index2.php:

index.php

<?php
/**
* @version $Id: index.php,v 1.16 2004/09/02 02:31:28 eddieajau Exp $
* @package Mambo_4.5.1
* @copyright (C) 2000 - 2004 Miro International Pty Ltd
* @license [url]http://www.gnu.org/copyleft/gpl.html[/url] GNU/GPL
* Mambo is Free Software
*/

/** Set flag that this is a parent file */
define( "_VALID_MOS", 1 );

if (!file_exists( "../configuration.php" )) {
	header( "Location: ../installation/index.php" );
	exit();
}

require_once( "../configuration.php" );
require_once( "../includes/mambo.php" );
include_once ( $mosConfig_absolute_path."/language/".$mosConfig_lang.".php" );
$database = new database( $mosConfig_host, $mosConfig_user, $mosConfig_password, $mosConfig_db, $mosConfig_dbprefix );
$database->debug( $mosConfig_debug );
$acl = new gacl_api();

$option=mosGetParam($_REQUEST,'option',null);
// mainframe is an API workhorse, lots of 'core' interaction routines
$mainframe = new mosMainFrame( $database, $option, '..', true );

if (isset( $_POST['submit'] )) {
	/** escape and trim to minimise injection of malicious sql */
	$usrname = $database->getEscaped( trim( mosGetParam( $_POST, 'usrname', '' ) ) );
	$pass = $database->getEscaped( trim( mosGetParam( $_POST, 'pass', '' ) ) );

if (!$pass) {
	echo "<script>alert('Please enter a password'); document.location.href='index.php';</script>\n";
} else {
	$pass = md5( $pass );
}

$database->setQuery( "SELECT COUNT(*)"
. "\nFROM #__users"
. "\nWHERE (usertype='administrator' OR usertype='superadministrator')"
);
$count = intval( $database->loadResult() );
if ($count < 1) {
	echo "<script>alert(\""._LOGIN_NOADMINS."\"); window.history.go(-1); </script>\n";
	exit();
}

$database->setQuery( "SELECT * FROM #__users WHERE username='$usrname' AND block='0'" );
$my = null;
$database->loadObject( $my );

/** find the user group (or groups in the future) */
$grp = $acl->getAroGroup( $my->id );
$my->gid = $grp->group_id;
$my->usertype = $grp->name;

if ($my->id) {
	if (strcmp( $my->password, $pass )
	|| !$acl->acl_check( 'administration', 'login', 'users', $my->usertype )) {
		echo "<script>alert('Incorrect Username, Password, or Access Level.  Please try again'); document.location.href='index.php';</script>\n";
		exit();
	}

	session_name( 'mosadmin' );
	session_start();

	$logintime = time();
	$session_id = md5( "$my->id$my->username$my->usertype$logintime" );
	$database->setQuery( "INSERT INTO #__session"
	. "\nSET time='$logintime', session_id='$session_id', "
	. "userid='$my->id', usertype='$my->usertype', username='$my->username'"
	);
	if (!$database->query()) {
		echo $database->stderr();
	}

	$_SESSION["session_id"] = $session_id;
	$_SESSION["session_user_id"] = $my->id;
	$_SESSION["session_username"] = $my->username;
	$_SESSION["session_usertype"] = $my->usertype;
	$_SESSION["session_gid"] = $my->gid;
	$_SESSION["session_logintime"] = $logintime;
	$_SESSION["session_userstate"] = array();

	session_write_close();
	/** cannot using mosredirect as this stuffs up the cookie in IIS */
	echo "<script>document.location.href='index2.php';</script>\n";
	exit();
} else {
	echo "<script>alert('Incorrect Username and Password, please try again'); document.location.href='index.php';</script>\n";
	exit();
}
} else {
	initGzip();
	$path = $mosConfig_absolute_path . '/administrator/templates/' . $mainframe->getTemplate() . '/login.php';
	require_once( $path );
	doGzip();
}
?>

index2.php

<?php
/**
* @version $Id: index2.php,v 1.25 2004/09/04 10:47:42 rcastley Exp $
* @package Mambo_4.5.1
* @copyright (C) 2000 - 2004 Miro International Pty Ltd
* @license [url]http://www.gnu.org/copyleft/gpl.html[/url] GNU/GPL
* Mambo is Free Software
*/

/** Set flag that this is a parent file */
define( "_VALID_MOS", 1 );

if (!file_exists( "../configuration.php" )) {
	header( "Location: ../installation/index.php" );
	exit();
}

require_once( "../globals.php" );
require_once( "../configuration.php" );
require_once( $mosConfig_absolute_path . "/includes/mambo.php" );
include_once( $mosConfig_absolute_path . "/language/".$mosConfig_lang.".php" );
require_once( $mosConfig_absolute_path . "/administrator/includes/admin.php" );

$database = new database( $mosConfig_host, $mosConfig_user, $mosConfig_password, $mosConfig_db, $mosConfig_dbprefix );
$database->debug( $mosConfig_debug );
$acl = new gacl_api();

$option = strtolower( mosGetParam( $_REQUEST, 'option', '' ) );
if ($option == '') {
	$option = 'com_admin';
}
// must start the session before we create the mainframe object
session_name( 'mosadmin' );
session_start();

// mainframe is an API workhorse, lots of 'core' interaction routines
$mainframe = new mosMainFrame( $database, $option, '..', true );

// initialise some common request directives
$task = mosGetParam( $_REQUEST, 'task', '' );
$act = strtolower( mosGetParam( $_REQUEST, 'act', '' ) );
$section = mosGetParam( $_REQUEST, 'section', '' );
$no_html = strtolower( mosGetParam( $_REQUEST, 'no_html', '' ) );

if ($option == 'logout') {
	require 'logout.php';
	exit();
}

// restore some session variables
$my = new mosUser( $database );
$my->id = mosGetParam( $_SESSION, 'session_user_id', '' );
$my->username = mosGetParam( $_SESSION, 'session_username', '' );
$my->usertype = mosGetParam( $_SESSION, 'session_usertype', '' );
$my->gid = mosGetParam( $_SESSION, 'session_gid', '' );

$session_id = mosGetParam( $_SESSION, 'session_id', '' );
$logintime = mosGetParam( $_SESSION, 'session_logintime', '' );

// check against db record of session
if ($session_id == md5( $my->id.$my->username.$my->usertype.$logintime )) {
	$database->setQuery( "SELECT * FROM #__session"
	. "\nWHERE session_id='$session_id'"
	//. "\n	AND (usertype='administrator' OR usertype='superadministrator')"
	);
	if (!$result = $database->query()) {
		echo $database->stderr();
	}
	if ($database->getNumRows( $result ) <> 1) {
		echo "<script>document.location.href='index.php'</script>\n";
		exit();
	}
} else {
	echo "<script>document.location.href='$mosConfig_live_site/administrator/index.php'</script>\n";
	exit();
}

// update session timestamp
$current_time = time();
$database->setQuery( "UPDATE #__session SET time='$current_time'"
. "\nWHERE session_id='$session_id'"
);
$database->query();

// timeout old sessions
$past = time()-1800;
$database->setQuery( "DELETE FROM #__session WHERE time < '$past'" );
$database->query();

// start the html output
if ($no_html) {
	if ($path = $mainframe->getPath( "admin" )) {
		require $path;
	}
	exit;
}

initGzip();

$path = $mosConfig_absolute_path . "/administrator/templates/" . $mainframe->getTemplate() . "/index.php";
require_once( $path );

doGzip();
?>

[AstroTeg]

Without looking fully at the code or understanding your entire process, I think you need to enable cookies in curl. Its pretty easy to do this:

curl_setopt($curl, CURLOPT_COOKIEFILE, 'cookie.txt');
curl_setopt($curl, CURLOPT_COOKIEJAR, 'cookie.txt');

Check out php.net's curl docs for a lot more info. In short, CURLOPT_COOKIEFILE points CURL to a file containing the cookies. The CURLOPT_COOKIEJAR tells CURL to save any cookies it gets to a file. Make sure the script has read/write permissions on the cookie file (you may have to create a dummy file and then set the rights). By setting up cookies, you allow curl to store temporary session data between page 1 and page 2 (your script goes to page 2 ok, but your page 2 code sees curl isn't logged in any more since it is probably checking the session/cookies).

[marcnyc]

will this work regardless if the script uses sessions or cookies?

[AstroTeg]

Yup. A session is just a cookie that expires immediately when the user closes their browser.

[marcnyc]

You're right it seems to have worked like a charm... I thought sessions are in the database and are not cookie-based...
By the way if somebody runs into the same issue please note that you would have to put the COOKIEJAR option in the first curl operation and the (to store the session/cookie contents) and COOKIEFILE in the second curl operation (to retrive it).
Thanks again for your time and generosity!

[marcnyc]

Unfortunately I have to report that it stopped working today when I made a pre-launch test.
The website's launch is due tomorrow so I have a deadline. Therefore, if the price is right, I will pay anybody who can solve the problem.
I don't know what the problem is, but with the same code it simply doesn't work anymore.

Here is my code (I've removed username and password):

<?
	// MAMBO admin login
	$url2 = 'http://www.chaindlk.com/content/administrator/index.php';
	$urlstring2 = 'usrname=XXXXXX&pass=XXXXXX&submit=Login';
	$ch2 = curl_init();
	curl_setopt($ch2, CURLOPT_URL,$url2);
	curl_setopt($ch2, CURLOPT_HEADER, 0);
	curl_setopt($ch2, CURLOPT_POST, 1);
	curl_setopt($ch2, CURLOPT_POSTFIELDS, $urlstring2);
	curl_setopt($ch2, CURLOPT_FOLLOWLOCATION, 0);
	curl_setopt($ch2, CURLOPT_REFERER, 'http://www.chaindlk.com/content/administrator/index.php');
	curl_setopt($ch2, CURLOPT_RETURNTRANSFER, 1);
	curl_setopt($ch2, CURLOPT_COOKIEJAR, 'registration.txt');
	$data2 = curl_exec($ch2);
// ///////////////////////////////////////////////////////////////////////////// TROUBLESHOOTING
if ( substr_count(strtolower($data2), strtolower("<script>document.location.href='index2.php';</script>")) >= 1 )
	echo '<script>alert( "MOS admin login DONE" );</script>';
// ///////////////////////////////////////////////////////////////////////////// TROUBLESHOOTING

if ( substr_count(strtolower($data2), strtolower("<script>document.location.href='index2.php';</script>")) >= 1 ) {
	// COMMUNITY BUILDER user tables synchronization
	$url3 = 'http://www.chaindlk.com/content/administrator/index2.php?option=com_comprofiler&task=syncUsers';
	$urlstring3 = 'option=com_comprofiler&task=syncUsers';
	$ch3 = curl_init();
	curl_setopt($ch3, CURLOPT_URL,$url3);
	curl_setopt($ch3, CURLOPT_HEADER, 0);
	curl_setopt($ch3, CURLOPT_POST, 1);
	curl_setopt($ch3, CURLOPT_POSTFIELDS, $urlstring3);
	curl_setopt($ch3, CURLOPT_FOLLOWLOCATION, 0);
	curl_setopt($ch3, CURLOPT_REFERER, 'http://www.chaindlk.com/content/administrator/index2.php?option=com_comprofiler&task=tools');
	curl_setopt($ch3, CURLOPT_RETURNTRANSFER, 1);
	curl_setopt($ch3, CURLOPT_COOKIEFILE, 'registration.txt');
	$data3 = curl_exec($ch3);
	curl_close($ch3);
curl_close($ch2);
// ///////////////////////////////////////////////////////////////////////////// TROUBLESHOOTING
if ( substr_count(strtolower($data3), strtolower("now in sync")) >= 1 )
	echo '<script>alert( "sync DONE" );</script>';
// ///////////////////////////////////////////////////////////////////////////// TROUBLESHOOTING
}
exit;
?>

At this point this is urgent to me so if you can help please do, I have no idea what changed or why it would be different now... It simply doesn't work. The problem is the same I had initially: it doesn't stay logged in. I have double-checked this by trouble shooting my script.

Thanks in advance for any help.

PS I never get to the second alert (sync DONE)...
PPS as you might get from the example to make sure I am actually logged in I check to see if the log in page directs me to the login area (which is index2.php), and I looked at the code, it does it with a js document.location, so that is NOT the issue...

[marcnyc]

ANY ideas?

[marcnyc]

I am still tackling with this monster... Can anybody help?
Could there be any server settings that wouldn't let me use Curl's cookie-related options?
I just can't explain why the rest of the Curl actions seem to work...
Thanks

[tsinka]

Hi,

one question:

you wrote "pre-launch test". Do you mean that the script works on your development server and not on the target server or that it doesn't work on your development server where it worked earlier ?

Thomas

[marcnyc]

That's a good question. I can't recall if that time I saw it working it was actually working on the remote server where it is now or on my local machine... I think it was already uploaded but I might be wrong... I do know that I have php5 on my machine and php 4.x on the remote server...

Do you have any idea what might be preventing it from working?

Thanks for your time.

[tsinka]

Did you check if the cookie file exists at all on the remote server and if the cookie file get's updated each time you reload the script ?

Thomas

[tsinka]

One test:

Move the line
curl_close($ch2);
so that it comes just after the
$data2 = curl_exec($ch2);

line. I don't know what curl does if you open a curl session which inits the cookie jar file and then open another curl session that uses that cookie file before you closed the first session ... just a guess ... I can't test it now.

EDIT: I somewhere read that you need to use the full path to the cookie file not only a relative path on some systems/setups but I can't remeber where.

Thomas

[marcnyc]

Ok I have just made some addiitional tests and I am not sure what the problem is because the same code behaves differently if placed in the page where it is supposed to be or if placed in a blank php document...
In a blank document it seems to work and the content of the cookie file registration.txt is different every time (a new session everytime).
Is there a way to delete the contents of that file after I am done, so it will be clean for the next use?
I also seem to have some issues with caching (some values of the form I am sending values to don't seem to be updated when I re-submit a form)...

Iin reply to your second post (that I just saw after posting this reply of mine)...
moving that curl closing statement around was just me doing tests, I have tested both ways...
I have also read this thing about full urls but I thought it is not mandatory because the same code in a blank file without the template seems to work (ie writes to that file), but I'll test it anyway. Thanks for the suggestion

[tsinka]

Use e.g. the unlink function to delete the cookie file after you're done. Did you already think about concurrent requests (if two different users access the script(s) at the same time overwriting the cookie file of each other) ?

Please post the code of the script that doesn't work if you put the curl code in.

Thomas

[tsinka]

I modified your script a little bit. First of all you can tell curl to use the session cookie data only within the startet curl session by providing an empty COKIEFILE option. That enables curl to parse cookies but they will be detroyed by curl_close. You can send more than one request so just use curl_init once like this:

<?PHP
  // MAMBO admin login
  $url2 = 'http://www.chaindlk.com/content/administrator/index.php';
  $urlstring2 = 'usrname=XXXXXX&pass=XXXXXX&submit=Login';
  $ch2 = curl_init();
  curl_setopt($ch2, CURLOPT_URL,$url2);
  curl_setopt($ch2, CURLOPT_HEADER, 0);
  curl_setopt($ch2, CURLOPT_POST, 1);
  curl_setopt($ch2, CURLOPT_POSTFIELDS, $urlstring2);
  curl_setopt($ch2, CURLOPT_FOLLOWLOCATION, 0);
  curl_setopt($ch2, CURLOPT_REFERER, 'http://www.chaindlk.com/content/administrator/index.php');
  curl_setopt($ch2, CURLOPT_RETURNTRANSFER, 1);
  curl_setopt($ch2, CURLOPT_COOKIEFILE, "");
  $data2 = curl_exec($ch2);

  if ( substr_count(strtolower($data2), strtolower("<script>document.location.href='index2.php';</script>")) >= 1 ) {
    echo '<script>alert( "MOS admin login DONE" );</script>';

$url3 = 'http://www.chaindlk.com/content/administrator/index2.php?option=com_comprofiler&task=syncUsers';
$urlstring3 = 'option=com_comprofiler&task=syncUsers';
curl_setopt($ch2, CURLOPT_URL,$url3);
curl_setopt($ch2, CURLOPT_HEADER, 0);
curl_setopt($ch2, CURLOPT_POST, 1);
curl_setopt($ch2, CURLOPT_POSTFIELDS, $urlstring3);
curl_setopt($ch2, CURLOPT_FOLLOWLOCATION, 0);
curl_setopt($ch2, CURLOPT_REFERER, 'http://www.chaindlk.com/content/administrator/index2.php?option=com_comprofiler&task=tools');
curl_setopt($ch2, CURLOPT_RETURNTRANSFER, 1);
$data3 = curl_exec($ch2);
if ( substr_count(strtolower($data3), strtolower("now in sync")) >= 1 ) {
  echo '<script>alert( "sync DONE" );</script>';
}
  }

  if ($ch2) {
    curl_close($ch2);
  }
?> 

I tested that with two little script and it worked perfectly.

Thing to have in mind:
- if the scripts check the referer-> make sure that the referer submitted is accepted by the scripts.
- echo $data2 and $data3 to see how the data looks like.
- does the second script expect index2.php?option=com_comprofiler&task=syncUsers as post or as get parameters ?

I tested that you can do that with just one curl session. But I couldn't test if that works with exactly the two scripts you request in the curl session(s). You might need to tweak it a little bit.

The advantage of the code above is that you don't need to store any cookie data in files since curl stores them temporarily in memory until the curl session is closed.

Thomas

[marcnyc]

it seems to have worked...
I wanted to come back and post this to let you know and to thank you and to avoid that your efforts would go un-noticed, un-thanked etc...
Unfortunately I got cought in some heavy work so I haven't been able to further develop my script yet, but it seems that your code fixed it.
As soon as I have some more time again I will get down and dirty and keep working on this code and on this curl section of it.
THANK YOU very much for your help. I do really appreciate your time and efforts.