Sreen scraping with CURL login and DOM

andywilliams99

I need to check a suppliers website for stock availability and pricing but this requires a login.

To get around this I am attempting to implement CURL to log me in and redirect me to the product page in order to then scrape the availability and price etc.

The issue I have is that the login appears not to work. I get the error:

Warning: curl_setopt_array() [function.curl-setopt-array]: Unable to access cookies_path.txt in (path to folder specified)

I am using a class obtained from another forum which has good feedback and therefore should work.

Full code is as follows:

class Secure_Crawler {

private $loginUrl = 'this is the page the login form posts to';

private $options = array( );

private $connected = false;

function __construct () {

    $cookies = 'cookies_path.txt';

    $this->options = array(
        CURLOPT_USERAGENT => 'Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0)',
        CURLOPT_HEADER => false, // Add response headers to return, for testing
        CURLOPT_RETURNTRANSFER => true, // Add html to return 
        CURLOPT_COOKIEJAR  => $cookies,
        CURLOPT_COOKIEFILE => $cookies,
     );

// Reset cookies
    @ unlink($cookies);
}

function login ($username, $password) {

    $ch = curl_init();

$options = $this->options;
    $options[CURLOPT_URL] = $this->loginUrl;
    $options[CURLOPT_POST] = true;
// Login form fields
    $options[CURLOPT_POSTFIELDS] = $this->getPostFields(
	array(
		'login.login' =>  $username, 
		'login.password' => $password,
		'smalllogin.path' => '/main/en/page.html'
	));
    $options[CURLOPT_FOLLOWLOCATION] = false;
    curl_setopt_array($ch, $options);
    curl_exec($ch);

    //Close curl session
    curl_close($ch);

    $this->connected = true;
}

function get ($url) {

    if (!$this->connected)
        throw new Excetion("Not connected");

    $ch = curl_init();

    //Get
    $options = $this->options;
    $options[CURLOPT_URL] = $url;
    curl_setopt_array($ch, $options);
    $results = curl_exec($ch);

    //Close curl session
    curl_close($ch);

return $results;
}

private function getPostFields ($data) {

    $return = array();

    foreach ($data as $key => $field) {
        $return[] = $key . '=' . urldecode($field);
    }

    return implode('&', $return);
}
}



$crawler = new Secure_Crawler();

// Login to website
$crawler->login('my username', 'my password');

// Get Content
$content = $crawler->get('this is the product page');

// modifications...

echo $content;

I had assumed there may be an issue writing the cookies_path.txt file to the directory however I think I have correct permissions and it appears that the file is being created but then deleted. In any case the script appears to be unable to read the file to perform log on.

johanafm

If there are ever more than one instance running, you should name your cookie file uniquely for each instance.
Since you unlink the cookiejar file, you do not need it (assuming you reuse one single curl instance)?
But if you really do need the cookiejar functionality, then I'd use a specific directory for this rather than save these files in the current working directory with my script files. Also, I'd make sure I know the permissions are ok rather than just think they are. But then you have to remove the unlink or you still do not use the cookiejar.

andywilliams99;10963601 wrote:
In any case the script appears to be unable to read the file to perform log on.

There is no file when you try to login. Assuming you do have permissions to create the cookiejar file, there will be one when you

$content = $crawler->get('this is the product page');

But since your ::getPostFields function does exactly this

    private function getPostFields ($data) {

    $return = array();

}

I'd say you never manage to login, so wether you have a cookiejar or not doesn't matter.

Create and reuse one single instance curl instance. This way you would not need to worry about cookies at all. All you need to do is private $ch, and in construct $this->ch = curl_init(); And of course remove the curl_close etc.

Have you checked the response to your login request? Have you checked what's actually sent in the request, for example CURLINFO_HEADER_OUT?

andywilliams99

Unfortunately as far as CURL goes I am a newbie. I only came across it while trying to solve this particular issue.

There should never be more than one instance running at a time so my guess is that the cookie file naming should not be an issue.

In terms of permissions the directory which contains this file (currently the same as that in which the script is running) is set to (CHMOD) 7777

I had previously removed the unlink but with no change.

Unfortunately I am lost with the rest your advice - can you simplify it for me (sorry).

johanafm

This is my general idea for handling curl. Combine with the rest of your code... Stuff I feel you really should remove is commented away with #, while code that should be there has been cut away

class Secure_Crawler {
	private $ch;

function __construct () {
# can be taken out from $this->options = ...
#            CURLOPT_COOKIEJAR  => $cookies,
         );

#        @ unlink($cookies);

$this->ch = curl_init();
}

function login ($username, $password) {

#        $ch = curl_init();

# all calls to curl now uses $this->ch instead of $ch

$result = curl_exec($this->ch);
if ($result === false) {
    # various request errors
    $this->connected = false;
}
else {
    # no certainty that you are logged in unless you actually parse
    # the response (found in $result)
    if (login_is_ok($result)) {
        $this->connected = true;
        }
    else {
        $this->connected = false;
        }
    }

# no need to close your curl handle
#        curl_close($ch);

# Also, this line is somewhat fishy, since you never checked your curl_exec return
# value, nor what was actually the request result. Deal with this in the if/else above
# instead.
#        $this->connected = true;
    }

function get ($url) {

    if (!$this->connected)
        throw new Excetion("Not connected");

# Allready have a resource handle...
#        $ch = curl_init();

# If you really will make no more requests, closing your handle here is ok
# else, do it in a separate function and call this as needed
        curl_close($this->ch);

return $results;
}

public function curl_close() {
    curl_close($this->ch);
    }


private function getPostFields ($data) {
# why are you returning an empty array directly?
# the rest of the code will never be executed...
#        $return = array();

# This should use urlencode(), not urldecode
        foreach ($data as $key => $field) {
            $return[] = $key . '=' . urlencode($field);
        }

    return implode('&', $return);
}

andywilliams99

OK but I see 2 things missing - the $data array for the post data, and the login page?

johanafm

Yes, everything you need to keep unchanged was removed. You get to piece things together. Try yourself, then ask if you run into any errors and we'll take it from there.