[RESOLVED] cURL complaining about cert...how to fix?

sneakyimp

I have a script that uses cURL to access a remote HTTPS server. It was working fine for a long time but apparently something has changed. I in this function the curl_exec call is returning false:

function fetchPage($url, $truncateCookieFile=TRUE, $verifySSL=TRUE) {
	// create a new cURL resource
	$ch = curl_init($url);

// set URL and other appropriate options
//curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_AUTOREFERER, TRUE);
curl_setopt($ch, CURLOPT_FRESH_CONNECT, TRUE);
curl_setopt($ch, CURLOPT_HEADER, TRUE);
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, TRUE);
curl_setopt($ch, CURLOPT_VERBOSE, TRUE);
curl_setopt($ch, CURLOPT_HEADER, FALSE);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, TRUE); // return the results of the curl exec
curl_setopt($ch, CURLOPT_USERAGENT, 'Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.14) Gecko/20080404 Firefox/2.0.0.14');

if (!$verifySSL) {
	/* this is added in v2 of this function */
	curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, FALSE);
}


/* because the remote site may use sessions, we need a cookie jar*/
$cookieFile = SERVER_PATH . DIRECTORY_SEPARATOR . 'cookies.txt';
curl_setopt($ch, CURLOPT_COOKIEFILE, $cookieFile);
curl_setopt($ch, CURLOPT_COOKIEJAR, $cookieFile);

/* grab URL and put it in $result */
$result = curl_exec($ch);
if ($result === false) {
	throw new Exception('CURL failed to fetch page:' . curl_error($ch));
}

$code = curl_getinfo($ch, CURLINFO_HTTP_CODE);
if ($code != 200) throw new Exception('CURL returned non-OK code:' . $code);

/* close cURL resource, and free up system resources */
curl_close($ch);

/* if $truncateCookieFile is true, blank the cookie file and eliminate any sessions */
if ($truncateCookieFile) {
	if (!($fp = fopen($cookieFile, 'w'))) {
		throw new Exception('Could not truncate cookie file');
	}
	fclose($fp);
}

/* return the result */
return $result;
}

As you might have noticed, I have written my function so you can ignore SSL problems but I'd rather fix the cert issue so that I can be sure my connection is encrypted properly and that all the certs check out as they should (or i fail because there's an imposter).

The exception that's getting thrown says CURL failed to fetch page:SSL certificate problem, verify that the CA cert is OK. Details:
error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed

I have googled this and found a page that seems pretty informative here:
http://curl.haxx.se/docs/sslcerts.html

However, I don't really follow what needs to be done. As I understand it, a given site hosting https requires a certificate. This certificate must be 'signed' by a certificate authority. This is where my understanding gets kinda fuzzy. Any given browser (or cURL or other means of accessing https) must compare the CA signature on a given cert to some bundle of 'trusted' CAs. Or something like that.

Basically, I am essentially at a loss as to how to proceed here. I'm guessing (I can visit that site using a browser and find out which certificate authority has signed the SSL cert for this troublesome remote server, but I'm not sure how to do that. I'm even less sure how to find out if that CA cert is valid. I'd really appreciate it if someone could walk me through this.

dagon

whats the url of the page your fetching?

sneakyimp

It varies. There are a number of domains. Here is one:
https://www.broadwayoffers.com/

sneakyimp

Can anyone give me some instruction on how to update the cert bundle or otherwise prevent this problem?

dagon

you cant update some one else's cert.

bretticus

Okay, here's how it works in a nutshell (please feel free to correct any misinformation I give.)

Browsers/operating systems are pre-installed/upgraded with the latest public keys from various certificate authorities (thawte, verisign, geotrust, etc.) These public keys are called root certificates. Because of the magic of public/private key crytology, your browser can verify that a given website's keys have been signed by one of your root certificates (trusted authority.)

The issue is that whatever path your installation of cURL uses does not have access to one or more root keys. In the past, I have updated my bundle and been able to get around this issue. Since I do not remember just how at this moment (nor do I know if this is your server either), I will give you another suggestion. You can tell cURL not to validate the certificate:

curl_setopt ($ch, CURLOPT_SSL_VERIFYHOST, 0);
curl_setopt ($ch, CURLOPT_SSL_VERIFYPEER, 0);

I suggest only doing this if you do not care about a possible man in the middle attack (if they're credit card numbers or other sensitive information.)

sneakyimp

Thanks bretticus!

I do understand that most SSL-capable software gets delivered with a 'Certificate Bundle' which serves as a starting point for validating SSL. For instance, Firefox when installed has a list of 'root certificates'. Whenever you go to some new site trying to connect via SSL, the server coughs up a certificate for that new site. This certificate must be 'signed'. If it is not signed by one of the guys in my root certificate bundle, then I get some kind of warning which is the problem I'm having. At least I think that's how it works.

I want to be sure in this application that there is no man-in-the-middle attack so I need to fix it. Perhaps I should update my certificate bundle? Where does one find a new/comprehensive certificate bundle?

bretticus

sneakyimp;10899243 wrote:
I want to be sure in this application that there is no man-in-the-middle attack so I need to fix it. Perhaps I should update my certificate bundle? Where does one find a new/comprehensive certificate bundle?

Man, now you're really testing my memory!? 🙂 BTW, what operating system is your cURL binary on?

bretticus

Here is a link that points to the ca bundle with some basic instructions on how to find them (pull out your nix find command assuming you are booting nix.)

sneakyimp

Okey doke I have wrestled with this and have some information for those interested.

Firstly, the cURL homepage appears to be here: http://curl.haxx.se

The documentation on that site has a message:
"I just want to mention that the CA cert bundle we bundle in the curl release
archives is very out of date and lacks a lot."

A page on the site does offer a certificate bundle file in PEM format (see X.509 for more info). This page also describes the various ways you might extract a PEM formatted certificate bundle from Mozilla -- either the mozilla you have installed on your *nix machine or the certificates that are included in a file in the Mozilla source tree.

The basic idea is that you can supply a text file containing certificates in PEM format (just like the one linked above) and this constitutes a 'certificate bundle' as far as cURL is concerned. The whole trick here is to include certificates only from reputable certificate authorities but include enough certs so you don't get paranoid security warnings at every site you visit. I don't really know everything about how the public key infrastructure (PKI) works but it is my understanding that when you visit a website that offers up a certificate for an SSL/TLS connection, that certificate must be signed by someone. Either this signing someone themself must be included in your browser's certificate bundle OR somebody who signed their certificate must be in your certificate bundle. I'm not certain but I believe that PKI-aware browsers will recursively check up the certificate signing hierarchy until they find a self-signed certificate or they find a certificate that is in your bundle. If among all the signers in a certificate chain your browser can't find ANYONE who is in your certificate bundle, then you'll get a warning and the little lock icon will probably have something indicating the connection is not truly secure. In the case of cURL, a security error will be thrown like the one with which I started this thread. If anyone can offer a clearer explanation then I'd love to read it. It seems like a lot of hand waving.

SO if you want to try and get a new certificate bundle because cURL comes with a crappy one and is throwing security errors, you'll need to get one in PEM format. When you finally have it (let's assume it's located at /home/user/cacert.pem) then do this in your PHP code:

// $ch is the curl connection you created with curl_init
curl_setopt($ch, CURLOPT_CAINFO, '/home/user/cacert.pem');

How to get a PEM-formatted certificate bundle:
Please keep in mind that you are trusting whatever source you use to give you a cert bundle with only valid certs. This does not actually guarantee security against evil cert spoofers.
1) You can download the PEM-formatted certificate bundle file freely supplied at the cURL website. It's ready to rock. NOTE: Some tests revealed this cert bundle to be more permissive than one extracted from mozilla source tree. Use at your own risk. I have no idea at all if this version is any different than that supplied with the cURL distribution.
2) You can run this perl script which downloads a file ostensibly containing trusted certificates from the mozilla website and reformats it as a PEM file. I don't know if that's the latest version of the mozilla source file or not. I recommend getting the perl script from CVS because it will be the latest version with no line numbers. I downloaded the file to my iMac as mk-ca-bundlep.pl and ran this command and it created a file called ca-bundle.crt

perl mk-ca-bundle.pl

.
3) Another way is to run this shell script on a *nix machine which is supposed to extract the certificates off your local firefox installation. I couldn't get this one to run.

For a small collection of websites that I was testing, I found that the cert bundle you can download from the curl website will validate more websites than the one I extracted from the mozilla source tree. I have no idea why. Ironically, the sites I was hoping to cURL in the first place are not validated by either. I still have no idea why.

I hope this helps someone.

EDITS: Miscellaneous readability things.

sneakyimp

It occurred to me that folks might want to connect to a site which simply isn't validated by whatever certificate bundle they may have downloaded/extracted/borrowed/stolen. If you would like cURL to validate some website of your choosing, you can manually extract a certificate using Firefox and add it yourself to whatever PEM-formatted certificate bundle file you happen to be using.

I assume here that you have begged/borrowed/stolen some PEM-formatted certificate file using the methods supplied in the previous post and have set cURL to use the file by setting the CURLOPT_CAINFO option. Let's assume you call this file cacert.pem

I ran these steps on a windows machine with Firefox 3 but the process is probably nearly identical on a mac or *nix.

1) Open firefox and visit the securely hosted page that you want cURL to visit. E.g., https://somemachine.somedomain.com.
2) Double-click the lock icon in the bottom right hand corner of your Firefox window. You can also right-click a blank area of the page and select 'Page Info' and then select the Security tab. A window pops up.
3) In the section of the popup window labeled 'Web Site Identity' click the button that says 'View Certificate'. Another popup window should appear.
4) In this popup window, click the Details tab.
5) In the pane labeled 'Certificate Hierarchy' you will see a nested hierarchy of text labels representing the website you have visited, the organization that signed its cert, the organization that signed that cert, etc. The website you have visited (in this case somemachine.somedomain.com) should be listed at the bottom. Click whatever object appears as the immediate parent of the site you have just visited.
6) Click the button at the bottom of the window that says 'Export'. You will be prompted to save a file somewhere on your hard drive.
7) Make sure that you select 'X.509 Certificate (PEM)' on the dialog where it says 'Save as Type'. If you don't do this, it will be saved in the wrong format.
8) Select a location and filename for the file where you are sure you can find it later and save the file. This will result in a text file. On my machine, firefox automatically adds the file extension '.crt' to whatever filename I choose. Let's assume you call this file foo.crt]. You can close all the firefox windows now.
9) Open the resulting file (foo.crt) with a text editor and copy the contents of the file. The contents of that file should look something like this:

Paste the contents at the very end of the certificate bundle file you are using with curl (cacert.pem) and save it to the right location on your server (/home/user/cacert.pem in the example in the previous post).

If you have done everything properly, cURL should now successfully validate the site's security certificate without complaints. NOTE: In adding this certificate to your certificate bundle, you have chosen to trust any certificate signed by it which may include more sites than just the one that interests you.