cURL doesn't work... But only on Yahoo servers

bradgrafelman · Jul 9, 2012

Matt.Denton;11007883 wrote:
And finally... gethostbyname does NOT fail when using google.com. But it DOES FAIL for clients.mindbodyonline.com.

That's interesting; that suggests that at least some DNS resolution can take place.

One curiosity I note is 'clients.mindbodyonline.com' is really just an alias for 'clients_cdn.mindbodyonline.com' which is yet another alias for 'cs15.adn.edgecastcdn.net' where we finally reach an A record with an address of '72.21.92.35'.

What did the cURL error message look like when you tried to use the IP address instead? Note that you'd want to use [man]curl_setopt/man with the CURLOPT_HTTPHEADER option to define the 'Host' HTTP header since you're no longer using 'clients.mindbodyonline.com'. (This is why using the IP address in your browser won't work; the browser is still able to contact the server, but it no longer knows that it should tell the server it's trying to reach 'clients.mindbodyonline.com' - the same physical server is apparently hosting several virtual ones.)

Derokorian;11007901 wrote:
Maybe it just sees the request as screen-scraping and is rejecting it because there is no browser-agent string?

Impossible - the cURL error is pointing to a DNS resolution issue, meaning that PHP wasn't even able to attempt to contact the remote server, let alone get rejected.

Matt_Denton · Jul 9, 2012

Thanks for the reply. Sorry I don' know much about TCP/IP and HTTP, but doing this

<?php
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL,"http://72.21.92.35");
//curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_HTTPHEADER, array('Host: clients.mindbodyonline.com'));

if(curl_exec ($ch) === FALSE) {
   echo 'Curl error: ' . curl_errno($ch) . ', ' . curl_error($ch);
   die();
}

curl_close ($ch);
?>

...gives me a 302... Here are the response headers:

HTTP/1.1 302 Found
Date: Mon, 09 Jul 2012 19:43:33 GMT
Location: https://clients.mindbodyonline.com/
Server: BigIP
Content-Length: 0

So I uncommented the "curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);" and got...

Curl error: 6, Couldn't resolve host 'clients.mindbodyonline.com'

NogDog · Jul 9, 2012

It's possible they're looking at the HTTP headers that your cURL request is sending, and they're deciding it does not look like a legitimate browser request, and so are sending the 302 redirect header. If that's the case, you may need to send some additional headers via curl_setopt(), such as to emulate a "valid" user-agent header and so forth.

Matt_Denton · Jul 10, 2012

NogDog;11007919 wrote:
It's possible they're looking at the HTTP headers that your cURL request is sending, and they're deciding it does not look like a legitimate browser request, and so are sending the 302 redirect header. If that's the case, you may need to send some additional headers via curl_setopt(), such as to emulate a "valid" user-agent header and so forth.

I tried

curl_setopt($ch, CURLOPT_USERAGENT, 'Mozilla/5.0 (Windows NT 5.1; rv:13.0) Gecko/20100101 Firefox/13.0.1');

But I get all the same errors. I must stress again this ALL works from localhost server AND other web hosting services. And it's a web service I'm trying to access. The WSDL file is here: http://clients.mindbodyonline.com/api/0_5/ClassService.asmx?wsdl

But again, I appreciate the replies everyone.

bradgrafelman · Jul 10, 2012

NogDog;11007919 wrote:
It's possible they're looking at the HTTP headers that your cURL request is sending, and they're deciding it does not look like a legitimate browser request, and so are sending the 302 redirect header. If that's the case, you may need to send some additional headers via curl_setopt(), such as to emulate a "valid" user-agent header and so forth.

I doubt this is what's happening (although you're right - it is certainly possible).

Instead, I suspect what is happening is what would happen for all similar requests (automated or not); it's being redirected to use the https:// protocol.

@.Denton: Try changing the protocol from 'http://' to 'https://' in your test script that uses the remote server's IP address and manually sets the Host HTTP header. (Note it's been a while since I've used cURL - you may need to play around with more of the cURL options to get SSL enabled properly.)

Matt.Denton;11007925 wrote:
But I get all the same errors. I must stress again this ALL works from localhost server AND other web hosting services.

Right - not doubting that at all. It's starting to look like whatever host you're on is somehow disrupting the DNS resolution process (intentionally or not), perhaps due to all of those canonical names I described above.

Have you tried contacting your provider's customer/technical support department? (You'd want to include the bits and pieces we've learned in this thread; the DNS resolution works just fine on other servers, and connecting to the remote server directly via its IP address is working - it's just using the 'clients.mindbodyonline.com' DNS name that seems broken at the present.)

Matt_Denton · Jul 10, 2012

This

<?php
$ch = curl_init();

//These two lines produced the same error code, cURL error 6 can't resolve host
//curl_setopt($ch, CURLOPT_URL,"https://72.21.92.35");
curl_setopt($ch, CURLOPT_URL,"https://clients.mindbodyonline.com");

curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_HTTPHEADER, array('Host: clients.mindbodyonline.com'));
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, FALSE);   

curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 2);

if(curl_exec ($ch) === FALSE) {
   echo 'Curl error: ' . curl_errno($ch) . ', ' . curl_error($ch);
   die();
}

curl_close ($ch);
?>

Gives the same error, cURL 6, can't resolve host 'clients.mindbodyonline.com'

This

<?php
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL,"https://72.21.92.35");
//curl_setopt($ch, CURLOPT_URL,"https://clients.mindbodyonline.com");

curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_HTTPHEADER, array('Host: clients.mindbodyonline.com'));
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, FALSE);   

curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 2);

if(curl_exec ($ch) === FALSE) {
   echo 'Curl error: ' . curl_errno($ch) . ', ' . curl_error($ch);
   die();
}

curl_close ($ch);
?>

gives

Curl error: 6, SSL: certificate subject name '*.mindbodyonline.com' does not match target host name '72.21.92.35'

So I changed it to curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, FALSE); and got the cURL error 6 can't resolve host error again.

And this

<?php
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL,"https://72.21.92.35");
//curl_setopt($ch, CURLOPT_URL,"https://clients.mindbodyonline.com");

curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_HTTPHEADER, array('Host: clients.mindbodyonline.com'));

if(curl_exec ($ch) === FALSE) {
   echo 'Curl error: ' . curl_errno($ch) . ', ' . curl_error($ch);
   die();
}

curl_close ($ch);
?>

Gives

Curl error: 77, error setting certificate verify locations: CAfile: /usr/local/share/certs/ca-root-nss.crt CApath: none

I called them yesterday and they couldn't do a thing. I could perhaps call them again with some more information but the guys I called don't have any more control over the server than you guys do.

Matt_Denton · Jul 10, 2012

I can also give you guys info on successful requests. Here's some headers from a successful request. I doubt they help, but hey, why not.

request headers

GET /api/0_5/ClassService.asmx?wsdl HTTP/1.1
Host: clients.mindbodyonline.com
Accept: */*

response

HTTP/1.1 200 OK
Cache-Control: private, max-age=0
Content-Type: text/xml; charset=utf-8
Date: Tue, 10 Jul 2012 00:53:03 GMT
P3P: CAO DSP COR CUR TAIa OUR NOR UNI STA
Server: Microsoft-IIS/7.5
Vary: Accept-Encoding
X-AspNet-Version: 4.0.30319
X-Powered-By: ASP.NET
Transfer-Encoding: chunked

Matt_Denton · Jul 10, 2012

Anyone have any more ideas?

NogDog · Jul 10, 2012

Maybe try:

curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);

Matt_Denton · Jul 11, 2012

NogDog;11008075 wrote:
Maybe try:
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);

I did, see my above post:p

Matt_Denton · Jul 13, 2012

You know what I ended up doing? I created an entire proxy on the Google App Engine site. Goodbye 7 hours of my life...

evannmr · Jan 14, 2014

Yahoo servers appear to have a problem with redirects. In the case of cURL that will result in an error (6). [fsockopen has the same problem but obviously throws a different errno.]

A workaround that seems to get over the problem is of the form -
$rec = dns_get_record("example.com");
$ipAddr = $rec[0]['ip'];
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL,$ipAddr);

A complex site may need a little more in that you may have to find the correct member of the array $rec

dalecosp · Jan 15, 2014

Matt.Denton;11008369 wrote:
You know what I ended up doing? I created an entire proxy on the Google App Engine site. Goodbye 7 hours of my life...

I would include this thread, or that specific post, or at least a line-item mention of this on my next job interview/resume. That 7 hours might be worth it, after all.

cURL doesn't work... But only on Yahoo servers

Bbradgrafelman

MMatt_Denton

NogDog

MMatt_Denton

Bbradgrafelman

MMatt_Denton

MMatt_Denton

MMatt_Denton

NogDog

MMatt_Denton

MMatt_Denton

Eevannmr

dalecosp