cURL doesn't work... But only on Yahoo servers

Matt_Denton

I'm having a huge problem with cURL. The long story is I'm trying to consume a web service at <http://clients.mindbodyonline.com/api/0_5/ClassService.asmx?wsdl>. The Yahoo servers don't support native SOAP, so I am using nuSOAP (which I know is no longer actively developed, but that's fine by me). fsockopen isn't supported on the Yahoo servers either, so I'm forcing nuSOAP to use cURL.

Problem is cURL just won't connect to the service. I get CURL error 6: Couldn't resolve host 'clients.mindbodyonline.com'

cURL is enabled on Yahoo servers. The following code works fine and dandy:

    <?php
    $ch = curl_init();
    curl_setopt($ch, CURLOPT_URL,"http://www.google.com/");

if(curl_exec ($ch) === FALSE) {
   echo "FALSE!!!!!";
   echo 'Curl error: ' . curl_errno($ch) . ', ' . curl_error($ch);
}
curl_close ($ch); 
?>

...and shows me the Google page. So cURL is enabled and works. But this code, with the URL changed, does not:

    <?php
    $ch = curl_init();
    curl_setopt($ch, CURLOPT_URL,"http://clients.mindbodyonline.com/api/0_5/ClassService.asmx?wsdl");

if(curl_exec ($ch) === FALSE) {
   echo "FALSE!!!!!";
   echo 'Curl error: ' . curl_errno($ch) . ', ' . curl_error($ch);
}
curl_close ($ch); 
?>

Can anyone help me here? The code works perfectly fine on my localhost server, AND on http://codepad.viper-7.com, which runs the PHP code for free... It seems to be specifically this URL and specifically on the Yahoo web hosting. Anyone know why?

Weedpacket

Just out of interest: does [man]gethostbyname[/man] succesfully resolve the name? (If it does, and if no-one else has any ideas, perhaps giving cURL the IP can serve as a workaround.)

Matt_Denton

Weedpacket;11007871 wrote:
Just out of interest: does [man]gethostbyname[/man] succesfully resolve the name? (If it does, and if no-one else has any ideas, perhaps giving cURL the IP can serve as a workaround.)

Thanks for the reply! I'm sorry but I'm not completely sure what you mean: I ran this code:

<?
$ip = gethostbyname('clients.mindbodyonline.com');
echo $ip;
?>

And it printed out

clients.mindbodyonline.com

I've already tried giving it the IP address, but sadly it didn't work. The IP address is 72.21.92.35, but I can't use the IP address in the URL bar to access the site either... It seem clients.mindbodyonline.com, preview.mindbodyonline.com, and www.mindbodyonline.com all use the same IP address. cURL can't access any of them. However if I try to use cURL to access mindbodyonline.com, I get a 301 permanently moved... Using

curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);

then redirects me and I get

cURL error 6: couldn't resolve host 'www.mindbodyonline.com'

Again, this all works on my localhost server as well as other web hosting services.

Matt_Denton

Yeah, so now I see that if gethostbyname returns an unmodified string, it failed. So yes, it failed. All other information I can think of is in the above post.

Matt_Denton

And finally... gethostbyname does NOT fail when using google.com. But it DOES FAIL for clients.mindbodyonline.com.

And is there a way to edit posts on this forum? Because that would help a lot.

Derokorian

Matt.Denton;11007883 wrote:
And is there a way to edit posts on this forum? Because that would help a lot.

Yes you can edit posts here but only once you've been an active member for X amount of time (not sure what the exact rule is).

As for your problem I have no input sorry. Have you tried setting more options? Maybe it just sees the request as screen-scraping and is rejecting it because there is no browser-agent string?

bradgrafelman

Matt.Denton;11007883 wrote:
And finally... gethostbyname does NOT fail when using google.com. But it DOES FAIL for clients.mindbodyonline.com.

That's interesting; that suggests that at least some DNS resolution can take place.

One curiosity I note is 'clients.mindbodyonline.com' is really just an alias for 'clients_cdn.mindbodyonline.com' which is yet another alias for 'cs15.adn.edgecastcdn.net' where we finally reach an A record with an address of '72.21.92.35'.

What did the cURL error message look like when you tried to use the IP address instead? Note that you'd want to use [man]curl_setopt/man with the CURLOPT_HTTPHEADER option to define the 'Host' HTTP header since you're no longer using 'clients.mindbodyonline.com'. (This is why using the IP address in your browser won't work; the browser is still able to contact the server, but it no longer knows that it should tell the server it's trying to reach 'clients.mindbodyonline.com' - the same physical server is apparently hosting several virtual ones.)

Derokorian;11007901 wrote:
Maybe it just sees the request as screen-scraping and is rejecting it because there is no browser-agent string?

Impossible - the cURL error is pointing to a DNS resolution issue, meaning that PHP wasn't even able to attempt to contact the remote server, let alone get rejected.

Matt_Denton

Thanks for the reply. Sorry I don' know much about TCP/IP and HTTP, but doing this

<?php
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL,"http://72.21.92.35");
//curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_HTTPHEADER, array('Host: clients.mindbodyonline.com'));

if(curl_exec ($ch) === FALSE) {
   echo 'Curl error: ' . curl_errno($ch) . ', ' . curl_error($ch);
   die();
}

curl_close ($ch);
?>

...gives me a 302... Here are the response headers:

HTTP/1.1 302 Found
Date: Mon, 09 Jul 2012 19:43:33 GMT
Location: https://clients.mindbodyonline.com/
Server: BigIP
Content-Length: 0

So I uncommented the "curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);" and got...

Curl error: 6, Couldn't resolve host 'clients.mindbodyonline.com'

NogDog

It's possible they're looking at the HTTP headers that your cURL request is sending, and they're deciding it does not look like a legitimate browser request, and so are sending the 302 redirect header. If that's the case, you may need to send some additional headers via curl_setopt(), such as to emulate a "valid" user-agent header and so forth.

Matt_Denton

NogDog;11007919 wrote:
It's possible they're looking at the HTTP headers that your cURL request is sending, and they're deciding it does not look like a legitimate browser request, and so are sending the 302 redirect header. If that's the case, you may need to send some additional headers via curl_setopt(), such as to emulate a "valid" user-agent header and so forth.

I tried

curl_setopt($ch, CURLOPT_USERAGENT, 'Mozilla/5.0 (Windows NT 5.1; rv:13.0) Gecko/20100101 Firefox/13.0.1');

But I get all the same errors. I must stress again this ALL works from localhost server AND other web hosting services. And it's a web service I'm trying to access. The WSDL file is here: http://clients.mindbodyonline.com/api/0_5/ClassService.asmx?wsdl

But again, I appreciate the replies everyone.

bradgrafelman

NogDog;11007919 wrote:
It's possible they're looking at the HTTP headers that your cURL request is sending, and they're deciding it does not look like a legitimate browser request, and so are sending the 302 redirect header. If that's the case, you may need to send some additional headers via curl_setopt(), such as to emulate a "valid" user-agent header and so forth.

I doubt this is what's happening (although you're right - it is certainly possible).

Instead, I suspect what is happening is what would happen for all similar requests (automated or not); it's being redirected to use the https:// protocol.

@.Denton: Try changing the protocol from 'http://' to 'https://' in your test script that uses the remote server's IP address and manually sets the Host HTTP header. (Note it's been a while since I've used cURL - you may need to play around with more of the cURL options to get SSL enabled properly.)

Matt.Denton;11007925 wrote:
But I get all the same errors. I must stress again this ALL works from localhost server AND other web hosting services.

Right - not doubting that at all. It's starting to look like whatever host you're on is somehow disrupting the DNS resolution process (intentionally or not), perhaps due to all of those canonical names I described above.

Have you tried contacting your provider's customer/technical support department? (You'd want to include the bits and pieces we've learned in this thread; the DNS resolution works just fine on other servers, and connecting to the remote server directly via its IP address is working - it's just using the 'clients.mindbodyonline.com' DNS name that seems broken at the present.)

Matt_Denton

This

<?php
$ch = curl_init();

//These two lines produced the same error code, cURL error 6 can't resolve host
//curl_setopt($ch, CURLOPT_URL,"https://72.21.92.35");
curl_setopt($ch, CURLOPT_URL,"https://clients.mindbodyonline.com");

curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_HTTPHEADER, array('Host: clients.mindbodyonline.com'));
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, FALSE);   

curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 2);

if(curl_exec ($ch) === FALSE) {
   echo 'Curl error: ' . curl_errno($ch) . ', ' . curl_error($ch);
   die();
}

curl_close ($ch);
?>

Gives the same error, cURL 6, can't resolve host 'clients.mindbodyonline.com'

This

<?php
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL,"https://72.21.92.35");
//curl_setopt($ch, CURLOPT_URL,"https://clients.mindbodyonline.com");

curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_HTTPHEADER, array('Host: clients.mindbodyonline.com'));
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, FALSE);   

curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 2);

if(curl_exec ($ch) === FALSE) {
   echo 'Curl error: ' . curl_errno($ch) . ', ' . curl_error($ch);
   die();
}

curl_close ($ch);
?>

gives

Curl error: 6, SSL: certificate subject name '*.mindbodyonline.com' does not match target host name '72.21.92.35'

So I changed it to curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, FALSE); and got the cURL error 6 can't resolve host error again.

And this

<?php
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL,"https://72.21.92.35");
//curl_setopt($ch, CURLOPT_URL,"https://clients.mindbodyonline.com");

curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_HTTPHEADER, array('Host: clients.mindbodyonline.com'));

if(curl_exec ($ch) === FALSE) {
   echo 'Curl error: ' . curl_errno($ch) . ', ' . curl_error($ch);
   die();
}

curl_close ($ch);
?>

Gives

Curl error: 77, error setting certificate verify locations: CAfile: /usr/local/share/certs/ca-root-nss.crt CApath: none

I called them yesterday and they couldn't do a thing. I could perhaps call them again with some more information but the guys I called don't have any more control over the server than you guys do.

Matt_Denton

I can also give you guys info on successful requests. Here's some headers from a successful request. I doubt they help, but hey, why not.

request headers

GET /api/0_5/ClassService.asmx?wsdl HTTP/1.1
Host: clients.mindbodyonline.com
Accept: */*

response

HTTP/1.1 200 OK
Cache-Control: private, max-age=0
Content-Type: text/xml; charset=utf-8
Date: Tue, 10 Jul 2012 00:53:03 GMT
P3P: CAO DSP COR CUR TAIa OUR NOR UNI STA
Server: Microsoft-IIS/7.5
Vary: Accept-Encoding
X-AspNet-Version: 4.0.30319
X-Powered-By: ASP.NET
Transfer-Encoding: chunked

Matt_Denton

Anyone have any more ideas?

NogDog

Maybe try:

curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);

Matt_Denton

NogDog;11008075 wrote:
Maybe try:
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);

I did, see my above post:p

Matt_Denton

You know what I ended up doing? I created an entire proxy on the Google App Engine site. Goodbye 7 hours of my life...

evannmr

Yahoo servers appear to have a problem with redirects. In the case of cURL that will result in an error (6). [fsockopen has the same problem but obviously throws a different errno.]

A workaround that seems to get over the problem is of the form -
$rec = dns_get_record("example.com");
$ipAddr = $rec[0]['ip'];
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL,$ipAddr);

A complex site may need a little more in that you may have to find the correct member of the array $rec

dalecosp

Matt.Denton;11008369 wrote:
You know what I ended up doing? I created an entire proxy on the Google App Engine site. Goodbye 7 hours of my life...

I would include this thread, or that specific post, or at least a line-item mention of this on my next job interview/resume. That 7 hours might be worth it, after all. 😉