CURL and file returning blank result with some sites.

The-Master

My server online returns blank pages with certain sites. One example of a site that it returns a blank page with is proxy.org.

What is the difference between these sites and normal sites.

Please note I have tried using the exact same script to view proxy.org on my home server and it works.

Is it a php extension or something that hasn't been enabled?

Thanks in advance.🙂

dougal85

I'm not sure if this relates to your problem exactly. However, Some websites block traffic from programming languages.

I'm not sure how they do it, I guess its to do with detection of the user agent or something. For example, I think you might find a search result page for google doesn't work but www.google.com does work.

The-Master

Thanks for the response. Yeah. Myspace is like that, but I set a fake user agent for it and it worked. Even when I install a proxy script some sites just don't work. I get "Failed to connect to the specified host. Possible problems are that the server was not found, the connection timed out, or the connection refused by the host."

Could it be some sort of firewall on the server blocking some sites?

The sites that return nothing do work on my home server so I don't think it is anything to do with the script. I think it is to do with the server settings and maybe the php extensions installed or not installed.

dougal85

I doubt it would be a firewall. Since the connection would still be going through port 80. However this depends on what type of firewall they are using of course...

Perhaps they just have a more sophisticated way of determining if its a real browser or not?

I'm not really sure to be honest...

mikell

so you're saying you can't pull up http://proxy.org/index.shtml using your script right? as in you can't even view the proxy.org homepage...if you meant to say you can't use proxy.org with your script (as in view other websites via a proxy via your script) then stop reading now and let me know cause that's very different.

if your php environment is good enough so that your script works on some sites, i would think it would be good enough to work on all sites...

could be that your hosting company is blocking access to proxy.org... i don't know much about hosting company policies but i wouldn't be surprised if they discouraged their clients from using proxies for some lame reason. if so they would block any requests to that url.

if you have shell access to that server, log in and try to download the homepage with telnet. if you can't then your hosting company is blocking you or the proxy site has banned your server's ip for some reason...cause otherwise you should be able to download the homepage via telnet since i was able to. if you don't know how, here is a print out of my screen:

gateway# telnet proxy.org 80
Trying 74.53.126.157...
Connected to proxy.org.
Escape character is '^]'.
GET / HTTP/1.1
Host: proxy.org
<hit enter again>

type the stuff in bold...if a bunch of text starts flying down your screen then it worked. and if it worked, then the problem is likely to be your php settings, like you said, or a problem with the script itself. thats my guess anyways, good luck.