curl screen scrape problem

damiantaylor

Hi,
I've been struggling for days trying to get a screen scrape to work which just goes off to a carrier's tracking website and looks to see if a parcel I have posted has been delivered yet - I'm trying to stop it being such a time consuming manual process!

It all seems to work ok, except I can only get the results from the last 22 scrapes I attempt.
If I send 30 requests using curl_multi_exec the first 8 always come back 'couldn't connect to host' and the last 22 work.
I have tried reversing the order of the requests and I get the same reuslt - 8 failed, 22 worked but because I did the requests in reverse, the 8 that failed first time around worked this time and 8 of the requests that worked first time around failed this time!

I've no idea what is going on but am guessing it's the carrier's server blocking me?
Here's the options I have set for curl:
CURLOPT_USERAGENT, "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.1) Gecko/20061204 Firefox/2.0.0.1");
CURLOPT_MAXCONNECTS, 15);
CURLOPT_FAILONERROR, true);
CURLOPT_FOLLOWLOCATION, true);
CURLOPT_AUTOREFERER, true);
CURLOPT_RETURNTRANSFER,true);

I was hoping the MAXCONNECTS would do the job but unfortunately not 🙁
Could it be my server blocking the attempts? If that sounds possible, how can I check?
Can you help me get this working?

damiantaylor

I've tried this on a different server and am getting a different response.
I just get completely random failures now. If I refresh my browser a different set of connections will fail each time.

The error I now get is 'Failed connect to website.com:80; Operation now in progress'

Any ideas on how to fix this?
I've set extra options as follows.....

CURLOPT_CONNECTTIMEOUT, 0
CURLOPT_TIMEOUT, 120

...but still getting failures 😕

bradgrafelman

damiantaylor;10996468 wrote:
Any ideas on how to fix this?

Yeah; don't do it. 🙂

Seriously, though, don't do it. Screen scraping is one of the worst possible methods you could use to gather information from 3rd party websites (not to mention that it's often prohibited by the sites' AUPs/TOSs/etc.).

Instead, see if you can't find some sort of API (SOAP, REST, etc.) or XML feed that will give you the information you're after. It'll be simpler to work with plus it probably won't be broken with every single change made to the website itself (which was designed for consumption by a user's web browser, not another web service trying to extract pieces of information).

Weedpacket

I agree with bradgrafelman's suggestion of finding an API or XML feed; basically, see if the carrier provides a website designed for machine consumption rather than for humans.

Obligatory XKCD link

damiantaylor

Thanks for the replies guys,
I'd love to do it the ethical way but this is the Royal Mail we're talking about and as far as I know they don't have such an api.

I'd love to be proved wrong on that one though!

Weedpacket

damiantaylor wrote:
I'd love to be proved wrong on that one though!

Assuming (a) you're doing this often enough to make automating the process worthwhile, and (b) this is the UK Royal Mail, then you can arrange with them to send you regular reports via ftp.

http://www.royalmail.com/royalmailtracked