CURL question

[netfrugal]

As I'm becoming more familiar with curl, I was wondering one thing.

I like to transfer data via curl and will be setting up a cron job to auto transfer things to another server, as a backup. Are there any issues that would cause the transfer to slow down, or cause problems - IF - there's a lot of data to transfer?

I know this is a broad question, because I'm just looking for any possible hiccups when I initiate the curl statements. Hiccups that I may not know about, or understand. Anything that may involve timeouts, or the like.

I don't expect any particular answer, but I'd like any thoughts that may help me keep my data flowing smoothly.

thanks!

[etully]

While curl can do that job, it's not really the right tool for the job. Using a tool like rsync, scp, or even ftp will allow you to:

1. move the traffic faster (because the protocol can dynamically change packet size)
2. move traffic faster (in the case of rsync) because it can handle compression
3. you can authenticate traffic more securely than SSL (because you can authenticate in both directions)
4. you can move traffic faster (in the case of rsync) because you only move the files and data that you need (it skips the data you already have from the last time cron did a backup)
5. you can resume a transfer that fails in the middle (which is important when you are handling gigabytes of data)
6. You can transfer data that you would never allow Apache to expose to the Internet

Curl was really designed for a different job. Those are the only hiccups that I know of.

[netfrugal]

I see your point. Then what was curl originally designed to do?
And here's another question. Concerning credit cards, I need to transact them through my gateway, and I've been using curl as the medium. Typically there are not very many, and very sporatic. But I've been wondering about high volume for the future - such as continuity programs. If I had thousands of accounts to rebill (not that I do!!) how would curl handle the issue? It would seem that there's no real way to catch a failure during a transfer - and pickup where it left off.

[etully]

Curl is essentially a web browser. It's there so that your web page can go look at other web pages. For casual needs, it's no big deal if it fails every now and then. Curl has the ability to do a really good job of pretending it's a web browser. It can handle cookies, follow web redirects, manage passwords, and deal with SSL sessions. Rsync, SCP and FTP don't do any of those things because they are built to be data movers, not web browsers.

Curl is often the right tool, however, for managing credit card authorizations because many of the gateway companies communicate over HTTP. As for dropped connections with high volume CC auth traffic, you don't want Curl to try to be your safeguard. You need to write your own error checking in PHP (or whatever language you like) to make sure that when you send a message, you get a response from the gateway.

To illustrate why you should do your own error checking instead of relying on a program like Curl to manage the connection, imagine that a bomb blew up the gateway company. Curl tries to make a connection and, of course, fails. If Curl is responsible for guaranteeing that the CC auth happens, it's not going to give up trying. It will try over and over until built in time outs in PHP and Curl make the session collapse in a mess. If, however, you write PHP scripts that are aware of the need for the CC authorization and they keep telling Curl to try until it's done, then if Curl fails, there is no messy collapse... PHP can optionally (A) send email warnings to you, (😎 write a record of the failed transaction to a database for a later attempt, (C) tell the user of the failed auth, or (D) it can opt to make Curl try again if you want. It's far more elegant to let PHP decide what to do if Curl can't make the connection than to let Curl decide how to handle the situation.