My reading comprehension has always been a bit poor, and this discussion is a bit abstract for me (channels?), but I think I can contribute a bit to this conversation as I have been using a database table to delegate jobs to other virtual servers allocated in the rackspace cloud. The operation of these machines in my case is massively parallel -- multiple machines running scores of simultaneous processes each, all working on the same db table. There's more detail here.
While my approach seems much more specific than this general-purpose discussion here, I have a bit of input.
Unless you can find some atomic operation to locate a job record and mark it off limits to other consumer processes in one indivisible action, you have a risk of two different processes choosing the same job. I don't know if this is called 'race conditions' or what the technical term is, but it is a problem you encounter with concurrent programming and must be accounted for or you can get hard-to-reproduce bugs that are really hard to eradicate. In my application, I resort to db transactions and had to take special care to use BEGIN TRANSACTION, SELECT...FOR UPDATE, COMMIT, ROLLBACK, and so on.
As a practical matter, I had to incorporate record lock columns into my job table to prevent separate machines from working on the same data. My jobs involve fetching images from remote servers, generating numerous resized versions of the original image, and cramming them into a CDN. As you might imagine, I might fail to fetch the remote image or the CDN gateway might be too busy to accept my upload request. These failures are even more likely because of the massively parallel construction of my image fetching cluster. It might very well seem like DDOS attack to a remote server and could either formulate too many concurrent requests to be served or I might be blocked by the remote server for defensive reasons. I therefore structured my application so that each consuming process would lock a batch of 500 records or so in the table by setting record_lock_microtime to the current unix timestamp and i would also set record_lock_name to some value which would uniquely identify which machine/process had locked it. In the event the download failed, the record would be unlocked and its failure_count would be incremented. After 10 failures, the record is dead and would not be attempted again. In the event the consumer machine crashed or the process encountered some fatal error, the database record_lock_name would tell me which process had been working on it. The record_lock_name also helps to keep the work of the various consumer machines separate from each other.
The problem I face now is how to dynamically add nodes to my cluster when there is work to be done and how to scale it down when there is no work to be done. There are also questions of performance reporting and reporting to a dashboard and health checks and so on. I've made every effort to have the various consumer machines be as independent as possible, but it seems there's no way around the need to create a master/hub script of some kind to allocate new machines when there is work to be done.