suggestions for control of cluster nodes by a master node?

sneakyimp

I'm working on a project wherein we need to auto-scale our computing resources in response to a daily batch of tasks. Basically, the tasks arrive at midnight when a PHP script runs on a web server and inserts a bunch of records in a database table on a shared mysql server. When these job records have been inserted, I would then like to run another script which interacts with the Rackspace cloud to launch a bunch of virtual server instances to process these jobs.

My main question is Can anyone recommend an effective tool or technique for a master PHP script running on my primary web server to monitor and control these newly allocated virtual servers?

Having thought about this some, I can see that a few types of operations will be necessary:
once a virtual (slave) server has been launched, hand that slave a unique identifier that it will use to lock records in the job database and then fire up a script on the slave to work on the jobs
monitor the slaves as they work to keep an eye on load averages, memory usage, job completion rates, and other data related to slave health and performance.
* when the jobs have been completed, de-allocate the slave servers so we save money.

I expect I could do a roll-your-own approach to this but can't help but wonder if some helpful tool already exists that would make this easier.

dalecosp

Given that cloud is relatively "new", I'm not sure if something exists or not. I'm working with VM's now ... no cloud here yet.

That said, I wonder if something I was discussing with Weedpacket would have much bearing on your thoughts. [thread=10389707]See Post #4.[/thread]

sneakyimp

dalecosp, thanks for your input! I'm delighted to see someone wranging with the same type of problem and I've posted my thoughts there (hope they are helpful).

I think my task here is a subclass of the general idea you describe in that other thread. It's more specific in a few ways:
1) producer process creates jobs MUCH quicker than they can be completed. A process that runs in under an hour might create 400,000 jobs one night. A single consumer machine can only process about 12,000 jobs per hour.

2) once a job is created, there's only one thing to do to it -- finish it. I suppose that I might be able to make my application more complex and have different phases for each job, but it's been plenty complicated enough just to construct the controls necessary to attack it with concurrent processes/machines
3) in my case we are talking not just separate processes but separate machines. Between the latencies involved in requesting remote images and CDN gateways and the sheer amount of CPU power required to create 3 million images a day, an individual machine is just not up to the task. that said, I'm looking for some kind of command/control/monitoring protocol or system to coordinate actions between these machines.

dalecosp

As your consumer boxes handle 12K jobs/hour, I suppose the first thing you want to decide is how many hours you want to wait for things to be done?

Your master node would divide the jobs into chunks of 12K records, or 24K records ... etc. It would keep a record of which ones were assigned. Your slave nodes would be responsible for reporting back to the master when they were done, and the master would check the job queue again and either hand out another chunk of work or give the slave instructions to go home for the day.

Is that helpful? Possible? I have to admit you're somewhat over my head on this ;-)

sneakyimp

dalecosp;11028321 wrote:
As your consumer boxes handle 12K jobs/hour, I suppose the first thing you want to decide is how many hours you want to wait for things to be done?

To call these 'boxes' is not really accurate as they are but a virtual server running as a timeslice on actual hardware that is running a hypervisor -- at least I think so. Q.v. Rackspace Cloud Servers.

dalecosp;11028321 wrote:
Your master node would divide the jobs into chunks of 12K records, or 24K records ... etc. It would keep a record of which ones were assigned. Your slave nodes would be responsible for reporting back to the master when they were done, and the master would check the job queue again and either hand out another chunk of work or give the slave instructions to go home for the day.

I have designed things a bit differently than this inasmuch as I have set up each of the Cloud Servers (i.e., slaves, consumers, ImageDaemons) to use a query that will check my job table for some eligible records. E.g.,

SELECT * FROM jobs_table WHERE record_lock_microtime IS NULL AND record_lock_name IS NULL and fetch_failures < 10 ORDER BY blah blah blah LIMIT 500

. This seemed advantageous because it prevents the master node from being a bottleneck when slaves need work, eliminates the need to serialize all kinds of data for transmission to a slave, and also allows each slave to talk directly to the db servers as jobs are either completed or they fail. Basically, once the slave is in motion, it talks directly to the database and the jobs table is the clearing-house for all work completed.

dalecosp;11028321 wrote:
Is that helpful? Possible? I have to admit you're somewhat over my head on this ;-)

It's always helpful to have someone chime in as it stimulates thought. In explaining it, I understand it better.

The job works effectively at the moment, but I need to know better how to control the slaves from the central machine. I'm thinking each slave should be running a web server and the HTTPS protocol is how the master node will communicate with the slaves. This is a bit tricky as I have written my multi-threaded (multiprocessing?) PHP script that runs on each slave machine to launch and die by CLI (a cron job restarts it periodically). It seemed like a bad idea to try and combine the concurrent code type stuff with apache -- I still think this is true.

sneakyimp

Still looking for ideas about command-and-control techniques. Unless there's some magic protocol that permits a PHP script to directly issue authenticate and issue SSH commands to remote servers, I expect I'll need to concoct some authentication-and-control protocol that will require me to install PHP/Apache on each of the slave/worker/consumer machines.

Weedpacket

PECL does have an SSH2 extension.

sneakyimp

Weedpacket;11028421 wrote:
PECL does have an SSH2 extension.

Thanks, WP for pointing this out. Looks like the install might require some particulars but I am looking into this.

dalecosp

sneakyimp;11028411 wrote:
Still looking for ideas about command-and-control techniques. Unless there's some magic protocol that permits a PHP script to directly issue authenticate and issue SSH commands to remote servers, I expect I'll need to concoct some authentication-and-control protocol that will require me to install PHP/Apache on each of the slave/worker/consumer machines.

If it's running on CLI under a user's crontab, it's perfectly able to function with system() and friends. I do this frequently. Key-based auth is a must, at least in my book; and possibly passwordless keys, so if you're in a field/situation where that's not permitted ... it might not work.

sneakyimp

dalecosp;11028643 wrote:
If it's running on CLI under a user's crontab, it's perfectly able to function with system() and friends. I do this frequently. Key-based auth is a must, at least in my book; and possibly passwordless keys, so if you're in a field/situation where that's not permitted ... it might not work.

The problem is that the consumer-type scripts are running on entirely separate machines. It had occurred to me to try and use system or exec or some such beastie to communicate this way, but it seemed like a real chore to work up all the PHP to use those commands to connect to a remote script and parse out the results. I think this SSH2 pecl extension looks very promising and expect to test it some today.

One of the bigger problems I'm wrestling with now is how, from the Master Machine, to connect using the SSH2 extension to a remote machine and make sure that the consumer process (a PHP script) is running, healthy, and doing what it should be doing.

sneakyimp

So this command-and-control script seems to be shaping up a bit. As it turns out, I am keeping a database to list all of the Cloud Servers that I have running. My command-and-control script will run on a cron job every 5 minutes or so and will, based on the amount of work to do and the number of servers currently running (or currently jumping into action [the database keeps track]) will calculate whether or not we need to shut some servers down or fire more servers up.

If we need to shut them down, I'll use ssh2 to login, halt the process running on them (which should result in a nice orderly shut down thanks to prescient design of consumer processes), and then shut them down and terminate them so they are de-allocated by the cloud.

If we need to fire up new servers, I do a loop that forks off a process for each new server to be fired up. Once these new processes are all forked off, the primary command-and-control server exits.

Each forked off server-launching process does the following:
1) makes a request to the rackspace gateway for a new server
2) based on the API response, stores data about the new server in a database
3) waits, periodically polling the API for news on the new server
4) when the new server is finally ready, the process connects via SSH2 and fires up my consumer process
5) process writes any data we need to the db and exits.