load average for intervals measured in seconds rather than minutes?

sneakyimp

I was wondering if there might exist a linux command or php function that will tell me how busy my CPU is? I've gotten accustomed to typing 'w' which gives me a load average (not quite the same thing as CPU utilization but pretty good) or 'top' (which does actually calculate some kind of CPU % utilization).

Unfortunately, the 'w' command only shows load averages over 1, 5, and 15 minutes. I need something that looks at a shorter interval (e.g., 15 seconds). Can anyone tell me how I might get load average on the order of seconds rather than minutes?

Why do I need this? Because I'm writing a multi-threaded PHP script and would like it to be able to utilize all available resources up to a point -- I want to check the CPU utilization before deciding how many threads to spawn.

Weedpacket

The whole point of load averages is that it smooths out spikes or dips in CPU activity caused by things like processes starting up, blocking on I/O, and so on. Trying to work with a smaller window and your own efforts to measure and act on the results would start to skew and invalidate those results. For a machine that is supposed to be working flat out, a load average of 1×number of processor cores is good, and there's no point trying to maintain a level higher than that (because you haven't got the processors to run everything at once).

Since your "threads" obviously don't need to be running concurrently and you've got free rein over how many are running at a particular time, then more interesting is the number of processor cores you've got to work with (and multithreading is probably more trouble and overhead than it's worth): roughly, a CPU that is having to keep switching contexts to run several processes (or threads) concurrently is going to show poorer performance than one that can complete one task and get it out of the way before starting another.

sneakyimp

Weedpacket;11028049 wrote:
The whole point of load averages is that it smooths out spikes or dips in CPU activity caused by things like processes starting up, blocking on I/O, and so on. Trying to work with a smaller window and your own efforts to measure and act on the results would start to skew and invalidate those results. For a machine that is supposed to be working flat out, a load average of 1×number of processor cores is good, and there's no point trying to maintain a level higher than that (because you haven't got the processors to run everything at once).

I'm not looking for anything much smaller than one minute. My problem is that averaging over one minute seems so very sluggish. I observe roughly the following
launch some processes (using pcntl_ functions, so perhaps they are not threads but processes?)
see load average rise to some threshold, how quickly depends on how many threads I launch.
once the load average exceeds my configurable threshold (currently 4 on a dual-core machine), my script declines to launch any new processes, which results in the computer sitting completely idle while the sluggish one-minute load average slowly comes back down
* setting my threshold equal to the number of cores simply does not utilize very much of the computer's capacity. I observe about 15-25% utilization of CPU (I use the "top" command).

Weedpacket;11028049 wrote:
Since your "threads" obviously don't need to be running concurrently and you've got free rein over how many are running at a particular time, then more interesting is the number of processor cores you've got to work with (and multithreading is probably more trouble and overhead than it's worth): roughly, a CPU that is having to keep switching contexts to run several processes (or threads) concurrently is going to show poorer performance than one that can complete one task and get it out of the way before starting another.

I believe the application I'm using this for is a textbook example of where threads are useful because there is a lot of latency due to I/O requests. The script (which is launched via CLI and runs continuously all day) checks a database table for images that must be downloaded from our partners' websites, resized to seven distinct sizes, and then inserted into a CDN. This job was originally done by a single thread of execution and it was just too damn slow because the script would actually spend about 95% of its time waiting for a response to the HTTP download requests it would send out for the images. Writing this MT script has increased average performance from an image every few seconds or so to several images per second. Yes, it was a pain in the ass but I just couldn't think of any reasonable alternative.