how to calculate running averages over time from data arriving at irregular intervals

[sneakyimp]

I have some complex scripts that download images, process them, etc. I want to develop some analytical tools to monitor the performance of these scripts and am having some trouble determing how I might calculate some kind of moving average of the following:
total images successfully processed per hour (per second? per minute?) -- to monitor overall performance and inform dynamic resource allocation (i.e., help calculate how much work a single compute instance performs)
total bytes downloaded per hour (per second? per minute?) -- to monitor network performance and hopefully to diagnose any network bottleneck that might be encountered
* failed image fetch attempts

While I'm familiar with the concept of a moving average of the last n data points (add last n data points, divide by n), I am hoping instead to get an average over time from data points that occur at irregular intervals, hopefully without accumulating a bazillion data records. My intuition tells me it would be possible to continuously calculate a single moving average as long as I track enough information about the time elapsed since my last calculation, the number of images/bytes/failed attempts, etc. I'm having trouble sorting the math(s) of this concept and was hoping someone might help me determine the appropriate type of moving average to calculate and work out some kind of scheme.

The basic idea is that I have a main process that fetches some batch of jobs to process (typically 500 jobs) and then forks off a process to handle each job. Jobs are typically completed at a rate of 1-10 per second but the script seems to get work done in bursts, surging here and dragging there. In the interest of making this concept scalable, I expect I should try and calculate progress (and possibly update a database record with the result) with every batch of 500 jobs rather than 1-10 per second as each job-handling process finishes. When the batch completes, I think it's easy enough to divide the number of images successfully processed by the elapsed number of seconds and convert to per-hour or per-minute as needed, but it seems like this number might be fairly volatile? I'd like to average it out over some time period like the load average calculations you get when you use the w command in linux. I will then run a cron job every minute which can retrieve the current average value from the database and make a plot for the last 24 hours that can be made to make a meaningful graph.

Any thoughts are welcome.

[Weedpacket]

Well, you could build a histogram and get the moving averages off that.
If you're looking at checking every minute gather cumulative stats for each minute into a single bin, then take averages over the most recent n bins.
Now the intervals are regular.

[sneakyimp]

Thanks for your response, Weedpacket!

The problem is that the code to calculate the rates (bytes/interval, images/interval, etc.) does not live in the cron job which runs at regular intervals but lives rather in a long-running process in a particular subroutine that runs at very unpredictable intervals. The cron job's code is not privy to the data needed to make these rate calculations and, unless I'm missing something, the erratic subroutine can't simply average up the data from the past n intervals because these n intervals might reflect a lot of time or just a little time.

More specifically, the end-of-batch routine (which does have access to the number of images fetched and the number of bytes fetched) runs when a batch of 500 jobs has been completed. Given the variability of download sizes, response latencies, remote server load, etc., it might finish a 500-job batch in 50 seconds one time or 500 seconds the next.

The cron job, which runs every one minute or five, would simply be checking the averages calculated by the end-of-batch routine and stored to a database table.

Seems to me that, at the very least, the end-of-batch routine will need to take into account the time required to complete the most recent batch when calculating the weighted average. The number of images processed and the number of bytes downloaded is likely to be very similar for each 500-job batch. It's the time required to execute the batch that will likely tell most of the story about how fast things are going. The time required is at least half the story.

Seems pretty clear to me that the most recently calculated bytes/interval calculation should be weighted against the existing moving average in proportion to the time required to complete the batch. My calculus muscles are completely atrophied, but I suspect that, in order to determine the actual calculation, I need to make a decision about how much to weight our must recent batch against the accumulated moving average -- weighting the latest one lightly may inordinately favor historical values, making for a dull graph -- and I'm still not clear how the weighting over time should be labeled in such a graph. Favoring our most recent calculation over all the prior ones would likely make for a very jerky graph but could be called something like "bytes/sec of most recently executed batch" -- which might be fine.