- Edited
I'm working on a page which uses AJAX to display viewer counts for a video. I'm told we might have 1000 concurrent viewers watching a video, with each of their browsers polling my AJAX endpoint to check the viewer count every 10 or 15 seconds. That could easily result in 60-100 requests per second to my AJAX endpoint.
My AJAX endpoint, a PHP script, fetches the viewer counts by contacting the MUX[.]com API. I cache the API response by storing it in a file. This cache is considered stale after API_RESPONSE_CACHE_LIFETIME seconds (currently 15 seconds). This mostly works fine.
HOWEVER, concurrency, coupled with the modest latency of the MUX API has resulted in bursts of requests going out to the API. When we have enough viewers, the requests come in fast enough that multiple invocations of the script detect a stale cache and they all launch an API request. This is problematic, so I want to use some kind of locking to control access to the API. I have a variety of questions.
For starters, I want to use flock, which requires you to first call fopen. My testing (Ubuntu 20) indicates this will NOT alter the mtime of a file. Can I be sure of that for other file systems?
$fp = fopen($jwt_cache_file, "c");I think to be safe I need a more comprehensive understanding of the behavior of fopen with param 'c'.
I was also hoping for some critique of my approach with flock and a shared resource (i.e., a cache file). This function get_mux_jwt takes care to use flock to control access to the JWT cache file. This JWT is generated locally, but get cached in a file. I hope to use a nearly identical approach when hitting the MUX API. If anyone sees a problem, or room for improvement, I'd be grateful for advice. I'm especially concerned about deadlock gumming up the entire web server.
/**
* Attempts to retrieve JWT for the specified id_type and video_id, checks cache first
* and if that is missing, empty, or stale, will contact API to get fresh one
* @param string $id_type
* @param string $video_id
* @throws Exception
* @return NULL|string
*/
function get_mux_jwt($id_type, $video_id) {
if (!is_dir(JWT_CACHE_DIR)) {
if (!mkdir(JWT_CACHE_DIR)) {
throw new Exception('unable to create jwt cache dir');
}
}
if (!is_writable(JWT_CACHE_DIR)) {
throw new Exception('jwt cache dir is not writable');
}
// we have to generate a different JWT for each video id / id type
$jwt_cache_file = JWT_CACHE_DIR . $id_type . '_' . $video_id;
$retval = NULL;
$jwt_loop_start = microtime(TRUE);
$jwt_fp = NULL;
while (!$retval) {
// FIRST, check to see if we need to time out
$elapsed_time = microtime(TRUE) - $jwt_loop_start;
if ($elapsed_time >= JWT_RENEWAL_TIMEOUT) {
// TOO MUCH TIME ELAPSED important to free any locked resources to prevent deadlock
if (!is_null($jwt_fp)) {
@flock($jwt_fp, LOCK_UN); // fclose supposed to unlock, so this may be gratuitous but just to be safe
@fclose($jwt_fp);
// TODO perhaps we should return some JSON response here indicating error condition or N/A?
// as currently written, $retval will likely be NULL, and certainly empty
break; // exit the while loop
}
}
// SECOND, check for a fresh jwt in the cache file
$jwt_mtime = filemtime($jwt_cache_file); // returns FALSE if no such file exists or is in unreachable dir
if ($jwt_mtime) {
if (!is_readable($jwt_cache_file)) {
throw new Exception("$jwt_cache_file is not readable");
}
// check if cache file too old, leave 10 second buffer to give us time to hit API endpoint
$max_age = JWT_LIFETIME - 10;
$file_age = time() - $jwt_mtime;
if ($file_age < $max_age) {
// cache file is fresh, however this could be empty
$retval = trim(file_get_contents($jwt_cache_file));
//echo "CACHED JWT: $retval\n";
// TODO - validate the JWT...could be non-empty but still invalid
// rather than return or break, we continue, because retval might be empty
continue; // skip the rest of the while loop and start next iteration
}
}
// if we reach this point, the JWT cache is nonexistent/empty/stale, and we must generate a new jwt
// echo "JWT CACHE IS NONEXISTENT/EMPTY/STALE\n";
// IMPORTANT! To prevent a swarm on the API due to high concurrency, we implement this looping/locking
// open cache file usng "c" (write only, create if doesn't exist, DO NOT TRUNCATE!!!)
// IMPORTANT -- 'r+' will fail if file does not exist and testing suggests this 'c' fopen
// call will NOT affect the filemtime of the file
// which is good because we don't want other processes thinking the file is fresh
$jwt_fp = fopen($jwt_cache_file, "c");
// try to lock the JWT cache file
if (flock($jwt_fp, LOCK_EX | LOCK_NB)) { // USE LOCK_NB SO THIS DOESN'T BLOCK
// WE HAVE EXCLUSIVELY LOCKED THE JWT CACHE FILE. Let us proceed with API request
// IMPORTANT: these are *SIGNING KEYS* and are distinct from the API secret + key
$payload = array(
"sub" => $video_id,
"aud" => $id_type,
"exp" => time() + JWT_LIFETIME, // Expiry time in epoch - in this case now + 10 mins
"kid" => MUX_SIGNING_KEY_ID
);
// TODO might this return a nonempty-but-invalid jwt? should we validate the jwt? could this function call throw an exception?
$retval = JWT::encode($payload, base64_decode(MUX_SIGNING_PRIVATE_KEY), 'RS256');
//echo "NEW JWT: $jwt\n";
// truncate the file...NOTE there's a slim chance some process will jump in and read
// the truncated/empty file before we write the new contents...processes should
// take care to check for empty/degenerate JWT when reading this file
if (!ftruncate($jwt_fp, 0)) {
throw new Exception('Unable to truncate jwt_cache_file');
}
if (!fwrite($jwt_fp, $retval, 8192)) { // some googling says JWT shoudl be under 8K bytes
throw new Exception('Unable to write JWT to jwt_cache_file');
}
if (!fflush($jwt_fp)) {
throw new Exception('Unable to flush JWT to jwt_cache_file');
}
// unlock and close the cache file
flock($jwt_fp, LOCK_UN);
fclose($jwt_fp);
} else {
// unable to lock the file, which means that someone else has locked it
// NOTE that the jwt was empty/stale, so let's sleep and try again
// echo "UNABLE TO LOCK $jwt_cache_file\n";
fclose($jwt_fp); // close the file first
sleep(1);
}
}
return $retval;
} // get_mux_jwt
The only thing I've done recently that is vaguely similar is database-driven, storing a JWT with an expiration timestamp, and to-date not having to really worry if a couple or 3 processes all try to refresh it.)