Read-only polling failing due to esource contention when reading/writing a data file?

sneakyimp

I have a very very urgent request.

I'm trying to receive data via a POST operation in one PHP function and store it anywhere I can such that the data can be inspected by hordes of curious visitors via another PHP function.

I have tried writing the data to a super-simple MySQL db, to a file, and to a cache. In every case, there seems to be resource contention to get at the data quickly. Basically, when one script is writing to the data file once per second, it appears to lock out ALL visitors.

More specifically, I have a loop that performs a POST operation to my gateway.php script:

public function send()
		for($i=0; $i<10; $i++ ) {

		$this->send_one();

		// accelerometers will be sampling approx 250ms (ie: 4 times per second)
		usleep(1000 * 1000);
		echo $i . "\n";
	}
public function send_one() {

	$cu = new SMBLE_curl_utility();
	$cu->no_delay = TRUE; // sets CURLOPT to prevent nagle's algorithm


	$coords = array(); // some random data
	for($j = 0; $j<self::NUMBER_OF_POINTS; $j++){
		$coords[$j] = implode(",", $this->pick_random_coordinate());
	}

	$data = self::NUMBER_OF_POINTS . "," . implode(",", $coords); // just a string of data

	// this just makes a simple POST operation to the url which receives the data
	$cu->post("/gateway", array(
			"d" => $data
	));
}

When that loop runs, any scripts polling the data file are timing out.

Can anyone suggest what this might be? Is it that my curl POSTs are maintaining some consistent database/file/cache connection that is locking up the data file to prevent reads? Or does it have something to do with my server keeping the HTTP connection open for my send() function and this somehow causes a lock to be estagblished on the data?

Derokorian

I think its more important to see the file handling code to be able to offer suggestions. As a thought, try to make sure all your file operations are autonomous and non-blocking. look at http://www.grobmeier.de/performance-ofnonblocking-write-to-files-via-php-21082009.html#.VZwfGvlVhBc

sneakyimp

Ok well I've tried it a few different ways. The file approach looks like this:

	/**
	 * public method to receive data via POST from the trade show demo
	 */
	public function index()
	{

	try {

		if ($this->input->method() != "post") {
			show_error("This page must be accessed via POST");
		}

		$csv = $this->input->post("d");
		$input_data = explode(",", $csv);

		// blah blah blah...generate some data...not a lot

		$output_data = array(
			"microtime" => microtime(TRUE),
			"sensor_count" => $sensor_count,
			"coordinates" => $coords
		);

		// write this to the data file
		$this->write_sensor_data_file(json_encode($output_data)); // results in about 100-200 bytes of data


	} catch (Exception $e) {
		show_error("An unexpected internal error occurred");
		$this->debug_log->write("Exception caught:\n$e");
	}
} // index()

private function write_sensor_data_file($json){
	$data_file_path = dirname(BASEPATH) . self::DATA_FILE;
	$fp = fopen($data_file_path, "w"); // w, as I understand it, will just truncate the file and write it without bothering with locks, etc.
	if ($fp === FALSE) {
		show_error("Unable to open file");
	}
	fputs($fp, $json, 1000);
	fclose($fp); // close file immediately
}

The data polling operations are handled thusly:

	public function poll_file() {
		$json = $this->retrieve_latest_data_json_file();
		echo $json;
		die();
	}

private function retrieve_latest_data_json_file() {
	$data_file_path = dirname(BASEPATH) . self::DATA_FILE;
	$fp = fopen($data_file_path, "r"); // r should only request read access
	if ($fp === FALSE) {
		show_error("File failed to open");
	}
	$json_data = trim(fgets($fp, 1000)); // basically read the entire contents of the file up to 1000 bytes
	fclose($fp);

	return $json_data;
}

sneakyimp

I'm wondering if it might be possible to simply write to memory rather than a file. I'm trying to set up codeigniter's caching, but it seemingly requires APC on my machine. Seems to work, but also slow, which makes me think it's writing to a file. How to find out?

sneakyimp

I've tried altering my file routines that read and write to the data file as suggested in the article you linked but I still get the polling scripts timing out when I run a script that POSTs data once every second.

	private function retrieve_latest_data_json_file() {
		$data_file_path = dirname(BASEPATH) . self::DATA_FILE;
		$fp = fopen($data_file_path, "r");
		flock($fp, LOCK_UN); // unlock the file!
		if ($fp === FALSE) {
			show_error("File failed to open");
		}
		$json_data = trim(fgets($fp, 1000));
		fclose($fp);

	return $json_data;
}

private function write_sensor_data_file($json){
	$data_file_path = dirname(BASEPATH) . self::DATA_FILE;
	$fp = fopen($data_file_path, "w");
	flock($fp, LOCK_UN);
	if ($fp === FALSE) {
		show_error("Unable to open file");
	}
	if (flock($fp, LOCK_EX)) {
		fputs($fp, $json, 1000);
	} else {
		show_error("Unable to get file lock to write file");
	}

	flock($fp, LOCK_UN);
	fclose($fp);
}

dalecosp

I'm not really smart here, but if you want to read a file in real time, why do you call LOCK_EX to write to it?

And, if it is a file and a Nixen box, what about calling tail(1) on it via [man]system/man or its friends?

If it's Linux, could you write to a file in /dev/shm?

sneakyimp

dalecosp;11048935 wrote:
I'm not really smart here, but if you want to read a file in real time, why do you call LOCK_EX to write to it?

I'm using LOCK_UN to unlock it as much as possible -- unless I've made a mistake? Please do point out any mistake you see.

dalecosp;11048935 wrote:
And, if it is a file and a Nixen box, what about calling tail(1) on it via [man]system/man or its friends?

I've tried tail, will try it again. Any idea why this might improve things? I'm using "r" to open the file for reading. Also, the file is only one line (a few hundred bytes) long.

dalecosp;11048935 wrote:
If it's Linux, could you write to a file in /dev/shm?

This is new to me. No idea what you are suggesting?

Derokorian

sneakyimp;11048923 wrote:
I'm wondering if it might be possible to simply write to memory rather than a file. I'm trying to set up codeigniter's caching, but it seemingly requires APC on my machine. Seems to work, but also slow, which makes me think it's writing to a file. How to find out?

APC is incredibly fast, uses no disk access at all, and should only set to make things faster... However, if the system is misconfigured and you're relying on swap space then APC will be the same as disk R/W since swap is stored on disk.

sneakyimp

I tried a lot of methods:
writing to DB
writing to flat file
storing via APC
storing with memcached

All of these methods have the same result: any page that is polling for the data has all of its poll requests time out while the data simulation script is running. This is mystifying to me.

But I seem to have made a breakthrough. The original testing scheme works like this
1) I access example.com/controller/send in a browser. This send method runs a loop 100 times. Each iteration of the loop generates a random data payload of maybe 100 chars and POSTs it to my gateway url which is example.com/controller/gateway
2) example.com/controller/gateway receives the POST request and writes to db/file/apc/memcached
3) I access example.com/controller/chart in my browser which loads a chart depicting the data and starts polling example.com/controller/poll at some interval from 250-1000ms
4) example.com/controller/poll reads the last record written to the db/file/apc/memcached and returns it as AJAX to re-render the chart.

THIS IS SLOW AS CHRISTMAS. If the send method is running to generate data, it causes the data polling requests to time out until the data generation is complete -- despite the fact that the data generation script is probably sleeping for most of its execution lifetime. And also despite my best efforts to unlock files and/or prevent any script from hogging them.

!!!HOWEVER!!!, if I run the method send as a CLI script instead of accessing it with a browser via apache, then this thing blazes along. While the CLI script is running, I can have five different browsers open, each polling example.com/poll at 250ms and not one of them ever times out. This is how one runs a codeigniter method via CLI:

php /var/www/html/index.php controller/send

This is fast and flawless.

What is going on here?