Time to try multithreading ?

sneakyimp

I'm working on a distributed application. There's a center server (the Manager) that will handle socket connections from a variety of web-crawling clients. The basic idea is that the Manager will direct the crawling operations of the various clients.

I have a nice socket server for the centralized Manager machine. It seems to work well enough.

As I contemplate the client script that will be running on many machines, I have been wondering a couple of things.

First, I'm wondering if I should try to create some kind of multithreaded script. The clients will continually check the socket connection to the Manager to look for new instructions. At the same time, they'll use cURL to fetch web pages from various remote servers. I expect to be calling either [man]socket_select[/man] or [man]socket_read[/man] to determine when I need to read data from the socket just in case new instructions have arrived from the Manager. I also expect to be waiting around for cURL to respond with whatever page we happen to be crawling. Seems like a lot of waiting which will result in poor performance unless I can somehow get at least two threads running at once. Ideally, I'd have multiple threads for the page crawling bit so I can keep the computer busy fetching new pages while it's waiting for the requested ones to arrive.

I'm wondering if I might be able to set up multithreading for this client script but I've read an ominous warning about how PHP doesn't handle multithreading well in an apache mpm environment but am wondering if that applies to CLI/daemon applications. I have some experience writing multithreaded code but it's been awhile. I know I have to avoid race conditions, deadlock, i must be careful with shared resources, and i should try to make sure my functions are reentrant. Anyone have experience writing multithreaded php apps?

Failing some multithreaded magic in this client script, I was hoping for a little advice on [man]socket_set_nonblock[/man]. The docs are pretty scarce at php.net on this. I'm hoping to avoid loops that are constantly calling a socket_read command which simply fails -- that loop would consume tons of CPU time methinks. Is it possible to set the read failure timeout or something?

sneakyimp

OK this has now become much more confusing. Apparently [man]socket_set_nonblock[/man] does not affect [man]socket_read[/man]. I have written a super primitive client for testing and the call to socket_read blocks despite my use of socket_set_nonblock.

Here's the primitive server for testing:

// test script

// set some variables...use localhost for now
$host = "127.0.0.1";
$port = 1234;

// don't timeout!
set_time_limit(0);

// create socket
$socket = socket_create(AF_INET, SOCK_STREAM, 0) or die("Could not create socket\n");

// bind socket to port
$result = socket_bind($socket, $host, $port) or die("Could not bind to socket\n");

// start listening for connections
$result = socket_listen($socket, 3) or die("Could not set up socket listener\n");

// accept incoming connections and
// spawn another socket to handle communication
$spawn = socket_accept($socket) or die("Could not accept incoming connection\n");
echo "spawned socket\n";

sleep(5);

$msg = "This is message\n";
socket_write($spawn, $msg, strlen($msg)) or die("Could not write output for msg\n");
echo "wrote msg\n";


// close sockets
socket_close($spawn);
socket_close($socket);

Here is the client:

$host="127.0.0.1";
$port = 1234;

// create socket
$socket = socket_create(AF_INET, SOCK_STREAM, 0) or die("Could not create socket\n");

// connect to server
$result = socket_connect($socket, $host, $port) or die("Could not connect to server\n");

// prevent the socket from blocking so our loop continues to run
socket_set_nonblock($socket) or die("Could not set socket to non-blocking\n"); // DOESN'T WORK?


$i = 0;
do {
        $i++;
        echo "=== read iteration #$i ===\n";
        $incoming = socket_read($socket, 1024, PHP_NORMAL_READ);

    if ($incoming !== ""  && $incoming !== FALSE) {
            echo "msg $i:" . $incoming . "\n";
    }
    if ($incoming === "") {
            echo "msg is empty string\n";
    }
    if ($incoming === FALSE) {
            echo "msg is FALSE\n";
    }


    echo "loop $i complete\n";

    // we don't want the loop to iterate too quickly
    sleep(1);
} while ($incoming !== "" && $incoming !== FALSE);

echo "loop completed\n";

socket_close($socket);

I opened two SSH windows to my server and first ran the server script and in the 2nd window ran the client script. Here's the server output:

$php server.php
spawned socket
wrote msg

Here's the client output:

$ php client.php
=== read iteration #1 ===
msg 1:This is message

loop 1 complete
=== read iteration #2 ===
msg is empty string
loop 2 complete
loop completed

I was expecting to get about 5 iterations of the client loop before the message from the server finally arrived but I get only 2 iterations. The first iteration is for the one message sent by the server, the second one is when the server closes the socket connection.

WTF? Unfortunately, the docs are quite poor on these functions. More than ever this appears to necessitate that I write a multi-threaded program. I guess I'll work on socket_select in my client or something. Perhaps that will work.