[RESOLVED] ssh2_auth_pubkey_file(): Authentication failed, and Too many open files

[cbj4074]

I'm using PHP's ssh2 PECL extension to issue shell commands to a number of different servers in an automated fashion.

On each iteration of a loop, I create a new instance of my SSH2 class, open a new SSH connection using ssh2_connect(), and issue a command to one of any number of servers. Once the response is received, I disconnect (by issuing the "exit;" command via ssh2_exec()) and unset the variable that points to the SSH2 class instance. This has worked quite well in the past, and I was able to issue several thousand commands without issue.

But, as of late, I've introduced an issue that is causing PHP to report:

Warning: ssh2_auth_pubkey_file(): Authentication failed for my-user-name using public key in file.php on line XXX

This begins to happen after some 600 iterations of the loop have already completed.

After this warning is issued, I receive another: "Too many open files". Once these two warnings occur the first time, they both occur on every attempt thereafter until the loop has run its course.

As far as I'm aware, I'm not opening any file handles (e.g., via fopen() and family), so I'm not sure what the second warning is about.

I'm almost thinking that the limit, whatever it may be, is being reached on the remote side, e.g., I have too many SSH connections open to the remote host, and the remote host refuses additional connections.

Within each iteration of the loop, I'm checking if some files exist, e.g.:

if (file_exists('ssh2.sftp://' . $ssh->getSftp() . $fileName)) {

}

//where $ssh->getSftp() = ssh2_sftp($this->con) and $this->con = ssh2_connect($this->host, $this->port, array('hostkey', 'ssh-rsa'));

Does anything jump-out at anyone?

Thanks in advance for any insights!

[bradgrafelman]

Do you have shell access to the server where you're running this script? If so, which OS/distro are you running?

You could try letting it run for a few hundred iterations and then listing the open files for the PHP process. That might shed some light on what isn't being release properly.

Alternatively... you say "after some 600 iterations", but how many unique connections do you make? Is there a reason why you could keep a pool of connections open all the time and simply use whichever one is appropriate for any given iteration?

[cbj4074]

Hi, Brad; thanks for jumping in here!

I meditated on this a bit more, trying to discern how the two error messages could possibly be related.

I'm using public key authentication after connecting to the remote host, so PHP has to open the key files (both public and private) after establishing each connection. If there is a bug in the ssh2 library, and the file handles are not closed properly, this may explain why when the maximum number of files has been opened, public key authentication fails (PHP can't open the keys).

But then I took your sound advice and let the loop run for a few hundred iterations before executing "lsof -c php". I was appalled to find that every SSH connection remains open! And, I don't see any mention of the key files. The entries look like this:

php     12030 root  953u  IPv4         2742068636      0t0        TCP example.com:59326->example2.com:ssh (ESTABLISHED)

I had noticed before that ssh2_disconnect() is conspicuously absent from the library. How does one initiate an explicit disconnection with the ssh2 library? I thought that calling unset($ssh) on my object would suffice. That didn't fix this problem, so I tried issuing "exit;" via ssh2_exec(). No dice there either, although, I haven't made any effort to determine why that doesn't work just yet.

To answer your question, yes, I do have shell access to the server on which the script is running. The OS is Ubuntu 10.04.

Regarding your last point, what you describe is a much better implementation. The SSH connection will be made to one of only about four remote servers, so opening a pool of 4 connections and using each as needed seems like a much better idea. Thanks for the suggestion!

That said, I do still want to figure out how to implement proper disconnections; this isn't the only place in which I use this SSH2 class.

For what it's worth, if I kill the script (with Ctrl+C) once the warnings begin, all of those open SSH connections are closed immediately.

Thanks again!

[bradgrafelman]

Can you show us more of your code where you tried to execute the 'exit' command and unset the connection? Also, one more idea: [man]ssh2_connect/man returns a resource, so after you execute the 'exit' command (but before you try to unset the resource), try using [man]fclose/man.

[cbj4074]

Sure!

Once the SSH connection is established successfully, I request the sFTP subsystem:

function sftpCon()
{
	$sftp = ssh2_sftp($this->con);
	$this->sftp = $sftp;
}

Then I perform a file_exists() check, as demonstrated in my initial post.

Next, I attempt to disconnect by calling my custom function, e.g., $ssh->disconnect(). The two relevant functions are as follows:

function cmdExec()
{
	$argc = func_num_args();
	$argv = func_get_args();

$cmd = '';
for ($i = 0; $i < $argc; $i ++) {
	if ($i != ($argc - 1)) {
		$cmd .= $argv[$i] . ' && ';
	} else {
		$cmd .= $argv[$i];
	}
}

$stream = ssh2_exec($this->con, $cmd);
if ($stream !== FALSE) {
	stream_set_blocking($stream, true);

	$contents = '';
	while($line = fgets($stream)) {
		flush();
		$contents .= $line;
	}

	fclose($stream);

	return $contents;
}
else {
	$this->log[] = 'SSH command execution failed!';
	return FALSE;
}
}

function disconnect()
{
	$this->cmdExec('exit;');
	fclose($this->sftp);
}

As you can see, I have tried adding the call to fclose() after issuing the "exit" command. Unfortunately, PHP complains:

Warning: fclose(): supplied resource is not a valid stream resource in file.php on line XXX

If I var_dump($this->sftp), the output is "resource(XXX) of type (SSH2 SFTP)".

I know the cmdExec() method is viable, because if I replace "exit;" with something else, like "ls -l", the remote directory listing is indeed returned.

Please let me know if any other information would be helpful. Thanks again, Brad!

[cbj4074]

I finally figured it out after finding this thread:

http://stackoverflow.com/questions/5820089/disconnect-from-ssh2-connect

The key was to call unset($this->sftp). Once I added that bit, sending the "exit" command via ssh2_exec() seemed unnecessary.

My disconnect() method now looks like this:

function disconnect()
{
	unset($this->shell);
	unset($this->sftp);
	unset($this->con);
}

Now, there is only ever one connection at a time listed in the "lsof -c php" output.

Thanks for pointing me in the right direction here, Brad!

[Weedpacket]

Just in passing, note that [man]unset[/man] can take multiple arguments.