Try doing it this way:
Note, these are back ticks, from the upper left hand side of the keyboard (US keyboard that is) not single quotes...
$res=htpasswd -b pwdfile name password;
print $res.'<BR>';
Note you'll need to include the whole path in the pwdfile name, and httpd or nobody (depending on which id runs the web server daemon) will have to have write permission for that file/directory.