Had more or less the same problem on HP-UX 10.20.
I first tried to install the latest binary distribution (3.23.49 i think).
The mysql_install script terminated and reported a mysql core dump, which is probably what you have encountered.
I tried to remedy the situation by compiling the sources but ran into even more problems-you can see a whole bunch of relevant messages in the mysql.org mailing lists.
I am a total novice on HP-UX systems, so I maybe missing something here, but this is how I solved the problem:
Apparently, the setrlimit is just a warning-it is not the root of the problem. The setrlimit warning probably has to do with the max no of tables simultaneously open in mysql. Anyway, you can increase the limit if you enter SAM, under Kernel options. Increase the maxfiles variable, then rebuild kernel and restart your system.
You will still see the warning, but with the increased value.
This will not solve the coredump problem, however. I had to install a series of system patches which are supposed to fix DCE library problems:
PHCO_23684
PHSS_23672
PHSS_19739
(patches available from hp support site)
I also installed the XSW800GR1020 general release patch bundle (which contains a whole series of patches) but I think the above individual patches should be enough.
Check installation instructions for each patch-some may have prerequisites which are not installed on your system, and there are also post-installation instructions-in most cases you need to reboot your system.
After installing the patches, the hp-ux binary worked ok.
good luck
