OK, after finishing some more experiements I have found a solution that works on all my installations.
Regardless of what config tells you or the manual tells you, always specify the path to libs. Ex:
--with-gd=/usr/local --with-ttf=/usr/local
Not specifyng the path allowed php to build without error, but it would not load, and sometimes would load, but would cause apache to segfault when those functions were called.
However specifiing the path seems to have solved my problem, at least with GD, FreeType, Imap, and ldap. No more failures to load, no more segfaults.
Note. I had to do a make distclean, buildconf, first or apparently configure was not noticing the difference, you might be able to get away with simply removing config.cach etc. I didn't try it.
Is this a possible problem with php build process?
DAve