I've wanted something simliar myself, I dont have the money ($1500) for WHM/Cpanel but I've been hoping to learn enough PHP to possible do some tasks automaticlly... I know if possible to work with the filesystem, create a folder, its possible to edit a file so you could append information to the end of a configuration file like apaches http.conf for virtual hosting and possible a simliar thing for ftp.
I havent seen any systems, plesk, cpanel are all for paid/reseller hosting plans nothing free that would enable you to easily let customers signup and get instant access or anything.
You would know perhaps but i'm sure the interface for Cpanel uses CGI/Perl or something in combination with shell scripts (.sh) to do whatever it does and you might be able to learn something from looking at those. Also could be binary files that you wont be able to view.
I run a windows 2000 server so its a pain to setup a user with active directory/exchange email then G6 ftp server and finally email them back with the information.
If i could have a form to enter all the info once and have each item setup that would be great.
I've also thought about a user area where they could view online a directory list, perhaps edit files online, create/delete MySQL DB's. All of that is possible with php, just have to learn enough to do it.
As for what your looking for, Im not quite sure how to go about it, if you do figure out sometihng let me know. I'm working on setting up a few linux boxes to make it a bot more possible perhaps using the method i described above.