tablex,
Without knowing what the script is it is impossible to help.
Most of these tube sites will use a server-side software on a cloud to allow batch converting without affecting website performance.
One thing to note: Its HIGHLY doubtful you're allowed to do this on a shared web host.. so if that's what you're on, you may want to reconsider where its being hosted, if not, all is well.
Any script-based video conversion, would be PHP/Perl Based, and as it runs through an interpreter it may be why the quality is so poor. Again, any compiled application (written in C/C++, etc) would be much to your benefit.
For cost effectiveness, you may want to look into an online video conversion service, there are many out there which allow you to upload via an API.
This will save your server tons of bandwidth and may possibly be more cost effective.
However - to answer your question, we would need to know what script you are using.