1) Generally speaking, it's not 'safe' in the sense that once people know you have bank account data you become a target. It's a bit like living dangerously. The degree of safety depends on many things. I wouldn't recommend a windows box with the default install, for instance. The default LAMP install might also have vulnerabilities. It's important that the box be 'hardened' meaning that you use the latest available version of all your software components, turn off all unnecessary services, disable direct login by root, turn off all server signatures in APACHE and turn down your error messages to prevent hackers from getting helpful information about the software installed on your box as they attempt to exploit it. There's basically an endless list of precautions you can take and some are more helpful than others. Start here.
2) Not sure what the 'best' method is but I've heard blowfish is good. You can encrypt using blowfish with the mcrypt library. It may or may not be enabled on a given PHP install. PEAR also has a blowfish implementation. Real world security techniques also use one-way encryption (i.e., 'hashing') which is a different animal and useful in different circumstances such as password encryption. The question isn't really 'which encryption is best?' but rather 'when should i use encryption and what kind should i use?'
3) Like I said before, there's an endless list of precautions you can take. Try keeping up to date on exploits by reading security-related sites so you know if you choice of software might have problems. Read about exploitation techniques. Network topography is also important. Personally, I would try to make sure that my server with sensitive data is not directly exposed to the internet. You should only access the sensitive machine through a proxy or firewall machine which acts as a gatekeeper.
4) i can't think of any FOSS payroll/bank scripts. The very idea makes me nervous.
5) Read up on it a LONG time before you plan to actually implement anything. Perhaps you should talk to those more experienced programmers about their techniques.