Does that affect the performance of my server a lot?
This is a good question to ask in a forum... you will get lots of good answers... but no matter what answers you get, you really should test this for yourself.
Learn to use Apache Benchmark (ab). It's a program that will hit your web site hard and tell you how many simultaneous users can be using your site. You can find out whether or not your site will run out of memory, slow down, crash the database server, etc.
All the advice and opinions in the forum are just gut feelings that we have based on the information you give us. Without actually knowing what brand of motherboard, which version of PHP, Apache, MySQL, which Linux kernel, which nic card, speed of your RAM, which CPU, which modules you have installed in PHP and Apache, your site's traffic patterns and how you have all your software configured, our performance estimates are only going to be ballpark estimates.
And besides, if we say that you should be fine and then the site crashes, do you really want to tell your boss, "The people at phpbuilder said it would work". If you load test it yourself, then you will truly know whether or not you're safe.