I haven't tried CI, Kohana, or Cake; however, I have used Zend extensively.
Zend isn't your typical "framework". Each component is able to stand on its own which means you can use their ACL or DB or Gdata extensions without actually using the entire library. So while there are probably around 100 or so files, you really can get away with just a few.
Now, I've found a setup that works extremely well, and keeps the framework "light". I've got a dedicated server and I put the framework in a shared folder "/usr/share/php/frameworks" which has the Zend folder with the framework inside. So any site can use the framework just by saying "include('Zend/*.php');" and it will work. So your application can be fairly lightweight, while the framework is maintained server-side to be the most up-to-date version.
I like the ZF mainly because it can be used independently from the rest of itself. But CodeIgnitor seems interesting and seems to fair better for performance.