yeah it is related to the overhead associated with creating a PHP instance. This is one reason why you don't want to run PHP as CGI on a web server, because it is a bit slow since each request creates a new instance. FastCGI is better, since it keeps a pool of instances to reuse. Don't think there is anything like this for the CLI though.
I've got quite a bit compiled in (and loading in php.ini) as well, but get about the same time as bubblenut gets... but it nearly doubles if I turn Xdebug on, which I would expect given all that extension has to do. [PHP 5]
You can dynamically load libraries in PHP as well only when you need them, but most people don't use that feature since most hosts just compile everything in instead of compiling most extensions as standalone modules that people can load in when they need to.