The garbage collector runs randomly with a given probability, 1/100 by default. This means that every 100 times you end a session, it will run.
On a very quiet site, or one which uses sessions on only very few pages, it could take a long time before the cleanup happens.
The default session handler should not allow an expired session to be used (it will still be considered expired), but may not clean it up immediately.
This is alright.
If you are storing so much data in your sessions that the size of these files is a problem, then your design probably needs a rethink.
Mark