Ah, I had a Eureka! moment in the shower, but no, I did not run out naked to post this 🙂
I propose that you have one server to host the precious repository and another server to host the x_files repository.
The precious repository will be directly accessed, both read and write, by you and your trusted collaborators. The x_files repository will be a clone of the precious repository, periodically pulling from the precious repository (e.g., a cron job to pull every 5 minutes or something).
You can setup gitolite on the server hosting the x_files repository. Collaborator X will have read only access to master and all branches except for feature branches with a prefix that will not conflict with any of the feature branches of your collaborators (just in case), or maybe use personal branches (I'm not sure).
Whenever Collaborator X wants to work on a feature, he/she will branch from master into a feature branch and work on it, pushing to the corresponding feature branch on the x_files repository. When you are notified that the feature has been baked to perfection (in the opinion of Collaborator X), you will change your remote to point to the x_files, then pull the feature branch. Upon doing this, you can use git diff/difftool as per normal. When you are ready to merge into master, you do a squashed merge. I believe that when you commit the squashed merge, it will automatically use your username/email, but if not you can override with --author. Then, you change your remote to point back to precious, then push.
This way, precious will only see the changes committed with your username/email. master on x_files will be updated with that commit. The feature branch on x_files can be discarded instead of having to deal with the divergence, with a new feature branch created for the next feature based on the updated master.