That operates on the premise that spammers are legitimate business people that will follow proper rules. Unfortunately, the ones that are the problems (such as those that give you a "remove" option, only to sell your now confirmed email address to other spam lists) are the ones that will bypass your whitelist.
Does a spammer care if they can't get in to your personal account? No. They have millions of other addresses to use, and they don't have the time or money to devote to deal with your whitelist. In fact, spammers only need a few people to respond to their millions of spam messages sent out in order to make any sort of profit.
On top of that, most spammers have fake or third-party return addresses, so your email "inviting" them to join your whitelist will be returned as undeliverable (which of course adds yet another email in your inbox you don't want).
For a related link, and some fun, check out www.thespamletters.com 🙂