Obviously you can't just go in Hotmail's servers and grab people's contacts, but are you talking about services where the site asks you to enter your Yahoo e-mail/password and it grabs the data from Yahoo's servers.
If so, I doubt anyone's going to want to enter their username/passwords unless they're on a very trusted site. In fact, I never use these services just because I don't like the concept of entering my Yahoo e-mail/password anywhere except yahoo.com .
NOTE: The 'Yahoo! Mail' references can be substituted with various other e-mail hosts I'm sure... Hotmail, Gmail, etc. ... I was just using Yahoo as an example.