Yeh I've done that, and yep it all depends on how big the return data is, and where your server is located. If it is local then the speed won't be effected too much, but if it is a remote database server then the speed will be greatly increased. Another factor is the server load, how many people are accessing it at any given moment, we ran into this problem when I first programmed up the news page for my site (oglchallenge.otri.net), we'd have something like 10,000 people visit the site in a day and the server just went nutz and was slow as hell all day, after that I went and selected only the data I needed and we havn't run into that problem since.
So it's all up to those 3 little factors, data size, server location and load.