Well, you have to rely on something. Look, all your vars are going to take up memory however they are defined. Between calls, the session vars are stored as a disc file, and the larger that is the more disk read/write you'll have. If you need them, and there are a lot or they are large, then you just have to take the performance hit in memory and disk IO.
That said, how else are you going to maintain state between user calls? Whomever told you it was a 'bad' idea to rely on session vars should have told you why and what the 'better' option was; assuming such an expert has one.