Indeed, if you sort a number that is stored as a character, it will be sorted by character.
But, if you format your dates properly, that means: including leading zeros, the character order and the numeric order are the same.
nevertheless, you'd have to build a custom ordering query to order by year/month/day. Because, like you say, if the order is changed to month/day/year, the ordering changes.
If the field is a dat however, MySQL will order it correctly by itself.