Yes it is possible, but not pretty. You could do something like:
UPDATE tablename SET num = SUBSTRING(LPAD(num+1, 5, 0) FROM -3);
This will perform like you asked, ie, once it gets 999+1 it will return to 0, but, like I said, it's not pretty. Effective yes, pretty no. (shrug)
As far as making it a MySQL function, yes it is possible, but only with the MAX release of the newer versions... They are supposed to have support for User Defined Functions, but I've never used it before.