what types of files are being uploaded?
if it is something like images, then you only have to set the images directory as writable.
sure there is a bit of a security risk, but you can reduce it by building a good admin authentication.
if we worried too much about such risk, then php may as well not have functions like copy() and unlink() 😉
and for god sakes.. don't make them or yourself take down things manually!!! hehe .. isn't this why we program? 😃
really though, .. i don't know the specifics of your program or how sensitive those 'down/uploads' are, ... but if it's not very high, just make sure the files get backed up regularly and you shouldn't have a difficult time recovering if anything does happen.
recovering will take a lot less time than manually managing files over time.