Dunno if your still looking for this, but you can try a multi-stage regular expression... ie: something like:
$tmp = $string
$tmp =~ s/[a-zA-Z-]//g; # remove all the valid characters from a temp copy of the string
$tmp =~ s/(.)/\$1/g; # Create a "pattern" of characters we don't want
$string =~ s/[$tmp]//g; # remove unwanted characters...
