ceanderson,
Although I have never done this, here is a thought...
If your site has relatively low-volume, you can have it save the failed login attempt in a Database (with the timestamp).
If you are using sessions, just save the number of attempts, once it reaches 5, save the login name in the DB, with a timestamp.
Additioanlly, if they close their browser, or log whatever, it will not allow logins from that the login name, or however you would like to limit it)
Once they successfully log in, or if they attempt again (after the 10 minutes) just delete the previously stored record (to prevent it from getting too bulky with junk data).
Although I am extremely sure there is a much better way to do it, in theory, this should work.