I think I found out what is causing it, and it looks like an odd bug in Safari.
When the dialog appears, even though it looks like it does, it doesn't actually have proper focus, so the first click (anywhere on the alert dialogue, not just the button, try it) focuses it, and then clicking on the button works as expected.
I'm not sure whether this is some sort of 'feature' though. For example, once the alert dialogue shows, I can highlight stuff on the page behind it, but only once. If I try to interact with the page after that first time, the dialogue flashes at me as I would expect a modal dialogue to do.