how to keep bots and mischief out of one's site

sneakyimp

I built a site about 10 years ago that has been running fairly happily but the old codebase is groaning under the strains of growth. We are building a new version slowly but have need for a short term fix.

We got a notification from Amazon SES that we have an unacceptably high hard bounce rate. It would appear that mischevious users (or bots perhaps) are signing up on our site using nonexistent email addresses. That is they are valid in the sense that filter_var will pass them, but they don't correspond to any real email address. When you send email to them (e.g., our 'thanks for registering' email) it bounces because the email address doesn't exist. The vast majority of them have gmail.com for the domain.

I recently had to give up trying to run a PHPBB forum because CAPTCHA has apparently been soundly defeated. I checked on the PHPBB forum and they recommended Question and Answer CAPTCHA instead of image-based CAPTCHA. I eventually started to have bots cracking my Q&A captcha too.

I cannot screen all the users who sign up manually to make sure each application is valid, there are simply too many each day. I also don't think I can implement any kind of process that would send a text message to a phone or call you on your phone.

I'm wondering what the start-of-the-art approach is for preventing this sort of abuse? Any discussion would be appreciated.

johanafm

sneakyimp;11045629 wrote:
When you send email to them (e.g., our 'thanks for registering' email) it bounces because the email address doesn't exist.

Send an activation email before you activate the account.

The problem does not necessarily go away, but you leave dealing with it to email providers such as gmail.

Derokorian

I would take a look at http://portal.areyouahuman.com/installation as the new age of human verification.

sneakyimp

johanafm;11045631 wrote:
Send an activation email before you activate the account.

The problem does not necessarily go away, but you leave dealing with it to email providers such as gmail.

We do send an email when users register and these are the emails that are bouncing. It is these bounces which are the problem. Ideally I would be able to verify that the email address actually exists before sending the email. Since I cannot do this, I want a way to make sure it's a person entering the email address.

sneakyimp

Derokorian;11045635 wrote:
I would take a look at http://portal.areyouahuman.com/installation as the new age of human verification.

I believe we have privacy concerns. Our services are targetted toward students, some of whom might be minors. We also do not sell any information about our users to advertisers, spam lists, etc. I'm a bit concerned about the mention of Advertisers and Agencies, Publishers and Media Companies, and Ad Tech Companies on the home page. I expect our clientele (high schools and universities) might not be exactly thrilled about the use of this service.

EDIT: THANKS for these suggestions, guys.

sneakyimp

I'm still kicking this around. Apparently recaptcha (previously at http://recaptcha.net) has been claimed by google and is now found at

I've been told that I should not be using any system run by google or other people-tracking groups. As I mentioned previously, privacy is a very important concern for this particular website I'm working on. I found a few CAPTCHA options that might work, but I suspect the bots will be pretty good at beating them:
https://www.keycaptcha.com/
https://www.phpcaptcha.org/
http://www.abeautifulsite.net/a-simple-php-captcha-script/

Does anyone have thoughts on Q&A captcha or other novel captcha techniques?

schwim

I use AreYouHuman CAPTCHA. So far, 100% success in blocking, 0 complaints.

sneakyimp

schwim;11046807 wrote:
I use AreYouHuman CAPTCHA. So far, 100% success in blocking, 0 complaints.

There's no doubt in my mind that such systems will work better than the old image-based captchas (which bots can defeat) or any system I might concoct or find to install myself. The issue is one of constraints. My collaborator has made it clear that both AreYouHuman and reCAPTCHA (now run by Google) present an unacceptable compromise of privacy. I've tried to point out that these systems are going to be much easier and work much better, but so far it's not having much effect.

Bonesnap

I have seen weird, off the wall solutions like "What is the third item in our menu?" and then have a text field. In your validation you'd have a predefined value that you can check against.

sneakyimp

Bonesnap;11046855 wrote:
I have seen weird, off the wall solutions like "What is the third item in our menu?" and then have a text field. In your validation you'd have a predefined value that you can check against.

I've implemented this very thing in a PHPBB project (they call it Q&A CAPTCHA) and it was my experience that this is not particularly effective either. The bot programmers have created filters to recognize the most common types of questions so you ultimately spend a lot of time trying to think of more clever questions but this in turn makes it harder on you and your users.

The Keycaptcha link I posted looks pretty interesting, but I don't think it accommodates blind people -- the old image captcha plugins would allow you to hear an audio file of the char sequence.

schwim

The way it works is a real person shows up at your site and answers a couple of the Q & A and then programs the script to output the answers. They don't have to answer all of them, just enough to come up with the right answer during a rotation in which they use different IPs, so they don't end up with the same unanswerable question showing persistently. As more bot herders answer the questions and program their scripts, the more successful bot registrations. That's why you get just a couple initially then later a flood if the questions aren't changed.

If you are ok with them successfully registering but catching them at the posting portion of their visit, you might consider running new users' content through Akismet. I've tied in all my custom scripts as well as forums into the service and the success rate of catching the spam is almost 100%. I just check the content of anyone that has less than 5 clean posts and has been a member for longer than a month. Once Akismet catches it, you can just write a function that will wipe them off the face of the site with one click.

sneakyimp

In the current case I'm dealing with, there is only one point where I need to catch them: when they register. There is no posting or anything like that (which makes me wonder why these stupid bots sign up). Additionally, I have been told in no uncertain terms that userdata of any kind is not to be shared with a third party.

schwim

If you're not allowed to use any of the services that are successful, then I don't see a solution for you aside from devoting your life to developing the next successful technology. The days of throwing up a picture of an orange and asking what it is or asking people to count to 6 are over.

sneakyimp

schwim;11046873 wrote:
If you're not allowed to use any of the services that are successful, then I don't see a solution for you aside from devoting your life to developing the next successful technology. The days of throwing up a picture of an orange and asking what it is or asking people to count to 6 are over.

I think this is kind of what I was thinking and I think I sort of expected it to be the case. I think it's important to point out that using some offering from Google or from AreYouHuman means admitting a pathetic dependency on those-who-would-track-us-and-destroy-our-privacy. This makes me sad but I suppose I will assimilate...resistance is futile.

sneakyimp

So, sadly, I'm looking at implementing old-school captcha in the short term. Serving visually impaired users has been mandated for our site so the only non-borg solution I've found that might do that is securimage. I can't help but wonder if this code is secure or whether it might be some crafty hacker's way of infiltrating servers. I've been searching the code for fopen and curl and so on as a tepid attempt at looking for security holes, but wonder if there might be some better way to check if this library is acceptable for use. It's about 5500 lines of code but I wonder if there might be some kind of Angie's List for code or some automated means of checking the security of a PHP library one might want to use.

schwim

Well, I don't know if it's safe, but I do know that it's been cracked. I had to swap it out for a previously mentioned system for a client that was having an issue with spam floods.

sneakyimp

schwim;11047119 wrote:
Well, I don't know if it's safe, but I do know that it's been cracked. I had to swap it out for a previously mentioned system for a client that was having an issue with spam floods.

SIGH.

Derokorian

You could make a dynamic question field, add 2 number inputs or concatenate 2 string fields together... something like that.

schwim

I feel your pain. I don't mean this flippantly, but bots have gotten better at filling out registrations than those requiring accessibility solutions.

Is there a chance you could offer registration assistance via contact to those needing it and then use a test that's harder on the bots? If yes, I imagine we could come up with something that would stymy the majority of the registration scripts. If not, I really think you're in the situation to where you will have to decide to allow both those needing assistance and the bots or neither.

sneakyimp

Derokorian;11047123 wrote:
You could make a dynamic question field, add 2 number inputs or concatenate 2 string fields together... something like that.

This is called Q&A CAPTCHA by the phpBB community. Apparently bots can also get past this kind of thing. There's an art form to Q&A CAPTCHA but I don't really know what that is. I'm leaning toward this solution.