let's say I have a website with an account creation form. How can I stop people from writing a program that automatically submits the form thousands of times, which would fill up my database with junk and cause my system to come to a standstill?
I don't think you can do anything clever with cookies or "session variables", as these can be automatically obtained.
I am guessing that you need rate limiting:(https://en.wikipedia.org/wiki/Rate_limiting)? Is this it?
Most of the techniques which will help you are anti-automation tools. The following may be a helpful resource in this regard:
http://projects.webappsec.org/w/page/13246938/Insufficient%20Anti-automation
This is the reason many websites deploy tools like re-Captcha and that may even be one of the easiest quick-fixes to your problem but it's not foolproof. There are even anti-anti-automation tools just to get around defenses like re-Captcha.
Another tool you will definitely want to look at is Fail2Ban but you will need to enable the specific configuration to protect your website. This is easy to do but by default, it only protects SSH. In many cases, Fail2Ban by itself will take care of your problem.
If you have a more persistent attacker you may need to look into what is unique about their attack technique and either stop it by writing a custom IDS rule for something like Snort or by using a tool like mod_security. Sometimes rate limiting tricks in IPTables works too if you are either on Linux or can place a Linux box in front of your website. There are a lot of ways to solve these but you may need to dig deeper into how their attacks work. Another low-tech way to handle this is by changing your "submit form" url semi-frequently via a cron job, this isn't the easiest solution but it can be very effective. There are many other techniques as well including things like using GEO-IP locations to block the source of the attacks but the key is looking for anti-automation tools which work with the stack you are currently using. Finally, search if the tools or frameworks you currently use have anti-automation tools built-in, some times these ship with things like web-frameworks but need to be enabled/configured by the user of these tools.
One problem you need to be aware of is the attackers likely already know your current public IP address so simply adding a cloud-based WAF solution, like CloudFlare, may not be a practical or useful option to quickly stop active attacks especially if you have other sites linking to your current website via IP (this shouldn't be the case but it happens). Likewise, you will have to make firewall changes to enforce that ONLY connections from CloudFlare can access your application otherwise it will have no effect at all on attacks directly to your current public IP address. This is still a useful security control for future attacks but may create other short-term problems if not implemented correctly and these won't protect from all types of attacks either. It should be noted that there are a number of WAF-bypass techniques which will allow attackers to continue to submit form requests even with one of these in place.
More information about these problems can also be found here:
OWASP Automated Threats to Web Applications
Finally, in case the attacker changes to brute force in response to your implementation of anti-automation tools you may want to go ahead and read the following too:
OWASP Blocking Brute Force Attacks
Hopefully, this helps.
I would suggest considering one or more of the below mentioned measures :
End-point specific rate limiting
Put rate limiting on the account creation url on your website.
reCAPTCHA
Assuming users create account by visiting account_creation page on your website, you can introduce reCAPTCHA during this process. Google's reCAPTCHA
It helps you detect abusive traffic on your website.
Web Application Firewall (WAF)
Consider using any web-application-firewall.
Example : Open source WAF - Modsecurity
Using Third Party Providers
Use any third party solution to be protected against such attack.
Example : Cloudflare also provides web application firewall. Take a look at this article.
Have you heard of completely automated public Turing test to tell computers and humans apart CAPTCHA? I'm sure you've clicked on [ ] I'm not a robot on many sites using Google's free reCAPTCHA. That's how you stop robots from using your form.
In addition, you should require email address validation. Surprisingly many sites still allows account creation with any email address.
Email them a confirmation link and don't create the account until the link has been followed.
This is not foolproof but it'll take a somewhat more determined attacker to automate this.
