I’ve experienced a DDoS attack on my web service for the first time
Many things happen while you are building a SaaS. They sometimes make you happy, sad, angry, grateful, depressed, excited... In retrospect, you can see those happenings were fun memories after all. Last weekend, I experienced a DDoS attack against my small web service for the first time. Obviously, it was a bad happening. I’d like to talk about this incident and how I handled it.
Hi, it’s Takuya. I’ve been developing and operating Inkdrop — a cross-platform Markdown note-taking app — for more than 3 years alone. It’s a very niche SaaS with less than 20k users, which I’m living off of. You can see how I built it through my other blog posts. The server load is basically low and calm because it’s not for mass consumers. I don’t get a traffic spike because of getting covered by huge media or something. So, I can focus on improving the app instead of spending a lot of time on operating servers. But then it happened.
30k fake accounts have been created by an attacker
It was Saturday, (and worse,) my wife’s birthday. We had late lunch at a restaurant and took a nap at home. I didn’t check emails. After getting up from the happy afternoon nap, I noticed an alert email from AWS CloudWatch which reported the high CPU utilization on the CouchDB server on EC2. I jumped into the console and saw the server was much busier than usual. What’s going on? Breathed deeply. 😨
I found that too many accounts are being created (34k as a result) despite the fact that Google Analytics does not indicate any high volume of traffic. Then I recognized it’s an attack.
I decided to tweet about this as an incident first.
I stopped the API server right away so that the attacker couldn’t create more fake accounts. Then I looked into those fake account data and found that they are attacking from various IPs, which tells me it’s a DDoS(distributed denial-of-service) attack. In this case, you can’t deny requests from the attacker based on the source IP address. So, what is the purpose of their attack? I found that all the accounts have the same name like this:
"firstName": "ПОЛУЧИТЕ ДЕНЬГИ https://bit.ly/xxxxxx ",
"lastName": "ПОЛУЧИТЕ ДЕНЬГИ https://bit.ly/xxxxxx ",Okay, it’s Russian and a shortened URL. “ПОЛУЧИТЕ ДЕНЬГИ” means “Cash in” and the link was to a phishing page:
After a short while, I realized that their purpose is not to claim money from me but from all the people who would get the signup-confirmation emails. I understood my server was exploited to send bulk emails to the target people who seem to speak Russian. The victims got an email something like this:
Terrible. But one thing that relieved me was that their purpose was not to destroy/steal my user data. The existing users basically don’t get any big issues with this attack. Fortunately, my SES(Simple Email Service) account didn’t get banned by AWS because most people who received the scam email didn’t complain about it:
Well, how can I stop that?
Implemented reCAPTCHA v3
The problem is that bots can call my user registration API as many times as they want. Let’s restrict it. As we know they want to send phishing URLs to people, it’d be good to decline new accounts whose name includes ‘http’. That’s very easy to do. But it will not work for unknown attacks. So, I decided to implement Google’s reCAPTCHA v3. Luckily, I already know about it.
reCAPTCHA v3 returns a score for each request without user friction. The score is based on interactions with your site and enables you to take an appropriate action for your site.
Unlike CAPTCHA, it doesn’t require a user to correctly evaluate and enter a sequence of letters or numbers perceptible in a distorted image displayed on their screen, which would impact some conversion rate. Besides, it’s free and the implementation is very easy. I’ve done it in 30 minutes. You will see the logo of reCAPTCHA on the signup page:
Check out the reCAPTCHA’s website to learn how to use it:
I don’t see any false negatives at the moment. It scores humans correctly as humans (≥ 0.5):
So, it works fine. A few hours later, the attack stopped and it didn’t happen again. It was a..tough lesson.
By the way, Patrick told me another option:
It looks like a very simple request validation like oAuth. You can increase the complexity by using a hash algorithm like MD5(maddr + nonce + password) where nonce is a random number or Unix timestamp. But if you do so, I’d recommend you to use reCAPTCHA v3 instead because you can implement it with the same amount of effort. Other options would be:
Having a honeypot field looks promising for such form submissions. It’d be nice to also check other vulnerabilities on the APIs. I also recommend you to set a monthly budget limit for your AWS account or something like that just in case so that you won’t be charged for the massive usage by attackers. Thank you for all the advice!
And now, I’m cleaning up the 30k garbage accounts while writing this article :)
Thank you the attacker for ruining our weekend and my wife’s birthday. But we’ve already celebrated the day before the attack.
Thank you for all your support!
Homepage: https://www.inkdrop.app/
Send feedback: https://forum.inkdrop.app/
Contact us: contact@inkdrop.app
Twitter: https://twitter.com/inkdrop_app
Instagram: https://www.instagram.com/craftzdog/
