What happened
In addition to general bot traffic which increased system load, a variety of eFiling hosted websites also sustained a DDOS (Distributed Denial Of Service) attack - thus far, there’s no evidence to suggest these were targeted attacks.
How it happened
Typical bot traffic was safely handled by a variety of counter measures already in place such as network and application firewalls. This attack was spread across a wide range of IP addresses that ultimately circumvented eFiling’s rate limit protection and overloaded multiple servers on which eFiling runs. The attack didn’t contain any malicious payloads thus also not being blocked effectively by eFiling’s application firewall.
The attack started as early as 9:35am but the effects were not felt until much later at approx. 9:55am.
Between 10:00am and 11:00am, eFiling was able to identify and successfully block 226,111 requests, while processing 200+ requests per second – during this time users may have experienced dropped connections due to high server utilisation - until eFiling went completely offline between 10:35 and 11:00am while updates were being applied.
While updates were applied at 11:00am, it took until 11:10am for those changes to take effect across all eFiling websites.
Resolution
The immediate resolution was to identify and block the originating IP addresses as well as adjust settings within our firewalls which took a lot longer than expected and resulted in multiple server restarts.
While updating the local firewall settings and testing the changes, it became clear we needed more effective protection against this size of attack – in response we expanded our existing Google Armour rules to stop traffic at a network level and applied an aggressive short term rate limit to specific pages.
What next
We’re going to be conducting a review of our existing DDOS protection and mitigating services, as well as review other preventative measures.
We’ll be looking to further integrate into Google Armour which is a service designed to stop this type of attack. During this process we’ll be working with our server management company and liaising with google engineers and support team.
A review of internal infrastructure documentation will also be carried out to ensure that our teams are responding to such outage in the most efficient way.