While Netlify provides a variety of automated services for DDoS detection, rate limiting, and traffic blocking to keep your sites and your accounts in good health, customers on our enterprise plan can now take additional preemptive measures to prevent abuse and control how your sites respond to different traffic patterns through our highly-customizable Rate Limiting feature.
TL;DR
In this guide, we’ll look at some common patterns of malicious traffic that can impact sites and applications on the web, and examine the ways that you can take extra steps to safeguard against these using our Rate Limiting controls.
Why these controls are necessary
At Netlify, we aim to simplify the work of building and serving sites and applications as much as possible by providing developers with carefully designed and engineered defaults. There are many aspects of web development that can adopt the same default behaviors and characteristics letting you concentrate your efforts on the aspects of your content and user experience that differentiates you and makes your sites valuable.
Sometimes though, particularly for more exotic or high profile sites and applications, the ability to override or customize the default services and behaviors that you inherit by using Netlify can be important. Our defaults are a good fit for the majority of cases, but not every case, so we also offer the ability for power users to take a little more control.
Rate limiting is a good example of such a feature available to those on our enterprise plan. Some sites might have unusual traffic characteristics that you need to accommodate, or elevated attack risks that manifest in a specific way. Over-and-above our built-in safeguards and defaults, you can now take even more hands-on control.
Let’s look at a couple of examples:
- webscraping
- API abuse and overload
Gathering rules for reuse across teams
To make the creation, use, and management of these rate limit rules more convenient across the sites and applications in your team, you can bundle these rules in a ruleset with descriptive names like “General abuse prevention” and then apply this ruleset to all existing and new sites as a Rate Limiting team policy.
If you ever need to customize a ruleset for an individual site, you can still create and apply a different ruleset to a site and override the team policy. This lets you customize rate limit rules for certain sites that may be having a security incident or just need different types of protection.
Ok, with that in mind, to those examples…
Example: Preventing web scraping
Web scraping is the process of collecting data from websites using automated tools. It involves fetching and parsing your website content to gather information for various purposes like research, analysis, or aggregation. While you might be delighted for your content to be indexed and made discoverable by search engines that do this in a responsible way, malicious or poorly implemented web scrapers often saturate your site with large numbers of requests for your resources and can result in some undesirable loads and bandwidth usage.
Setting up a rate limit rule can help to prevent web scraping.
The result of setting a rate limit rule
Once this rate limit rule is triggered, Netlify will make use of an HTTP response code intended for this exact scenario. Returning a 429 status code blocks the relevant requests for 60 seconds, effectively throttling the offending automated requester and in some cases making it timeout, fail, and give up.
How to set a rate limit rule to prevent web scrapers
This is managed in the Netlify admin.
-
As a Team Owner, go to Team settings > Access & security > Rate Limiting > Team Rulesets. Choose to create a new ruleset or edit an existing ruleset.
-
Add a rule to your ruleset. Give your rule a descriptive name, such as “Prevent web scraping.”
- Define when the rule will trigger with at least one rule condition that can detect web scraping. What might detect that? There are some telltale user agents that we can look out for.
In this example, the rule will trigger if a header matches different
User-Agentvalues. You can create a rule condition for each unique value or just one rule condition with the regex expression(node|python|okhttp|fasthttp|Go).\*.
-
Define the rate limit for this rule. The optimal number of requests will depend on your use case. For this example the rate limit is 2500 requests for a 60 second interval.
-
Decide the optimal way to calculate requests for your use case. Since web scraping is usually done by a small set of IP addresses, the Per domain and address option could be the best option so that you target specific IP addresses that have exceeded the limit.
If you want to protect your site against DDoS attacks and control bandwidth usage, we recommend calculating requests with the Per domain option.
-
Be sure to save your changes. This rule will apply to any sites where this ruleset is applied.
Example: Prevent API overload
Some sites provide APIs and services to drive their user interfaces or they may even exist purely to provide an API for others. Netlify Functions are excellent for this!
To keep the usage of such APIs under control and to prevent API abuse, we recommend you set up a rate limit rule for your APIs.
The result of setting a rate limit rule
As with the previous example, when this rate limit rule is triggered, Netlify will begin returning a 429 status code to appropriate requests which blocks the relevant requests for the the duration you specify.
How to set a rate limit rule to prevent API overload and abuse
Add a rule condition where the rule is triggered when a request path matches your API with a regex expression like /api/v1/.\* :
This rule becomes active once you save it in a ruleset, apply it to a site, or set this ruleset as your team policy.
You have the ability to get deeper and more granular with your rule conditions than in these examples. Take a look at the documentation for more.
Further reading
- More details in the documentation
- Launch announcement on the Netlify blog