We restrict the use of the public dnswl.org nameservers to 100’000 queries per day for all organisations using it for free. With this limitation, we want to keep the traffic for all public mirrors (some of which are donated) at an acceptable level (currently 100 to 200 GByte per month).
Those with a higher load are intended to get a paid subscription for an rsync download and access the data locally.
Unfortunately, it is not straightforward to enforce these limits. DNS does not make it easy to identify an administrative contact behind a query source, and many DNS setups make it difficult even for the actual administrator to identify current query behaviour.
Dnswl.org has historically taken a “light” approach for the enforcement of the 100’000 queries per day limit, basically counting on the honesty of all users and spending considerable time to identify and contact administrators of query sources going way over the stated limit. Given a growing number of such abusive query sources, this manual approach does not scale well.
The “light” approach to enforcement also means that we err on the side of caution: we do only aggregate usage data from a selection of nameservers, and we only sample the data from the nameservers where we collect the usage data. Thus, we sincerely underestimate the actual usage by design.
Detecting and limiting abusive query rates
We are now taking additional action to ensure that our public nameserver infrastructure remains accessible for the tens of thousands of free users. These steps include:
- If nameservers can easily be linked to an organisation (rDNS exists and is non-generic, whois on the IP points to a promisingly identifiable organisation), we will send one notification to the abuse contact for the domain (using whois.abuse.net lookups).
- If no meaningful response is received to the above notification within a couple of days, the querying nameserver will receive a “REFUSED” return code instead of the actual DNS answer. Due to the way some DNS resolvers work, this may result in a higher query rate, since the resolver just tries it again and again to get an answer.
- If still no action is taken, and if the abusive rate of queries goes on for a longer time, the querying nameserver may get a “listed, hi” response.
- Those querying nameservers doing more than 1 mio queries per day for a longer time may get a “listed, hi” response.
- In counting the amount of queries, we aggregate networks belonging together (eg across multiple /24-sized networks, or having rDNS in the same domain etc).
- There are some organisations trying to play a “whack-a-mole” game by hopping between multiple public nameservers or similar techniques. We are not interested to play such games, but reserve the right to act accordingly.
Note that when a nameserver gets a “REFUSE” message from dnswl.org, it will likely get a similar response from other black- and whitelists as well, and the spamfilter will seriously underperform. It should therefore be in the administrators won interest to fix such a situation.
“listed, hi” response
In the extreme cases listed above, and for a limited time until query rates have gone down to the acceptable limits, we may return a special answer code to all queries, 127.0.10.3. The “10” indicates that this is a “special” return code, the “3” stands for “high trust” level.
This will cause that a spam filter will mark all mails as coming from a highly trusted server, and will thus result in some spams coming through to users.
While we do not appreciate having to cause such negative effects, the carelessness of the administrators concerned leaves us no other choice.
Out of the 50k to 60k nameserver IPs querying our public nameservers every day, less than 0.1% are affected by this stricter enforcement of our acceptable use limits. The number is even lower when considering the (estimated) number of organisations. However, some “big” nameserver providers are sometimes affected (eg Google public DNS).
What to do when affected by “listed, hi”
1. Contact us at admins@dnswl.org. As soon as we have a working administrative/operative contact, there is no reason to continue the “listed, hi” response. Note that while we are spread over multiple timezones and try to act fast on these issues, we do not have guaranteed 24×7 operations.
2. Switch away from a heavily used public DNS server, and use a local resolver.
3. If you are doing more than 100’000 queries per day, you must either get an rsync subscription, or stop using the dnswl.org public nameserver infrastructure.
[Update]
Some statistics on abusive query sources
It should be noted that DNS statistics are not always straightforward, so the numbers should be taken with a grain of salt. All our numbers therefore are heavily erring on the side of caution. Since we only collect and aggregate logs from a selection of our nameservers, and since we are only sampling the data (throwing away about a quarter of all collected logs), the real numbers are about three times as high as we report them.
All query sources which are deemed “abusive” are doing the high query rates for weeks and months. We may take a single day as an example, October 14th 2011, which happens to be a Friday (weekends generally have different patterns, but usually have the same Top N names).
Overall 78k unique IPs were querying the public nameservers on that day, each doing on average of 2’500 queries per day. The Top 100 Query Sources are all doing way above 100k queries each, the Top 20 Query Sources are way above 1 mio queries each. Some examples (aggregated by organisation, as far as this is possible based on rDNS):
Google DNS (8.8.8.8 etc) 33 mio queries (spread over ~ 50 IPs)
Dimenoc.com 6.7 mio (on a single IP)
Dyndns.com 5 mio (on 4 IPs)
Cogentco.com 4.5 mio
Safesecureweb.com 1.2 mio
Bluehost.com 750k queries (spread over 12 IPs)
Of those sources, only one (Dyndns) has promised action after being contacted.
[/Update]