Category Archives: news

1’000 Self Service users, 3’000 Change Requests

Since the Self Service went live in mid December 2015, we got over 1’000 Self Service user signups, and these users created over 3’000 Change Requests. Of course we are happy about the great up-take of the Self Service in the community.

If you also want to manage your own IP addresses and associated data at, please go to

For those that are already registered, a few points to keep in mind:

  • Just adding your domain name will not provide much value. We work mainly on IP addresses of mail servers, so you should add these IP addresses.
  • When you add IP addresses, make sure they have meaningful reverse DNS entries (PTR records), and that the reverse and forward DNS are consistent.
  • Yes, you can add IPv6 addresses.
  • You will receive regular notifications from about DNS becoming inconsistent and other signs of stale data. In the future, we may expand this to cover eg information from our spam traps and similar mechanisms.
  • We intend to add more features to the Self Service. Check back regularly to see what is new.


Users behind TOR abusing resources

For years, an unknown number of users behind TOR exit nodes have been abusing resources. Apparently for every email they receive, they query via the web, using the search interface open to all users. This causes considerable load on our webservers, the database servers, and our network infrastructure, sometimes causing significant delays for legitimate users.

Despite numerous attempts to a) get these TOR users to notice the abusive nature of their behavior and b) returning empty data to them using the exit address list provided by the TOR project, the abuse did not stop. On the contrary, for every action on our side, there was a workaround on the abusers side.

Obviously, there is only a limited amount of things we can do about it, as long as some people and institutions willingly accept anti-social behavior by providing TOR exit nodes, and we only have a limited amount of time to waste — especially to access data which is readily available in a variety of ways and formats.

As of today, we are redirecting accesses from TOR users to our search interface to, hoping that they will realize the abusive and anti-social behavior and start using our data in one of the accepted and supported methods, which require far, far less resources.

Dear TOR users, if the data is so valuable to you, then why do you insist on using the least efficient way to use it? Self Service Portal

News from the team:

For the past years we used an e-mail based approach for user requests to add, change or remove data. This was rather time-consuming and we decided that we need a better process to handle such requests.

That new process is in the new Self Service Portal. Users can use it to “claim” a DNSWL Id (or create a new DNSWL Id and claim this one right away), to request changes to the meta-data associated with such a DNSWL Id, and most important to add/remove IP addresses.

It should be noted that merely “adding a domain” is pointless. You need to add IP addresses, since the reputation that publishes relies only on IP addresses. No IP address, no reputation data.

Users who successfully claimed a DNSWL Id will get regular reports about changes observed that may negatively impact the reputation of “their” IP addresses, eg DNS becoming inconsistent, and about the approval (or rejection) of change requests. While users can request certain changes, the admin team reserves full editorial independence on any and all changes.

We intend to add further features and functions to the Self Service Portal over time, so stay tuned 🙂

You can give the Self Service Portal a try here:

Change to PGP key

As of Nov 30 2015, there is a slight change to the PGP key we use for signing the download files – there is no more expiration date on that key. Private key and fingerprint remain unchanged.

The updated key can be downloaded here.

“Am I Blocked?”

Starting in early November 2014, we took some additional measures to enforce the query limit on our free nameserver infrastructure of 100’000 queries per 24 hours per organisation. As a result, some people may get a “blocked” result in their spamfilter setup.

In SpamAssassin, this is shown by a “RCVD_IN_DNSWL_BLOCKED” rule in the spam report, and likely a link to which is generic to all DNSxL with similar limits.


Get an rsync subscription if you are doing more than 100k queries per day yourself.

Why am I blocked?

In order to ensure free access to the vast majority of users, and in order to avoid overloading the often donated resources, many black- and whitelists have limits on how much the public infrastructure may be used. At, we set the limit at 100’000 queries per 24 hours. With the caching of DNS records, this equals about 300’000 to 500’000 emails.

Why has this suddenly changed?

We do not have a strict enforcement which would block a query source immediately once it passes 100’000 queries. There is usually a relatively lax enforcement, and when feasible we try to contact the operator before we block it. However, this is often difficult since there is no identification needed to use the services.

Also, some users are using a shared, third-party nameserver. This is fine as long as all the users of such a nameserver combined stay below 100’000 queries. As soon as a user joins such a nameserver with a different useage pattern or overall higher volume, it may consistently bypass the 100’000 mark and may thus become blocked some time in the future.

Starting in early November 2014, we got more coverage of logs from our various public nameserver mirrors, so we now see (some heavy) users which had gone unnoticed before.

What do I do now?

First, find out what is actually being blocked:

  • If you are using a shared nameserver (eg Google’s, they are likely blocked.
  • If you are using a local nameserver, see whether it’s configured to do full name resolution, or if it simply forwards queries to a shared nameserver which is then blocked.

The easiest way to find out if a particular nameserver is blocked is by asking:

$ dig -t txt

This should return (among other lines, omitted here for clarity):

;; ANSWER SECTION: 86347 IN TXT "no"
;; Query time: 36 msec

The “ANSWER SECTION” should return either “no” or “yes”, and the “SERVER” line indicates which nameserver provided this answer. Be sure to run this on the box(es) where your spamfilter resides, and if possible using the same user account as used for the spamfilter.

If you are legitimately doing more than 100’000 queries per 24 hours, you must get a subscription to download the rsync file for local useage. If you can not afford a subscription, and / or if you are active in the email or the anti-spam community, please contact for a “community” subscription.

OK, so I’ll evade the block!

We can’t stop this from happening. But we will eventually detect those playing games,  eg those moving nameserver IPs around. Our goal is not to chase, our goal is to keep the infrastructure – part of which is donated – free to use for the vast majority of users.

But using a shared nameserver improves caching!

In theory, this is correct. In the real world, there will always be free-riders who hide behind such shared nameservers.


Improving IPv6 support has gradually improved the level of IPv6 support over the past months. The easy part is offering services over IPv6:

  • Most public websites are accessible over IPv6 (since well over a year)
  • Incoming and outgoing mails may pass through IPv6
  • Nameservers for both the and zones have a healthy mix of IPv4 and IPv6 (and georedundancy etc)
  • Monitoring adapted to include (hopefully) all services which are also offered over IPv6
  • As a general rule, and wherever available, internal communication between servers happens over IPv6

A bit more work was to enable our internal data handling systems to support IPv6:

  • Database: the main driver for the move to Postgres was it’s stellar support for IPv4 and IPv6 address calculations/storage/manipulation. The migration gave us instant IPv6 support in many important areas (>> 1 year ago).
  • GUI tools: Our internal admin tools had to be adapted in order to display and parse IPv6 addresses (eg screen real estate, validation rules etc)
  • Log collection and parsing tools: Most of our log parsing has been improved to accept IPv4 and IPv6. While the tools have been updated, they have not been rolled out to all mirrors yet. Sufficiently performance for log collection by sniffing on network interface requires some advanced IP packet trickery in Perl 🙂

Work in progress:

  • DNSxL for IPv6 queries has only been defined in RFC 5782 basically identical to IPv4. This is considered a risk, as the vast IPv6 space may lead to amplified DDoS attacks on DNS cache infrastructure. We therefore have some trials for “DNS tree walking”, in addition to the “naive” RFC5782-way. The implications of this will be subject to a separate posting.
  • As the query format has not been sufficiently stabilized yet, we do not yet collect statistics about query content (from which we eg identify new mailservers) to IPv6.
  • Some of our export formats (eg for SpamAssassin) now also include IPv6 addresses.
  • Rsync access for subscribers only runs on IPv4. We may gradually start to add AAAA addresses to the rsync hostname in the near future. We are not yet sure whether this will negatively impact subscribers.

What’s next?

Overall, the amount of traffic over IPv6 is minimal, but we do believe that this will significantly grow over time. All new services will be designed to support both IPv4 and IPv6 from the start, and the few remaining services will either be upgraded or discontinued. Eventually IPv6 is becoming the standard, and IPv4 the after-thought.

Abusive use of infrastructure – new method to enforce limits

Our previous method of enforcing limits caused some concern, both in public and private conversations. The main argument is that causing false negatives is not acceptable in principle, not even for cases of obvious abusive use.

We listened to these thoughts, and have now changed our approach. The criteria for blocking such abusive nameservers are still the same: repeated use way above the 100k / 24 hours limit and no response to reasonable attempts at contacting them. Also, most of the things in our previous news item referenced above are still valid.

However, the technical handling has changed to reduce the number of queries that a legitimate client will actually perform. The new handling has the following technical effects:

  • Affected nameservers see a different “view” of the data.
  • This view will not return any useful data other than the regular entries for “” etc.
  • Some additional changes to make it as easy as possible to identify that your nameserver is blocked.

To see whether you are affected, use “dig -t txt” on the mailserver (or other machine which uses the same nameserver setup).

There is an additional change under discussion with the SpamAssassin team to define a dedicated return value to indicate “blocked for excessive usage” (see this discussion on Bugzilla). With this specific return value, the application (SA in this case) will know to not attempt any more queries until the TTL has expired.

We will continue to watch the situation on our public nameserver infrastructure and will work to ensure that it remains accessible and usable for free for most users.

Abusive use of infrastructure – enforcing limits

We restrict the use of the public nameservers to 100’000 queries per day for all organisations using it for free. With this limitation, we want to keep the traffic for all public mirrors (some of which are donated) at an acceptable level (currently 100 to 200 GByte per month).

Those with a higher load are intended to get a paid subscription for an rsync download and access the data locally.

Unfortunately, it is not straightforward to enforce these limits. DNS does not make it easy to identify an administrative contact behind a query source, and many DNS setups make it difficult even for the actual administrator to identify current query behaviour. has historically taken a “light” approach for the enforcement of the 100’000 queries per day limit, basically counting on the honesty of all users and spending considerable time to identify and contact administrators of query sources going way over the stated limit. Given a growing number of such abusive query sources, this manual approach does not scale well.

The “light” approach to enforcement also means that we err on the side of caution: we do only aggregate usage data from a selection of nameservers, and we only sample the data from the nameservers where we collect the usage data. Thus, we sincerely underestimate the actual usage by design.

Detecting and limiting abusive query rates

We are now taking additional action to ensure that our public nameserver infrastructure remains accessible for the tens of thousands of free users. These steps include:

  • If nameservers can easily be linked to an organisation (rDNS exists and is non-generic, whois on the IP points to a promisingly identifiable organisation), we will send one notification to the abuse contact for the domain (using lookups).
  • If no meaningful response is received to the above notification within a couple of days, the querying nameserver will receive a “REFUSED” return code instead of the actual DNS answer. Due to the way some DNS resolvers work, this may result in a higher query rate, since the resolver just tries it again and again to get an answer.
  • If still no action is taken, and if the abusive rate of queries goes on for a longer time, the querying nameserver may get a “listed, hi” response.
  • Those querying nameservers doing more than 1 mio queries per day for a longer time may get a “listed, hi” response.
  • In counting the amount of queries, we aggregate networks belonging together (eg across multiple /24-sized networks, or having rDNS in the same domain etc).
  • There are some organisations trying to play a “whack-a-mole” game by hopping between multiple public nameservers or similar techniques. We are not interested to play such games, but reserve the right to act accordingly.

Note that when a nameserver gets a “REFUSE” message from, it will likely get a similar response from other black- and whitelists as well, and the spamfilter will seriously underperform. It should therefore be in the administrators won interest to fix such a situation.

“listed, hi” response

In the extreme cases listed above, and for a limited time until query rates have gone down to the acceptable limits, we may return a special answer code to all queries, The “10” indicates that this is a “special” return code, the “3” stands for “high trust” level.

This will cause that a spam filter will mark all mails as coming from a highly trusted server, and will thus result in some spams coming through to users.

While we do not appreciate having to cause such negative effects, the carelessness of the administrators concerned leaves us no other choice.

Out of the 50k to 60k nameserver IPs querying our public nameservers every day, less than 0.1% are affected by this stricter enforcement of our acceptable use limits. The number is even lower when considering the (estimated) number of organisations. However, some “big” nameserver providers are sometimes affected (eg Google public DNS).

What to do when affected by “listed, hi”

1. Contact us at As soon as we have a working administrative/operative contact, there is no reason to continue the “listed, hi” response. Note that while we are spread over multiple timezones and try to act fast on these issues, we do not have guaranteed 24×7 operations.
2. Switch away from a heavily used public DNS server, and use a local resolver.
3. If you are doing more than 100’000 queries per day, you must either get an rsync subscription, or stop using the public nameserver infrastructure.


Some statistics on abusive query sources

It should be noted that DNS statistics are not always straightforward, so the numbers should be taken with a grain of salt. All our numbers therefore are heavily erring on the side of caution. Since we only collect and aggregate logs from a selection of our nameservers, and since we are only sampling the data (throwing away about a quarter of all collected logs), the real numbers are about three times as high as we report them.

All query sources which are deemed “abusive” are doing the high query rates for weeks and months. We may take a single day as an example, October 14th 2011, which happens to be a Friday (weekends generally have different patterns, but usually have the same Top N names).

Overall 78k unique IPs were querying the public nameservers on that day, each doing on average of 2’500 queries per day. The Top 100 Query Sources are all doing way above 100k queries each, the Top 20 Query Sources are way above 1 mio queries each. Some examples (aggregated by organisation, as far as this is possible based on rDNS):

Google DNS ( etc) 33 mio queries (spread over ~ 50 IPs) 6.7 mio (on a single IP) 5 mio (on 4 IPs) 4.5 mio 1.2 mio 750k queries (spread over 12 IPs)

Of those sources, only one (Dyndns) has promised action after being contacted.



Sometime today, Nov 27 2010, amidst the hardware problems with one of our servers, we silently passed the milestone of 100’000 active entries in the database (it’s slightly more IP addresses, because there are also some ranges of IP addresses in our database). That data is used by about 50’000 organisations world-wide.

Based on our statistics, we cover about 90% of the volume of e-mail, and about two thirds of the number of IPs who send e-mail. We are still missing a “long tail” of about one third smaller mailservers in our database.

You may be curious to know how we arrive at these numbers. We obviously can not look into everyone’s e-mail logfiles, but we can look into the DNS traffic on our nameservers. There, we do not only look into who is querying our data, but we also look into what they are querying.

We aggregate this data on a daily and monthly basis to reduce the volume. We then filter out the “noise”: those with extremely low query volumes, those which are clearly dynamic/end-user space (“dynamic” in the hostname etc), and some other tests. The remaining IPs are added into a queue of IPs to be reviewed and assigned to appropriate DNSWL records.

From all IPs (including the filtered and those already assigned), we compute “magnitudes”. These magnitudes indicate basically the percentage of an IP from total world-wide e-mail traffic. Now, we do not directly measure e-mail traffic, but DNS lookups, from which we infer e-mail traffic based on the assumption that those with many DNS lookups are those with a lot of e-mail.

Given the caching mechanisms in DNS, our setup has a tendency to under-estimate the volume of the big senders. It’s a flaw we are willing to accept, especially as this effect is distributed over a very large number of IPs and so does not heavily distort the analysis.

Since an individual IP generally has a very low percentage of overall e-mail traffic, we would have very small numbers. We therefore use logarithmic magnitudes (see table at the bottom of this posting). All the “unassigned” IPs together are usually in the area of magnitude 9.0, ie about 10%. These 10% are (with some daily/weekly fluctuation) about 100’000 IPs. A considerable number of these IPs is later found to be snow-shoe spam or otherwise spammish and are thrown away, so we estimate that there are still between 50’000 and 75’000 IPs which we have not covered (ie total of 150’000 to 175’000 e-mail sending IPs).

Notes about our data

  • We collect DNS usage data from six of our 14 mirrors. While it can be rather safely assumed that the large senders will be covered in any case, there may be regional differences which we do not account for (there is some regional bias in the geographical distribution of our nameservers).
  • There is a considerable number of outright broken DNS queries to our nameservers. RFC1918 IPs, hostnames instead of IPs, Multicast IPs, IPs from before it was being assigned – and most likely there are much more erroneous setups which we can not distinguish from regular traffic. We assume that this will not unduly distort our data in aggregated form.
  • Some IPs are observed only for a short period of time. This may be typical snow-shoe behavior or other similar usage patterns. Strictly speaking, we should remove those IPs. We are however deliberately slow in removing them, because they may only be used sporadically (newsletters once a month, …).
  • We only started to collect full magnitude data a short while ago when we move the database to a dedicated, big, fat machine. The monthly magnitudes seem to be pretty stable, but the time before christmas may have unusual traffic patterns.
  • We do not have such usage data for traffic between the big players in the e-mail space (eg between Yahoo and Hotmail) or internal to such and other organisations. Our data is based on the observation of DNSWL queries from the mostly small- to mid-size organisations that use our whitelisting information.


Magnitude Percent of overall lookups
10.0 100%
9.0 10%
8.0 1%
7.0 0.1%
6.0 0.01%
5.0 0.001%
4.0 0.0001%
3.0 0.00001%
2.0 0.000001%
1.0 0.0000001%