Category Archives: news

Do you manage the reputation of your IPs?

TL;DR: Sign up to the dnswl.org Self Service and start managing the reputation of your IPs.

dnswl.org has been managing reputation of mailservers since over ten years. Since last year, you can provide input for your IPs directly and in a structured manner to the project.

When you properly manage the reputation of your IPs on the dnswl.org Self Service, you get…

  • improved reputation with over 100’000 spam filtering organisations using our data
  • improved reputation with blacklist operators who use our data to fine-tune their lists
  • hints about how a neutral third-party (that’s us) view your reputation, independent of user and/or sender influence

What does “properly manage the reputation” mean?

  1. You sign up to the Self Service and start to claim your domain.
  2. You create change requests to edit the meta data associated with this domain.
  3. You create change requests to add/remove IP addresses.
  4. Once we have an established channel to the responsible for a domain or a range of IPs (that is: to you), we may in some cases increase the reputation of your IPs in the database.
  5. You will receive feedback if we do not accept one of your IP addresses (eg due to too bad reputation, missing or inconsistent naming).
  6. You regularly review the data for consistency, accuracy and completeness.

There are over 10’000 users managing their own data on the Self Service. It’s important for the efficiency of the anti-spam community that we get a good coverage – so please join the effort to maintain a healthy email eco-system!

 

Enforcement of dnswl.org access limits

dnswl.org limits the use of the public nameserver infrastructure to those doing fewer than 100’000 lookups per 24 hours and to non-commercial use (see here for details).

In the past years, we have not been very strict in enforcing these limits. Some users have been constantly ignoring the limits and the warnings we sent out once in a while in an approach of “soft enforcement”.

Since this did not really improve the excessive load generated by some users, we will now move to a more “strict enforcement” mode. We will send out warnings if the IPs of nameservers can easily be traced back to a responsible party. However that is often not the case – in these situations, we will simply turn off access to our public infrastructure for those who repeatedly ignore our warnings.

Subscription website moved into Self Service

For historical reasons, the website to manage subscriptions to dnswl.org data was using a separate website (and “separate-everything”). Over the past months we worked to integrate this into the more generic Self Service.

This has now been finalized, and all subscription functions (get a new subscription, renew a subscription, print invoice, manage download IPs) are available from the “Subscription” tab in the Self Service.

Existing users, their subscriptions and their download IPs have been migrated, but for security reasons passwords have been reset. To log in, these users should follow the following steps:

  • Go to the Self Service portal https://www.dnswl.org/selfservice/ and select “Forgot password”. The email address is the same as the one previously used.
  • “Forgot password” will send an email to the registered address with a link. Clicking the link will display the new password.
  • Once logged in, all data is available under the “Subscription” tab.

For all questions, please contact us at office (at) dnswl.org.

 

1’000 Self Service users, 3’000 Change Requests

Since the dnswl.org Self Service went live in mid December 2015, we got over 1’000 Self Service user signups, and these users created over 3’000 Change Requests. Of course we are happy about the great up-take of the Self Service in the community.

If you also want to manage your own IP addresses and associated data at dnswl.org, please go to https://www.dnswl.org/selfservice/.

For those that are already registered, a few points to keep in mind:

  • Just adding your domain name will not provide much value. We work mainly on IP addresses of mail servers, so you should add these IP addresses.
  • When you add IP addresses, make sure they have meaningful reverse DNS entries (PTR records), and that the reverse and forward DNS are consistent.
  • Yes, you can add IPv6 addresses.
  • You will receive regular notifications from dnswl.org about DNS becoming inconsistent and other signs of stale data. In the future, we may expand this to cover eg information from our spam traps and similar mechanisms.
  • We intend to add more features to the Self Service. Check back regularly to see what is new.

 

Users behind TOR abusing dnswl.org resources

For years, an unknown number of users behind TOR exit nodes have been abusing dnswl.org resources. Apparently for every email they receive, they query dnswl.org via the web, using the search interface open to all users. This causes considerable load on our webservers, the database servers, and our network infrastructure, sometimes causing significant delays for legitimate users.

Despite numerous attempts to a) get these TOR users to notice the abusive nature of their behavior and b) returning empty data to them using the exit address list provided by the TOR project, the abuse did not stop. On the contrary, for every action on our side, there was a workaround on the abusers side.

Obviously, there is only a limited amount of things we can do about it, as long as some people and institutions willingly accept anti-social behavior by providing TOR exit nodes, and we only have a limited amount of time to waste — especially to access data which is readily available in a variety of ways and formats.

As of today, we are redirecting accesses from TOR users to our search interface to https://www.torproject.org/docs/faq-abuse.html.en, hoping that they will realize the abusive and anti-social behavior and start using our data in one of the accepted and supported methods, which require far, far less resources.

Dear TOR users, if the data is so valuable to you, then why do you insist on using the least efficient way to use it?

 

dnswl.org Self Service Portal

News from the dnswl.org team:

For the past years we used an e-mail based approach for user requests to add, change or remove data. This was rather time-consuming and we decided that we need a better process to handle such requests.

That new process is in the new dnswl.org Self Service Portal. Users can use it to “claim” a DNSWL Id (or create a new DNSWL Id and claim this one right away), to request changes to the meta-data associated with such a DNSWL Id, and most important to add/remove IP addresses.

It should be noted that merely “adding a domain” is pointless. You need to add IP addresses, since the reputation that dnswl.org publishes relies only on IP addresses. No IP address, no reputation data.

Users who successfully claimed a DNSWL Id will get regular reports about changes observed that may negatively impact the reputation of “their” IP addresses, eg DNS becoming inconsistent, and about the approval (or rejection) of change requests. While users can request certain changes, the dnswl.org admin team reserves full editorial independence on any and all changes.

We intend to add further features and functions to the Self Service Portal over time, so stay tuned 🙂

You can give the Self Service Portal a try here: https://www.dnswl.org/selfservice/

Change to dnswl.org PGP key

As of Nov 30 2015, there is a slight change to the PGP key we use for signing the download files – there is no more expiration date on that key. Private key and fingerprint remain unchanged.

The updated key can be downloaded here.

“Am I Blocked?”

Starting in early November 2014, we took some additional measures to enforce the query limit on our free nameserver infrastructure of 100’000 queries per 24 hours per organisation. As a result, some people may get a “blocked” result in their spamfilter setup.

In SpamAssassin, this is shown by a “RCVD_IN_DNSWL_BLOCKED” rule in the spam report, and likely a link to https://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block which is generic to all DNSxL with similar limits.

TL;DR

Get an rsync subscription if you are doing more than 100k queries per day yourself.

Why am I blocked?

In order to ensure free access to the vast majority of users, and in order to avoid overloading the often donated resources, many black- and whitelists have limits on how much the public infrastructure may be used. At dnswl.org, we set the limit at 100’000 queries per 24 hours. With the caching of DNS records, this equals about 300’000 to 500’000 emails.

Why has this suddenly changed?

We do not have a strict enforcement which would block a query source immediately once it passes 100’000 queries. There is usually a relatively lax enforcement, and when feasible we try to contact the operator before we block it. However, this is often difficult since there is no identification needed to use the services.

Also, some users are using a shared, third-party nameserver. This is fine as long as all the users of such a nameserver combined stay below 100’000 queries. As soon as a user joins such a nameserver with a different useage pattern or overall higher volume, it may consistently bypass the 100’000 mark and may thus become blocked some time in the future.

Starting in early November 2014, we got more coverage of logs from our various public nameserver mirrors, so we now see (some heavy) users which had gone unnoticed before.

What do I do now?

First, find out what is actually being blocked:

  • If you are using a shared nameserver (eg Google’s 8.8.8.8/8.8.4.4), they are likely blocked.
  • If you are using a local nameserver, see whether it’s configured to do full name resolution, or if it simply forwards queries to a shared nameserver which is then blocked.

The easiest way to find out if a particular nameserver is blocked is by asking:

$ dig -t txt amiblocked.dnswl.org

This should return (among other lines, omitted here for clarity):

;; ANSWER SECTION:
amiblocked.dnswl.org. 86347 IN TXT "no"
;; Query time: 36 msec
;; SERVER: 10.0.1.1#53(10.0.1.1)

The “ANSWER SECTION” should return either “no” or “yes”, and the “SERVER” line indicates which nameserver provided this answer. Be sure to run this on the box(es) where your spamfilter resides, and if possible using the same user account as used for the spamfilter.

If you are legitimately doing more than 100’000 queries per 24 hours, you must get a subscription to download the rsync file for local useage. If you can not afford a subscription, and / or if you are active in the email or the anti-spam community, please contact admins@dnswl.org for a “community” subscription.

OK, so I’ll evade the block!

We can’t stop this from happening. But we will eventually detect those playing games,  eg those moving nameserver IPs around. Our goal is not to chase, our goal is to keep the infrastructure – part of which is donated – free to use for the vast majority of users.

But using a shared nameserver improves caching!

In theory, this is correct. In the real world, there will always be free-riders who hide behind such shared nameservers.

 

Improving dnswl.org IPv6 support

dnswl.org has gradually improved the level of IPv6 support over the past months. The easy part is offering services over IPv6:

  • Most public websites are accessible over IPv6 (since well over a year)
  • Incoming and outgoing mails may pass through IPv6
  • Nameservers for both the dnswl.org and list.dnswl.org zones have a healthy mix of IPv4 and IPv6 (and georedundancy etc)
  • Monitoring adapted to include (hopefully) all services which are also offered over IPv6
  • As a general rule, and wherever available, internal communication between dnswl.org servers happens over IPv6

A bit more work was to enable our internal data handling systems to support IPv6:

  • Database: the main driver for the move to Postgres was it’s stellar support for IPv4 and IPv6 address calculations/storage/manipulation. The migration gave us instant IPv6 support in many important areas (>> 1 year ago).
  • GUI tools: Our internal admin tools had to be adapted in order to display and parse IPv6 addresses (eg screen real estate, validation rules etc)
  • Log collection and parsing tools: Most of our log parsing has been improved to accept IPv4 and IPv6. While the tools have been updated, they have not been rolled out to all list.dnswl.org mirrors yet. Sufficiently performance for log collection by sniffing on network interface requires some advanced IP packet trickery in Perl 🙂

Work in progress:

  • DNSxL for IPv6 queries has only been defined in RFC 5782 basically identical to IPv4. This is considered a risk, as the vast IPv6 space may lead to amplified DDoS attacks on DNS cache infrastructure. We therefore have some trials for “DNS tree walking”, in addition to the “naive” RFC5782-way. The implications of this will be subject to a separate posting.
  • As the query format has not been sufficiently stabilized yet, we do not yet collect statistics about query content (from which we eg identify new mailservers) to IPv6.
  • Some of our export formats (eg for SpamAssassin) now also include IPv6 addresses.
  • Rsync access for subscribers only runs on IPv4. We may gradually start to add AAAA addresses to the rsync hostname in the near future. We are not yet sure whether this will negatively impact subscribers.

What’s next?

Overall, the amount of traffic over IPv6 is minimal, but we do believe that this will significantly grow over time. All new services will be designed to support both IPv4 and IPv6 from the start, and the few remaining services will either be upgraded or discontinued. Eventually IPv6 is becoming the standard, and IPv4 the after-thought.