Wednesday, June 19. 2013
dnswl.org has gradually improved the level of IPv6 support over the past months. The easy part is offering services over IPv6:
A bit more work was to enable our internal data handling systems to support IPv6:
Work in progress:
Overall, the amount of traffic over IPv6 is minimal, but we do believe that this will significantly grow over time. All new services will be designed to support both IPv4 and IPv6 from the start, and the few remaining services will either be upgraded or discontinued. Eventually IPv6 is becoming the standard, and IPv4 the after-thought.
Wednesday, February 27. 2013
E-mail marketing provider epsilon.com (DNSWL Id 3108) let one of their customers continuously spam the dnswl.org admin address – abuse reports have been ignored for months, and they make it very hard to find an abuse reporting address to begin with. In fact, the logical choice of email@example.com bounces.
We therefore suspended all epsilon.com and related IP in dnswl.org effective immediately (expect some delay until DNS fuly updates).
It should be noted that epsilon.com appears under varying domain names in your logs, eg epsiloninteractive.com, bigfootinteractive.com, bfi0.com, epidm.com, and possibly more. Also, they sometimes use their customer domain names as reverse DNS for their IP space, often in the form of “mta.news.
Saturday, November 24. 2012
Sunday, November 4. 2012
Wednesday, September 19. 2012
At dnswl.org we serve over 80’000 organisations with our database which contains 150’000 (and growing) entries of “good mailservers”.
In order to maintain the quality of the data and of our infrastructure, we are looking for additional volunteers to support the community and the project. While there is a lot of work to do, we have the most pressing needs in three categories:
If you would like to work in one of the areas, or in some combination, please contact the admin team or Matthias. This is mostly volunteer work (but financing for infrastructure is assured), but it would be good if you could spend a handful of hours a week on the project.
Below is a description of what we see as the priorities in the three areas. This list is not exhaustive, and you have all the freedom to tinker as long as it serves the goal of the project :)
We operate a number of servers for public access via DNS, and for editing the data. There is a significant amount of code (Perl and PHP) involved for editing data, aggregating and enriching log and usage data, and operating the infrastructure. We need to “keep the stuff running” and fix occasional bugs. In order to simplify the systems management tasks, we would need to introduce new tools or improve the existing ones.
We use a mostly standard tool chain (Apache, Perl, PHP, Postgres, Bind, rbldnsd, rsync, Nagios, Smokeping, Request Tracker, Postfix and so on).
dnswl.org data editing
Although we use a number of tools to help with the maintenance and the growth of our data, most of our actions are manual in nature (ie, involve a manual verification click or some similar action). We interact with requestors and other stakeholders, we assess the trustworthiness of entries, we maintain the quality of our data.
We want to expand the number and diversity of our editors. If you maintain your own local reputation list which you want to offer for import into dnswl.org data, or if you are willing to spend a few hours a week for data editing and related tasks, please get in touch with us.
Sunday, December 18. 2011
Our previous method of enforcing limits caused some concern, both in public and private conversations. The main argument is that causing false negatives is not acceptable in principle, not even for cases of obvious abusive use.
We listened to these thoughts, and have now changed our approach. The criteria for blocking such abusive nameservers are still the same: repeated use way above the 100k / 24 hours limit and no response to reasonable attempts at contacting them. Also, most of the things in our previous news item referenced above are still valid.
However, the technical handling has changed to reduce the number of queries that a legitimate client will actually perform. The new handling has the following technical effects:
To see whether you are affected, use “dig -t txt amiblocked.dnswl.org” on the mailserver (or other machine which uses the same nameserver setup).
There is an additional change under discussion with the SpamAssassin team to define a dedicated return value to indicate “blocked for excessive usage” (see this discussion on Bugzilla). With this specific return value, the application (SA in this case) will know to not attempt any more queries until the TTL has expired.
We will continue to watch the situation on our public nameserver infrastructure and will work to ensure that it remains accessible and usable for free for most users.
Monday, October 17. 2011
We restrict the use of the public dnswl.org nameservers to 100’000 queries per day for all organisations using it for free. With this limitation, we want to keep the traffic for all public mirrors (some of which are donated) at an acceptable level (currently 100 to 200 GByte per month).
Those with a higher load are intended to get a paid subscription for an rsync download and access the data locally.
Unfortunately, it is not straightforward to enforce these limits. DNS does not make it easy to identify an administrative contact behind a query source, and many DNS setups make it difficult even for the actual administrator to identify current query behaviour.
Dnswl.org has historically taken a “light” approach for the enforcement of the 100’000 queries per day limit, basically counting on the honesty of all users and spending considerable time to identify and contact administrators of query sources going way over the stated limit. Given a growing number of such abusive query sources, this manual approach does not scale well.
The “light” approach to enforcement also means that we err on the side of caution: we do only aggregate usage data from a selection of nameservers, and we only sample the data from the nameservers where we collect the usage data. Thus, we sincerely underestimate the actual usage by design.
Detecting and limiting abusive query rates
We are now taking additional action to ensure that our public nameserver infrastructure remains accessible for the tens of thousands of free users. These steps include:
Note that when a nameserver gets a “REFUSE” message from dnswl.org, it will likely get a similar response from other black- and whitelists as well, and the spamfilter will seriously underperform. It should therefore be in the administrators won interest to fix such a situation.
“listed, hi” response
In the extreme cases listed above, and for a limited time until query rates have gone down to the acceptable limits, we may return a special answer code to all queries, 127.0.10.3. The “10” indicates that this is a “special” return code, the “3” stands for “high trust” level.
This will cause that a spam filter will mark all mails as coming from a highly trusted server, and will thus result in some spams coming through to users.
While we do not appreciate having to cause such negative effects, the carelessness of the administrators concerned leaves us no other choice.
Out of the 50k to 60k nameserver IPs querying our public nameservers every day, less than 0.1% are affected by this stricter enforcement of our acceptable use limits. The number is even lower when considering the (estimated) number of organisations. However, some “big” nameserver providers are sometimes affected (eg Google public DNS).
What to do when affected by “listed, hi”
1. Contact us at firstname.lastname@example.org. As soon as we have a working administrative/operative contact, there is no reason to continue the “listed, hi” response. Note that while we are spread over multiple timezones and try to act fast on these issues, we do not have guaranteed 24×7 operations.
Some statistics on abusive query sources
It should be noted that DNS statistics are not always straightforward, so the numbers should be taken with a grain of salt. All our numbers therefore are heavily erring on the side of caution. Since we only collect and aggregate logs from a selection of our nameservers, and since we are only sampling the data (throwing away about a quarter of all collected logs), the real numbers are about three times as high as we report them.
All query sources which are deemed “abusive” are doing the high query rates for weeks and months. We may take a single day as an example, October 14th 2011, which happens to be a Friday (weekends generally have different patterns, but usually have the same Top N names).
Overall 78k unique IPs were querying the public nameservers on that day, each doing on average of 2’500 queries per day. The Top 100 Query Sources are all doing way above 100k queries each, the Top 20 Query Sources are way above 1 mio queries each. Some examples (aggregated by organisation, as far as this is possible based on rDNS):
Google DNS (18.104.22.168 etc) 33 mio queries (spread over ~ 50 IPs)
Of those sources, only one (Dyndns) has promised action after being contacted.
Wednesday, February 2. 2011
The Goodmail accreditation service is shutting down, as numerous sources on Twitter and the Blogosphere are reporting.
If you are a current Goodmail customer and would like to register your IPs in the dnswl.org whitelist, please fill in the request form at http://www.dnswl.org/request.pl and write “Goodmail” in the comment box.
Saturday, November 27. 2010
Sometime today, Nov 27 2010, amidst the hardware problems with one of our servers, we silently passed the milestone of 100’000 active entries in the dnswl.org database (it’s slightly more IP addresses, because there are also some ranges of IP addresses in our database). That data is used by about 50’000 organisations world-wide.
Based on our statistics, we cover about 90% of the volume of e-mail, and about two thirds of the number of IPs who send e-mail. We are still missing a “long tail” of about one third smaller mailservers in our database.
You may be curious to know how we arrive at these numbers. We obviously can not look into everyone’s e-mail logfiles, but we can look into the DNS traffic on our nameservers. There, we do not only look into who is querying our data, but we also look into what they are querying.
We aggregate this data on a daily and monthly basis to reduce the volume. We then filter out the “noise”: those with extremely low query volumes, those which are clearly dynamic/end-user space (“dynamic” in the hostname etc), and some other tests. The remaining IPs are added into a queue of IPs to be reviewed and assigned to appropriate DNSWL records.
From all IPs (including the filtered and those already assigned), we compute “magnitudes”. These magnitudes indicate basically the percentage of an IP from total world-wide e-mail traffic. Now, we do not directly measure e-mail traffic, but DNS lookups, from which we infer e-mail traffic based on the assumption that those with many DNS lookups are those with a lot of e-mail.
Given the caching mechanisms in DNS, our setup has a tendency to under-estimate the volume of the big senders. It’s a flaw we are willing to accept, especially as this effect is distributed over a very large number of IPs and so does not heavily distort the analysis.
Since an individual IP generally has a very low percentage of overall e-mail traffic, we would have very small numbers. We therefore use logarithmic magnitudes (see table at the bottom of this posting). All the “unassigned” IPs together are usually in the area of magnitude 9.0, ie about 10%. These 10% are (with some daily/weekly fluctuation) about 100’000 IPs. A considerable number of these IPs is later found to be snow-shoe spam or otherwise spammish and are thrown away, so we estimate that there are still between 50’000 and 75’000 IPs which we have not covered (ie total of 150’000 to 175’000 e-mail sending IPs).
Notes about our data
Monday, October 25. 2010
As announced earlier, dnswl.org is changing it’s operating model. Users with high query volumes (> 100’000 queries/24 hours) and commercial filter vendors of anti-spam products and services are required to purchase a subscription (see here for full access requirements).
Over the next couple of days, we will start the enforcement of this model. “Enforcement” means that we may block your rsync access or your access to the DNS servers without further warning, since we usually do not know who you are.
If you use the rsync access today, please register early for the subscription. If you register before Nov 1st 2010, you will get an additional 3 months free with your 12 months subscription.
Subscribe today at https://subscription.dnswl.org/
Friday, October 8. 2010
The shiny new dnswl.org subscription management website is now in public beta.
Who could and should use this website?
All users who are expected to need a subscription (doing more than 100’000 queries/day on the public nameservers, commercial vendors of anti-spam solutions, rsync access for other reasons) can use the subscription mechanisms starting today. The site is still in beta mode, meaning that things may break. However, the actual rsync distribution mechanism is considered to be in production.
What is the benefit of using it at this early stage?
You get a longer transition period to the new rsync facility, and three months “early adopter” subscription for free (together with a regular subscription). Eventually, all current users of “rsync1.dnswl.org” will need to migrate. The earlier you start, the less you will have to rush towards the end. Enforcement of the new rules will start in about four weeks.
What does it cost?
We will publish the final pricing on www.dnswl.org as soon as we get out of beta. Current pricing is 120 Euros / 100 users, 300 Euros / 1’000 users, 1’200 Euros / for each 10’000 users. There is a reduced rate for education/academic institutions and not-for-profit organisations with a 50% discount.
For any questions, please contact us at email@example.com.
Link to the dnswl.org subscription management website: https://subscription.dnswl.org/
Saturday, October 2. 2010
dnswl.org has successfully provided whitelisting data for anti-spam filters since 2006. This is how we will ensure the continued success of dnswl.org:
The landscape for anti-spam solutions in 2011 is different from the landscape in 2006. dnswl.org must also adapt to these changes in order to remain a relevant player beyond 2011. Luckily, this will have little to no impact for most of our 50’000 users. We will provide the same independent level of data editing to live up to our promise:
Improve the reliability of the email system.
In a more mature and competitive market with big challenges ahead (adapting the anti-spam toolchain for IPv6), dnswl.org will also require a solid organisational and financial basis. However, dnswl.org has not and will not charge money to get on the list, as this would create adverse incentives.
We decided that dnswl.org will change to a subscription model for “heavy” users and sellers of anti-spam services or products. We consider those who do more than 100’000 queries per 24 hours on the public nameservers to be heavy users.
Prices have not yet been finalized. We plan to have a regular and a reduced rate; educational institutions, not-for-profits etc will qualify for a reduced rate. Those who contribute resources, data or time to the project will get free subscriptions.
For those who run a small- to medium-scale anti-spam solution (eg based on SpamAssassin) nothing will change, provided you stay below the 100’000 queries/24 hours limit. Also, we will not rush to cut someone’s access off just because he has a high-traffic day.
The financial basis for dnswl.org is not the only change that is needed to ensure the continued success of the project. We will need to broaden the reach of our editors, try to get more imports of existing whitelisting projects, and invest time and resources into improving our tools and internal data.
Getting ready for IPv6 and a somewhat improved infrastructure will be the first priorities as soon as the subscription model is in place.
Implementation of the subscription model
Migrating from our current model will be a challenge: We have no contact data for the current rsync users, and also not for the “heavy” users on the public nameservers. Further, the infrastructure and processes for handling the subscription needs yet to be built and tested. The introduction schedule looks roughly like this:
Please note that this schedule is subject to change depending on how things evolve. We will communicate regularly on the progress and next steps. You can always contact us at admins /at/ dnswl.org.
Contact for journalist enquiries: Matthias Leisi firstname.lastname@example.org, phone +41 79 207 31 08
Monday, September 27. 2010
Today, Spamhaus announced the public release of their whitelist. This is an important step for Spamhaus and for the concept of whitelisting in spamfiltering in general, and thus for us at dnswl.org as well.
With better whitelisting, the reliability of the overall email system can be greatly enhanced. Mails from senders with a good reputation are guaranteed to be delivered to it’s destination, and more stringent fitlers can be applied to mails from senders with unknown or outright bad reputation.
This concept has been at the core of dnswl.org in the almost four years since the start, and it has been understood and applied by our 50’000 users worldwide. We are glad that more players are increasing their efforts towards whitelisting, which will benefit all players in that area.
dnswl.org is aware that the landscape for spamfiltering (and thus for whitelisting) is changing. We have a number of steps planned to ensure that dnswl.org is fit to cope with this evolving landscape, and with the growing importance of whitelisting.
The three most important items on our roadmap are 1) to create a transparent and sustainable financing for our operation, 2) to encourage more partners to share their whitelisting data and 3) to add IPv6:
We will have a public announcement about the financial support for dnswl.org in a few weeks.
If you want to partner with dnswl.org, or share your whitelisting data with dnswl.org, or if you want to support our data editing, please contact us at email@example.com.
Very basic support for IPv6 is already included in the backend systems. However, there will be an intense testing phase before the first IPv6 addresses are published. dnswl.org will work with other players in the industry to ensure that the whole “toolchain” for DNSxL operations and use are available for a mixed IPv4/IPv6 world.
Tuesday, May 18. 2010
We don’t expect a lot of donations via Flattr, but it surely is an interesting approach to social payments.
Friday, March 26. 2010
(Page 1 of 2, totaling 29 entries) » next page