dnswl.org News

Reporting abuse to dnswl.org now made easier

nospam@example.com (news.dnswl.org) — Wed, 23 Dec 2009 08:21:27 +0100

We are currently building up a new method to report spam from dnswl.org-listed IP addresses. It will allow you to simply copy&paste (in the future: forward by email) spam you received.

If you would like to participate, please contact us at admins /at/ dnswl.org to get a login to the reporting interface. You can also join our (low volume) public mailing list for discussions around this feature.

Not every single spam may result in a direct action (eg lowering a score) but it is important for us to get unambiguous feedback to judge the trustworthiness of an entry in our database.

Please note that we are not, like eg Spamcop, a generic abuse reporting service, but can only act on entries listed in our database.

GnuPG key expired and updated

nospam@example.com (news.dnswl.org) — Sat, 21 Nov 2009 08:34:03 +0100

Unfortunately, we missed that the GnuPG key we use to sign the exported files expired on 2009-11-17. This may have caused some of your scripts to do unexpected things. We apologize for any inconvenience caused.

We updated the key, it again is valid for three years (expires 2012-11-20). How to upgrade:

If you use the standard script, it is sufficient if you delete the old key (gpg —delete-keys 1B4A29A2) and re-run the script. It will automatically fetch the new key.
For all other purposes, the new key is available from the website here

Who is using dnswl.org data?

nospam@example.com (news.dnswl.org) — Sat, 18 Jul 2009 15:27:04 +0200

We reported about 12’000 users and 250’000 queries per minute back in August 2007, which grew to about 100’000 to 200’000 users in November 2008. Since then, especially the “indirect” usage of dnswl.org data has grown (through rsync data).

While we are not in a position to reveal exact names since we never asked for permission, there are a couple of things we can say (unless noted otherwise, data is as of July 2009):

We have over 65’000 unique /24 networks querying our public nameservers (up from 55’000 in August 2008).
There are about 500 unique /24 networks retrieving our data via rsync (up from 350 in August 2008).
- The rbldnsd-files are downloaded most often, but the Postfix-formatted files are not far behind. And five sites are downloading the Lotus-Notes-formatted files :-)
The public nameservers handle about 250 to 300 mio queries per day.
- This number depends on the world-wide spam activity, and is about the same as in July/August 2008 (before the drop in spam levels as a consequence of the shutdown of McColo).
- OpenDNS.com and UltraDNS.com serve their users locally cached data, thus greatly reducing the load on our public nameservers.
Monthly traffic for a DNS-only server is about 100 Gbyte.

User categories:

Many big players in the managed email and email appliance markets retrieve our data through rsync.
Some fellow DNSBL admins seem to use our data for various purposes. Hi, guys :-)
A lot of educational institutes around the world (eg from Turkey, Germany, Argentina)
Many small and large ISPs query our data through the public nameservers (and we constantly chase them in order to switch to rsync transfer…).
There are only few large commercial users in the Top 500 users of the public nameservers, but there are many of the provider names one would expect to serve such customers directly.

50'000!

nospam@example.com (news.dnswl.org) — Tue, 27 Jan 2009 08:45:24 +0100

As of yesterday, Jan. 26th 2009 shortly before midnight (CET), we reached 50’000 active network entries in our database. According to some rough estimate we are halfway there!

Just kidding — it is of coure an illusion to hope to be able to list every mailserver in existence. Already today it is somewhat of a challenge to keep up with daily, regular changes to the existing data (and we are building tools to make that more efficient).

However, I would like to thank all contributors and supporters of the dnswl.org project for their past (and hopefully future :-) ) effort.

— Matthias

Thanks for your help

nospam@example.com (news.dnswl.org) — Sat, 29 Sep 2007 14:10:55 +0200

Two months ago, we wrote a “Call for Help” for dnswl.org – and got more response than one would expect!

We got offered four additional nameservers, of which two are already in production (and due to which we now have to improve our nameserver operation handling – one of the more comfortable problems to have…)
There is a good number of co-administrators being active. If you contacted the admins via mail or via the request form, you should have noticed a shorter turn-around time.
A number of great contacts with the makers of anti-spam tools or system operators with very specific situations, which are still ongoing.
Data of various nature was offered to us.

So, thanks a lot to all who have offered their help – and we’re always glad if you bring along your ideas!

dnswl.org - calling for your support

nospam@example.com (news.dnswl.org) — Sat, 04 Aug 2007 12:00:00 +0200

[This article is cross-posted to the two blogs at http://news.dnswl.org/ and http://matthias.leisi.net/ and to the dnswl-users mailing list

Currently, most of dnswl.org is run by myself with the help of some volunteers (“backup” administrators should I ever be run over by a bus) and some organisations.

dnswl.org has gained traction since it’s inception in October 2006, and it’s data is now being used by a number of anti-spam solutions (eg filtering applications, blocklist providers, reputation aggregation services). It has reached a point where more resources are required. Since a commercial business model may create unfavourable incentives (see the last section of this for some rationale), I try to stick to a volunteer-based organisation. The volunteer, that would be you.

Please let me briefly explain how dnswl.org works so that you can better understand the areas where you could be of help.

How dnswl.org works

dnswl.org has basically three parts:

Data for dnswl.org (ie, identifying domains and their associated IP addresses used to send email) comes from log files, requests through the website and by mail, and, most importantly, from various import sources (all data is manually verified before being added for distribution).
Managing the data: We aggregate IP addresses / network ranges by “owner”. The owner is signified by a domain (with multiple secondary domains possibly added). All IP addresses are regularly checked for RBL “appearances” and DNS inconsistencies. Over time, scores may be adjusted up- or downwards, categories re-assigned, and IP addresses may be added or removed. This is usually based on one of the regular checks above or when we receive feedback.
Using the data: We export data in various formats (for rbldnsd and BIND nameservers, in formats usable by Postfix and Lotus Notes, and some others), and distribute it via DNS, rsync and HTTP (the latter transfer will be discontinued at a not-yet-determined point in time, since it is highly inefficient for that purpose).

These things are glued together by a public website (mostly static), a request tracker (RT3), a web-based administration interface (based on PHP and MySQL), a number of batch jobs (Perl) and standard Unix-based tools (rsync, rbldnsd, mrtg, Apache etc).

What should be improved

Given the increased use of dnswl.org, the number and (net-) geographical distribution of nameservers should be increased. Specifically, DNS mirrors in the following locations would be great (but other mirrors would help as well):

US east coast
Northern Europe
Asia/Pacific

CPU and IO load for typical a rbldnsd-based nameserver are minimal (can most likely be run in a VMWare/Xen/however virtualised environment), but it has some bandwidth requirements (a couple of GB per month).

A DNS mirror must be able to regularly update it’s data via rsync, and preferrably run rbldnsd (generic DNS servers such as BIND can basically be used as well, since our data is rather small – it will hardly every be more than 100k entries). Ideally, a nameserver will do minimal logging (”-s” switch to rbldnsd) and regularly run a little Perl script to participate in the statistics

Handling the automated “import queue” and the manual “request queue” would be a full-time job. Additional volunteers to manage data that can spend at least an hour or two per week would allow to speed up the whole process. It would also be worthwhile to have someone chasing large providers (of the likes of Yahoo, Postini et al.) to let us know their full ranges of mail-sending IP addresses. Possible tasks/focus:

Working on the import and request queues
Manual additions (eg from logfiles or personal knowledge)
Establish relations with large providers
Review missing/incomplete/old data (eg assign categories, cleanup CIDRs, ...)

Volunteers must be trustworthy individuals, and preferrably have a background and/or current work experience in handling spam or security issues in general.

Having more volunteers for the queues would allow myself to spend more time on the improvement of the architecture and implementation. Can you spell “historically grown codebase”? ;) Some clearly definable work could well be done by a knowledgeable person, but it may be difficult/take a lot of time for a new person to learn the overall environment. Possible tasks include:

Regular RBL checks for large ranges, where it is not feasible to query each individual IP address at each “big” blacklist.
ASN/riswhois/other checks for large ranges
Automated feedback loop on spam from dnswl.org-listed sources, eg as a plugin for popular spamfilters or something similar.
Improve features for domain-level whitelisting.

All dnswl.org features are built on Perl (for regular jobs etc) and PHP (the admin interface), so some knowledge in either or both of them is required (however, rewriting certain parts in eg Java/JSP could be considered).

One can probably tell from looking at the public website that no talented designer has been involved yet. The following seem the most obvious tasks for improvements in design:

Create a logo for dnswl.org (hey, dnswl.org should be present in the logo-bar on karmasphere.com/ as well!)
General design, typographics etc for the public website.
Since I’m not a native english speaker, the wording and general content of the public website may use some improvement.

The future of dnswl.org

I do not regard dnswl.org as an ego-project. It is my goal to establish the project as a long-term, stable endeavour. If you are willing and able to take responsibility for some parts of dnswl.org, you are more than welcome to participate. It’s no problem if you want to start “small” – and grow your duties and responsibilities over time, if you wish to do so.

Besides the additional help mentioned above, I intend to evolve and stabilise the overall organisation over time. I’d be glad to share and pass on responsibility.

I’m convinced that “enumerating goodness” (as opposed to the “enumerating badness” still prevalent in today’s world of fighting spam) will become more important over the coming years, and you can help make it work!

dnswl.org - usage increasing

nospam@example.com (news.dnswl.org) — Sat, 04 Aug 2007 10:52:22 +0200

Two weeks ago I wrote that dnswl.org has more than 5’000 users. Well, that’s a thing of the past. As of today, August 4th 2007, there are more than 12’000 users of dnswl.org (see the original article on the caveats with such numbers).

The increase is “bumpy”, indicating lower useage on weekends (parallel with the lower email traffic usually seen outside working days).

This page shows the charts and numbers for three of the nameservers for the list.dnswl.org zone. Assuming that all nameservers have roughly equal traffic patterns, the current rate of queries to dnswl.org nameservers is roughly 250’000 queries per minute (based not on peak, but on 30-minute average traffic); of these, roughly 10’000 queries per minute are for an entry listed in our database.

The 250k/10k are only queries that reach our nameservers. Due to the caching inherent in DNS, the actual number of e-mails being “checked” by dnswl.org is much higher (but nobody will be able to tell how much higher).

Announcement: New categories in dnswl.org

nospam@example.com (news.dnswl.org) — Tue, 17 Jul 2007 17:34:02 +0200

Over the past weeks, we continued to enhance dnswl.org data not only by adding more “good” mailservers, but also by refining the categories we assign to these entries.

The latest additions are the “Manufacturing/Industrial” (127.0.13.* response code) and “Retail” (127.0.14.* response code) categories. If we add a new category, existing entries may take some time until they are re-assigned.

Whitelisting, taken to the next step

nospam@example.com (news.dnswl.org) — Tue, 17 Jul 2007 17:26:44 +0200

The Karmasphere project is an “open platform for sharing reputation data”. Amongst other data sources, it uses dnswl.org data to determine the reputation of IP addresses and domain names.

Karmasphere is still in beta, but looks quite promising – and it started to use dnswl.org data in a way we didn’t even think was possible, namely based on the domains we publish alongside the individual network ranges. As a consequence, we will slightly enhance the data we process for dnswl.org to put additional focus on domain names.

No changes are required for the current IP-based usage &ndash we will continue to manage and add to this data as we have since November 2006.

dnswl.org now has more than 5'000 users

nospam@example.com (news.dnswl.org) — Tue, 17 Jul 2007 17:09:42 +0200

Yesterday, July 16th 2007, dnswl.org passed the mark of 5’000 users. And as of mid last week, we list more than 10’000 individual IP addresses and networks as “good” mailservers.

You can see the number of networks and hosts querying dnswl.org here (and at the very bottom of that page, you will find a link to the metadata statistics)

Continue reading "dnswl.org now has more than 5'000 users"

Metadata changes and additions

nospam@example.com (news.dnswl.org) — Thu, 17 May 2007 09:42:17 +0200

Starting from today, DNSWL records include a country code. Not all entries are already “enriched” with this information. It is planned to make this information available through DNS in the future. Currently it is only available through the search on the main website.

Three category descriptions have been changed:

“Bank, (Re-) Insurance” → “Financial Services”
“Medical, Doctors, Clinics” → “Healthcare”
“Government” → “Public Sector”

Note that this change only applies to the descriptions, and not to the numeric codes, which remain unchanged.

New News for dnswl.org

nospam@example.com (news.dnswl.org) — Sat, 12 May 2007 19:56:19 +0200

News for dnswl.org website and data used to be located at http://www.dnswl.org/news. To make matters easier to edit, we now switched to a blog-based system for these news. The blog is available at http://news.dnswl.org/, including RSS/Atom feeds for consumption in your favourite newsreader.

dnswl.org data and SpamAssassin 3.2.0

nospam@example.com (news.dnswl.org) — Mon, 07 May 2007 12:00:00 +0200

SpamAssassin version 3.2.0 includes rules to query dnswl.org. Unfortunately, the rules as distributed are not complete — while the rules to distinguish low, medium and high scores are present, the actual lookup is missing. Add the following to your local.cf (or equivalent) to enable the rules:

header __RCVD_IN_DNSWL eval:check_rbl('dnswl-firsttrusted', 'list.dnswl.org.')

See the How to use section if you use an earlier version of SpamAssassin and want to enable dnswl.org lookups.