Two months ago, we wrote a “Call for Help” for dnswl.org – and got more response than one would expect!

• We got offered four additional nameservers, of which two are already in production (and due to which we now have to improve our nameserver operation handling – one of the more comfortable problems to have…)
• There is a good number of co-administrators being active. If you contacted the admins via mail or via the request form, you should have noticed a shorter turn-around time.
• A number of great contacts with the makers of anti-spam tools or system operators with very specific situations, which are still ongoing.
• Data of various nature was offered to us.

So, thanks a lot to all who have offered their help – and we’re always glad if you bring along your ideas!

[This article is cross-posted to the two blogs at http://news.dnswl.org/ and http://matthias.leisi.net/ and to the dnswl-users mailing list

Currently, most of dnswl.org is run by myself with the help of some volunteers (“backup” administrators should I ever be run over by a bus) and some organisations.

dnswl.org has gained traction since it’s inception in October 2006, and it’s data is now being used by a number of anti-spam solutions (eg filtering applications, blocklist providers, reputation aggregation services). It has reached a point where more resources are required. Since a commercial business model may create unfavourable incentives (see the last section of this for some rationale), I try to stick to a volunteer-based organisation. The volunteer, that would be you.

Please let me briefly explain how dnswl.org works so that you can better understand the areas where you could be of help.

How dnswl.org works

dnswl.org has basically three parts:

1. Data for dnswl.org (ie, identifying domains and their associated IP addresses used to send email) comes from log files, requests through the website and by mail, and, most importantly, from various import sources (all data is manually verified before being added for distribution).
2. Managing the data: We aggregate IP addresses / network ranges by “owner”. The owner is signified by a domain (with multiple secondary domains possibly added). All IP addresses are regularly checked for RBL “appearances” and DNS inconsistencies. Over time, scores may be adjusted up- or downwards, categories re-assigned, and IP addresses may be added or removed. This is usually based on one of the regular checks above or when we receive feedback.
3. Using the data: We export data in various formats (for rbldnsd and BIND nameservers, in formats usable by Postfix and Lotus Notes, and some others), and distribute it via DNS, rsync and HTTP (the latter transfer will be discontinued at a not-yet-determined point in time, since it is highly inefficient for that purpose).

These things are glued together by a public website (mostly static), a request tracker (RT3), a web-based administration interface (based on PHP and MySQL), a number of batch jobs (Perl) and standard Unix-based tools (rsync, rbldnsd, mrtg, Apache etc).

What should be improved

Given the increased use of dnswl.org, the number and (net-) geographical distribution of nameservers should be increased. Specifically, DNS mirrors in the following locations would be great (but other mirrors would help as well):

• US east coast
• Northern Europe
• Asia/Pacific

CPU and IO load for typical a rbldnsd-based nameserver are minimal (can most likely be run in a VMWare/Xen/however virtualised environment), but it has some bandwidth requirements (a couple of GB per month).

A DNS mirror must be able to regularly update it’s data via rsync, and preferrably run rbldnsd (generic DNS servers such as BIND can basically be used as well, since our data is rather small – it will hardly every be more than 100k entries). Ideally, a nameserver will do minimal logging (”-s” switch to rbldnsd) and regularly run a little Perl script to participate in the statistics

Handling the automated “import queue” and the manual “request queue” would be a full-time job. Additional volunteers to manage data that can spend at least an hour or two per week would allow to speed up the whole process. It would also be worthwhile to have someone chasing large providers (of the likes of Yahoo, Postini et al.) to let us know their full ranges of mail-sending IP addresses. Possible tasks/focus:

• Working on the import and request queues
• Manual additions (eg from logfiles or personal knowledge)
• Establish relations with large providers
• Review missing/incomplete/old data (eg assign categories, cleanup CIDRs, ...)

Volunteers must be trustworthy individuals, and preferrably have a background and/or current work experience in handling spam or security issues in general.

Having more volunteers for the queues would allow myself to spend more time on the improvement of the architecture and implementation. Can you spell “historically grown codebase”? ;) Some clearly definable work could well be done by a knowledgeable person, but it may be difficult/take a lot of time for a new person to learn the overall environment. Possible tasks include:

• Regular RBL checks for large ranges, where it is not feasible to query each individual IP address at each “big” blacklist.
• ASN/riswhois/other checks for large ranges
• Automated feedback loop on spam from dnswl.org-listed sources, eg as a plugin for popular spamfilters or something similar.
• Improve features for domain-level whitelisting.

All dnswl.org features are built on Perl (for regular jobs etc) and PHP (the admin interface), so some knowledge in either or both of them is required (however, rewriting certain parts in eg Java/JSP could be considered).

One can probably tell from looking at the public website that no talented designer has been involved yet. The following seem the most obvious tasks for improvements in design:

• Create a logo for dnswl.org (hey, dnswl.org should be present in the logo-bar on karmasphere.com/ as well!)
• General design, typographics etc for the public website.
• Since I’m not a native english speaker, the wording and general content of the public website may use some improvement.

The future of dnswl.org

I do not regard dnswl.org as an ego-project. It is my goal to establish the project as a long-term, stable endeavour. If you are willing and able to take responsibility for some parts of dnswl.org, you are more than welcome to participate. It’s no problem if you want to start “small” – and grow your duties and responsibilities over time, if you wish to do so.

Besides the additional help mentioned above, I intend to evolve and stabilise the overall organisation over time. I’d be glad to share and pass on responsibility.

I’m convinced that “enumerating goodness” (as opposed to the “enumerating badness” still prevalent in today’s world of fighting spam) will become more important over the coming years, and you can help make it work!

Two weeks ago I wrote that dnswl.org has more than 5’000 users. Well, that’s a thing of the past. As of today, August 4th 2007, there are more than 12’000 users of dnswl.org (see the original article on the caveats with such numbers).

The increase is “bumpy”, indicating lower useage on weekends (parallel with the lower email traffic usually seen outside working days).

This page shows the charts and numbers for three of the nameservers for the list.dnswl.org zone. Assuming that all nameservers have roughly equal traffic patterns, the current rate of queries to dnswl.org nameservers is roughly 250’000 queries per minute (based not on peak, but on 30-minute average traffic); of these, roughly 10’000 queries per minute are for an entry listed in our database.

The 250k/10k are only queries that reach our nameservers. Due to the caching inherent in DNS, the actual number of e-mails being “checked” by dnswl.org is much higher (but nobody will be able to tell how much higher).

Over the past weeks, we continued to enhance dnswl.org data not only by adding more “good” mailservers, but also by refining the categories we assign to these entries.

The latest additions are the “Manufacturing/Industrial” (127.0.13.* response code) and “Retail” (127.0.14.* response code) categories. If we add a new category, existing entries may take some time until they are re-assigned.

The Karmasphere project is an “open platform for sharing reputation data”. Amongst other data sources, it uses dnswl.org data to determine the reputation of IP addresses and domain names.

Karmasphere is still in beta, but looks quite promising – and it started to use dnswl.org data in a way we didn’t even think was possible, namely based on the domains we publish alongside the individual network ranges. As a consequence, we will slightly enhance the data we process for dnswl.org to put additional focus on domain names.

No changes are required for the current IP-based usage &ndash we will continue to manage and add to this data as we have since November 2006.

Yesterday, July 16th 2007, dnswl.org passed the mark of 5’000 users. And as of mid last week, we list more than 10’000 individual IP addresses and networks as “good” mailservers.

You can see the number of networks and hosts querying dnswl.org here (and at the very bottom of that page, you will find a link to the metadata statistics)

Continue reading "dnswl.org now has more than 5'000 users"

Starting from today, DNSWL records include a country code. Not all entries are already “enriched” with this information. It is planned to make this information available through DNS in the future. Currently it is only available through the search on the main website.

Three category descriptions have been changed:

• “Bank, (Re-) Insurance” → “Financial Services”
• “Medical, Doctors, Clinics” → “Healthcare”
• “Government” → “Public Sector”

Note that this change only applies to the descriptions, and not to the numeric codes, which remain unchanged.

News for dnswl.org website and data used to be located at http://www.dnswl.org/news. To make matters easier to edit, we now switched to a blog-based system for these news. The blog is available at http://news.dnswl.org/, including RSS/Atom feeds for consumption in your favourite newsreader.

SpamAssassin version 3.2.0 includes rules to query dnswl.org. Unfortunately, the rules as distributed are not complete — while the rules to distinguish low, medium and high scores are present, the actual lookup is missing. Add the following to your local.cf (or equivalent) to enable the rules:

header __RCVD_IN_DNSWL eval:check_rbl('dnswl-firsttrusted', 'list.dnswl.org.')

See the How to use section if you use an earlier version of SpamAssassin and want to enable dnswl.org lookups.