[This article is cross-posted to the two blogs at http://news.dnswl.org/ and http://matthias.leisi.net/ and to the dnswl-users mailing list
Currently, most of dnswl.org is run by myself with the help of some volunteers (“backup” administrators should I ever be run over by a bus) and some organisations.
dnswl.org has gained traction since it’s inception in October 2006, and it’s data is now being used by a number of anti-spam solutions (eg filtering applications, blocklist providers, reputation aggregation services). It has reached a point where more resources are required. Since a commercial business model may create unfavourable incentives (see the last section of this for some rationale), I try to stick to a volunteer-based organisation. The volunteer, that would be you.
Please let me briefly explain how dnswl.org works so that you can better understand the areas where you could be of help.
How dnswl.org works
dnswl.org has basically three parts:
- Data for dnswl.org (ie, identifying domains and their associated IP addresses used to send email) comes from log files, requests through the website and by mail, and, most importantly, from various import sources (all data is manually verified before being added for distribution).
- Managing the data: We aggregate IP addresses / network ranges by “owner”. The owner is signified by a domain (with multiple secondary domains possibly added). All IP addresses are regularly checked for RBL “appearances” and DNS inconsistencies. Over time, scores may be adjusted up- or downwards, categories re-assigned, and IP addresses may be added or removed. This is usually based on one of the regular checks above or when we receive feedback.
- Using the data: We export data in various formats (for rbldnsd and BIND nameservers, in formats usable by Postfix and Lotus Notes, and some others), and distribute it via DNS, rsync and HTTP (the latter transfer will be discontinued at a not-yet-determined point in time, since it is highly inefficient for that purpose).
These things are glued together by a public website (mostly static), a request tracker (RT3), a web-based administration interface (based on PHP and MySQL), a number of batch jobs (Perl) and standard Unix-based tools (rsync, rbldnsd, mrtg, Apache etc).
What should be improved
Given the increased use of dnswl.org, the number and (net-) geographical distribution of nameservers should be increased. Specifically, DNS mirrors in the following locations would be great (but other mirrors would help as well):
- US east coast
- Northern Europe
- Asia/Pacific
CPU and IO load for typical a rbldnsd-based nameserver are minimal (can most likely be run in a VMWare/Xen/however virtualised environment), but it has some bandwidth requirements (a couple of GB per month).
A DNS mirror must be able to regularly update it’s data via rsync, and preferrably run rbldnsd (generic DNS servers such as BIND can basically be used as well, since our data is rather small – it will hardly every be more than 100k entries). Ideally, a nameserver will do minimal logging (”-s” switch to rbldnsd) and regularly run a little Perl script to participate in the statistics
Handling the automated “import queue” and the manual “request queue” would be a full-time job. Additional volunteers to manage data that can spend at least an hour or two per week would allow to speed up the whole process. It would also be worthwhile to have someone chasing large providers (of the likes of Yahoo, Postini et al.) to let us know their full ranges of mail-sending IP addresses. Possible tasks/focus:
- Working on the import and request queues
- Manual additions (eg from logfiles or personal knowledge)
- Establish relations with large providers
- Review missing/incomplete/old data (eg assign categories, cleanup CIDRs, ...)
Volunteers must be trustworthy individuals, and preferrably have a background and/or current work experience in handling spam or security issues in general.
Having more volunteers for the queues would allow myself to spend more time on the improvement of the architecture and implementation. Can you spell “historically grown codebase”? ;) Some clearly definable work could well be done by a knowledgeable person, but it may be difficult/take a lot of time for a new person to learn the overall environment. Possible tasks include:
- Regular RBL checks for large ranges, where it is not feasible to query each individual IP address at each “big” blacklist.
- ASN/riswhois/other checks for large ranges
- Automated feedback loop on spam from dnswl.org-listed sources, eg as a plugin for popular spamfilters or something similar.
- Improve features for domain-level whitelisting.
All dnswl.org features are built on Perl (for regular jobs etc) and PHP (the admin interface), so some knowledge in either or both of them is required (however, rewriting certain parts in eg Java/JSP could be considered).
One can probably tell from looking at the public website that no talented designer has been involved yet. The following seem the most obvious tasks for improvements in design:
- Create a logo for dnswl.org (hey, dnswl.org should be present in the logo-bar on karmasphere.com/ as well!)
- General design, typographics etc for the public website.
- Since I’m not a native english speaker, the wording and general content of the public website may use some improvement.
The future of dnswl.org
I do not regard dnswl.org as an ego-project. It is my goal to establish the project as a long-term, stable endeavour. If you are willing and able to take responsibility for some parts of dnswl.org, you are more than welcome to participate. It’s no problem if you want to start “small” – and grow your duties and responsibilities over time, if you wish to do so.
Besides the additional help mentioned above, I intend to evolve and stabilise the overall organisation over time. I’d be glad to share and pass on responsibility.
I’m convinced that “enumerating goodness” (as opposed to the “enumerating badness” still prevalent in today’s world of fighting spam) will become more important over the coming years, and you can help make it work!
