Tuesday, January 20, 2009

New Clamav-milter for ClamAV 0.95

ClamAV 0.95, which is currently scheduled for release by Sourcefire in March 2009, will include a redesigned and completely rewritten clamav-milter.

Developers and keen users of ClamAV may have noticed that the version of clamav-milter within the SVN repository has changed a lot. We want to let you know what we’ve done and why.

The most notable difference in the new clamav-milter is that the internal mode has been dropped which means that now you will need to run clamd. This has not only allowed us to keep clamav-milter compact and readable, but also it avoids a lot of code duplication. With the old clamav-milter, internal mode was almost the same as having an outdated clamd with a milter interface because we were not keeping the code up-to-date with clamd’s API.

The second important difference is that now clamav-milter has its own configuration and log files that replace the large number of command line switches in the previous version. To ease the difficulty of learning another configuration file, the new clamav-milter comes with a program that will generate a configuration file from your existing command line options and clamd.conf file.

Some features are no longer supported:

  • Notifications
  • Black-listing
  • Phish false positive prevention by use of a subset of SPF
  • Scanning information is no longer added to the email headers by default
  • Scanning and other information can no longer be added to message bodies

So Why Has This Been Done?

Nigel Horne, the program’s previous author, is no longer a member of ClamAV’s engineering team – he is now ClamAV’s product manager. The milter program did not support many new features included within ClamAV and hence clamav-milter was starting to lag behind and bugs were not being addressed. The code was over 7500 lines and we felt it was a great opportunity to rewrite the code from scratch to be more closely coupled with the rest of ClamAV.

As a result we have been able to support new features, including:

  • Clamav-milter can now run as a completely unprivileged user (e.g. nobody)
  • Quarantine has been reworked to use the native milter interface on later versions of Sendmail and Postfix that support it
  • White-listing now uses regular expressions, replacing strict matching
  • Support for Postfix has been added in addition to Sendmail
  • Full IPv6 support.

The new milter’s configuration file is designed to be consistent with the configuration file for clamd, allowing you to fine-tune specific configurations and to route log messages to a dedicated file.

The new milter supports load balancing to copies of clamd in a round-robin fashion. Should one instance of clamd temporarily go down, clamav-milter will issue probe requests every few minutes and the instance will be re-entered into the pool as soon as it becomes available again. Scan requests to remote clamds are performed via the STREAM command, while requests to a local scanner are (preferably) sent via a FILDES command (file descriptor passing over a UNIX socket). This allows systems’ administrators to run clamav-milter and clamd as different users.

And last, but not least, if you prefer, you can continue to use the old version which is kept under …/contrib/old-clamav-milter.
The new clamav-milter will supersede the old one in ClamAV 0.95. Whilst the previous version will still be available, it will no longer be supported.

Installation Instructions

Run ./configure --enable-milter and make as usual.

There are two ways to configure the new clamav-milter:

  • Use the example clamav-milter.conf that we have provided as a template for your configuration. It's well documented, but if something's not clear please report it to bugs.clamav.net and we’ll fix it.
  • Run the make-clamav-milter-conf.pl script with the same run-time arguments you currently pass to clamav-milter and a configuration file will be generated automatically.

Known Issues

The white-list format now uses regular expressions; the old format was a list of strings wrapped in “<>”, the white-list file will therefore need to be edited. We plan to add automated conversion of the file to later versions of the configuration converter script, in the meantime the file will need to be edited by hand.

The round-robin clamd selection requires more work. It works well with either one or a high number of instances of clamd, but the round-robin strategy is limited when the number of scanners is as low as two or three.

And Finally…

The new milter is currently a work in progress. Although we’ve tested it with several hundred GBs of emails, real-life situations are usually more complex than inside the lab. We’re working hard to ensure that clamav-milter is portable to more operating systems.

Please send us your feedback on the new program by adding a comment to this blog; we’re really interested to know what you think!

13 comments :

  1. Well, having used the legacy milter since starting with ClamAV at ver 0.88, I welcome the change. Having duplicate code is almost ALWAYS a cause of problems. Having the fewest number of duplicate functions makes maintaining code much easier.

    I look forward to a RC to try the new milter.

    ReplyDelete
  2. You wrote that you are no longer a member of ClamAV’s engineering team.
    I just saw a tweet about a release of clamav-redirector by "nigelhorne". Is that you or are there 2 different Nigel in the ClamAV team?

    ReplyDelete
  3. I began work on clamav-redirector some time ago when I was still in the Engineering team. I thought it of benefit to the Open Source Community to finally get around to publishing it.

    To avoid any doubt though, clamav-redirector isn't part of ClamAV, it's a separate project. So there's no conflict of my time, I published it in my spare time.

    ReplyDelete
  4. I switched from 0.94.2 to the beeding edge due to reliability problems: almost certainly a problem with the limitations of the server, not the code I hasten to add.

    Had a load of problems getting it to work at all, resulting in a fair few code upgrades. Now it's working, it's lighter, faster, and certainly more reliable on my server.

    ReplyDelete
  5. I just tried the clamav-milter in version 0.95 on Solaris 8 and was unable to get it to talk to the clamd local socket. Each mail message generated log entries of: 'Failed to initiate streaming/fdpassing' followed by 'Timed out while reading clamd reply' Has there been any testing on Solaris 8?

    ReplyDelete
  6. I'd be happier if the perl script worked. It aborts with an error about being unable to determine the clamd socket. It also seems determined to use /usr/local/etc, despite my configuration being in /etc.

    ReplyDelete
  7. Maybe I'm missing something...
    Recently I installed 0.94 and everything was fine. Now with 0.95 I have a feeling clamav-milter is a loop, no scan is performed, no SMTP headers are modified.

    ReplyDelete
  8. Sorry - my setup is as follows:
    i386/FreeBSD 7.1 / Sendmail 8.14.3

    ReplyDelete
  9. still wrestling with this new config file. same issue as Mark (above) but getting nowhere fast (all sockets, no TCP listeners)

    ~ Fedora 5 ~ been keeping up for at least 2 or 3 years but this change has me befuddled

    Mar 31 13:09:14 pegasus clamav-milter[6154]: send failed: Bad file descriptor
    Mar 31 13:09:14 pegasus clamav-milter[6154]: Failed to flush STREAM

    Getting this every time I attempt to route sendmail through

    ReplyDelete
  10. I'm having a couple of issues with the 0.95 milter, no matter if using via inet: or local:
    Are there any instructions to compile the contrib/old-clamav-milter ?

    ReplyDelete
  11. make-clamav-milter-conf.pl -l -o -q /path/to/clamd.socket

    returns FAIL: Unable to determine clamd socket

    I've put the path to both the milter socket (old format) and clamd's socket, but neither work.

    ReplyDelete
  12. It would be helpful if someone could explain how to handle quarantined messages. In my particular case, what sendmail commands can I use to resolve them? For instance, is there a way to forward them to an admin account for disposition?

    ReplyDelete
  13. I like the new milter interface and the fact that things are now in a configuration file. I would however like to have the IP address of the connecting host logged along with the name of the virus in the milter's log.

    I have written a small patch to that end but I am unsure of who/where to send it.

    ReplyDelete