Wednesday, March 30, 2016

Here at ClamAV, we continue to welcome signature submissions to improve community detection. As a thanks to our community, we like to reward individuals with some cool “ClamAV swag” items such as our ClamAV t-shirts, battery chargers, Snort calendars, and other goodies for signature submissions accepted.

Standard rules for submission criteria:

We are accepting signatures into ClamAV via the Community-Sigs mailing list, which anyone may join here:  http://lists.clamav.net/cgi-bin/mailman/listinfo/community-sigs

When we receive a signature, we will follow our standard internal procedures (which involves heavy QA of the signature, testing, optimization for performance, and running it against our bank of clean files).

You may reference the ClamAV Signatures Manual for general signature questions, as well as discussing it among fellow ClamAV Signature writers on the above list.

The signatures are released in the Official ClamAV Signature Set and are available to the ClamAV community via our normal mirror rule distribution process, published approximately every four hours!

We will provide you feedback about how to improve your signatures such as what you should or should not do, tips and tricks involved with the latest versions of ClamAV, as well as giving the author full attribution for their submissions.

If you’d like to submit to the ClamAV signature set, please email the above list with your signatures and research behind it (pcap, ascii dump, references, anything!)

As always False positive reports belong here: http://www.clamav.net/reports/fp

The highest submitter for accepted signatures for each month will receive some goodies.  Keep in mind that we must accept the signatures.

We thank the community in advance for community submissions, as well as continued submission of false positive reports.

Tuesday, March 29, 2016

A Brief History

With the release of ClamAV 0.99, users saw a slew of new features aimed at enhancing ClamAV’s core functionalities. Included in these features were the improvements to ClamAV’s long-neglected on-access scanning capabilities. Ages ago, ClamAV supported on-access scanning via the now-defunct Dazuko kernel module. When development on Dazuko ended in 2011, the ClamAV team began searching for a suitable replacement. The team soon landed on the recently stable fanotify API that first shipped with the 2.6.37 Linux kernel. By November of 2011 a barebones fanotify-based on-access scanner had been completed, but it wouldn’t see public release until ClamAV 0.98.

Fast forward to ClamAV version 0.99four years after the creation of that first prototypeand users found themselves presented with a completely reworked on-access scanning system that now leveraged not only fanotify, but inotify as well. While ClamAV’s documentation had been updated to help navigate users through the transition, a number of people reported problems they were having with ClamAV’s on-access scanner. Almost invariably, these issues came down to either improper configuration or system-level limitations. This guide seeks to provide users interested in ClamAV’s on-access scanning with a definitive resource on the topicstraight from the horse’s mouth, so to speak.

System Requirements

fanotify
On-access scanning, requires a system running a Linux kernel (version >= 3.8) with fanotify compiled in. Check the kernel configuration for fanotify’s presence by running:
$ cat /boot/config-<kernel_version> | grep FANOTIFY

Results if fanotify is available:
CONFIG_FANOTIFY=y

Results if fanotify can prevent malicious file access attempts:
CONFIG_FANOTIFY_ACCESS_PERMISSIONS=y

If this is seen:
CONFIG_FANOTIFY=y
# CONFIG_FANOTIFY_ACCESS_PERMISSIONS is not set

Then on-access scanning will be constrained to notify-only mode and will be unable to prevent access to malicious files, since fanotify lacks the ability to block events on the system.

inotify
To use the dynamic directory determination (DDD) system introduced in 0.99, ensure inotify is configured with enough watchpoints. By default, inotify can watch for changes in 8192 directories at any one time. If more directories than that need protection, run the following commandreplacing <NUM> with the maximum number of directories to be watched:
$ echo <NUM> | sudo tee -a /proc/sys/fs/inotify/max_user_watches

Overview of Configuration Options

The clamd.conf configuration file contains a number of options which modify the behaviour of ClamAV’s on-access scanner. Each configuration option is listed below alongside its intended purpose and any caveats of which users should remain cognizant.

ScanOnAccess
Purpose: Enables on-access scanning under clamd if set to “yes”. If set to ”no”, all other on-access configurations options will be ignored.

Caveats: clamd must be run as root.

OnAccessMountPath
Purpose: Watches the entirety of one or more specified mount points and notifies the user of any malicious events caught by fanotify.

Caveats: Users must specify a valid mount point. If this option is enabled, clamd will disable fanotify prevention to keep users from accidentally locking up their system. Enabling this option will also disable the DDD system and all configuration options which rely on that system.

OnAccessMaxFileSize
Purpose: Allows the user to stop clamd from scanning files larger than the specified size. If set to “0”, clamd will scan all files regardless of size.

Caveats: On some kernel versions and architectures, fanotify is known to fail when blocking access to files larger than 4GB. ClamAV will alert users if this problem occurs.

OnAccessIncludePath
Purpose: The files within the any specified path(s) will be monitored for access attempts. If the DDD system is enabled, clamd will watch this path recursively, i.e., the specified directory and all subdirectories will be monitored. If the DDD system is disabled, only the topmost directory will be watched and all subdirectories will be ignored.

Caveats: This option will be ignored if OnAccessMountPath is enabled. As an additional precaution, if DDD is enabled, clamd will prevent users from specifying “/” as an include path.

OnAccessExcludePath
Purpose: Recursively removes the given directory from the set of directories monitored by clamd.

Caveats: If the DDD system is disabled, this option is ignored.

OnAccessExcludeUID
Purpose: When fanotify catches an event, clamd will check given UIDs against the captured pid. If a match is found, the event is ignored and no scanning is performed.

Caveats: There aren't any known caveats with this option.

OnAccessDisableDDD
Purpose: This option allows users to turn off the DDD system if they prefer the legacy on-access scanning behaviour that was available in 0.98. If DDD is disabled OnAccessIncludePath will not recursively watch paths, making OnAccessExcludePath a redundant option.

Caveats: If OnAccessMountPath is enabled, clamd will act as though DDD is disabled.

OnAccessPrevention
Purpose: Flags fanotify to block any triggered events on monitored files, which allows ClamAV to scan affected files to determine if those events should be allowed to proceed.

Caveats: The kernel must be compiled with “CONFIG_FANOTIFY_ACCESS_PERMISSIONS=y” for this option to function. As a precaution, if OnAccessMountPath is enabled, clamd automatically disables this option to stop the user from accidentally locking up their system. Only access and open events can be blocked by fanotify.

OnAccessExtraScanning
Purpose: Provides DDD users with inotify event coverage, which allows for scans on newly created, moved, or renamed files.

Caveats: Since this option relies on the inotify framework maintained by the DDD system, it is ignored if OnAccessDisableDDD is “yes”.

Configuration Examples

This guide ends with a few simple clamd.conf configuration examples which will satisfy a number of general use cases. Though there are only four, these examples can readily be expanded upon or adapted to fit more targeted use cases.

Watch entire filesystem
ScanOnAccess yes
OnAccessMountPath /
OnAccessExcludeUID 0

Details: This will set fanotify to watch the entire filesystem in real-time and trigger ClamAV to run scans on any files opened, accessed, or closed except by the root user. While clamd will report any viruses found during this scanning, fanotify will not perform any blocking or prevention.

Watching select users’ home directories
ScanOnAccess yes
OnAccessIncludePath /home
OnAccessExcludePath /home/user2
OnAccessExcludePath /home/user4
OnAccessExtraScanning yes

Details: Like the previous example, fanotify is not flagged to prevent events which occur on malicious files. However, further scanning coverage is provided thanks to OnAccessExtraScanning leveraging the inotify events caught by the DDD system.

Protecting a directory recursively
ScanOnAccess yes
OnAccessIncludePath /home/user/Downloads
OnAccessPrevention yes
OnAccessExtraScanning yes

Details: Here fanotify will block any triggered events, allowing only clean files to be opened, or otherwise accessed. This configuration will also immediately alert the user if a malicious file is inadvertently downloaded.

Protecting multiple dirs non-recursively
ScanOnAccess yes
OnAccessIncludePath /home/user/dir1
OnAccessIncludePath /home/user/dir2
OnAccessPrevention yes
OnAccessDisableDDD yes

Details: Similarly to legacy behaviour, only the top level of dir1 and dir2 will be protected in this case. The contents of all subdirectories will be ignored and extra scanning via inotify events is not possible since DDD is disabled.

Wednesday, March 16, 2016

Our ClamAV Signature Interface maintenance is now complete.  While we apologize for the delay, the rollout of the the new Signature Interface inside of ClamAV will result in several new features for the community, and I wanted to tell you about some of them:

First, the first new “main.cvd” in about two years.  This main.cvd has been completely re-written from scratch, and while the function of the “main” is largely the same, it’s been rewritten to not only enforce order to the signatures, but naming convention as well.  For example:

W97M.Ethan.AK-1 has moved to Doc.Trojan.Ethan
Worm.Padowor.A-zippwd has moved to Win.Worm.Padowor
Adware.Smshoax has moved to Win.Adware.Smshoax

Re-naming of the signatures may affect a local user’s whitelist.  If you have excluded certain signatures in the past that are now firing, we ask that you both submit the file to us for false positive remediation (if you believe it to be a false positive), and rename the signature whitelist on your side.

This new main is 109Mb in size, and contains 4 million signatures for ClamAV.  Now that the main.cvd has been rewritten, it is now easier for us to create diffs, which means upgrading the main more often, and making the “daily.cvd” smaller more often.

Second,  we now have the ability to offer different types of CVDs.  For instance, we now have the ability to distribute 3rd party signatures that are officially signed by ClamAV, but updated through the ClamAV global mirror network.  If we wanted to separate out “policy” type signatures from the daily.cvd into their own cvd, we can now do that.

Third, while we have not removed some of the older signature formats, we did convert those older signatures to the newer formats to empty those older “cvd”s out.

For example:
“db" signatures were consolidated into “ndb" signatures
“zmd" and “rmd"  archive signatures we moved to the “cdb" container signature format

These formats are not new, they simply have never been published before. This includes other formats such as “hsb", “msb", “sfp", and “crb".  The older formats are supported for now, we are simply no longer publishing them.

Fourth, newer features, like the ability to write signatures based on the SHA256 of a file have been added to the system, and we can now publish that type of detection.

We’d like to thank you for your patience.

Wednesday, March 9, 2016

ClamAV added support for a new family, or perhaps old family of documents, in the 0.99.1 release: Hangul Word Processor (HWP) documents. HWP documents are document files specialized in the Korean language and developed by Hancom Inc.. The documents’ specialization in Korean make it a highly popular format used in South Korea with the government being a notable example. As popular format, it is subjected to possibly malicious content.

For this release, we primarily targeted the word-specific documents: HWP 2.x, HWP 3.x, HWP 5.x, and HWPX. The other file formats developed by Hancom Inc. which cover the spreadsheet and presentation formats are already handled by pre-existing methods in ClamAV; the only exception is HPT (an old presentation format) which will not be in this release.
  • HWP 2.x, also known as HWPML
    • XML-based document format similar to Microsoft’s older XML document format
    • Contents of the document is stored in the XML including all embedded content
    •  Embedded content is usually base64-encoded and normally uses zlib compression
    • General embedded content is stored in OLE2 containers
    • File  property collection: document’s attributes and metadata fields
  • HWP 3.x, also known as HWP
    • Custom binary file format. For additional information on the format, the documentation can be retrieved from Hancom’s website (note that it’s in Korean)
    • Contents of HWP 3.x are stored in a file segment that uses optional password encryption and normally uses zlib compression
    • Embedded content is stored in the content stream with general embedded content stored in HWP-styled* OLE2 containers
    • File property collection: data from various file headers
  • HWP 5.x, also known as HWP
    • OLE2-based document format similar to Microsoft’s 97-2003 document formats
    • Contents of HWP 5.x are stored in individual streams with zlib compression normally used on specific content streams including embedded content
    • Embedded content is stored in individual streams under the BinData directory with general embedded content stored as HWP-style* OLE2 containers
    • File property collection: data from the fileheader stream which appears to be the HWP legacy header and the /x005HwpSummaryInformation stream which uses the same property method as 97-2003 Microsoft documents
  • HWPX
    • OOXML-compliant document format
    • Contents of the document are stored in XML documents within a ZIP archive
    • Embedded content is stored in the BinData directory with general embedded content stored as HWP-style* OLE2 containers
    • File property collection: data from the content.hpf document which is an xml document with the legacy file header.

*Hwp-styled OLE2 container are identical to normal OLE2 container with the exception that a 32-bit value is prepended to file/stream/data segment


    Monday, March 7, 2016

    As we periodically do, we will be releasing a new main.cvd and daily.cvd this coming weekend (March 12-13).

    We have completely re-written our signature management interface internally here at ClamAV, to provide us with more features and a better management process for cvd releases.  Part of the process will be to create these two new files.

    Releasing these files will have an impact, not only to the end user who will have to download these files, but also to our dedicated mirror operators that will be serving these files to the end users.

    The estimated size of these files are 100 MB and 10 MB respectively.

    Please be ready to receive these files this weekend.  It is our hope that everything goes smoothly and, other than bandwidth, users shouldn't experience any problems.  For any questions, please do not hesitate to contact us.


    Wednesday, March 2, 2016

    Join us in welcoming ClamAV 0.99.1 to the family!  It is ready for immediate download at ClamAV.net's download site.  As a reminder, this is the last release that will be done on SourceForge.net, and we encourage most people to move their downloads over to ClamAV.net immediately. 

    Below are the Release Notes:

    0.99.1
    ------

    ClamAV 0.99.1 contains a new feature for parsing Hancom Office files
    including extracting and scanning embedded objects. ClamAV 0.99.1
    also contains important bug fixes. Please see ChangeLog for details.

    Thanks to the following community members for code submissions used in
    ClamAV 0.99.1:

    Jim Morris
    Andreas Cadhalpun
    Mark Allan
    Sebastian Siewior


    Below is the ChangeLog:

    Wed, 24 Feb 2016 18:45:00 -0500 (Steven Morgan)
    ------------------------------------------
     * ClamAV 0.99.1 release build.

    Wed, 24 Feb 2016 13:29:42 -0500 (Kevin Lin)
    ------------------------------------------
     * hwp5.x: fix for streams without names

    Sat, 20 Feb 2016 15:53:48 +0100 (Sebastian Andrzej Siewior)
    ------------------------------------------
     * libclamav: yara: avoid unaliged access to 64bit variable

    Thu, 11 Feb 2016 15:56:47 -0500 (Mickey Sola)
    ------------------------------------------
     * bb11455 - patch by Mark Allan to add show-progress option to freshclam.

    Tue, 16 Feb 2016 14:15:18 -0500 (Kevin Lin)
    ------------------------------------------
     * added 'CustomXML' as trigger for likely OOXML

    Tue, 3 Feb 2016 17:30:00 -0500 (Steven Morgan)
    ------------------------------------------
     * ClamAV 0.99.1 beta1 release.

    Tue, 2 Feb 2016 14:23:13 -0500 (Kevin Lin)
    ------------------------------------------
     * add scanning options for scanning xml-based documents
    (MSXML, OOXML, HWPML) and HWP3

    Fri, 29 Jan 2016 16:17:40 -0500 (Kevin Lin)
    ------------------------------------------
     * add dconfs for XDP, MBR, GPT, APM, OOXML, MSXML, and HWP formats

    Tue, 2 Feb 2016 12:38:27 -0500 (Kevin Lin)
    ------------------------------------------
     * hwp: scan decompressed data on limits exceeded

    Sun, 31 Jan 2016 15:41:54 -0500 (Steven Morgan)
    ------------------------------------------
     * Fix for signature name length<3 in .ign & .ign2 CVD files.

    Fri, 29 Jan 2016 11:21:46 -0500 (Steven Morgan)
    ------------------------------------------
     * Change RTF file magic from '{\rtf' to '{\rt'

    Wed, 27 Jan 2016 14:17:47 -0500 (Mickey Sola)
    ------------------------------------------
     * bb11444 - zeroing out buffer at allocation to avoid writing uninitialized
    bytes to a file

    Wed, 20 Jan 2016 16:21:58 -0500 (Mickey Sola)
    ------------------------------------------
     * bb11446 - adding check for compressed input stream

    Wed, 13 Jan 2016 14:56:46 -0500 (Kevin Lin)
    ------------------------------------------
     * clean up and boost accuracy to detecting OOXML documents

    Thu, 14 Jan 2016 17:01:01 -0500 (Steven Morgan)
    ------------------------------------------
     * 11477 - Fix crash when using pcre statistics due to inconsistent memory
    management function calls

    Thu, 14 Jan 2016 15:31:02 -0500 (Mickey Sola)
    ------------------------------------------
     * bb11444: adding check to verify a valid number of rounds for rijndael aes
    encryption/decryption

    Wed, 13 Jan 2016 13:18:01 -0500 (Mickey Sola)
    ------------------------------------------
     * bb11441 - fixing edge case revealed by unit testing

    Tue, 12 Jan 2016 17:20:26 -0500 (Steven Morgan)
    ------------------------------------------
     * bb11466 - let html be scanned raw when --scan-html=no is in effect.

    Tue, 12 Jan 2016 16:14:40 -0500 (Steven Morgan)
    ------------------------------------------
     * bb11475: fix crash due to memory overlay with crafted 7z files.

    Tue, 12 Jan 2016 14:30:41 -0500 (Mickey Sola)
    ------------------------------------------
     * bb11441 - fixing off by one OOB write in htmlnorm

    Mon, 11 Jan 2016 14:47:30 -0500 (Mickey Sola)
    ------------------------------------------
     * bb11446 - fixing autoit OOB bufferread

    Wed, 6 Jan 2016 14:57:48 -0500 (Mickey Sola)
    ------------------------------------------
     * sigtool: adding support for decoding cdb sigs

    Tue, 5 Jan 2016 13:13:47 -0500 (Steven Morgan)
    ------------------------------------------
     * bb11467 - fix embedded http links.

    Tue, 22 Dec 2015 19:04:42 -0500 (Steven Morgan)
    ------------------------------------------
     * bb11451 - fix guard macro name collision on AIX.

    Tue, 22 Dec 2015 13:36:51 -0500 (Steven Morgan)
    ------------------------------------------
     * fix regression crash on sid 1005597703 due to uninitialzed
    (i.e., garbage initialized) structure.

    Mon, 21 Dec 2015 17:39:47 -0500 (Steven Morgan)
    ------------------------------------------
     * bb11456 - corrects mso stream prefix value for big endian processors.
    Patch by Jim Morris.

    Thu, 17 Dec 2015 16:16:55 -0500 (Kevin Lin)
    ------------------------------------------
     * ooxml_hwp: add support for filetyping and preclassification

    Thu, 17 Dec 2015 11:04:00 -0500 (Kevin Lin)
    ------------------------------------------
     * hwpole2: new filetype and handler for hwp embedded ole2 files

    Thu, 17 Dec 2015 12:46:38 -0500 (Mickey Sola)
    ------------------------------------------
     * bb11443 - fixing possible oob dereference when parsing mbox files

    Tue, 15 Dec 2015 16:12:01 -0500 (Mickey Sola)
    ------------------------------------------
     * bb11442 - adding explicit wwunpack oob checks

    Tue, 15 Dec 2015 10:53:05 -0500 (Mickey Sola)
    ------------------------------------------
     * bb11445 - fixing edge case where a null terminator was unintentionally
    written into an adjacent buffer.

    Wed, 16 Dec 2015 16:13:05 -0500 (Kevin Lin)
    ------------------------------------------
     * hwpml: use msxml_parser callback scanner for binary data

    Wed, 16 Dec 2015 16:12:20 -0500 (Kevin Lin)
    ------------------------------------------
     * msxml_parser: add callback-based scanning mechanism

    Tue, 15 Dec 2015 13:01:40 -0500 (Kevin Lin)
    ------------------------------------------
     * HWPML: added hwpml_keys for hwpml parsing

    Tue, 15 Dec 2015 10:54:20 -0500 (Kevin Lin)
    ------------------------------------------
     * add HMPML filetype, tab fixes in filetype.c

    Mon, 14 Dec 2015 16:34:11 -0500 (Kevin Lin)
    ------------------------------------------
     * Hwp3.x: inflate compressed segment and offset tracking

    Fri, 11 Dec 2015 17:50:40 -0500 (Kevin Lin)
    ------------------------------------------
     * Hwp3.x file header parsing and preclass

    Wed, 16 Dec 2015 14:56:56 -0500 (Kevin Lin)
    ------------------------------------------
     * pcre: fixed minimum pcre version check

    Fri, 11 Dec 2015 17:41:38 -0500 (Steven Morgan)
    ------------------------------------------
     * bb11452 - endianess correction for swf files on sparc -
    patch supplied by Jim Morris.

    Wed, 9 Dec 2015 12:15:16 -0500 (Mickey Sola)
    ------------------------------------------
     * bb11424 - correcting buffer alignemnt for riff detection on sparc systems;
    patch by Jim Morris

    Wed, 9 Dec 2015 00:17:41 +0100 (Andreas Cadhalpun)
    ------------------------------------------
     * fix detection of libcheck without pkg-config file

    Wed, 9 Dec 2015 11:43:35 -0500 (Kevin Lin)
    ------------------------------------------
     * add HWP5 filetype tracking to preclassification

    Tue, 8 Dec 2015 15:05:30 -0500 (Kevin Lin)
    ------------------------------------------
     * add HWP 3.x internal filetypes

    Tue, 8 Dec 2015 14:47:12 -0500 (Kevin Lin)
    ------------------------------------------
     * add hwp5 contents to preclass set

    Fri, 4 Dec 2015 11:55:52 -0500 (Kevin Lin)
    ------------------------------------------
     * proper identification of HWP file format

    Tue, 8 Dec 2015 18:36:48 -0500 (Steven Morgan)
    ------------------------------------------
     * bb11438 - strengthen file typing for OOXML.

    Wed, 2 Dec 2015 14:06:32 -0500 (Mickey Sola)
    ------------------------------------------
     * bb-11099 - normalization of whitelist signatures

    Wed, 2 Dec 2015 16:23:48 -0500 (Steven Morgan)
    ------------------------------------------
     * github issue #9 - wrong length & offset.