Tuesday, August 18, 2020

ClamAV 0.103.0 release candidate

Today, we are pleased to announce the ClamAV 0.103.0 release candidate.

Please help us validate this release. We need your feedback, so let us know what you find and join us on the ClamAV mailing list, in #clamav on irc.freenode.net, or on our Discord, which is bridged with our IRC.

Please submit bugs to our Bugzilla. 

ClamAV 0.103.0 includes the following improvements and changes.

 Major changes

  • clamd can now reload the signature database without blocking scanning. This multi-threaded database reload improvement was made possible thanks to a community effort.
    • Non-blocking database reloads are now the default behavior. Some systems that are more constrained on RAM may need to disable non-blocking reloads as it will temporarily consume two times as much memory. We added a new clamd config option ConcurrentDatabaseReload, which may be set to no.
Special thanks to who made this feature a reality:
  • Alberto Wu
  • Alexander Sulfrian
  • Arjen de Korte
  • David Heidelberg
  • Ged Haywood
  • Julius Plenz
  • Michael Orlitzky

Thank you all for your patience in waiting for this feature.
 

Notable changes

  • The DLP module has been enhanced with additional credit card ranges and a new engine option that allows ClamAV to alert only on credit cards (and not, for instance, gift cards) when scanning with the DLP module. John Schember developed this feature, with input from Alexander Sulfrian.
  • Support for Adobe Reader X PDF encryption, an overhaul of PNG scanning to detect PNG specific exploits, and a major change to GIF parsing, which makes it more tolerant to problematic files and adds the ability to scan overlays, all thanks to work and patches submitted by Aldo Mazzeo.
  • clamdtop.exe now available for Windows users. Functionality is somewhat limited when compared to clamdtop on Linux. PDCurses is required to build clamdtop.exe for ClamAV on Windows.
  • The phishing detection module will now print "Suspicious link found!" along with the "Real URL" and "Display URL" each time phishing is detected. In a future version, we would like to print out alert-related metadata like this at the end of a scan, but for now, this detail will help users understand why a given file is being flagged as phishing.
  • Added new *experimental* CMake build tooling. CMake is not yet recommended for production builds. Our team would appreciate any assistance improving the CMake build tooling so we can one day deprecate Autotools and remove the Visual Studio solutions.
    • Please see the new CMake installation instructions found in INSTALL.cmake.md for detailed instructions on how to build ClamAV with CMake.
  • Added --ping and --wait options to the clamdscan and clamonacc client applications.
    • The --ping (-p) command will attempt to ping clamd up to a specified maximum number of attempts at an optional interval. If the interval isn't specified, a default 1-second interval is used. It will exit with status code `0` when it receives a PONG from clamd or status code `21` if the timeout expires before it receives a response.
Example:
  • clamdscan -p 120 will attempt to ping clamd 120 at a 1 second interval.
    • The --wait (-w) command will wait up to 30 seconds for clamd to start. This option may be used in tandem with the --ping option to customize the max # of attempts and the attempt interval. As with --ping, the scanning client may exit with status code 21 if the timeout expires before a connection is made to clamd.
Example:
  • clamdscan -p 30:2 -w <file> will attempt a scan, waiting up to 60 seconds for clamd to start and receive the scan request.
    • The ping-and-wait feature is particularly useful for those wishing to start clamd and start clamonacc at startup, ensuring that clamd is ready before clamonacc starts. It is also useful for those wishing to start clamd immediately before initiating scans with clamdscan rather than having the clamd service run continuously.
  • Added Excel 4.0 (XLM) macro detection and extraction support. Significantly improved VBA detection and extraction as well. Work courtesy of Jonas Zaddach.
    • This support not yet added to sigtool, as the VBA extraction feature in sigtool is separate from the one used for scanning and will still need to be updated or replaced in the future.
  • Improvements to the layout and legibility of temp files created during a scan. Improvements to legibility and content of the metadata JSON generated during a scan.
To review the scan temp files and metadata JSON, run:

  clamscan --tempdir=<path> --leave-temps --gen-json <target>

Viewing the scan temp files and metadata.json file provides some insight into how ClamAV analyzes a given file and can also be useful to analysts for the initial triage of potentially malicious files.
 

Other improvements

  • Added ability for freshclam and clamsubmit to override default use of OpenSSL CA bundle with a custom CA bundle. On Linux/Unix platforms (excluding macOS), users may specify a custom CA bundle by setting the CURL_CA_BUNDLE environment variable. On macOS and Windows, users are expected to add CA certificates to their respective system's keychain/certificate store. Patch courtesy of Sebastian A. Siewior
  • clamscan and clamdscan now print the scan start and end dates in the scan summary.
  • The clamonacc on-access scanning daemon for Linux now installs to sbin instead of bin.
  • Improvements to the freshclam progress bar so the width of the text does not  shift around as information changes and will not spill exceed 80 characters, even on very slow connections. Time is now displayed in Xm XXs (or Xh XXm) for values of 60 seconds or longer. Bytes display now changes units at the proper 1024 B/KiB instead of 2048 B/KiB. Patch courtesy of Zachary Murden.
  • Improve column alignment and line wrap rendering for clamdtop. Also fixed an issue on Windows where clamdtop would occasionally disconnect from clamd and fail to reconnect. Patch courtesy of Zachary Murden.
  • Improvements to the AutoIT parser.
  • Loosened the curl version requirements in order to build and use clamonacc. You may now build ClamAV with any version of libcurl. However clamonacc's file descriptor-passing (FD-passing) capability will only be available with libcurl 7.40 or newer. FD-passing is ordinarily the default way to perform scans with clamonacc as it is significantly faster than streaming.
  • Added LZMA and BZip2 decompression routines to the bytecode signature API.
  • Disabled embedded type recognition for specific archive and disk image file types. This change reduces file type misclassification and improves scan time performance by reducing duplicated file scanning.

Bug fixes

  • Fixed issue scanning directories on Windows with clamdscan.exe that was introduced when mitigating against symlink quarantine attacks.
  • Fixed behavior of freshclam --quiet option. Patch courtesy of Reio Remma.
  • Fixed behavior of freshclam's OnUpdateExecute, OnErrorExecute, and OnOutdatedExecute config options on Windows when in daemon-mode so it can handle multiple arguments. Patch courtesy of Zachary Murden.
  • Fixed an error in the heuristic alert mechanism that would cause a single detection within an archive to alert once for every subsequent file scanned, potentially resulting in thousands of alerts for a single scan.
  • Fixed clamd, clamav-milter, and freshclam to create PID files before dropping privileges, to avoid the possibility of an unprivileged user from changing the PID file so that a service manager will kill a different process. This change does make the services unable to clean up the PID file on exit.
  • Fixed the false positive (.fp) signature feature. In prior versions, the hash in a false positive signature would be checked only against the current layer of a file being scanned. In 0.103, every file layer is hashed, and the hashes for each in the scan recursion list are checked. This ensures that .fp signatures containing a hash for any layer in the scan leading up to the alert will negate the alert.
    • As an example, a hash for a zip containing the file which alerts would not prevent the detection in prior versions. Only the hash of the embedded file would work. For some file types where the outermost is always an archive, eg. docx files, this made .fp signatures next to useless. For certain file types where the scanned content was a normalized version of the original content, eg. HTML, the normalized version was never hashed and this meant that .fp signatures never worked.
  • Fixed Trusted & Revoked Windows executable (PE) file signature rules (.crb) maximum functionality level (FLEVEL) which had been being treated as the minimum FLEVEL. These signatures enable ClamAV to trust executables that are digitally signed by trusted publishers, or to alert on executables signed with compromised signing-certificates. The minimum and maximum FLEVELS enable or disable signatures at load time depending on the current ClamAV version.
  • Fixed a bug wherein you could not build ClamAV with --enable-libclamav-only if curl was not installed on the system.
  • Various other bug fixes, improvements, and documentation improvements.
 

New Requirements

  • Autotools (automake, autoconf, m4, pkg-config, libtool) are now required in order to build from a Git clone because the files generated by these tools have been removed from the Git repository. To generate theses files before you compile ClamAV, run autogen.sh. Users building with Autotools from the release tarball should be unaffected.
  • Flex and Bison are now required in order to build from a Git clone. Flex and Bison are also required to build with CMake.  Users building with Autotools from the release tarball should be unaffected.
 

Acknowledgments

The ClamAV team thanks the following individuals for their code submissions:

  • Aldo Mazzeo
  • Ángel
  • Antonino Cangialosi
  • Clement Lecigne
  • Jamie Biggar
  • Jan Smutny
  • John Schember
  • Jonathan Sabbe
  • lutianxiong
  • Reio Remma
  • Sebastian A. Siewior
  • Zachary Murden 

at