ClamAV, CVDs, CDIFFs and the magic behind the curtain

The amount of malicious files that ClamAV can detect has increased immensely over the past few years, but with this increase in efficacy comes some challenges with scale.  

Some of these challenges have required drastic measures to ensure the effective operation of the ClamAV infrastructure, including blocking certain methods of downloading the official ClamAV signature sets. To give the community more insight into these matters, we’d like to discuss some of these challenges in-depth and provide insight into future changes and optimizations coming to the product.

ClamAV EOL versions prior to 0.100

ClamAV community, we want to inform you that, effective March 1, ClamAV 0.99.0 (and all minor versions) will no longer be supported in accordance with ClamAV's EOL policy.  For clarification, anything prior to 0.100.0.

End of life (EOL) for ClamAV essentially means that we will no longer be testing against that version when we write signatures, and we may break something with a future release. So, while signatures may work past March 1, we are no longer testing that configuration. 

Please upgrade to the newest version of ClamAV, currently at 0.103.1, available for download now!

As always, thank you for using ClamAV.

ClamAV 0.103.1 patch release

ClamAV 0.103.1 is out now. Users can head over to clamav.net/downloads to download the release materials.

 

The latest version of ClamAV contains the following fixes and improvements:

Notable changes

•  Added a new scan option to alert on broken media (graphics) file formats.
  
  This feature mitigates the risk of malformed media files intended to exploit vulnerabilities in other software. At present, media validation exists for JPEG, TIFF, PNG and GIF files. To enable this feature, set AlertBrokenMedia yes in clamd.conf for use with ClamD, or use the --alert-broken-media option when using ClamScan. These options are disabled by default in this patch release but may be enabled in a subsequent release.
  
  Application developers may enable this scan option by enabling CL_SCAN_HEURISTIC_BROKEN_MEDIA for the heuristic scan option bit field.
  

• Added CL_TYPE_TIFF, CL_TYPE_JPEG types to match GIF and PNG typing behavior. BMP and JPEG 2000 files will continue to detect as CL_TYPE_GRAPHICS because ClamAV does not yet have BMP or JPEG 2000 format-checking capabilities.
  

 Bug fixes

• Fixed PNG parser logic bugs that caused an excess of parsing errors and fixed a stack exhaustion issue affecting some systems when scanning PNG files. PNG file type detection was disabled via signature database update for ClamAV version 0.103.0 to mitigate the effects from these bugs.
  

• Fixed an issue where PNG and GIF files no longer work with Target:5 graphics signatures if detected as CL_TYPE_PNG or CL_TYPE_GIF rather than as CL_TYPE_GRAPHICS. Target types now support up to 10 possible file types to make way for additional graphics types in future releases.
  

• Fixed ClamOnAcc's --fdpass option.
  
  File descriptor passing (or "FD-passing") is a mechanism by which ClamOnAcc and ClamDScan may transfer an open file to ClamD to scan, even if ClamD is running as a non-privileged user and wouldn't otherwise have read-access to the file. This enables ClamD to scan all files without having to run ClamD as root. If possible, ClamD should never be run as root to mitigate the risk in case ClamD is somehow compromised while scanning malware.
  
  Interprocess file descriptor passing for ClamOnAcc was broken since version 0.102.0 due to a bug introduced by the switch to cURL for communicating with ClamD. On Linux, passing file descriptors from one process to another is handled by the kernel, so we reverted ClamOnAcc to use standard system calls for socket communication when FD-passing is enabled.
  

• Fixed a ClamOnAcc stack corruption issue on some systems when using an older version of libcurl. Patch courtesy of Emilio Pozuelo Monfort.
  

• Allow ClamScan and ClamDScan scans to proceed even if the realpath lookup failed. This alleviates an issue on Windows scanning files hosted on file-systems that do not support the GetMappedFileNameW() API, such as on ImDisk RAM-disks.
  

• Fixed FreshClam's --on-update-execute=EXIT_1 temporary directory cleanup issue.

• ClamD's log output and VirusEvent feature now provide the scan target's file path instead of a file descriptor. The ClamD socket API for submitting a scan by FD-passing doesn't include a file path. This feature works by looking up the file path by the file descriptor. This feature works on Mac and Linux but is not yet implemented for other UNIX operating systems. FD-passing is not available for Windows.
  

• Fixed an issue where FreshClam database validation didn't work correctly when run in daemon mode on Linux/Unix.

• Fixed scan speed performance issues accidentally introduced in ClamAV 0.103.0 caused by hashing file maps more than once when parsing a file as a new type, and caused by frequent scanning of non-HTML text data with the HTML parser.
  

Other improvements

• Scanning JPEG, TIFF, PNG and GIF files will no longer return "parse" errors when file format validation fails. Instead, the scan will alert with the "Heuristics.Broken.Media" signature prefix and a descriptive suffix to indicate the issue, provided that the "alert broken media" feature is enabled.
  

• GIF format validation will no longer fail if the GIF image is missing the trailer byte, as this appears to be a relatively common issue in otherwise functional GIFs.
  

• Added a TIFF dynamic configuration (DCONF) option that was missing. This will allow us to disable TIFF format validation via signature database update in the event that it proves to be problematic. This feature already exists for many other file types.
  

Acknowledgments

The ClamAV team thanks Emilio Pozuelo Monfort for their code submissions.

ClamAV 0.103.0 released

With your feedback on previous candidates, the official release of ClamAV version 0.103.0 is available now on our Downloads page.

ClamAV 0.103.0 highlights

With your feedback on the previous candidates, we've fixed these additional issues:

• The freshclam PID file was not readable by other users in previous release candidates but is now readable by all.
• An issue with how freshclam was linked with the autotools build system caused SysLog settings to be ignored.
• The real-path checks introduced to clamscan and clamdscan in 0.102.4 broke scanning of some files with Unicode filenames and files on network shares for Windows users.

Thanks to the users for your help in fixing these bugs.

Major changes

• clamd can now reload the signature database without blocking scanning. This multi-threaded database reload improvement was made possible thanks to a community effort.
• Non-blocking database reloads are now the default behavior. Some systems that are more constrained on RAM may need to disable non-blocking reloads, as it will temporarily consume double the amount of memory. We added a new clamd config option ConcurrentDatabaseReload, which may be set to no.

Special thanks to those who made this feature a reality:

• Alberto Wu
• Alexander Sulfrian
• Arjen de Korte
• David Heidelberg
• Ged Haywood
• Julius Plenz
• Michael Orlitzky

Notable changes

• The DLP module has been enhanced with additional credit card ranges and a new engine option that allows ClamAV to alert only on credit cards (and not, for instance, gift cards) when scanning with the DLP module. John Schember developed this feature, with input from Alexander Sulfrian.

• We added support for Adobe Reader X PDF encryption and overhauled the PNG-scanning tool to detect PNG-specific exploits. We also made a major change to GIF parsing that now makes it more tolerant of problematic files and adds the ability to scan overlays, all thanks to work and patches submitted by Aldo Mazzeo.

• clamdtop.exe is now available for Windows users. The functionality is somewhat limited when compared to clamdtop on Linux. PDCurses is required to build clamdtop.exe for ClamAV on Windows.

• The phishing detection module will now print "Suspicious link found!" along with the "Real URL" and "Display URL" each time ClamAV detects phishing. In a future version, we would like to print out alert-related metadata like this at the end of a scan, but for now, this detail will help users understand why a given file is being flagged as phishing.

• Added new *experimental* CMake build tooling. CMake is not yet recommended for production builds. Our team would appreciate any assistance improving the CMake build tooling so we can one day deprecate autotools and remove the Visual Studio solutions.
• Please see the new CMake installation instructions found in INSTALL.cmake.md for detailed instructions on how to build ClamAV with CMake.

• Added --ping and --wait options to the clamdscan and clamonacc client applications.
• The --ping (-p) command will attempt to ping clamd up to a specified maximum number of attempts at an optional interval. If the interval isn't specified, a default one-second interval is used. It will exit with status code `0` when it receives a PONG from clamd or status code `21` if the timeout expires before it receives a response.

Example:

• clamdscan -p 120 will attempt to ping clamd 120 at a one-second interval.
• The --wait (-w) command will wait up to 30 seconds for clamd to start. This option may be used in tandem with the --ping option to customize the max # of attempts and the attempt interval. As with --ping, the scanning client may exit with status code 21 if the timeout expires before a connection is made to clamd.

Example:

• clamdscan -p 30:2 -w <file> will attempt a scan, waiting up to 60 seconds for clamd to start and receive the scan request.
• The ping-and-wait feature is particularly useful for those wishing to start clamd and start clamonacc at startup, ensuring that clamd is ready before clamonacc starts. It is also useful for those wishing to start clamd immediately before initiating scans with clamdscan rather than having the clamd service run continuously.

• Added Excel 4.0 (XLM) macro detection and extraction support. Significantly improved VBA detection and extraction as well. Work courtesy of Jonas Zaddach.
• This support not yet added to sigtool, as the VBA extraction feature in sigtool is separate from the one used for scanning and will still need to be updated or replaced in the future.

• Improvements to the layout and legibility of temp files created during a scan. Improvements to legibility and content of the metadata JSON generated during a scan.

To review the scan temp files and metadata JSON, run:

  clamscan --tempdir=<path> --leave-temps --gen-json <target>

Viewing the scan temp files and metadata.json file provides some insight into how ClamAV analyzes a given file and can also be useful to analysts for the initial triage of potentially malicious files.

Other improvements

• Added ability for freshclam and clamsubmit to override default use of OpenSSL CA bundle with a custom CA bundle. On Linux/Unix platforms (excluding macOS), users may specify a custom CA bundle by setting the CURL_CA_BUNDLE environment variable. On macOS and Windows, users are expected to add CA certificates to their respective system's keychain/certificate store. Patch courtesy of Sebastian A. Siewior.

• clamscan and clamdscan now print the scan start and end dates in the scan summary.

• The clamonacc on-access scanning daemon for Linux now installs to sbin instead of bin.

• Improvements to the freshclam progress bar to ensure the text does not shift around as information changes. The bar will not exceed 80 characters, even on very slow connections. Time is now displayed in Xm XXs (or Xh XXm) for values of 60 seconds or longer. The bytes display now changes units at the proper 1024 B/KiB instead of 2048 B/KiB. Patch courtesy of Zachary Murden.

• Improve column alignment and line wrap rendering for clamdtop. Also fixed an issue on Windows where clamdtop would occasionally disconnect from clamd and fail to reconnect. Patch courtesy of Zachary Murden.

• Improvements to the AutoIT parser.

• Loosened the curl version requirements in order to build and use clamonacc. You may now build ClamAV with any version of libcurl. However, clamonacc's file descriptor-passing (FD-passing) capability will only be available with libcurl 7.40 or newer. FD-passing is ordinarily the default way to perform scans with clamonacc, as it is significantly faster than streaming.

• Added LZMA and BZip2 decompression routines to the bytecode signature API.

• Disabled embedded type recognition for specific archive and disk image file types. This change reduces file type misclassification and improves scan time performance by reducing duplicated file scanning.
  

• Use pkg-config to detect libpcre2-8 before resorting to pcre2-config or pcre-config.
  Patch courtesy of Michael Orlitzky.
  

Bug fixes

• Fixed issue scanning directories on Windows with clamdscan.exe that was introduced when mitigating against symlink quarantine attacks.

• Fixed behavior of freshclam --quiet option. Patch courtesy of Reio Remma.

• Fixed behavior of freshclam's OnUpdateExecute, OnErrorExecute, and OnOutdatedExecute config options on Windows when in daemon-mode so it can handle multiple arguments. Patch courtesy of Zachary Murden.

• Fixed an error in the heuristic alert mechanism that would cause a single detection within an archive to alert once for every subsequent file scanned, potentially resulting in thousands of alerts for a single scan.

• Fixed clamd, clamav-milter, and freshclam to create PID files before dropping privileges, to avoid the possibility of an unprivileged user from changing the PID file so that a service manager will kill a different process. This change does make the services unable to clean up the PID file on exit.

• Fixed the false positive (.fp) signature feature. In prior versions, the hash in a false positive signature would be checked only against the current layer of a file being scanned. In 0.103, every file layer is hashed, and the hashes for each in the scan recursion list are checked. This ensures that .fp signatures containing a hash for any layer in the scan leading up to the alert will negate the alert.
• As an example, a hash for a zip containing the file which alerts would not prevent the detection in prior versions. Only the hash of the embedded file would work. For some file types where the outermost is always an archive, eg. docx files, this made .fp signatures next to useless. For certain file types where the scanned content was a normalized version of the original content, eg. HTML, the normalized version was never hashed and this meant that .fp signatures never worked.

• Fixed Trusted & Revoked Windows executable (PE) file signature rules (.crb) maximum functionality level (FLEVEL) which had been being treated as the minimum FLEVEL. These signatures enable ClamAV to trust executables that are digitally signed by trusted publishers or to alert on executables signed with compromised signing-certificates. The minimum and maximum FLEVELS enable or disable signatures at load time depending on the current ClamAV version.

• Fixed a bug wherein you could not build ClamAV with --enable-libclamav-only if curl was not installed on the system.

• Various other bug fixes, improvements and documentation improvements.

New Requirements

• Autotools (automake, autoconf, m4, pkg-config, libtool) are now required to build from a Git clone because the files generated by these tools have been removed from the Git repository. To generate these files before you compile ClamAV, run autogen.sh. Users building with autotools from the release tarball should be unaffected.

Acknowledgments

The ClamAV team thanks the following individuals for their code submissions:

• Aldo Mazzeo
• Ángel
• Antonino Cangialosi
• Clement Lecigne
• Jamie Biggar
• Jan Smutny
• Jim Klimov
  
• John Schember
• Jonathan Sabbe
• lutianxiong
• Michael Orlitzky
  
• Reio Remma
• Sebastian A. Siewior
• Zachary Murden 

ClamAV 0.103.0 second release candidate

Thanks to everyone who looked at our first release candidate for ClamAV 0.103.0 over the past few weeks. With your help, we found and fixed a few major issues. 

To make sure everything is in order, we published a second release candidate for ClamAV 0.103.0 Tuesday evening.

This second release candidate fixes a couple major issues and a handful of other issues that were reported these past two weeks. Some notable bug fixes and improvements added since our first release candidate include:

• Fixed clamd and clamav-milter service/daemon start issue when starting as root and switching users. This issue discussed in the mailing list and reported on Bugzilla.
  
• Fixed a build issue when libcheck is not installed. This issue reported on Bugzilla.
  
• Fixed a Windows issue using the clamscan.exe & clamdscan.exe's --remove option. This issue reported on Bugzilla.
  
• Added pkg-config support for finding pcre2 and ncurses when using the Autotools build system. pcre2 pkg-config support contributed by Michael Orlitzky via GitHub.com.
  
• Reverted the change that requires Bison & Flex when building from a Git clone. Bison & Flex-generated sources will remain in our Git repository. When building with CMake, they can be re-generated by enabling "maintainer mode."
• Fixed bugs in the ARJ and XAR archive parsers.
  

To see the full list of changes new in ClamAV 0.103, please see the 0.103.0-rc1 announcement.

Once again, we would very much appreciate your time evaluating this second release candidate. Please share your experience with us on the ClamAV mailing list, in #clamav on irc.freenode.net, or on our Discord, which is bridged with our IRC. Please continue to submit bugs to our Bugzilla. 

ClamAV 0.103.0 release candidate

Today, we are pleased to announce the ClamAV 0.103.0 release candidate.

Please help us validate this release. We need your feedback, so let us know what you find and join us on the ClamAV mailing list, in #clamav on irc.freenode.net, or on our Discord, which is bridged with our IRC.

Please submit bugs to our Bugzilla. 

ClamAV 0.103.0 includes the following improvements and changes.

Freshclam, cdiffs and bandwidth are your friends

During a recent review of file downloads from our ClamAV CDN network, we've noticed hundreds of IPs that seem to be downloading the daily.cvd and the main.cvd thousands of times a day.

There are about a dozen IPs that are downloading those to files more than 40,000 times a day. This is causing us to transfer about 250TB of data a day. We would encourage any users still doing this to cease as soon as possible. Not only does it waste our bandwidth — as we have much more efficient ways of downloading the updates — but it also wastes your bandwidth, as well.

Freshclam has the ability to download partial files of updates (called cdiffs).  Which are smaller, more incremental updates to the database. This allows users, and us, to manage our downloads in a much more efficient manner. We often receive the complaint, "I have to download the daily.cvd and main.cvd with Python and move the updates to an off-internet system."  That's fine — it's a use case we support. However, you can do the same with freshclam and the small cdiffs.

Furthermore, we also only release updates once a day.  Reducing the number of updates you check for (and, subsequently, download we assume through a crontab or periodic job of some type) would also alleviate this issue.

We will be constantly monitoring this in hopes that people migrate to using freshclam.  Over-abusers (for instance, the top 10 IPs that are downloading main.cvd 40,000 times a day), will be immediately blocked.  Further abusers may also be blocked, without notice.

To mitigate, please complete the following tasks:

1. Use Freshclam instead of Python or whatever downloading script you have cron'd.
2. Reduce the checks to once or twice a day.

Thank you for helping keep the ClamAV network healthy.

Any questions, please see us over on the ClamAV-Users list.

Writing a ClamAV signature for Masslogger

By Nikhil Hegde.

MassLogger is a .NET executable and an info-stealer having the ability to take screenshots, log keystrokes, etc. The binary features two obfuscated loaders. The first loader decrypts the second using the Rijndael algorithm. The second loader decodes the payload using the values in a Bitmap image. The presence of multiple loaders and obfuscation made it a good choice for a walkthrough of ClamAV signature creation. In this post, we’ll walk through the creation of Clam signatures for this malware. It is our hope that this gives you a closer look at the work we do and reverse-engineer malware.

I picked up the sample hash, 2b7455d2a9434cfe516d9d886248b45f1073c0cc9fef73b15e9a1ef187fe4677 from a tweet by Nocturnus. The sample is available on VirusTotal.

ClamAV 0.102.4 security patch released

ClamAV 0.102.4 is out now. Users can head over to clamav.net/downloads to download the release materials.

ClamAV 0.102.4 is a bug patch release to address the following issues:

CVE-2020-3350

Fixed a vulnerability a malicious user could exploit to replace a scan target's directory with a symlink to another path to trick clamscan, clamdscan, or clamonacc into removing or moving a different file (such as a critical system file). The issue would affect users that use the --move or --remove options for clamscan, clamdscan and clamonacc.

For more information about AV quarantine attacks using links, see RACK911 Lab's report.

CVE-2020-3327

Fixed a vulnerability in the ARJ archive-parsing module in ClamAV 0.102.3 that could cause a denial-of-service (DoS) condition. Improper bounds checking resulted in an out-of-bounds read that could cause a crash. The previous fix for this CVE in version 0.102.3 was incomplete. This fix correctly resolves the issue.

CVE-2020-3481

Fixed a vulnerability in the EGG archive module in ClamAV 0.102.0 - 0.102.3 that could cause a denial-of-service (DoS) condition. Improper error handling could cause a crash due to a NULL pointer dereference. This vulnerability is mitigated for those using the official ClamAV signature databases because the file type signatures in daily.cvd will not enable the EGG archive parser in affected versions.

We will be publishing a release candidate for version 0.103.0 in the next couple of weeks. Stay tuned!

The future of the ClamAV safebrowsing database

ClamAV has provided a signature database using Google's Safebrowsing API to provide advanced protection against emails with links to suspicious websites.

As of Nov. 11, 2019, we have stopped updating the Safebrowsing signature database because Google announced changes to their Safebrowsing API terms of service.

Google now requires commercial users to use the Google Web Risk API, a for-profit feature, instead of the Safebrowsing API. Though ClamAV itself is free and open-source, we cannot continue to provide Google Safebrowsing data to the general public.

Today, we're pleased to open-source a new tool for users to generate their own safebrowsing signature databases using their personal Safebrowsing API credentials for use in accordance with the policies of the Safebrowsing API user agreement.

Please check out the clamav-safebrowsing repository on Github to get started.