Tuesday, March 26, 2019

ClamAV 0.101.2

ClamAV 0.101.2 is a patch release to address a handful of security related bugs.

This patch release is being released alongside the 0.100.3 patch so that users
who are unable to upgrade to 0.101 due to libclamav API changes are protected.

This release includes 3 extra security related bug fixes that do not apply to
prior versions.  In addition, it includes a number of minor bug fixes and
improvements.

- Fixes for the following vulnerabilities affecting 0.101.1 and prior:
  - CVE-2019-1787:
    An out-of-bounds heap read condition may occur when scanning PDF
    documents. The defect is a failure to correctly keep track of the number
    of bytes remaining in a buffer when indexing file data.
  - CVE-2019-1789:
    An out-of-bounds heap read condition may occur when scanning PE files
    (i.e. Windows EXE and DLL files) that have been packed using Aspack as a
    result of inadequate bound-checking.
  - CVE-2019-1788:
    An out-of-bounds heap write condition may occur when scanning OLE2 files
    such as Microsoft Office 97-2003 documents. The invalid write happens when
    an invalid pointer is mistakenly used to initialize a 32bit integer to
    zero. This is likely to crash the application.

- Fixes for the following vulnerabilities affecting 0.101.1 and 0.101.0 only:
  - CVE-2019-1786:
    An out-of-bounds heap read condition may occur when scanning malformed PDF
    documents as a result of improper bounds-checking.
  - CVE-2019-1785:
    A path-traversal write condition may occur as a result of improper input
    validation when scanning RAR archives. Issue reported by aCaB.
  - CVE-2019-1798:
    A use-after-free condition may occur as a result of improper error
    handling when scanning nested RAR archives. Issue reported by David L.

- Fixes for the following assorted bugs:
  - Added checks to prevent shifts from causing undefined behavior in HTML
    normalizer, UPX unpacker, ARJ extractor, CPIO extractor, OLE2 parser,
    LZW decompressor used in the PDF parser, Xz decompressor, and UTF-16 to
    ASCII transcoder.
  - Added checks to prevent integer overflow in UPX unpacker.
  - Fix for minor memory leak in OLE2 parser.
  - Fix to speed up PDF parser when handling truncated (or malformed) PDFs.
  - Fix for memory leak in ARJ decoder failure condition.
  - Fix for potential memory and file descriptor leak in HTML normalization code.

- Removed use of problematic feature that converted file descriptors to
  file paths. The feature was intended to improve performance when scanning
  file types, notably RAR archives, for which the API requires a file path.
  This feature caused issues in environments where the ClamAV engine is run
  in a low-permissions or sandboxed process. RAR archives are still supported
  with this change, but performance may suffer slightly if the file path is not
  provided in calls to `cl_scandesc_callback()`.
  - Added filename and tempfile names to scandesc calls in clamd.
  - Added general scan option `CL_SCAN_GENERAL_UNPRIVILEGED` to treat the scan
    engine as unprivileged, meaning that the scan engine will not have read
    access to the file. Provided file paths are for logging purposes only.
  - Added ability to create a temp file when scanning RAR archives when the
    process does not have read access to the file path provided (i.e.
    unprivileged is set, or an access check fails).

Thank you to the Google OSS-Fuzz project for identifying and reporting many of
the bugs patched in this release.

Additional thanks to the following community members for submitting bug reports:

- aCaB
- David L.

ClamAV 0.100.3

ClamAV 0.100.3 is a patch release to address a few security related bugs.

This patch release is being released alongside the 0.101.2 patch so that users
who are unable to upgrade to 0.101 due to libclamav API changes are protected.

The bug fixes in this release are limited to security-related bugs only.
Users are encouraged to upgrade to 0.101.2 for additional improvements.

- Fixes for the following vulnerabilities:
  - CVE-2019-1787:
    An out-of-bounds heap read condition may occur when scanning PDF
    documents. The defect is a failure to correctly keep track of the number
    of bytes remaining in a buffer when indexing file data.
  - CVE-2019-1789:
    An out-of-bounds heap read condition may occur when scanning PE files
    (i.e. Windows EXE and DLL files) that have been packed using Aspack as a
    result of inadequate bound-checking.
  - CVE-2019-1788:
    An out-of-bounds heap write condition may occur when scanning OLE2 files
    such as Microsoft Office 97-2003 documents. The invalid write happens when
    an invalid pointer is mistakenly used to initialize a 32bit integer to
    zero. This is likely to crash the application.

Thank you to the Google OSS-Fuzz project for identifying and reporting the bugs
patched in this release.

Both of these can be found on ClamAV's downloads site, with 0.101.2 as the main release and 0.100.3 under "Previous Stable Releases"

Wednesday, February 20, 2019


The ClamAV Development team is looking to add a new C programmer to the team, located in Fulton, Maryland USA!

Our team is a small and friendly one. At present, we have two software engineers and one quality assurance engineer. All three of us work in the Cisco office in Fulton, MD where we occupy a comfortable corner complete with our own snack area and Nerf gun arsenal. And we're not simply legacy software maintainers: we actively design and engineer new features while improving existing ones. If you’re seeking a long-term, head-in-the-sand, maintenance job, look elsewhere. Our team is constantly engaged. From discussing new ways to improve our processes, to opening ourselves to new concepts, our willingness to learn and explore new ideas leads us to write better code, test more effectively, and ultimately build a better product.

As a member of the ClamAV team, you wouldn’t be working in a vacuum, and you wouldn’t get stuck fixing bugs in legacy software day-in and day-out. The team has an ever-evolving list of cool, new ideas to work on. Project planning is a collaborative effort. We derive requirements from community requests as well as from our partners in Talos Malware Research, Talos Web Team, and stakeholders across a handful of Cisco product groups. We work as a team to find the best way to tackle the more difficult challenges. The ClamAV Dev team is part of a larger group of skilled developers, researchers, and analysts, with whom we collaborate daily. Working on ClamAV, you'll always have a friendly and highly-skilled talentpool from which to draw upon and share ideas.

Our team performs for a wide audience. ClamAV is oft regarded as an anti-malware product, but our tools do more than protect endpoint devices. In fact, ClamAV is run on most mail servers and many web servers on the internet, protecting the world from malware transmitted via email and file upload. What’s more, our open source scanning and matching technology is baked into many of Cisco's core products. As a consequence, our team is responsible for maintaining a healthy relationship not only with the open source community but also with those Cisco product development teams that integrate ClamAV into their software.

Speaking of Talos, the ClamAV team is a core component of the largest and coolest commercial threat intelligence organization on the planet. Our offices are new and shiny. Our workforce is skilled, but also casual. We enjoy our work, and we know how to relax, too. That means our office has an active roster of video gamers, board gamers, ping pong players, and more.

Ok … so the work sounds good to you, and the office sounds great too… but your schedule can be a little crazy, and sometimes you need to stay home in your pajamas. No problem. Our team generally works from home once a week. Usually we work from home on Fridays, but things can be adjusted as needed to accomodate life's inescapable work-week appointments.

Does this job sounds too good to be true? Don't take my word for it. This year, Cisco placed #6 in Fortune’s rankings of the 100 best companies to work for! Our offices in Fulton, MD and in San Jose, CA are no exception.

To see the required skills, please check out the job listing on our careers page on TalosIntelligence.com.

Friday, February 15, 2019

We are currently experiencing issues with the safebrowsing publishing system.  This means that we are not able to currently publish a new safebrowsing.cvd file.

We are working on resolving the issue and will post an update when this is corrected.


ClamAV has adopted the use of Clang-Format for the purposes of improving the readability and maintainability of our code base.

Contributors to ClamAV, and those wishing to adopt the same format rules for their own projects, are advised to use the latest version of Clang-Format available. Some of the rules we have selected require version 7 as a minimum. Details are included below on how to install Clang-Format, including recommended plugins for popular text editors.

Some of our readers may be concerned that auto-formatting the code base will make it difficult to identify when a bug was introduced. The concern is valid but we have decided that the benefits in code readability outweighed the extra steps that may be needed when viewing the git history. As a reminder, `git blame` and `git diff` have the `-w` option allowing you to ignore changes to white-space which can help tremendously when analyzing changes. The ignore-whitespace feature can also be enabled in most Git plugins for text editors, such as VSCode’s GitLens extension.

Auto-formatting exemptions

In a few instances, we had to modify the code to ensure that Clang-Format would not incorrectly format the code. This is because there are still some cases where manually formatted code is more legible than Clang-Format auto-formatted code.

Situations where formatting requires some manual finesse:

  1. The `try` keyword may not be used as a variable name because Clang-Format will interpret it in the context of exception handling. Avoid the naming variables with reserved words such as `try` to alleviate such issues.
  2. Programmers often choose to align consecutive macros to make related constants easier to identify at a glance. The AlignConsecutiveMacros Clang-Format feature is still a work in progress so we are not yet able to automatically align consecutive macros.
  3. A similar issue to the alignment of consecutive macros is the alignment of large array and large struct declarations. Clang-Format generally does an okay job formatting single and multi-dimensional array and/or structure declarations. However, manual formatting in this case is generally better.

To address items 2 and 3 above, we have resorted to using the `// clang-format on/off` markers to protect specific blocks of code from being modified by Clang-Format.

ClamAV code examples where manual formatting was better than Clang-Format:
Preparatory steps taken to ensure correct formatting when working with Clang-Format can be found here.

This commit in our Git history implemented our Clang-Format rules for the first time. You may find it interesting to peruse to see how the rules affected the code base.

Clang-Format Usage

To support our chosen code format style, we have included the following in the clamav-devel dev/0.102 branch:


Formatting options:
  • Run the following in a directory containing a “.clang-format” ruleset. It will apply the rules in-place to the file or directory you specified:

    ~/workspace/clamav-devel$ clang-format -i <filepath>
  • If you’re working regularly with ClamAV, or another project configured to use Clang-Format, you may wish to set the workspace settings for your editor to format-on-save automatically for you.
    IMPORTANT: Be sure that you do NOT enable format-on-save globally for all projects, or you may inadvertently make large changes to code in projects that do not provide a Clang-Format ruleset.
  • If you have made many changes across the ClamAV code base, you may wish to run our clam-format convenience script:

    ~/workspace/clamav-devel$ ./clam-format
    This script does three things:
    • Re-generate the .clang-format config, specifying the specific format style rules our team has chosen to enforce (as described in the next section).
    • Update the format for all ClamAV-owned code in the repository in-place.
    • Undo changes from step (2) to specific files that should not be reformatted.

The Clam C/C++ Format Style

ClamAV’s format style was chosen not only because we find it attractive and legible, but also because it aligned best with the existing ClamAV code prior to enacting the new rules. We have done our best to minimize frivolous changes to the code base using the options that Clang-Format provides.

The Clam C/C++ Format Style uses the following Clang-Format options:

Language: Cpp, 

These rules only apply to C & C++ code.

UseTab: Never,
IndentWidth: 4,

Code must not use tabs for indentation, and must use 4 spaces for indentation.

AlignTrailingComments: true,

Comments after code on consecutive lines will be aligned.

Example:
int a;     // My comment a
int b = 2; // comment b


AlignConsecutiveAssignments: true,

Variable assignments on consecutive lines will be aligned.

Example:
int aaaa = 12;
int b    = 23;

int ccc  = 23;

AlignAfterOpenBracket: true,

Function arguments split onto multiple lines will be aligned.

Example:

someLongFunction(argument1,
                 argument2);


AlignEscapedNewlines: Left, 

Backslashes for escaped newlines will be aligned as far left as possible.

Example:
#define A     \
    int aaaa; \
    int b;    \
    int dddddddddd;


AlignOperands: true,

Operands split onto multiple lines will be aligned.

Example:
int aaa = bbbbbbbbbbbbbbb +
          ccccccccccccccc;


AllowShortFunctionsOnASingleLine: Empty,

Only empty functions may be defined on a single line.

Example:
void f() {}
void f2() {
    bar2();
}


AllowShortIfStatementsOnASingleLine: true,
Short if-statements may be written on a single line.

Example:
true, if (a) return;

AllowShortLoopsOnASingleLine: true,
Short loops may be written on a single line.

Example:
true, while (true) continue;

BreakBeforeBraces: Linux,

The Linux style of breaking before braces will be used. In this style, there is a new line before the opening brace on function, namespace and class definitions, but control statements such as if, else, for, do, while, and try and braces variable declarations such as enum and struct do not get a new line before the opening brace.

Example:
try {    foo();
} catch () {
}

void foo() { bar(); }

class foo
{
};

if (foo()) {
} else {
}

enum X : int { A, B };


BreakBeforeTernaryOperators: true,

Multiline ternary statements will place the ternary operators before the values, otherwise clang-format will reformat the statement into a single line.

Example:

veryVeryVeryVeryVeryVeryVeryVeryVeryVeryVeryLongDescription
    ? firstValue
    : SecondValueVeryVeryVeryVeryLong;


ColumnLimit: 0,

A column limit of 0 means that there is no column limit. In this case, Clang-Format will respect the input’s line breaking decisions within statements unless they contradict other rules.

FixNamespaceComments: true,

The closing brace of a namespace will include a comment providing the namespace name.

Example:
namespace a {
foo();
} // namespace a;


SortIncludes: false,

Header #include statements will not be automatically rearranged by alphabetical order, because the specific order of header includes may inadvertently or intentionally affect compilation.

MaxEmptyLinesToKeep: 1,

Only 1 empty line may exist between lines of code.

SpaceBeforeParens: ControlStatements,

Control statements will have space before the opening paren. Functions and macros will not.

Example:
void f() {    if (true) {        f();
    }
}


IndentCaseLabels: true,

Case labels in switch statements will be indented.

Example:
switch (fool) {
    case 1:
        bar();
        break;
    default:        plop();
}

DerivePointerAlignment: true

Pointer alignment preference is up to the author of the file but must be consistent across the file. Inconsistencies in a specific file will be reformatted to match the rest of the file.

Example pointer alignment possibilities:
  • int * a;
  • int* b;
  • int *c;
Install Clang-Format


Clang-Format is easy to install on most operating systems, although we will note that due to choices we’ve made in our format specification. Clang-Format version 7 or higher is required for ClamAV. If the Clang-Format tool is not provided as a separate package, it can be obtained by installing Clang or LLVM.


CentOS:

sudo yum install epel-release && sudo yum install clang

Fedora:

sudo dnf install clang

Ubuntu & Debian:

sudo apt install clang-format

macOS:

First, install Homebrew.

Then:

brew install clang-format

Windows:

First, install Chocolatey.

Then:

C:\> choco install llvm

Plugins for popular text editors and IDEs


In addition to the above manual command-line usage, clang-format plugins are readily available for most popular text editors, enabling you to auto-format as you edit using commands in your editor, or to auto-format each time you save a file.

VSCode:

https://marketplace.visualstudio.com/items?itemName=xaver.clang-format

Sublime Text:

https://packagecontrol.io/packages/Clang%20Format

VIM & Emacs:

http://clang.llvm.org/docs/ClangFormat.html#vim-integration

https://github.com/rhysd/vim-clang-format

Atom Editor:

https://atom.io/packages/clang-format

Visual Studio:

https://marketplace.visualstudio.com/items?itemName=LLVMExtensions.ClangFormat

Emacs:

https://www.reddit.com/r/emacs/comments/7uq9w1/replace_emacs_c_autoformatting_with_clangformat/

Conventions and recommendations beyond Clang-Format

In addition to the rules enforced by Clang-Format, we have some additional guidelines to use when developing code for ClamAV.

Disclaimer: As project with a history dating back almost two decades, not all code in ClamAV adheres to these guidelines. New code should, however. During the course of working with older code, it should be updated to follow these as best as is feasible and as time permits.

Integer Types


Integer variables should use variable types from stdint that are specific about variable width (int8_t, uint32_t, int64_t, etc.) wherever possible. These types should be included using #include “clamav-types.h”, for portability. Not all operating systems or toolchains provide stdint.h or inttypes.h.

Notable exceptions to the above:

ptrdiff_t may be used to hold the signed numerical value of the difference of two pointers. Similarly, intptr_t and uintptr_t may be used to store and perform arithmetic on pointers specifically because they match the width of a pointer on a given machine.

size_t may be used to store unsigned size values as needed.

off_t, ssize_t, int, and long and other poorly defined types of yore may be required (at least temporarily) by library API's. These should be avoided in general, favoring ptrdiff_t (signed), size_t (unsigned), or an even more explicit type, such as uint32_t or int64_t. The older integer types are not well defined and its size may vary by compiler, OS, and system architecture. These types should be properly converted to better defined types as fast as is practical.

Integer Types in Format Strings

The format characters for stdint variable types are defined in clamav-types.h as of ClamAV 0.101.1.

Example format string usage:
int64_t obj_id = 0;
printf("Object ID: " PRIi64 "\n", obj_id);


In addition to these types, many people forget how to print size_t, ptrdiff_t, ssize_t, and off_t in a way that is portable across all systems.

For reference, these are:
  • size_t val
    • “%zu”, val
  • ptrdiff_t val
    • “%zd”, val
  • off_t val
    • “%lld”, (long long)val
  • ssize_t val
    • “%lld”, (long long)val
Inline Documentation / Function Comments
Functions should have Doxygen-style comment blocks. These comment blocks should be located where other developers can easily read them when working with the functions.

That is to say that:

  • For library API's, the Doxygen comment block should appear above the function prototype in the .h header file. For an example, see clamav.h.
  • For almost all other function types, it is best to place the doxygen comment block above the implementation in the .c / .cc / .cpp file.
Tip: Many text editors offer Doxygen extensions that significantly assist in writing function comments. For VSCode, the Doxygen Documentation Generator extension is quite helpful.

Example comment block including a brief, description, in, out, and in/out parameters:

/**
 * @brief Scan a file, given a file descriptor.
 *
 * This callback variant allows the caller to provide a context structure that caller provided callback functions can interpret.
 *
 * @param desc File descriptor of an open file. The caller must provide this or the map.
 * @param filename (optional) Filepath of the open file descriptor or file map.
 * @param[out] virname Will be set to a statically allocated (i.e. needs not be freed) signature name if the scan matches against a signature.
 * @param[out] scanned The number of bytes scanned.
 * @param engine The scanning engine.
 * @param scanoptions Scanning options.
 * @param[in/out] context An opaque context structure allowing the caller to record details about the sample being scanned.
 * @return cl_error_t CL_CLEAN, CL_VIRUS, or an error code if an error occurred during the scan.
 */
extern int cl_scandesc_callback(int desc, const char *filename, const char **virname, unsigned long int *scanned, const struct cl_engine *engine, struct cl_scan_options *scanoptions, void *context);

Packed Struct Definitions

Some structs require packing to ensure that they correctly represent data the same when written to or read from disk or the network.

Take the example struct:
struct example {
    short y;
    int x;
    char z;
}

Without packing, the compiler might optimize execution time by representing the struct on disk as the following bytes:
y
y


x
x
x
x
z




In reality, the programmer probably envisioned the struct without all that extra padding:
y
y
x
x
x
x
z


To force the compiler to properly pack a struct so that it is portable, surround the struct definition with the following:

#ifndef HAVE_ATTRIB_PACKED
#define __attribute__(x)
#endif

#ifdef HAVE_PRAGMA_PACK
#pragma pack(1)
#endif

#ifdef HAVE_PRAGMA_PACK_HPPA
#pragma pack 1
#endif

/* struct definition */
struct example_struct
{
    ...
} __attribute__((packed));

#ifdef HAVE_PRAGMA_PACK
#pragma pack()
#endif

#ifdef HAVE_PRAGMA_PACK_HPPA
#pragma pack
#endif


Note that all THREE preprocessor directive forms are necessary for cross-platform compatible code, as per https://bugzilla.clamav.net/show_bug.cgi?id=1752.

Monday, February 11, 2019

Notice to all ClamAV users, our bugzilla instance at bugzilla.clamav.net will be experiencing some downtime tomorrow at 9am EST, February 14th, 2019, for about 90 minutes.  We are moving to the data to a new machine at that time.


Monday, January 7, 2019

ClamAV 0.101.1 is an urgent patch release to address an issue in 0.101.0 specifically for developers that depend on libclamav. The issue in 0.101.0 is that clamav.h required supporting headers that were not provided on make install. To address this issue, the internal cltypes.h header has been replaced by a clamav-types.h that is generated on ./configure and will be installed alongside clamav.h.

Other changes

Increased the default CommandReadTimeout to reduce the chance of mail loss if using clamav-milter with the TCP socket. Contribution by Scott Kitterman. Fixes for --with-libjson and --with-libcurl to correctly accept library install path arguments.

Acknowledgements

 The ClamAV team thanks the following individuals for their code submissions: Scott Kitterman

Known Issues

Some users have observed crashes the first time running freshclam after upgrading from 0.100 to 0.101. We haven't yet tracked down the source of the issue, but have found that the issue resolves itself and that subsequent calls to freshclam work as expected.

Please download and update to 0.101.1, send us your feedback on ClamAV-Users

Friday, December 21, 2018

Earlier this month we wrote to the ClamAV users and developers mailing lists to disclose an issue with ClamAV 0.101.0. To reach a larger audience, we are re-posting those details here.

Shortly after the 0.101.0 release was published, we discovered an issue in ClamAV 0.101.0 impacting software developers and software package maintainers that integrate libclamav into other products on Unix/Linux systems. This issue causes an error when a user attempts to compile against a system-installed libclamav. 

We have a fix staged and nearly ready to be published. The fix will be published shortly after the start of 2019 in the ClamAV 0.101.1 patch release. This delay is necessary in order to fully test the fix and complete the due diligence required when creating a new build, accounting for delays that the holiday season brings. We apologize for the inconvenience.

Here are some additional details about this patch:
  • ClamAV uses a lot of mixed integer variable types internally. The preferred types are standard int types (eg int8_t, uint64_t, size_t, ptrdiff_t, etc).
  • As an ongoing effort to make variable types more consistent throughout the code base, we made the mistake of including the `cltypes.h` header file in `clamav.h`, which is not only used internally, but defines the public libclamav API. Because neither the `cltypes.h` header, nor the supporting `clamav-config.h` header are provided when installing libclamav to a system, applications built with a system-installed libclamav from version 0.101.0 will fail to compile.
  • To resolve the issue, we will be replacing `cltypes.h` with a new `clamav-types.h` file that is generated when the user runs `./configure.` This will be installed alongside `clamav.h` when the user runs `make install.` This will ensure that `clamav.h` has access to the necessary integer types on all operating systems and architectures.
  • To mitigate similar issues in the future, we have added a test to our build-acceptance suite to test building an application against a system-installed libclamav library.


Monday, December 3, 2018

We are pleased to announce the release of ClamAV 0.101.0!  Please take a look at the below release notes for further information.

0.101.0 Release Notes

ClamAV 0.101.0 is a feature release with an assortment of improvements that we've cooked up over the past 6 months.

Highlighted Changes:

  • Our user manual has been converted from latex/pdf/html into Markdown! Markdown is easier to read & edit than latex, and is easier to contribute to as it eliminates the need to generate documents (the PDF, HTML).  The user manual is now shipped in:docs/UserManual[.md].  However, the most up to date version at any time will be on ClamAV.net (This is not live right now, but will be shortly)
  • Support for RAR v5 archive extraction! We replaced the legacy C-based unrar implementation with RarLabs UnRAR 5.6.5 library. Licensing is the same as before, although our libclamunrar_iface supporting library has changed from LGPL to the BSD 3-Clause license.
  • Libclamav API changes:
    • The following scanning functions now require a filename argument.
    • This will enable ClamAV to report more details warning and error information in the future, and will also allow for more sensible temp file names. The filename argument may be NULL if a filename is not available.
      • cl_scandesc
      • cl_scandesc_callback
      • cl_scanmap_callback
  • Scanning options have been converted from a single flag bit-field into a structure of multiple categorized flag bit-fields. This change enabled us to add new scanning options requested by the community. In addition, the name of each scan option has changed a little. As a result, the API changes will require libclamav users to modify how they initialize and pass scan options into calls such as cl_scandesc() .
  • With our move to openssl versions >1.0.1, the cl_cleanup_crypto() function has been deprecated. This is because cleanup of open-ssl init functions is now handled by an auto-deinit procedure within the openssl library, meaning the call to EVP_cleanup() may cause problems to processes external to Clam.
  • CL_SCAN_HEURISTIC_ENCRYPTED scan option was replaced by 2 new scan options:
    • CL_SCAN_HEURISTIC_ENCRYPTED_ARCHIVE
    • CL_SCAN_HEURISTIC_ENCRYPTED_DOC
  • clamd.conf and command line interface (CLI) changes:
    • As in 0.100.2, the clamd.conf OnAccessExtraScanning has been temporarily disabled in order to prevent resource cleanup issues from impacting clamd stability. As noted below, OnAccessExtraScanning is an opt-in minor feature of on-access scanning on Linux systems and its loss does not significantly impact the effectiveness of on-access scanning.  The option still exists, but the feature will not be enabled and a warning will show if LogVerbose is enabled.
    • For details, see: https://bugzilla.clamav.net/show_bug.cgi?id=12048
  • "Heuristic Alerts" (aka "Algorithmic Detection") options have been changed to make the names more consistent. The original options are deprecated in 0.101, and will be removed in a future feature release.
  • In addition, two new scan options were added to alert specifically on encrypted archives or encrypted docs. Previous functionality did both, even though it claimed to be specific to archives:
  • Scan option details:


    | Old `clamd.conf` option          | *New* `clamd.conf` option    |

    | -------------------------------- | ---------------------------- |

    | `AlgorithmicDetection`           | `HeuristicAlerts`            |

    | `DetectBrokenExecutables`        | `AlertBrokenExecutables`     |

    | `PhishingAlwaysBlockCloak`       | `AlertPhishingCloak`         |

    | `PhishingAlwaysBlockSSLMismatch` | `AlertPhishingSSLMismatch`   |

    | `PartitionIntersection`          | `AlertPartitionIntersection` |

    | `BlockMax`                       | `AlertExceedsMax`            |

    | `OLE2BlockMacros`                | `AlertOLE2Macros`            |

    | `ArchiveBlockEncrypted`          | `AlertEncrypted`             |

    |                                  | `AlertEncryptedArchive`      |

    |                                  | `AlertEncryptedDoc`          |



    | Old `clamscan` option        | *New* `clamscan` option          |

    | ---------------------------- | -------------------------------- |

    | `--algorithmic-detection`    | `--heuristic-alerts`             |

    | `--detect-broken`            | `--alert-broken`                 |

    | `--phishing-cloak`           | `--alert-phishing-cloak`         |

    | `--phishing-ssl`             | `--alert-phishing-ssl`           |

    | `--partition-intersection`   | `--alert-partition-intersection` |

    | `--block-max`                | `--alert-exceeds-max`            |

    | `--block-macros`             | `--alert-macros`                 |

    | `--block-encrypted`          | `--alert-encrypted`              |

    |                              | `--alert-encrypted-archive`      |

    |                              | `--alert-encrypted-doc`          |


Some more subtle improvements:


  • Logical signatures have been extended with a new sub-signature type which allows for numerical byte sequence comparison. For those familiar with Snort, this byte comparison feature works similarly to the byte_extract and byte_test feature, in that it allows signature writers to extract and compare a specified number of bytes (offset from a match) against another numeric value. You can read more about this feature, see how it works, and look over examples in our documentation.
  • Backwards compatibility improvements for detecting the OpenSSL dependency.
  • Freshclam updated to match exit codes defined in the freshclam.1 man page.
  • Upgrade from libmspack 0.5alpha to libmspack 0.7.1alpha. As a reminder, we support system-installed versions of libmspack. However, at this time the ClamAV-provided version of libmspack provides additional abilities to parse broken or non-standard CAB files beyond what the stock libmspack 0.7.1alpha provides. We are working with the upstream project to incorporate our modifications, and hopefully these changes will appear in a future release of libmspack.
  • Updated the bundled 3rd party library libxml2 included for Windows builds to version 2.9.8.
  • Updated the bundled 3rd party library pcre included for Windows builds to pcre2 version 10.31.
  • Upgraded Aspack PE unpacking capability with support up to version 2.42.
  • Improvements to PDF parsing capability.
  • Replaced the Windows installer with a new installer built using InnoSetup 5.
  • Improved curl-config detection logic
    • GitHub pull-request by Thomas Petazzoni.
  • Added file type CL_TYPE_LNK to more easily identify Windows Shortcut files when writing signatures.
  • Improved parsing of Windows executable (PE) Authenticode signatures. 
    • Pull request by Andrew Williams.
  • Added support for Authenticode signature properties commonly used by Windows system files. These files are now much more likely to be whitelisted correctly.
  • Signature parsing now works correctly on big endian systems.
  • Some simplification to freshclam mirror management code, including changes to reduce timeout on ignoring mirrors after errors, and to make freshclam more tolerant when there is a delay between the time the new signature database content is announced and the time that the content-delivery-network has the content available for download.
  • Email MIME Header parsing changes to accept argument values with unbalanced quotes. Improvement should improve detection of attachments on malformed emails.
    • GitHub pull request by monnerat.
  • Included the config filename when reporting errors parsing ClamAV configs.
    • GitHub pull request by Josh Soref.
  • Improvement to build scripts for clamav-milter.
    • GitHub pull request by Renato Botelho.

Other changes:

  • Removed option handler for AllowSupplementaryGroups from libfreshclam. This option was previously deprecated from freshclam in ClamAV 0.100.0 but remained in libfreshclam by mistake.
  • In older versions of pcre2 and in pcre, a higher PCRERecMatchLimit may cause
    clamd
    to crash on select files. We have lowered the default PCRERecMatchLimit to 2000 to reduce the likelihood of a crash and have added warnings to recommend using pcre2 v10.30 or higher to eliminate the issue.

Supporting infrastructure:

As you might imagine, ClamAV is much more than just the tarball or EXE you download and install. Here at Cisco Talos, we've been working hard on the support infrastructure that's so easy to take for granted.

  • Test Frameworks
    • Feature Testing:
      • Throughout the development of ClamAV 0.101, our quality assurance engineers have been hard at work rebuilding our QA automation framework in Python from the ground up to test ClamAV features on 32-and-64bit versions:
        • Linux: Ubuntu, Debian, CentOS, Fedora
        • FreeBSD 11
        • Windows 10
    • In addition to building out the framework, they've written over 260 individual feature tests to validate correctness of the new features going into 0.101 as well as to validate many existing features.
  •  Build Acceptance Testing:
    • Another major task accomplished during the development of 0.101 was the creation of a build acceptance test framework that we run from our Jenkins CI server.
  • Similar to the feature testing framework, our build acceptance framework tests across 64bit and 32bit (where available):
    • macOS 10 (.10, .11, .13)
    • Windows (7, 10)
    • Debian (8, 9), Ubuntu (16.04, 18.04), CentOS (6, 7)
    • FreeBSD (10, 11)

This pipeline creates our release materials including the Windows installers, and then validates that the basic install, update, start, scan, and stop procedures all work as expected each time commits are made to our development branches.

  • Signature Database Distribution:
    • During the course of ClamAV 0.101 development, our Community team has been able to migrate us from a network of third-party mirrors over to use the services of CloudFlare to provide a more unified content-delivery-network.
    • With CloudFlare, users in geographic regions that had few mirrors will notice much improved signature update speeds and reliability. In addition, we're excited to be able to finally see user metrics that will help us continue to improve ClamAV. We've observed a 57% improvement in download speeds (on average) across the globe. We are of course grateful to all of the community members who have donated their server bandwidth to mirror the ClamAV signature databases over the years. Thank you so much!  We have not called for an official shutdown of the old mirror network yet.
  • Development Processes:
    • As many of you know, ClamAV 0.100 was in development for a good two years. Not only was this frustrating for users awaiting new features and bug-fixes, it also made for a difficult transition for users that weren't expecting two years worth of change when 0.100 landed.
    • We have learned from the experience and are committed to providing shorter and more responsive ClamAV development cycles.

ClamAV 0.101 is the first of many smaller feature releases where we created a roadmap with distinct deadlines and with specific planned features. We based the feature list on both community requests and our own needs and then executed that plan.

We're very proud of ClamAV 0.101 and we hope you enjoy it.

Acknowledgements:

The ClamAV team thanks the following individuals for their code submissions:
  • Andrew Williams, Cisco Talos
  • Craig Andrews
  • Josh Soref
  • monnerat
  • Renato Botelho
  • tchernomax
  • Thomas Petazzoni
As always, we welcome discussion about ClamAV on our mailing lists.

Monday, November 19, 2018

The ClamAV 0.101.0 release candidate is here!

We have also made significant improvements to our User Manual (the user manual will be moved to clamav.net soon) and to the inline Doxygen documentation in clamav.h for libclamav users. Please take a peek, and consider submitting your own recommendations via GitHub pull-request. In particular, we'd love to expand the Usage section with details on how to integrate ClamAV with other software. Your input would be greatly appreciated.

A short summary of the improvements found in 0.101:

  • Changes to the libclamav API:
    • Those who build applications around our shared library will need to change how they declare and pass scanning options to libclamav. Please take a look at the change to our example code for details. 
    • Scanning functions now have a filename argument. The argument is optional, but improves the efficiency when parsing certain types that require a file on disk to open and read, and will allow for additional improvements in the future.
    • Many of the scanning option #defines have changed. These can be found in our clamav.h header.
    • The libclamav version number has changed.
  • Some of the clamd config and clamscan command line option names have changed. The original versions will still work for a time, but eventually they will be deprecated. These options in question are detailed in the NEWS document.
  • A new sub-signature type called "Byte Compare". Byte Compare sub-signatures can be used to evaluate a numeric value at a given offset from the start of another (matched) sub-signature within the same logical signature. That numerical value may be interpreted from signed ascii decimal, unsigned ascii hex, or unsigned binary data relative to a match offset. For details, see the signature writing documentation.
  • Changes to our documentation. Documentation is now in Markdown and should be easier to navigate and easier to contribute to.
  • Support for extraction/scanning of RAR v5.x archives.


For additional details on changes in 0.101, please read the notes in our NEWS document.

Thank you in advance for test-driving the ClamAV 0.101 release candidate! Bugs should be brought to our attention via the clamav-devel mailing list or via bugzilla.

Tuesday, October 30, 2018

Welcome to the ClamAV 0.101.0 beta!

Important notes about this release:


  • Changes to the libclamav API:
    • Those who build applications around our shared library will need to change how they declare and pass scanning options to libclamav. Please take a look at the change to our example code for details. 
    • Many of the scanning option #defines have changed. These can be found in our clamav.h header.
    • The libclamav version number has changed.
  • Some of the clamd config and clamscan command line option names have changed. The original versions will still work for a time, but eventually they will be deprecated. These options in question are detailed in the NEWS document.
  • A new sub-signature type called "Byte Compare". Byte Compare sub-signatures can be used to evaluate a numeric value at a given offset from the start of another (matched) sub-signature within the same logical signature. That numerical value may be interpreted from signed ascii decimal, unsigned ascii hex, or unsigned binary data relative to a match offset. Your participation in testing this feature during our beta period will be greatly appreciated. For details, see the signature writing documentation.
  • Changes to our documentation. Documentation is now in Markdown and should be easier to navigate and easier to contribute to.
  • Support for extraction/scanning of RAR v5.x archives.


For additional details on changes in 0.101, please read the notes in our NEWS document.

Thank you in advance for participating in the ClamAV 0.101 beta! Bugs should be brought to our attention via the clamav-devel mailing list or via bugzilla.

You may download the ClamAV 0.101.0 beta at the usual location on ClamAV.net.