Thursday, November 20, 2014

In ClamAV 0.98.5, a new feature provides for file property collection and analysis.  The feature is intended for software developers and analysts who want to include the collection and analysis of file properties in their applications in addition to scanning file content. ClamAV 0.98.5 collects properties on the following file types:
  • Microsoft Word
  • Microsoft Excel
  • Microsoft Powerpoint
  • Office Open XML (OOXML) Document
  • Office Open XML Presentation
  • Office Open XML Workbook
  • Microsoft Portable Executable
  • Adobe Portable Document Format

How does it work? There are three main areas to understand about the file properties scanning features.

The first area is about how to initiate the file properties collection through the ClamAV API. A program using the ClamAV API may indicate property scanning by setting an option. The option is required to invoke the file property collection scan mode. Additionally, the API provides a new callback function to enable custom processing such as tailoring the result of the property scan, or to write the json properties string to a file. Use of the callback function is optional.

By way of example, the command line program clamscan uses the following API call to scan a file:
if((ret = cl_scandesc_callback(fd, virpp, &info.blocks, engine, options, &chain) ...
If you are interested in more detail, clamscan/manager.c contains this API call, and libclamav/clamav.h contains the function prototype of cl_scandesc_callback. In this case, to indicate file properties scanning to the ClamAV engine from clamscan, use the following sequence:
options |= CL_SCAN_FILE_PROPERTIES;
if((ret = cl_scandesc_callback(fd, virpp, &info.blocks, engine, options, &chain) ...
That's it! clamscan will now collect file properties for analysis. In fact, this is basically what clamscan does to support the new --gen-json option.

That brings us to the second area, which involves how file properties are collected and stored. We use json for this purpose. As ClamAV scans a file, it gathers properties about the file as well. The properties are maintained as json objects. At the end of the file scan, and prior to the file properties analysis scan, ClamAV serializes the json objects to the json files properties string.

The file properties json is a recursive structure of other file property objects. At the top level, the file property object that may be thought of as representing the file as a container, The generic schema of the json file property object contains the following objects:
  • FileSize
  • FileType
  • FileMD5
  • ContainedObjects (Array)
  • Viruses (Array)
The ContainedObjects is an array of other embedded file objects, such as a spreadsheet embedded within a Word document. The whole pattern is repeated, in this case, where the embedded spreadsheet object may have its own ContainedObjects array. There are additional file properties that are dependent upon the particular file type. A complete list of file properties and their parent file types can be found in ClamAV_Document_Properties.xlsx within the ClamAV docs/ directory.

You can see the generated json file property string from the command line by using:
clamscan --gen-json --debug <some office, pdf, or pe file>
The third and last area is the analysis of the generated json file properties string.  After the original file scan, ClamAV performs a second scan on the resulting file properties string.  Both pattern signatures and bytecode programs may by used for analysis the file properties string. ClamAV handles these signatures identically to those used on normal files, except that the target file is the generated json file properties string. Signatures should specify the json properties file target type of 13 to ensure the signature operates on the file properties string. Bytecode signatures will be the most useful for the analysis phase. Several upcoming blog posts in this series will discuss in detail about writing bytecode programs for property scanning and using the bytecode json API. You can also see sample byte code signature in ClamAV 0.98.5 directory examples/fileprop_analysis/.

One final note: To use file property scanning, json-c must be installed and configured into ClamAV. If it is not already installed, you can obtain json-c from https://github.com/json-c/json-c or install it using your package manager. We support json-c versions 0.9 through 0.12, but recommend version 0.12. After installing json-c, include it into your ClamAV configuration using './configure --with-libjson'. See ./configure --help for additional information about ClamAV configure options. Note that json-c is not required by any other ClamAV facility.

Wednesday, November 19, 2014

Bytecode signatures are a specialized type of ClamAV signature which is able to perform additional processing of the scanned file and allow for more robust detection. Unlike the standard ClamAV signature types, bytecode signatures have a number of unique distinctions which need to be respected for their effective usage.

Bytecode Signature Generation
The major distinction between bytecode signatures and the other ClamAV based standard signature languages is that bytecode signatures are actually compiled from a user-written source file, similar to Java bytecode. The tool used to generate bytecode signatures from source is the clambc-compiler which is a separate project from ClamAV.

You can get it by using one of these commands:
git clone git://github.com/vrtadmin/clamav-bytecode-compiler (recommended for git <1.7)
git clone https://github.com/vrtadmin/clamav-bytecode-compiler (recommended for git 1.7+)

The repository can also be browsed online here:
https://github.com/vrtadmin/clamav-bytecode-compiler

Once the source is acquired, read the README in the project to compile and install the clambc-compiler. To compile the bytecode signature, simply run the command:

clambc-compiler [options] [source]

For information on how to write bytecode source, please refer to the clambc-compiler documentation or other blog posts.

Running Bytecode Signatures in ClamAV
Due to the nature of how bytecode signatures are ran in ClamAV, there are a number of pre-cautions taken to ensure safety of the bytecode signature execution.

Trust
Bytecode signatures, by default, are considered untrusted. In fact, only bytecode signatures published by Cisco, in the bytecode.cvd are considered “trusted”. This means that the ClamAV engine will, by default, never load, trigger or execute untrusted bytecodes. One can bypass this safety mechanism by specifying the bytecode unsigned option to the engine but it should be noted that it is up to the user’s discretion on using untrusted bytecode signatures.

For clamscan, the command line option is --bytecode-unsigned.
For clamd, one would need to specify BytecodeUnsigned yes to clamd.conf.

Timeout
Bytecode signatures are designed to only run for a limited amount of time designated by an internal timeout value. If execution time exceeds the value, the bytecode signature’s execution is terminated and the user is notified. The bytecode signature timeout value can be set by the user.

For clamscan, the command line is --bytecode-timeout=[time in ms].
For clamd, one would specify BytecodeTimeout [time in ms] to clamd.conf.

Bytecode Databases
Bytecode signatures are stored in a separate database from the standard ClamAV signatures. In fact, it is impossible to generate database files (with sigtool) that contain both bytecode signatures and standard signatures. Bytecode databases are generated by building a database that is named “bytecode.*” which triggers specialized handling by sigtool. Sigtool will add all bytecode signatures in the specified directory regardless if the name of the signature matches the name of the database. Bytecode signatures thus can be named anything provided the extension is “.cbc”.

Bytecode databases generated this way are still considered untrusted (unless published by Cisco) which means one needs to still specify the appropriate flags to use the database.

Issue Reporting
If anyone encounters issue with bytecode signatures, whether within the clambc-compiler or within ClamAV, they can report them to https://bugzilla.clamav.net/. Be sure to include the bytecode signature, bytecode source(if possible), and any other pieces of useful information.

Tuesday, November 18, 2014

Welcome to ClamAV 0.98.5! ClamAV 0.98.5 includes important new features
for collecting and analyzing file properties. Software developers and
analysts may collect file property meta data using the ClamAV API for
subsequent analysis by ClamAV bytecode programs. Using these features
will require that libjson-c is installed, but otherwise libjson-c is not
needed.

Look for our upcoming series of blog posts to learn more about using the
ClamAV API and bytecode facilities for collecting and analyzing file
properties.

ClamAV 0.98.5 also includes these new features and bug fixes:


  • Support for the XDP file format and extracting, decoding, and scanning PDF files within XDP files. Addition of shared library support for LLVM versions 3.1 - 3.5 for the purpose of just-in-time(JIT) compilation of ClamAV bytecode signatures. Andreas Cadhalpun submitted the patch implementing this support.
  • Enhancements to the clambc command line utility to assist ClamAV bytecode signature authors by providing introspection into compiled bytecode programs.
  • Resolution of many of the warning messages from ClamAV compilation.
  • Improved detection of malicious PE files.
  • Security fix for ClamAV crash when using 'clamscan -a'. This issue was identified by Kurt Siefried of Red Hat. 
  • Security fix for ClamAV crash when scanning maliciously crafted yoda's crypter files. This issue, as well as several other bugs fixed in this release, were identified by Damien Millescamp of Oppida.
  • ClamAV 0.98.5 now works with OpenSSL in FIPS compliant mode. Thanks to Reinhard Max for supplying the patch.
  • Bug fixes and other feature enhancements. See Changelog or git log for details.


Thanks to the following ClamAV community members for code submissions
and bug reporting included in ClamAV 0.98.5:

Andreas Cadhalpun
Sebastian Andrzej Siewior
Damien Millescamp
Reinhard Max
Kurt Seifried

Please download the latest release of ClamAV from 0.98.5 from our download page.

Tuesday, October 14, 2014

ClamAV 0.98.5-rc1 is now available for download.  Shown below are the notes for this release:

ClamAV 0.98.5 also includes these new features:

    - Support for the XDP file format and extracting, decoding, and
      scanning PDF files within XDP files.
    - Addition of shared library support for LLVM verions 3.1 - 3.4
      for the purpose of just-in-time(JIT) compilation of ClamAV
      bytecode signatures. Andreas Cadhalpun submitted the patch
      implementing this support.
    - Enhancements to the clambc command line utility to assist
      ClamAV bytecode signature authors by providing introspection
      into compiled bytecode programs.
    - Resolution of many of the warning messages from ClamAV compilation.
    - Bug fixes and other feature enhancements. See Changelog or
      git log for details.

Thanks to the following ClamAV community members for code submissions
and bug reporting included in ClamAV 0.98.5:

Andreas Cadhalpun
Sebastian Andrzej Siewior

----

The RC is available for download from:
http://www.clamav.net/download.html under the "Development Releases" section.

Please download, test, and provide feedback to the team here:

http://lists.clamav.net/mailman/listinfo/clamav-users

--

The ClamAV team (http://www.clamav.net/about.html#credits)

Tuesday, August 26, 2014

For the past several months we've been working diligently on a complete refresh of several Open Source websites, designs and logos.  The first website we rolled out a refresh of was Snort.org back in June.

At the same time, we've been working hard on ClamAV.net.  When the ClamAV project was acquired in 2007 by Sourcefire (now a part of Cisco), we retained the original website and hosting provider the website was built on, but we took this opportunity to start from scratch.

As with Snort.org, this wasn't just a facelift for the website, this was a complete rewrite.  Much of the content you are looking for is the same, for instance the virus submission forms are still on the site, but we've build some improvements:
  • Simple Navigation
    • Much like we tried to do with Snort.org, almost all content on ClamAV.net is one or two clicks away.
  • Faster
    • Not only is the site faster to load on the browser, but it is less load on the server side too.
  • Documentation
    • We now dynamically load the ClamAV FAQ from Github onto the site.  If someone would like to contribute to the FAQ, they may do so by submitting a pull request (at https://github.com/vrtadmin/clamav-faq.  Which, once accepted, will be rendered on the main clamav.net site for all to see.
  • Elimination of dead links and pages
We really hope that you enjoy on the new ClamAV.net, and are looking forward to hearing your feedback at snort-site@cisco.com

Please take a look at the new website over at: http://www.clamav.net

Tuesday, July 29, 2014

ClamAV Community,

This notice is to inform you that effective immediately ClamAV 0.96 (and all minor versions) is no longer supported in accordance with ClamAV's EOL policy which can be found here:

https://github.com/vrtadmin/clamav-faq/blob/master/faq/faq-eol.md

While the current CVD's being distributed will still work on ClamAV 0.96, we are enabling the functionality to actually make those versions no longer be able to update.  There is detection that we have written that cannot be shipped to the 0.96 branch.

Thank you for using ClamAV!

Wednesday, July 9, 2014

Introduction

In order to support more advanced features planned in future releases, ClamAV has switched to using OpenSSL for hashing. The ClamAV Visual Studio project included with ClamAV's source code requires the OpenSSL distributables to be placed in a specific directory. This article will teach you how to compile OpenSSL on a Microsoft Windows system and how to link ClamAV against OpenSSL.

This blog article is based on another article.


Prerequisites


You will need ActiveState ActivePerl installed. OpenSSL uses Perl-based Makefiles. You will also need Visual Studio 2010 or later.


Download OpenSSL


You can find OpenSSL's source here. Download the latest version. When this article was first authored, the latest version of OpenSSL was 1.0.1h. This article will reference that version throughout.

We will be using separate build and installation directories for 32bit and 64bit compilations. We'll extract the source code twice, to C:\openssl-src-32 and C:\openss-src-64. We'll then create two directories to hold the OpenSSL installation: C:\openssl-build-32 and C:\openssl-build-64.


Compiling OpenSSL


We will first build the 32bit libraries. You will need to open a Visual Studio 32bit command prompt. Please note that you cannot build OpenSSL 64bit from a 32bit Visual Studio command prompt and vice-versa.

We'll run these commands in the command prompt:
  1. C:
  2. cd \openssl-src-32
  3. perl Configure VC-WIN32 --prefix=C:\openssl-build-32
  4. ms\do_ms
  5. nmake -f ms\ntdll.mak
  6. nmake -f ms\ntdll.mak install
Once those commands have run, you'll need to create the 64bit OpenSSL distributables in a Visual Studio x64 comamnd prompt:
  1. C:
  2. cd \openssl-src-64
  3. perl Configure VC-WIN64A --prefix=C:\openssl-build-64
  4. ms\do_win64a
  5. nmake -f ms\ntdll.mak
  6. nmake -f ms\ntdll.mak install
You now have both the 32bit and 64bit builds of OpenSSL compiled and installed in their respective build directories.


ClamAV Dependencies Directory


Now that you have distributables for both 32bit and 64bit, we'll now need to create the directories where the ClamAV Visual Studio project expects the OpenSSL dependencies to be.

Create these directories:
  1. C:\clamdeps
  2. C:\clamdeps\win32
  3. C:\clamdeps\win32\openssl
  4. C:\clamdeps\win32\openssl\lib
  5. C:\clamdeps\win64
  6. C:\clamdeps\win64\openssl
  7. C:\clamdeps\win64\openssl\lib

Copy the 32bit directories and files over:
  1. C:\openssl-build-32\include to C:\clamdeps\win32\openssl
  2. C:\openssl-build-32\lib\libeay32.lib to C:\clamdeps\win32\openssl\lib
  3. C:\openssl-build-32\lib\ssleay32.lib to C:\clamdeps\win32\openssl\lib
  4. C:\openssl-build-32\bin\libeay32.dll to C:\clamdeps\win32\openssl\lib
  5. C:\openssl-build-32\bin\ssleay32.dll to C:\clamdeps\win32\openssl\lib
Now copy the 64bit directories and files over:
  • C:\openssl-build-64\include to C:\clamdeps\win64\openssl
  • C:\openssl-build-64\lib\libeay32.lib to C:\clamdeps\win64\openssl\lib
  • C:\openssl-build-64\lib\ssleay32.lib to C:\clamdeps\win64\openssl\lib
  • C:\openssl-build-64\bin\libeay32.dll to C:\clamdeps\win64\openssl\lib
  • C:\openssl-build-64\bin\ssleay32.dll to C:\clamdeps\win64\openssl\lib


Conclusion


We've now built the OpenSSL libraries that ClamAV now depends on. ClamAV requires them to be in C:\clamdeps\win{32,64}\openssl.