Resolving Issues With Freshclam

Certain users are experiencing database update issues due to the failed Authenticode database push. This blog post will show how to check if you're one of those affected users and how to fix freshclam.

Validate You're Affected

You can validate that you're having this particular issue by a number of ways:

1. Check the hash of your daily.cvd. You are affected if the hash matches the following:
1. MD5: 89dedb45609e59b0244fb5202ab6fa56
2. SHA1: 9947ec90e60499ab7c3331670d5b26b4eaac76e4
2. Check your freshclam log file for repeated errors that look like:
1. Ignoring mirror [Mirror's IP address here] (has connected too many times with an outdated version)
3. Check the version number and the functional level of the daily.cvd by using sigtool:
1. sigtool --info daily.cvd will show a version number of 16681 and a functionality level of 73

How To Fix Freshclam

If you are expereincing the problem, please do the following:  Stop the freshclam daemon if it's running, delete both mirrors.dat and daily.cvd, then restart the freshclam daemon. Freshclam will then download a new daily.cvd and will be up-to-date.

We apologize for any inconvenience this has caused and thank you for using ClamAV.  If you have any further issues, please send a message to the ClamAV user's list or contact us via IRC.

Planned Infrastructure Maintenance - 04 Mar 2013

In preparation for the ClamAV 0.98 release, we will be performing maintenance on the infrastructure beginning at 5:00 PM EST on 04 Mar 2013.

We will be pushing out a new signature database that does not have a corresponding cdiff file. This means that clients will pull down a full copy of the daily.cvd database, which will cause an increase in download traffic from the mirrors.

The maintenance is estimated to take one hour.  No impact to users, beyond the downloading of a new daily.cvd, is anticipated.

We would like to extend our thanks to the mirror providers for their contributions, and thank you for using ClamAV.

Post-Mortem Analysis Of Virus Database Push Issues

On Thursday, 14 Feb 2013, in preparation for the coming ClamAV 0.98 release, a new database was scheduled to be made available to users. We had a set of issues while performing this upgrade, and we feel that it is appropriate to let our users and mirror providers know what happened, what has done to fix the issues, and what is being done to prevent these issues from happening again.

So first, What Happened?

1. 14 Feb 2013 0800 EST: Start of our scheduled work on our infrastructure.
2. 14 Feb 2013 0815 EST: A new, custom daily.cvd (our virus definition database) was published. This database was generated with ClamAV 0.98, which in turn caused freshclam to think that a new version of ClamAV was available (not yet, but there will be).
3. 14 Feb 2013 0830 EST: Published a new daily.cvd, generated with ClamAV 0.97.6, the current version of ClamAV. This corrected the issue with incorrect notifications of a new version of ClamAV.
4. 14 Feb 2013 1100 EST: Clients report errors with updating. Investigation starts.
5. 14 Feb 2013 1130 EST: The problem was isolated. The new database wasn't copied into a critical directory on our internal Signature server. The database publishing infrastructure didn't know that a custom database had been published. The custom database was overwritten with a new database.  This resulted in some users being unable to use the .cdiff files (our incremental update files) for updating, leading to users who had downloaded the custom database to be unable to update.
6. 14 Feb 2013 1330 EST: A new database was published to resolve the issues. Issues should now be resolved for most users.
7. 19 Feb 2013 1700 EST: Issues resolved for all remaining users by modifying the set of available .cdiff files.

Fixes That Have Been Performed

We've deleted all database files that would cause errors. This should fix the remainder of issues for our users.  However, any users who are still seeing errors should delete the mirrors.dat file in their database directory to force a reset of mirror selection.

Prevention

We've put in place a workflow that will prevent issues like this from popping up. A full change-management process is in place, with an emphasis on peer-reviewed planning, comprehensive test plans and appropriate personnel assignments.  Change plans will be approved by a senior administrator, a ClamAV developer and a representative from the analyst team.

For the convenience of our mirror providers, there is now a set maintenance window for routine changes: Monday 5pm EST through midnight EST.  As always, we will aim to notify mirror providers a week in advance of any change.  In the case of emergent issues, a different time or a shorter notification may be required.

We apologize for any inconvenience caused by the problems outlined in this post.  We will continue to review our processes to ensure that we are providing the best experience for both our users and our mirror providers.

Authenticode Certificate Chain Verification

Introduction

Microsoft introduced digitally signing PE object files (authenticode) in Windows 98. Hardware drivers eligible for the Windows Logo Program are required to contain a valid authenticode signature. Since then, Microsoft has expanded the program to executable object files (EXEs) and DLLs.

Microsoft has its own public key infrastructure (PKI). There are four trusted root certificate authorities: two by Microsoft, Thawte, and Verisign. Microsoft's own executables for Windows are signed.

Authenticode At Work

The Problem

It's becoming more common for malware authors to sign their executables. Malware authors are known to steal existing certificates or forge certificates (the certificate forging link talks about SSL certificates; though you can't use certificates used for web browsing with authenticode, the concept still applies). Stealing and forging existing certificates can trick unsuspecting users into trusting the malware.

The Solution

ClamAV 0.98 now parses and validates the authenticode certificates. ClamAV now ships a database of trusted and revoked certificates. If the PE file being scanned is flagged as a virus, ClamAV follows this logic in validating the certificate chain contained inside the PE file:

1. Load the chain in full
2. Validate each certificate in the chain:
1. If no certificate matches one of the four trusted roots, the whole chain is considered invalid and is subsequently ignored. Further chain processing is stopped.
2. If one certificate matches a revoked entry in the database, the whole chain is considered invalid. Report PE file being scanned is a virus.
3. If no certificates match a revoked entry in the database, the PE file is marked as clean. No virus is reported for the PE file being scanned.

Generating A Certificate Revocation Entry

Since the authenticode chain is verified only after a file has been flagged as virus by the ClamAV engine, you have to first create a signature for the file if one doesn't already exist. If a signature exists, yet the file is reported as clean, chances are that the file is being whitelisted due to having a valid authenticode signature. You can verify that the file is being whitelisted by passing --debug --verbose to clamscan and looking for the word authenicode (yes, that is indeed misspelled, but that works out to our advantage). If you see that word, then the file is being whitelisted and a certificate revocation entry needs to be created.

First, we need to dump the certificate chain so that we can find the certificate to revoke. You can tell ClamAV to dump the certificate chain to stderr by passing --dumpcerts to clamscan. You will then match the certificate that is dumped with the public key of the lowest certificate in the chain that Windows shows. Windows shows a few bytes before the public key actually starts. Don't worry, if that confuses you, I have screenshots:

Windows Certificate Information

Validating and Dumping Authenticode Certificate Information via ClamAV

Certificate revocation entries go into a .crtdb file in your ClamAV database directory. Its format is as follows: name;trusted;subject;serial;pubkey;exponent;codesign;timesign;certsign;notbefore;comment[;minFL[;maxFL]]

Where:

1. Name: a descriptive name for the certificate entry
2. Trusted: a bit field (0 or 1) whether this entry is trusted or revoked
3. Subject: the subject reported by ClamAV
4. Serial: the serial reported by ClamAV
5. Pubkey: the public key reported by ClamAV
6. Exponent: The exponent of the certificate. Set this to 010001.
7. Codesign: A bit field (0 or 1) whether this certificate can sign code
8. Timesign: A bit field (0 or 1) whether this certificate can generate a timestamp signature
9. Certsign: A bit field (0 or 1) whether this certificate can sign other certificates
10. Notbefore: The notbefore field of the certificate. Defaults to 0 if empty.
11. Comment: any comments about this entry
12. MinFL: The minimum ClamAV feature level needed for this entry. Required only if MaxFL is set.
13. MaxFL: The maximum ClamAV feature level supported by this entry

Let's return back to the example. We now have enough information to create the certificate revocation entry. It would look like: 

Descriptive name here;0;fe72355be4b6893d8e5b628d1a9ae8863d202b6f;a58d94ce010afa9865e732870971428768c92d64;ba0532ff862861cfaf8c22601fe479e3697b6ab94ff01c3254d105018c93bdb47f4c3fc1fb1d20172c46a727fb589f310cf6b081517ee472d145dfd4939c7d4652aad06c3ee6722f15703e88bbd8bc4d56fe7030b21f105fa9817b625103273caf46072628207e81bc13ac6ba18cdd3e93b97c9761730eb14ce36464cc997075;010001;1;0;0;0;

After adding that revocation entry to our certs database, ClamAV now flags the sample as a virus:

clamscan Reporting Virus