PSA: FreshClam database download issue

We've received several reports of users experiencing issues with FreshClam triggered by last week's update to the main and daily CVD databases. 

FreshClam's primary advantage over tools like wget and curl, in addition to automatic testing of new databases, is the ability to perform an incremental update using database patch files. But when a database change is so large that a patch file is not efficient, we issue an empty patch file. This empty file will cause FreshClam to download the entire database instead. Unfortunately, our recent update using the empty patch file method has revealed a couple of issues. 

New Main & Daily CVD's are incoming

 Tomorrow, Wednesday July 14th, we are planning on publishing a brand new main.cvd and daily.cvd, as we do periodically to move more of the long term signatures into the main.cvd and make the daily.cvd smaller again.  

This will have an impact on your downloads of these files (as every ClamAV instance will have to re-download both files), so you may see a spike in your bandwidth usage.

We will monitor the situation on the mirror side and make any adjustments necessary, but we anticipate no issues.

ClamAV, CVDs, CDIFFs and the magic behind the curtain

The amount of malicious files that ClamAV can detect has increased immensely over the past few years, but with this increase in efficacy comes some challenges with scale.  

Some of these challenges have required drastic measures to ensure the effective operation of the ClamAV infrastructure, including blocking certain methods of downloading the official ClamAV signature sets. To give the community more insight into these matters, we’d like to discuss some of these challenges in-depth and provide insight into future changes and optimizations coming to the product.

Freshclam, cdiffs and bandwidth are your friends

During a recent review of file downloads from our ClamAV CDN network, we've noticed hundreds of IPs that seem to be downloading the daily.cvd and the main.cvd thousands of times a day.

There are about a dozen IPs that are downloading those to files more than 40,000 times a day. This is causing us to transfer about 250TB of data a day. We would encourage any users still doing this to cease as soon as possible. Not only does it waste our bandwidth — as we have much more efficient ways of downloading the updates — but it also wastes your bandwidth, as well.

Freshclam has the ability to download partial files of updates (called cdiffs).  Which are smaller, more incremental updates to the database. This allows users, and us, to manage our downloads in a much more efficient manner. We often receive the complaint, "I have to download the daily.cvd and main.cvd with Python and move the updates to an off-internet system."  That's fine — it's a use case we support. However, you can do the same with freshclam and the small cdiffs.

Furthermore, we also only release updates once a day.  Reducing the number of updates you check for (and, subsequently, download we assume through a crontab or periodic job of some type) would also alleviate this issue.

We will be constantly monitoring this in hopes that people migrate to using freshclam.  Over-abusers (for instance, the top 10 IPs that are downloading main.cvd 40,000 times a day), will be immediately blocked.  Further abusers may also be blocked, without notice.

To mitigate, please complete the following tasks:

1. Use Freshclam instead of Python or whatever downloading script you have cron'd.
2. Reduce the checks to once or twice a day.

Thank you for helping keep the ClamAV network healthy.

Any questions, please see us over on the ClamAV-Users list.

Want to improve your ClamAV experience? Here are some common mistakes we see with FreshClam

At Cisco Talos, we regularly get questions on how to get the most out of ClamAV. Therefore, we wanted to point out some common mistakes that users make in hopes that we can make using ClamAV easier, and more effective, for everyone.

Downloading the signatures

We want to emphasize the importance of using FreshClam to download the signatures instead of Wget, Python, or similar scripting libraries. A lot of folks from the community are attempting to download the daily, bytecode, and the main CVDs using scripting tools. While we understand the desire to do so, FreshClam has been engineered to take advantage of the fact that our publishing system provides "diff" files, or only what has changed since the last update.

This provides faster updates to the world, reduces our overhead on the server side, and lowers our overall bandwidth. This problem was not made apparent until we consolidated the mirror infrastructure to our CDN (Cloudflare). We are now able to observe download problems across our infrastructure.

For example, we have roughly 20 individual IPs that are attempting to download the whole main.cvd, daily.cvd, and bytecode.cvd files roughly, eight times a minute. Imagine what would happen if we were able to get users to stop doing that and use FreshClam. We are currently pushing about 54TB a day in updates to our users — you have to wonder how much that would drop if everyone was using FreshClam.

ClamAV supported versions

We took a look at the top 10 "User-Agents" that are downloading updates.

ClamAV/0.100.1 (OS: linux-gnu, ARCH: x86_64, CPU: x86_64)

ClamAV/0.99.4 (OS: linux-gnu, ARCH: x86_64, CPU: x86_64)

ClamAV/0.99.2 (OS: linux-gnu, ARCH: x86_64, CPU: x86_64)

ClamAV/0.98.7 (OS: linux-gnu, ARCH: x86_64, CPU: x86_64)

ClamAV/0.100.0 (OS: linux-gnu, ARCH: x86_64, CPU: x86_64)

ClamWin/0.99.1 (OS: win32, ARCH: i386, CPU: i386)

ClamWin/0.99.4 (OS: win32, ARCH: i386, CPU: i386)

ClamAV/0.99.3 (OS: linux-gnu, ARCH: x86_64, CPU: x86_64)

Wget/1.12 (linux-gnu)

ClamAV/0.96-exp (OS: linux-gnu, ARCH: i386, CPU: i686)

Since we already covered Wget and tools like that, let's move on to older versions of ClamAV.

As you can see, the top two versions are the most recent versions and essentially, the last major version back. These are Ubuntu/Debian users. We work very closely with the package maintainer for those distributions, and we can see the shift in versions when they update.

Let's shift our focus to No. 10 in the list above. ClamAV/0.96-exp, an experimental version of ClamAV back in the 0.96 days, before CVD diff'ing was released. Obviously, this is an errant cronjob (someone attempting to run FreshClam once a minute). Again, we are asking users to please update to a new version of ClamAV, which will, in turn, update your version of FreshClam (to support diff'ing).

We took the logs, removed Wget, curl and other scripting tools, only focusing on actual FreshClam installations, only focusing on versions of ClamAV.  Basically, asking our logs, "How many different versions of ClamAV are there out there?"

297

After ClamAV is installed, then what? How do I update the database?

You will want to edit freshclam.conf.example by commenting out the "example" text with a pound symbol (#).

Once that is done, run a 'freshclam' via command line and the database should update.

We also want to remind people that we do monitor IRC, so if you need help — give us a shout.

Community members and staff idle the channel and try to help to the best of our abilities.

IRC information

Service: irc.freenode.net
Channel: #clamav

How do you submit bugs to ClamAV?

Please visit https://bugzilla.clamav.net/.

If you are having Mirror issues

If you are running into an issue when using FreshClam to download the signatures, it could be the mirror.dat file. Try deleting this file, and then running FreshClam again.

In the "configure" file of FreshClam, you can set your mirror as 'database.clamav.net' instead of using a country code (db.xy.clamav.net).  In a future version of ClamAV, we will improve/remove the mentions of country based mirrors. Use database.clamav.net as the central point for your downloads.

If you want to contribute to documentation

Our documentation is also open-source. Feel free to edit it as need be, and submit the changes to our Git repository.