SigAnalyzer: Signature analysis with CASC

Executive summary

ClamAV Signature Creator (CASC) is an IDA Pro plugin that assists in the creation of ClamAV pattern signatures. We have enhanced this plugin to also analyze these signatures. The plugin highlights matching parts in a binary when its given a particular signature. This function is helpful when evaluating automatically generated signatures, e.g., from the BASS framework. As a larger number of signatures is automatically generated, it becomes ever more important to gain a quick understanding about the effects of these signatures. This functionality will allow us to check the accuracy of our signatures faster, and allow us to deliver a better product to our users.

Description

The signature analysis function integrates neatly into the existing CASC plugin. Because the signature analyzer (“SigAnalyzer”) function needs some extra libraries, the installation of the plugin changes slightly — you can install the library dependencies via your distribution if you use the system Python (e.g., on Linux on a 32-bit machine for any IDA version or on Linux with IDA 7.0 on a 64-bit machine), or you can use the provided ZIP archives with bundled library dependencies. Detailed installation instructions for both cases can be found in the project’s README.

If the library dependencies are satisfied, you’ll have a new tab “Analyze” next to the tab “Create” in the plugin’s pane. Here, you can paste a logical signature (ldb) or pattern signature (ndb), and press “Add signature.” The signature will appear in the “Signatures” list above if it parsed correctly. If nothing happens, keep an eye on error messages in IDA’s output window. Now you can double-click the signature in the list. For an ndb signature, this will directly take you to the match in the IDA-View window. The match is green-colored.

An ldb signature will show all sub-signatures on the right list instead. You can double-click on a sub-signature, and it will color the match and bring you to it. Keep an eye on the text row above the “Add signature” button — you can find more information on the match there, or a message if there is no match. Added signatures are stored inside the database, and will be available again in the signature selection window the next time you open the database.

Internally, the plugin relies on Yara-python to find matches. ClamAV signatures are translated to Yara rules, which are then matched against the binary data. The plugin’s source code is available on our GitHub page, as well as in installation packages for your convenience. Please go ahead and test the new functionality, but keep in mind that the functionality is in an early beta stage, and still may have a bug or two. If you come across a bug, or you want to comment, please don't hesitate to let us know through a GitHub issue.

We are excited to have our users use this new feature. Please continue to follow along on the blog for new ClamAV features as we develop them, and be sure to follow us on Twitter.

Want to improve your ClamAV experience? Here are some common mistakes we see with FreshClam

At Cisco Talos, we regularly get questions on how to get the most out of ClamAV. Therefore, we wanted to point out some common mistakes that users make in hopes that we can make using ClamAV easier, and more effective, for everyone.

Downloading the signatures

We want to emphasize the importance of using FreshClam to download the signatures instead of Wget, Python, or similar scripting libraries. A lot of folks from the community are attempting to download the daily, bytecode, and the main CVDs using scripting tools. While we understand the desire to do so, FreshClam has been engineered to take advantage of the fact that our publishing system provides "diff" files, or only what has changed since the last update.

This provides faster updates to the world, reduces our overhead on the server side, and lowers our overall bandwidth. This problem was not made apparent until we consolidated the mirror infrastructure to our CDN (Cloudflare). We are now able to observe download problems across our infrastructure.

For example, we have roughly 20 individual IPs that are attempting to download the whole main.cvd, daily.cvd, and bytecode.cvd files roughly, eight times a minute. Imagine what would happen if we were able to get users to stop doing that and use FreshClam. We are currently pushing about 54TB a day in updates to our users — you have to wonder how much that would drop if everyone was using FreshClam.

ClamAV supported versions

We took a look at the top 10 "User-Agents" that are downloading updates.

ClamAV/0.100.1 (OS: linux-gnu, ARCH: x86_64, CPU: x86_64)

ClamAV/0.99.4 (OS: linux-gnu, ARCH: x86_64, CPU: x86_64)

ClamAV/0.99.2 (OS: linux-gnu, ARCH: x86_64, CPU: x86_64)

ClamAV/0.98.7 (OS: linux-gnu, ARCH: x86_64, CPU: x86_64)

ClamAV/0.100.0 (OS: linux-gnu, ARCH: x86_64, CPU: x86_64)

ClamWin/0.99.1 (OS: win32, ARCH: i386, CPU: i386)

ClamWin/0.99.4 (OS: win32, ARCH: i386, CPU: i386)

ClamAV/0.99.3 (OS: linux-gnu, ARCH: x86_64, CPU: x86_64)

Wget/1.12 (linux-gnu)

ClamAV/0.96-exp (OS: linux-gnu, ARCH: i386, CPU: i686)

Since we already covered Wget and tools like that, let's move on to older versions of ClamAV.

As you can see, the top two versions are the most recent versions and essentially, the last major version back. These are Ubuntu/Debian users. We work very closely with the package maintainer for those distributions, and we can see the shift in versions when they update.

Let's shift our focus to No. 10 in the list above. ClamAV/0.96-exp, an experimental version of ClamAV back in the 0.96 days, before CVD diff'ing was released. Obviously, this is an errant cronjob (someone attempting to run FreshClam once a minute). Again, we are asking users to please update to a new version of ClamAV, which will, in turn, update your version of FreshClam (to support diff'ing).

We took the logs, removed Wget, curl and other scripting tools, only focusing on actual FreshClam installations, only focusing on versions of ClamAV.  Basically, asking our logs, "How many different versions of ClamAV are there out there?"

297

After ClamAV is installed, then what? How do I update the database?

You will want to edit freshclam.conf.example by commenting out the "example" text with a pound symbol (#).

Once that is done, run a 'freshclam' via command line and the database should update.

We also want to remind people that we do monitor IRC, so if you need help — give us a shout.

Community members and staff idle the channel and try to help to the best of our abilities.

IRC information

Service: irc.freenode.net
Channel: #clamav

How do you submit bugs to ClamAV?

Please visit https://bugzilla.clamav.net/.

If you are having Mirror issues

If you are running into an issue when using FreshClam to download the signatures, it could be the mirror.dat file. Try deleting this file, and then running FreshClam again.

In the "configure" file of FreshClam, you can set your mirror as 'database.clamav.net' instead of using a country code (db.xy.clamav.net).  In a future version of ClamAV, we will improve/remove the mentions of country based mirrors. Use database.clamav.net as the central point for your downloads.

If you want to contribute to documentation

Our documentation is also open-source. Feel free to edit it as need be, and submit the changes to our Git repository.