Generating ClamAV Signatures with IDAPython and MySQL

Covering malware is a constant fight and the more automation you can integrate, the easier life becomes. This post will go over a relatively easy setup for generating ClamAV signatures based on a set of samples.

I chose to work with OSX malware, specifically targeting Mach-O files. This would give me a relatively small sample set to work with. I downloaded the files from VirusTotal using the search type:macho positives:5+. At the time of download, this yielded 239 samples.

The first problem was grouping samples. Grouping the samples would allow to generate a single signature for multiple samples. One signature for each sample is costly and leads to a bloated signature set. For this, I set up three MySQL tables.

    binaries - stores information about each sample seen
    +-------+-------------+------+-----+---------+----------------+
    | Field | Type        | Null | Key | Default | Extra          |
    +-------+-------------+------+-----+---------+----------------+
    | id    | int(11)     | NO   | PRI | NULL    | auto_increment |
    | md5   | varchar(32) | NO   |     | NULL    |                |
    | size  | int(11)     | NO   |     | NULL    |                |
    +-------+-------------+------+-----+---------+----------------+

    functions - stores information about each function seen
    +-------+-------------+------+-----+---------+----------------+
    | Field | Type        | Null | Key | Default | Extra          |
    +-------+-------------+------+-----+---------+----------------+
    | id    | int(11)     | NO   | PRI | NULL    | auto_increment |
    | md5   | varchar(32) | NO   |     | NULL    |                |
    | size  | int(11)     | NO   |     | NULL    |                |
    +-------+-------------+------+-----+---------+----------------+

    link_table - associates each binary with a set of functions
    +---------+---------+------+-----+---------+-------+
    | Field   | Type    | Null | Key | Default | Extra |
    +---------+---------+------+-----+---------+-------+
    | prog_id | int(11) | NO   | PRI | NULL    |       |
    | fn_id   | int(11) | NO   | PRI | NULL    |       |
    +---------+---------+------+-----+---------+-------+

The table binaries stores a hash of each program, a unique id, and the program's size. The table functions stores the md5sum of the bytes comprising the function, a unique id, and the size of the function. The table link_table links each binary to the functions it contains. The grouping is done based on common functions between binaries.

In order to populate these tables I wrote an IDAPython script. It iterates through the functions of the program, calculates their md5sum, and then inserts that information into the functions table if its length is greater than 19. The value 19 was selected after some light analysis in order to filter out functions that only consisted of a few instructions. Here is the snippet that populates functions and link_table.

    # for all function offsets
    for fn_ea in Functions():
        if fn_ea == None:
            continue

        # get function from offset
        f = idaapi.get_func(fn_ea)

        # get function bytes
        start = f.startEA
        size = f.endEA - start
        bytes = GetManyBytes(start, size)

        # if the function is sufficiently long
        if bytes != None and len(bytes) > 19:
            fn_hash = md5(bytes).hexdigest().upper()
            fn_size = str(len(bytes))
            fn_data = (fn_hash, fn_size)

            # get function id, or insert and get function id
            fn_id = get_fn_id(cursor, cnx, fn_data)

            # link binary to function
            link_query = 'REPLACE INTO link_table (prog_id, fn_id) VALUES (%s, %s)'
            link_data = (prog_id, fn_id)
            cursor.execute(link_query, link_data)
            cnx.commit().


IDA and this script are called by a batch script for every target binary. Once these tables are populated another script is ran, generate_sigs.py. This script uses the MySQL functionality group_concat to group binaries, based on their common functions, into a list. The problem with this approach is that if binaries A, B, and C share functions x, y, and z, and binaries A and C share functions w, x, y, and z, then we will have duplicates in the list returned. To remedy this problem the script simply loops through the rows returned and if any list of binaries is completely contained in another list, it is removed. Any binary not in these groupings is marked to get its own signature.

Next, the md5sums of the functions common to each group are added to the table communicate. This was the best way for me to pass this information between scripts. Once this table is populated, another IDAPython script is called on the first binary in a group. This script iterates through the functions in the binary and if the function's md5sum matches one in the list of shared functions, its basic blocks are loaded into a table basic_blocks. This table stores the parent function's md5sum, the bytes that comprise the basic block, the basic block's md5sum, the size of the basic block, and its entropy. The byte_ prefix is used to differentiate between attributes of the raw data and the hex encoded version used in the ClamAV signatures.

    communicate - used to pass the md5s of common functions
    +--------+-------------+------+-----+---------+-------+
    | Field  | Type        | Null | Key | Default | Extra |
    +--------+-------------+------+-----+---------+-------+
    | fn_md5 | varchar(32) | NO   | PRI | NULL    |       |
    +--------+-------------+------+-----+---------+-------+

    basic_blocks - stores basic block information from functions
    +--------------+-------------+------+-----+---------+-------+
    | Field        | Type        | Null | Key | Default | Extra |
    +--------------+-------------+------+-----+---------+-------+
    | fn_md5       | varchar(32) | NO   | PRI | NULL    |       |
    | hex_bytes    | mediumtext  | NO   |     | NULL    |       |
    | bb_md5       | varchar(32) | NO   | PRI | NULL    |       |
    | byte_size    | int(11)     | NO   |     | NULL    |       |
    | byte_entropy | double      | NO   |     | NULL    |       |
    +--------------+-------------+------+-----+---------+-------+

Once the basic blocks are stored, the IDAPython script completes and returns the the signature generation script. The basic blocks are queried for, sorted by their parent function and a metric entropy * size. The script then iterates through the functions and selects the best basic block based on the previously mentioned metric. It continues to do this until it has a sufficient amount of bytes. It then constructs an LDB signature.

With my newly created signatures, I ran a test on all the samples I had downloaded.

----------- SCAN SUMMARY -----------
Known viruses: 107
Engine version: 0.98.1
Scanned directories: 1
Scanned files: 239
Infected files: 190
Data scanned: 78.35 MB
Data read: 81.36 MB (ratio 0.96:1)
Time: 2.332 sec (0 m 2 s)

The interesting lines are highlighted. Since this script should give near total coverage, a detection rate of 190/239, while impressive, did not meet my expectations. Something was amiss. My colleague Shaun Hurley noticed that 64 bit Mach-O files were being neglected. Thinking about it, this made sense. IDA has different versions for 32 bit and 64 bit files. I modified the scripts to use idaw64.exe and reran them on the 64 bit binaries. The combined signature set was more impressive.

----------- SCAN SUMMARY -----------
Known viruses: 155
Engine version: 0.98.1
Scanned directories: 1
Scanned files: 239
Infected files: 232
Data scanned: 78.82 MB
Data read: 81.36 MB (ratio 0.97:1)
Time: 2.535 sec (0 m 2 s)

Great success!

This method does have some drawbacks. Since I was running it in a VM, concerns about hard disk space influenced the choice to group based on functions rather than grouping based on basic blocks. This will be fixed by offloading MySQL to a more dedicated machine whose hard drive I can fill up. As well, only common functions between the binaries are considered when selecting basic blocks. This was an oversight on my part since other functions may not be exact matches but could share a lot of common code. With the extra database space, I do not think grouping based on basic blocks is an unreasonable task for these relatively small sets of samples. Building in automatic identification of 32 bit and 64 bit files would remove some manual effort from the process.

A good example of a signature generated for multiple samples is this one for Flashback:

Osx.Trojan.Flashback-16;Engine:51-255,Target:9;0&1&2&3&4&5&6&7&8;5531C089E58B550885D2740B;5589E583EC28895DF48B5D088975F88B750C897DFC8B7D1085DB0F94C285F60F94C031C908C20F858A000000;5589E58B450885C0740B;5589E583EC18C744240801000000C744240414000000C7042400000000E83F020000C74008000000;5589E557565383EC2C8B45100FB7550C8B5D188945E08B451489542404895C24088945E48B451C89;8D4208894424088B450C89142489442404E818FFFFFF0FBEC0;5589E557565383EC1C8B7D0885FF7432;8B450C894210B801000000;C744240801000000C74424040C000000C7042400000000E877010000893089C28B073B03751E

While that signature is just extracted x86, it alerts on the following 15 samples:

B5942F202930DAFF45C79BDC7871C088
548981EF3FCB813FCD3ED2EBAB8102D7
C067B84DC59C93C1363FD9FC56CD2918
B0199B369A3FCC71653ED8A9F7990AFC
4E855DD770680F80A30B9805262BBEE6
EF2DB2EEB040BDF1D0A9A18F2775149B
9272778BB6FBC00131FFCECE51388ACB
BE1B0DB89A4798E6C11E4EBFB6B479AE
CED7C97304BFFD932822565E99460213
B94BF524A537C02DDA4CD047F61E00C4
14DE914B0101C0E7A2C7CF521557E747
657E5A48CEC24F0C6F516CA55581550F
647AF7013D0DA77B6E74D3C692B1B6C3
84352BF4A2FA95FC51AD0781000AA864
93734AEBC1670C22A79F08D1A0FCBD8F

Overall, I'm very happy with these results. Since IDAPro is used to extract everything, this work will translate well to the other binary types that IDA is capable of parsing - most importantly, portable executables.

Careto: Covering unavailable samples

Kaspersky labs released a report that covers in detail a piece of malware known as "Careto" or "The Mask". The report included several MD5 hashes of samples and related files, IP addresses and domain information. Typically with ClamAV,  a hash signature targeting an entire file is formatted as following:

MD5:FileSize:Name

The samples for Careto and therefore their sizes were unavailable to us at the time of this blog post, making it impossible to release hash-based coverage. However, as of ClamAV 0.98, a hash signature can be written with a wildcard for the file size. The format for such a signature is:

MD5:*:Name:73

The 73 on the end will prevent the signature from being loaded by an older ClamAV engine that doesn't support this signature format.

The Mask is a combination of tools that cover 32-bit and 64-bit Windows, Mac OS X and Linux. Kaspersky also identified potential Android and Apple iOS variants. Their analysis indicates it can intercept many different forms of communication from the victim machine, exfiltrate data and provide remote access to the attacker.

This signatures file can be used to detect the sample discussed in the article. Just download it and put it in the same folder where you have your ClamAV signatures. If any alerts are generated from these please let us know by emailing research < at > sourcefire (dot) com.

ClamAV Mailing List Maintenance, Monday, February 10th, 2014

This notice is for the members of the ClamAV mailing lists found here:

http://lists.clamav.net/mailman/listinfo/clamav-users

On Monday, February 10th, 2014 starting at 10am EST, the ClamAV Mailing lists will be moving to new server hardware.  We anticipate this outage to last approximately four (4) hours.  We will be notifying everyone when the new server is up and operational.

Thank you for your patience.


Joel Esler
Threat Intelligence Team Lead
Open Source Manager
Vulnerability Research Team

Vote for ClamAV as the Sourceforge Project of the Month!

Sourceforge has fired up their monthly "Project of the Month" process again, and they were kind enough to choose ClamAV for this months vote!

You can read more about the process on their blog post here: https://sourceforge.net/blog/revival-of-weekly-featured-projects-and-project-of-the-month-voting/ 

And you can cast your vote here:  https://sourceforge.net/p/potm/discussion/vote/thread/7d522915/

Thanks to everyone who supports the ClamAV project, get out and vote!

(Note: You must be a member of Sourceforge, and must be logged in, to vote.)

ClamAV 0.98.1 has been released!

ClamAV 0.98.1 provides improved support of Mac OS X platform, support for new file types, and
quality improvements. These include:

- Extraction, decompression, and scanning of files within Apple Disk Image (DMG) format.
- Extraction, decompression, and scanning of files within Extensible Archive (XAR) format.
XAR format is commonly used for software packaging, such as PKG and RPM, as well as
general archival.
- Decompression and scanning of files in "Xz" compression format.
- Improvements and fixes to extraction and scanning of ole formats.
- Option to force all scanned data to disk. This impacts only a few file types where
some embedded content is normally scanned in memory. Enabling this option
ensures that a file descriptor exists when callback functions are used, at a small
performance cost. This should only be needed when callback functions are used
that need file access.
- Various improvements to ClamAV configuration, support of third party libraries,
and unit tests.

There are also fixes for other minor issues and code quality changes. Please
see the ChangeLog file for details.

--
The ClamAV team (http://www.clamav.net/team)

Cisco, Community and Open Source

In July we told you about Sourcefire’s agreement to be acquired by Cisco, and today that acquisition has closed – we are now one company. This also means that we are also now one community, and Cisco has reiterated its commitment to maintaining our innovation and support of Snort, ClamAV and other open source projects, as well as its own projects. As Marty Roesch wrote on our corporate blog:

"I can tell you with certainty that this is a great match for Sourcefire, for Cisco and, ultimately, for our customers, partners and open source communities… Beyond the technology, one of the things that is important to me is that Cisco and Sourcefire both share key values that transcend our company names, HQ locations and number of employees."

 I’m also happy to report that there will be no changes to how our communities are run or our communications, including mailing lists, snort.org, clamav.net or social media sites. Please visit the corporate blog for more details and, as always, reach out to me with questions. I will still be your community manager and I look forward to many more years of being a part of this community.

Article on Android Basic Block Signatures

Everyone that reads this blog may not read all of the other Sourcefire/Vulnerability Research Team (VRT) Blogs, so I thought I'd add a quick comment here about one of the Malware Research Teams articles over on our VRT Blog.

The article is entitled "Android Basic Block Signatures" and he goes over some good syntax for the ClamAV Signature Language.

Please check it out here: http://blog.talosintel.com/2013/10/android-basic-block-signatures.html

ClamAV 0.98 has been released!

ClamAV 0.98 includes many new features, across many different components
of ClamAV. There are new scanning options, extensions to the libclamav API,
support for additional filetypes, and internal upgrades.

- Signature improvements: New signature targets have been added for
PDF files, Flash files and Java class files. (NOTE: Java archive files
(JAR) are not part of the Java target.) Hash signatures can now specify
a '*' (wildcard) size if the size is unknown. Using wildcard size
requires setting the minimum engine FLEVEL to avoid backwards
compatibility issues. For more details read the ClamAV Signatures
guide.

- Scanning enhancements: New filetypes can be unpacked and scanned,
including ISO9660, Flash, and self-extracting 7z files. PDF
handling is now more robust and better handles encrypted PDF files. 

- Authenticode: ClamAV is now aware of the certificate chains when
scanning signed PE files. When the database contains signatures for
trusted root certificate authorities, the engine can whitelist
PE files with a valid signature. The same database file can also
include known compromised certificates to be rejected! This
feature can also be disabled in clamd.conf (DisableCertCheck) or
the command-line (nocerts). 

- New options: Several new options for clamscan and clamd have been
added. For example, ClamAV can be set to print infected files and
error files, and suppress printing OK results. This can be helpful
when scanning large numbers of files. This new option is "-o" for
clamscan and "LogClean" for clamd. Check clamd.conf or the clamscan
help message for specific details. 

- New callbacks added to the API: The libclamav API has additional hooks
for developers to use when wrapping ClamAV scanning. These function
types are prefixed with "clcb_" and allow developers to add logic at
certain steps of the scanning process without directly modifying the
library. For more details refer to the clamav.h file. 

- More configurable limits: Several hardcoded values are now configurable
parameters, providing more options for tuning the engine to match your
needs. Check clamd.conf or the clamscan help message for specific
details. 

- Performance improvements: This release furthers the use of memory maps
during scanning and unpacking, continuing the conversion started in
prior releases. Complex math functions have been switched from
libtommath to tomsfastmath functions. The A/C matcher code has also
been optimized to provide a speed boost. 

- Support for on-access scanning using Clamuko/Dazuko has been replaced
with fanotify. Accordingly, clamd.conf settings related to on-access
scanning have had Clamuko removed from the name. Clamuko-specific
configuration items have been marked deprecated and should no longer
be used.

There are also fixes for other minor issues and code quality changes. Please
see the ChangeLog file for details.

--
The ClamAV team (http://www.clamav.net/team)

Reminder: New Main.cvd is scheduled to be pushed tomorrow, September 17

As a reminder in case you didn't see the first warning, we are planning on pushing a new Main.cvd tomorrow, September 17th.  This will cause an increase in load on the mirror infrastructure as clients will have to download this new file.

Please see my previous blog post on the matter if you have any questions:
http://blog.clamav.net/2013/09/maincvd-scheduled-for-tuesday-september.html

Main.cvd scheduled for Tuesday September 17th

As many of you have written in and notified us, the "Daily.cvd" file is now bigger than the "Main.cvd".  This means on our side, it's time to make a new Main.cvd.

We are currently planning on cutting a new Main.cvd on Tuesday September 17th.  After the new Main.cvd is published the daily load on the mirrors and your networks should be much lighter.

We are estimating the new "Main" to be ~70MB.

We are also planning on making a new Main.cvd more periodically so this can be more easily predicted.  More information about this periodic process will be published as more details become available.