Tuesday, December 3, 2019

Introducing Mussels, an application dependency build automation tool

By Micah Snyder.

Today I'm very excited, and a little bit nervous, to unveil Mussels. Mussels is a cross-platform, general-purpose dependency build automation tool. You might compare it with Vcpkg, Conan, or Buildout. It serves a similar purpose, but the approach is a little different.


How to get Mussels

The Mussels project is hosted on GitHub under Cisco-Talos/Mussels.

By the time this blog post is published, you should be able to install Mussels from PyPI using Pip. You may also clone the Mussels Git repository and use Pip to install it locally.

Install Mussels from PyPI:

python3 -m pip install --user mussels

Install Mussels from a Git clone:

python3 -m pip install --user .

The origin of Mussels

Mussels is something I've been crafting as a hobby project in support of ClamAVⓇ. The need for a dependency management tool became obvious as we were actively engaged in upgrading ClamAV's external dependencies, specifically for Windows builds. Historically, ClamAV maintained a collection of third-party code that was copy-pasted into our own repository with custom Visual Studio project files created to build these libraries. This approach worked and made it simple enough to compile the project.

Maintenance, however, was a bit of a nightmare, so we decided to separate these dependencies out. This meant we'd have new things to build as dependencies of ClamAV, however, and things only got worse the moment we decided to add libcurl as a hard requirement in support of HTTP 1.1/2.0, and TLS/SSL.

All of a sudden, we went from ClamAV requiring just OpenSSL to be built separately from ClamAV, to having to build:
  • zlib
  • bzip2
  • pthread-w32
  • libjson-c
  • OpenSSL (depends on zlib)
  • libxml2 (depends on zlib)
  • libpcre2 (depends on bzip2, zlib)
  • libssh2 (depends on OpenSSL, zlib)
  • NGHTTP2 (depends on libxml2, OpenSSL, zlib)
  • libcurl (depends on OpenSSL, NGHTTP2, libssh2, zlib)
It clearly isn't feasible to build all of this without some sort of automation. Most of these libraries are actively maintained projects that see new releases every couple of months. And it certainly isn't something we wanted to maintain source code copies of inside the ClamAV source repository.

Mussels started out as something really simple that was largely focused on building dependencies for ClamAV on Windows. There were a couple of occasions, however, when I needed to build some new combination of libraries. I quickly threw together a few Mussels recipes to make it happen. With Mussels, things just worked. I'd type mussels build clamav_deps and it was off to the races while I'd wander away to grab some tea and marvel at how easy it all was. That was when I realized I really wanted to productize this Mussels-thing and make it available for the general public. I've spent as much time as I could afford to make Mussels ready for public consumption once I received approval to make it open-source.

How Mussels works

Mussels is intended to simplify the process of building complex applications that have lengthy dependency chains without having to write all new CMake, Meson, Bazel, XCode, or Visual Studio project files. Instead, you write (and share) simple recipes that leverage the original build systems intended by software authors of your external library dependencies.

Recipes are YAML files that detail how to build a given library or application. A recipe defines where to get the software source archive, what other recipes the software depends on for the build, what tools are required to build the software, and of course what commands to run to perform the build.

A simple example recipe:

name: zlib
version: "1.2.11"
url: https://www.zlib.net/zlib-1.2.11.tar.gz
mussels_version: "0.1"
type: recipe
platforms:
  Linux:
    host:
      build_script:
        configure: |
          cmake . -DCMAKE_INSTALL_PREFIX="{install}/{target}"
        make: |
          cmake --build . --config Release
        install: |
          make install
      dependencies: []
      install_paths:
        license/zlib:
          - README
      required_tools:
        - cmake
        - make
        - gcc


Like a recipe, developers may define tools in YAML as well. Tools describe how to identify if a given build tool (like GCC, Cmake, Python, Java, etc.) exists on the current machine.

A simple tool definition:

name: gcc
version: ""
mussels_version: "0.1"
type: tool
platforms:
  Posix:
    path_checks:
      - gcc
    command_checks:
      - command: "gcc --version"
        output_has: "gcc"
    file_checks:
      - /usr/local/bin/gcc
      - /usr/bin/gcc


Recipes and tools may be shared in Git repositories that we call "cookbooks."

Cookbooks can either be public Git repositories, private Git repositories, or simply a local directory containing recipe and tool YAML files. It's really easy to create a new recipe or tool by cloning existing recipes and tools to your local directory and then customizing them to suit your project's needs.

Mussels works on macOS, Linux/Unix, and Windows operating systems. Though it was originally written in support of building C-based application libraries, it's flexible and can be extended to build and assemble any software package.

Running a build with Mussels is as simple as this:
  1. Identify the recipe you wish to build using:
    mussels update
    mussels list -a
  2. Verify that the recipe will suit your needs with:
    mussels recipe show <recipe-name> -V
  3. Clone the recipe to your current directory:
    mussels recipe clone <recipe-name>

    Or, choose to trust the cookbook which provides the recipe:
    mussels cookbook trust <cookbook-name>
  4. Do a dry run, to see what all will be built:
    mussels build <recipe-name> --dry-run
    If you are missing tools required for the build, Mussels will tell you want you will need.
  5. Run the build!
    mussels build <recipe-name>
Hop over to the Mussels README page on GitHub to learn more.
 
You can also join us on Discord.

I hope you will give Mussels a try, and I hope you'll find a way to use Mussels to make development on your project a little bit easier.

For those who have been interested to build the latest version of ClamAV on Windows from source without having to build and assemble all of the dependencies by hand, we've also added the ClamAV Mussels cookbook to the Mussels bookshelf, making our own recipes publicly available for anyone to use or copy.