Today I'm very excited, and a little bit nervous, to unveil Mussels. Mussels is a cross-platform, general-purpose dependency build automation tool. You might compare it with Vcpkg, Conan, or Buildout. It serves a similar purpose, but the approach is a little different.
How to get Mussels
The Mussels project is hosted on GitHub under Cisco-Talos/Mussels.By the time this blog post is published, you should be able to install Mussels from PyPI using Pip. You may also clone the Mussels Git repository and use Pip to install it locally.
Install Mussels from PyPI:
python3 -m pip install --user mussels
Install Mussels from a Git clone:
python3 -m pip install --user .
The origin of Mussels
Mussels is something I've been crafting as a hobby project in support of ClamAVⓇ. The need for a dependency management tool became obvious as we were actively engaged in upgrading ClamAV's external dependencies, specifically for Windows builds. Historically, ClamAV maintained a collection of third-party code that was copy-pasted into our own repository with custom Visual Studio project files created to build these libraries. This approach worked and made it simple enough to compile the project.Maintenance, however, was a bit of a nightmare, so we decided to separate these dependencies out. This meant we'd have new things to build as dependencies of ClamAV, however, and things only got worse the moment we decided to add libcurl as a hard requirement in support of HTTP 1.1/2.0, and TLS/SSL.
All of a sudden, we went from ClamAV requiring just OpenSSL to be built separately from ClamAV, to having to build:
- zlib
- bzip2
- pthread-w32
- libjson-c
- OpenSSL (depends on zlib)
- libxml2 (depends on zlib)
- libpcre2 (depends on bzip2, zlib)
- libssh2 (depends on OpenSSL, zlib)
- NGHTTP2 (depends on libxml2, OpenSSL, zlib)
- libcurl (depends on OpenSSL, NGHTTP2, libssh2, zlib)
Mussels started out as something really simple that was largely focused on building dependencies for ClamAV on Windows. There were a couple of occasions, however, when I needed to build some new combination of libraries. I quickly threw together a few Mussels recipes to make it happen. With Mussels, things just worked. I'd type mussels build clamav_deps and it was off to the races while I'd wander away to grab some tea and marvel at how easy it all was. That was when I realized I really wanted to productize this Mussels-thing and make it available for the general public. I've spent as much time as I could afford to make Mussels ready for public consumption once I received approval to make it open-source.
How Mussels works
Mussels is intended to simplify the process of building complex applications that have lengthy dependency chains without having to write all new CMake, Meson, Bazel, XCode, or Visual Studio project files. Instead, you write (and share) simple recipes that leverage the original build systems intended by software authors of your external library dependencies.Recipes are YAML files that detail how to build a given library or application. A recipe defines where to get the software source archive, what other recipes the software depends on for the build, what tools are required to build the software, and of course what commands to run to perform the build.
A simple example recipe:
name: zlib
version: "1.2.11"
url: https://www.zlib.net/zlib-1.2.11.tar.gz
mussels_version: "0.1"
type: recipe
platforms:
Linux:
host:
build_script:
configure: |
cmake . -DCMAKE_INSTALL_PREFIX="{install}/{target}"
make: |
cmake --build . --config Release
install: |
make install
dependencies: []
install_paths:
license/zlib:
- README
required_tools:
- cmake
- make
- gcc
Like a recipe, developers may define tools in YAML as well. Tools describe how to identify if a given build tool (like GCC, Cmake, Python, Java, etc.) exists on the current machine.
A simple tool definition:
name: gcc
version: ""
mussels_version: "0.1"
type: tool
platforms:
Posix:
path_checks:
- gcc
command_checks:
- command: "gcc --version"
output_has: "gcc"
file_checks:
- /usr/local/bin/gcc
- /usr/bin/gcc
Recipes and tools may be shared in Git repositories that we call "cookbooks."
Cookbooks can either be public Git repositories, private Git repositories, or simply a local directory containing recipe and tool YAML files. It's really easy to create a new recipe or tool by cloning existing recipes and tools to your local directory and then customizing them to suit your project's needs.
Mussels works on macOS, Linux/Unix, and Windows operating systems. Though it was originally written in support of building C-based application libraries, it's flexible and can be extended to build and assemble any software package.
Running a build with Mussels is as simple as this:
- Identify the recipe you wish to build using:
mussels update
mussels list -a - Verify that the recipe will suit your needs with:
mussels recipe show <recipe-name> -V - Clone the recipe to your current directory:
mussels recipe clone <recipe-name>
Or, choose to trust the cookbook which provides the recipe:
mussels cookbook trust <cookbook-name>
- Do a dry run, to see what all will be built:
mussels build <recipe-name> --dry-run
If you are missing tools required for the build, Mussels will tell you want you will need. - Run the build!
mussels build <recipe-name>
You can also join us on Discord.
I hope you will give Mussels a try, and I hope you'll find a way to use Mussels to make development on your project a little bit easier.
For those who have been interested to build the latest version of ClamAV on Windows from source without having to build and assemble all of the dependencies by hand, we've also added the ClamAV Mussels cookbook to the Mussels bookshelf, making our own recipes publicly available for anyone to use or copy.
