Monday, June 28, 2021

New online documentation and other improved open-source processes

By Micah Snyder.

I'm excited to announce that we are making several changes that will improve how we work with the open-source community. We've also made several other changes to ClamAV.net over the past year that we are excited to formally unveil today, including a new security policy and new build advisories. 

New online documentation 

I'm pleased to unveil our new online documentation at docs.clamav.net. We've combined all of our documentation located on GitHub pages with the Rust language mdBook static site generator to provide a much-improved experience for navigating and contributing to our documentation. This new documentation uses GitHub Actions to automatically update the site whenever a commit is merged into the documentation source. 

Migrating from Bugzilla to GitHub

We are deprecating our old Bugzilla server. New tickets should be created using GitHub Issues. The old Bugzilla server will continue to exist as a reference but will not allow users to create new tickets. We believe that GitHub Issues provides a better experience for developers and reporters.

Bugzilla’s tickets were private by default, but GitHub Issues are always public. So, this change means that reporters will no longer depend on ClamAV developers to make a ticket public if it's not deemed a security issue.  


New security policy 

Any issue that may be a security risk should be reported to Cisco PSIRT and should never be reported on GitHub Issues. Please see our new Security Policy for more details.  

Revamped public test suite 

Earlier this year, we ported our public test suite to use CMake's CTest and a custom framework based on Python's unittest module in place of what used to be Check and a collection of Bash scripts.

This led to several positive outcomes:  
  • Our new CMake build system, introduced in 0.103, now has supports running our public test suite and is now ready to replace the old Autotools build system.   
  • We finally have support for running the public tests on Windows. Though our private test suite has supported Windows for some time and has a lot more tests, it does not test all the same things. Windows support in the public test suite means we are less likely to inadvertently introduce bugs that only affect Windows users.  
  • Our new Python-based test framework brings our public test suite more in line with how our internal QA test suite works. This means that we now have an easy option to migrate tests from our private test suite to our public test suite provided that the tests do not require access to network resources or other internal resources. 

Build system change advisory 

We now consider the CMake build system to be complete and stable in the 0.104 development branch. So, we have decided to remove our old Autotools build system. We previously intended to deprecate Autotools and keep it around for at least one more release. However, the work we’re doing to add Rust toolchain support into ClamAV has demonstrated that it isn’t worth the extra work required to maintain the Autotools build system. CMake and the Rust toolchain will be required to build ClamAV in the 0.104 release. 

GitHub pull-request automation  

  • We recently added GitHub automation to aid in pull-request review.   
  • We added GitHub Actions to build ClamAV and run our public test suite on Ubuntu, macOS and Windows.
  • We added the LGTM.com GitHub app for automatic static analysis review of pull-requests to the ClamAV and the clamav-bytecode-compiler repositories.   
  • We added a GitHub Action to validate the code matches our code style using Clang. 
 

Official ClamAV Docker images  

We now have official ClamAV images on Docker Hub for ClamAV and for the ClamAV bytecode compiler. Special thanks to Olliver Schinagl for their excellent work adding Docker support and documentation. 

However, we have not published a stable ClamAV release on Docker Hub. ClamAV 0.104.0 will be the first. For now, you can use the clamav/clamav:unstable for image builds that track with the latest changes added to the upcoming feature release. 

Revised Git workflow 

The Git workflow employed by the ClamAV project for the past two years used development branches (e.g., dev/0.104) to stage work for the next feature release. These branches ran additional tests after merging pull requests. The intention was that changes in a dev branch would only be merged into the master branch after proven stable.

However, effective use of the pull request process, along with significant improvements to our test automation, have negated the benefits of this extra layer. So, we are dropping this practice. As soon as we’re done with 0.104.0, the current feature version, we will be changing our Git workflow. Approved pull requests will be merged into our main branch, which will replace the feature development branch and the old master branch. Development branches will continue to be used only when working on patch versions (e.g., dev/0.104.1), but will be deleted after they are merged. This updated workflow will look something like this:










We may still create feature branches for larger collaborative works (e.g., feature/*), like feature/rust_integration, which adds Rust toolchain support. Feature branches are inherently unstable and will be rebased or even force-pushed as needed. 

For comparison's sake, here's what the old Git workflow looked like:


Working with GitHub directly 

The primary ClamAV repository used to live within the Cisco-Talos network. This is changing. Going forwards, our GitHub repository will be our primary repository. This move will have two major effects. 

First, reviewing and merging contributions from the community will be more natural. Although our internal Jenkins server for QA will not be publicly accessible, it will be set up to automatically test all PR's and not just those from the development team. This, in combination with our new GitHub actions for build testing, static analysis, and format checking, will ensure a more streamlined PR review experience. 

Second, the ClamAV development team will work in public forks on GitHub instead of forks on our corporate Git server. Our own code submissions will go through the same public pull-request process as any submission from the community. We are hopeful this will encourage developer growth within the community.

As you may have noticed in the above diagram, we will continue to maintain a private repository within the Cisco-Talos networks for staging and testing security-related fixes in advance of security patch version releases.