Software packaging and deployment

This is an opinionated overview of the available software packaging and deployment options, including those for commercial software and binary releases.

The problem

Proper and usable software installation may include placement of executable files, data files, documentation such as man pages and info manuals, configuration files with appropriate permissions, init system configurations (e.g., init scripts, service files) for daemons, freedesktop.org configurations for desktop applications. As well as creation of system users, a way to uninstall the software, proper handling of configuration files (not just overriding user configuration). And preferably it should be manageable using standard system tools, since it's a pain to use multiple package managers to maintain a system.

Furthermore, there are many nuances and incompatibilities among major GNU/Linux distributions (not to mention other UNIX-like systems, or different OS families), so it's not a straightforward task. For instance, it used to be a pain to write init files for different distributions because of the available software; now there is systemd on most of the major ones, yet the paths for service files still differ. Generally, automated and reliable integration of different software components is tricky.

Solutions

"Proper" flow

In the least hacky out of common scenarios, upstream developers release the software following the standards and conventions, which then gets packaged for different systems by maintainers, and necessary adjustments get introduced.

But this doesn't work for commercial software, or for binary releases of smaller projects. Neither does it guarantee that installation and maintenance wouldn't be a pain (cf. most of the issue trackers), though it doesn't have to be.

System-independent build systems

Build systems such as GNU autotools, or the language-specific ones such as Cabal for Haskell programs, can be used for packaging and installation on their own. Autotools even try to deal with system incompatibilities, but still don't cover all the tasks (such as user creation or portable service installation), dependency resolution and automatic installation are partial at best (for language-specific package managers), and of course the software installed that way isn't manageable with a package manager native to a given system.

It's even less suitable for binary distributions: build systems are mostly for building, as the name suggests.

Ad hoc mess

Custom shell scripts or Makefile, curl | sh installation, various other custom installers, manual code copying, lengthy and awkward installation instructions, DVCS-based deployment (with private keys and passwords occasionally being in a repository and/or hardcoded), and virtual machine images seem to be used rather often for in-house or "enterprise" commercial software. It's a mess and a nightmare to maintain, usually matching the software it is used for, but perhaps worth mentioning as a bad example.

Masked mess

There are projects that do roughly what's described in the previous section, but with dedicated websites full of marketing slogans and making those solutions not so custom by getting more people to use the same kind of a solution. For instance, Flatpak and AppImage (primarily for desktop applications), Docker. Their issues are not very different from those with the ad hoc approaches, though they introduce a possibility of at least non-standard package management, and may patch some of the issues that arise.

Upstream packaging

That's the one I've settled on so far: write a program as an upstream developer (following FHS and other standards and conventions, using portable libraries when available), then package it as a maintainer (following Debian New Maintainers' Guide, though I'm still cutting some corners, and with cabal copy --destdir=deb), then deploy and configure it as an administrator.

An upside of such an approach is that software properly integrates into the system, so the regular practices can be applied. Sometimes it also makes you to adjust the software to make it easier to package and maintain.

A downside is that properly maintaining a custom repository (with timely key rollover) is a responsibility, and the documentation seems to mostly aim FLOSS project inclusion into the primary repositories. Hence all the third-party repositories that break updates, apparently. I'm just using dpkg -i on Debian instead, but it doesn't quite solve the problem of distribution and updates via a system package manager.

Tips and tricks

Here are some tips and tricks for writing software and packaging it in such a way that it would be relatively painless to deploy and maintain (apart from the regular "follow the standards and conventions").

Database management

PostgreSQL authentication

For applications that use PostgreSQL, it is handy to default to an empty (but configurable, of course) connection string: it will just work with a local database and peer authentication, simplifying the necessary deployment steps. While providing a way to specify the connection string (and not, say, just credentials) keeps it very flexible.

Database initialisation

To prepare a database (create tables, define roles and security policies, stored procedures and aggregations, views, insert initial data, etc), a handy approach is to add an "init" mode into the application, which would simply read SQL files from a data directory (which should be packaged and installed there) and execute those in a single transaction, potentially prompting a user for an initial application administrator password. It seems straightforward, yet rarely gets done any nicely.