Pavel Panchekha


Share under CC-BY-SA.

Package Management for the 21.1st Century

A package manager is something any Linux user is familiar with – it's a piece of software tasked with installing, updating, and un-installing other software. Linux distributions tend to have battle-tested, production ready package managers. But with the new trend for "App Stores", a lot of other players have built their own package-manager-alikes, and its become evident that our experience with package managers for Linux distributions does not translate over.

Two specific blog posts underscore this issue very well. The first, "We've Packaged all of the Free Software… What Now?", describes the flaws in Debian and Ubuntu's model of software packages – the rough edges, so to speak. And "Why Platform-Specific Package Systems Exist and Won't Go Away" makes the case that the different scenarios we want a package system for are different enough that we simply can't take the existing options and extend them slightly to work – we need new thoughts about package systems, and a new approach to building them.

I think all of this talk is highly apropos: the package managers of the Linux style have some well-known and well-publicized flaws, while the App Stores currently being built solve only very simple cases; and, of course, nowadays every large piece of software has its own copy of a mediocre package system inside. I think we can build something better.

Here, as far as I see it, are the scenarios we need to support:

I think this is a tough list of requirements, but a doable one. But if we want this to work, we also need to make it transition-able: we want a solution that other tools can adopt as an alternate backend, then as the primary backend, and only then as the only backend. This is exceedingly difficult to do if we pick our own package format and installer, since there are so many different requirements. So I think it is necessary to start by separating package maintenance from installation. A package system should be able to find packages, search for updates, and buy software, but it should offload the task of installing these packages to other tools. Luckily, most existing package systems have some way of downloading the package as a file and installing it from this local file.

Furthermore, since we now have a vastly-reduced package space, and due to the needs to be cross-platform and cross-language, it seems most sensible that the package system of the 21.1st century ought to be a protocol. A well-designed protocol could be federated, and makes it easy to offload the work of installing to local tools.

Federation is a powerful requirement, because by adopting it as a central design feature of our protocol, we can subsume many other requirements. Here's how I imagine a protocol could work:

I think the model above also has the benefit of being simple to implement piece-by-piece, tool-by-tool. If just one language or community adopts the protocol, it could be proven to work, and in time people could set up repositories that mirror, say, the Debian archive, in a manner accessible by this system. Then over time apt could switch to using that mirror, or another tool using this mirror could be written that begins to replace apt.

The protocol itself needs very careful thought – the problem of shipping software is not trivial. A good first stab, for example, might be a protocol that allows for three operations: listing packages a repository has, querying for the available versions of a package, and perhaps other assorted metadata, and find all the dependencies of a package-version.

But there are flaws to this. For example, there's no good way to check for updates to just the installed packages, without downloading the full packages list or making multiple queries. Instead, we might imagine each repository having a vector clock of its neighbors and being able to list changes to its repository since a given point in time. For large archives that are not too-often updated (like OS package archives) this could be a good solution.

I think this aspect needs more work and more thought. But I'm confident a solution can be found that works for a much larger range of scenarios that current package systems, and I hope the above architecture spurs some thoughts that could lead us to this solution.