CLPM - Common Lisp Project Manager

WARNING: This software is BETA quality. I use it as my daily driver, but it is still a little rough around the edges and it may accidentally eat your files.

Description

CLPM is a project manager for Common Lisp. It can manage projects in both global and project specific contexts. It would be called a "package manager" in most other contexts, but the use of "package" unfortunately collides with the use of "package" in the Common Lisp spec as a namespace for symbols.

It consists of two major pieces. First is a standalone program that is responsible for all the heavy lifting of downloading and unpacking releases in the correct place, resolving dependencies, and managing project specific environments. This piece is generally referred to as CLPM, the CLPM core, or clpm and is distributed as a precompiled executable using SBCL (but it is possible to compile it from source). The second is a small client library written in portable Common Lisp that interfaces with ASDF and calls CLPM as necessary to find and install missing systems. This piece is generally referred to as the CLPM client or clpm-client and is meant to be loaded when you are interactively developing code.

CLPM is licensed under the two clause BSD license (BSD-2). So, basically do what you want with it (while providing appropriate attribution) and don't complain if it breaks things. CLPM is currently beta quality so expect some breakage and incompatibilites going forward.

To receive help or report issues please send email to clpm-devel@common-lisp.net. Additionally, feel free to join the email list at https://mailman.common-lisp.net/listinfo/clpm-devel. To join in the development of CLPM, you can find the project hosted on the common-lisp.net Gitlab server at https://gitlab.common-lisp.net/clpm/clpm.

Tutorial

It is highly recommended that you page through the tutorial. It will walk you through installing, configuring, and using CLPM. Abbreviated descriptions are below and the docs folder contains more in-depth technical discussions.

Installing

CLPM is distributed in both source and binary form. For either version, first install the dependencies:

A Lisp implementation
SBCL is currently required if you are installing from source. SBCL or CCL are the most tested implementations for groveling dependencies from .asd files (necessary if you are installing a development version of a project).

git
If you want to use development versions of your dependencies.

tar
Required.

Prebuilt binaries

To install CLPM in binary form, download the appropriate file from https://files.clpm.dev/clpm/. The current release files for each platform are:

clpm-0.4.1-darwin-amd64.tar.gz
CLPM compiled for 64bit Mac OS.

clpm-0.4.1-darwin-arm64.tar.gz
CLPM compiled for 64bit Mac OS with M1 processors.

clpm-0.4.1-linux-amd64.tar.gz
CLPM compiled for 64bit Linux.

clpm-0.4.1-linux-arm64.tar.gz
CLPM compiled for 64bit Linux on ARM processors.

clpm-0.4.1-windows-amd64.msi
CLPM installer for 64 bit Windows.

clpm-0.4.1.tar.gz
Source code, including git submodules.

clpm-v0.4.1.DIGESTS
Text file containing the SHA512 sums for every previously mentioned file.

clpm-0.4.1.DIGESTS.asc
Same as clpm-v0.4.1.DIGESTS, but signed with GPG key 0x10327DE761AB977333B1AD7629932AC49F3044CE.

GNU/Linux and Darwin

After downloading the tarball and validating the SHA512 sum, unpack it and run sh install.sh. By default, it is installed to /usr/local/, but that prefix con be changed using the INSTALL_ROOT environment variable.

Windows

After downloading the installer and validating the SHA512 sum, simply run the installer and follow the directions.

Source Install

See install for more details on installing from source.

Quickstart

Now that you have CLPM installed, this section walks you through how to set it up, using the packages provided by the primary Quicklisp distribution.

First, configure CLPM to use the primary Quicklisp distribution as a source for packages. Create a file called ~/.config/clpm/sources.conf (or %LOCALAPPDATA%\config\clpm\sources.conf on Windows) with the following contents:

("quicklisp"
 :type :quicklisp
 :url "https://beta.quicklisp.org/dist/quicklisp.txt")

See sources for more details on configuring sources.

Then, configure ASDF to find the CLPM client. Assuming you haven't modified your ASDF source registry too much, place the output of the following command at ~/.config/common-lisp/source-registry.conf.d/20-clpm-client.conf. If you've modified your source registry a lot, you probably know what to do with the output.

clpm client source-registry.d

Next, configure your favorite Lisp to load the client by placing the output of the following command in your Lisp's init file (such as ~/.sbclrc).

clpm client rc

Now you can start your favorite Common Lisp implementation and enter into the default CLPM context.

(clpm-client:activate-context "default" :activate-asdf-integration t)

When using Quicklisp metadata directly, you need to sync in order to get all the metadata locally (other types of sources are able to lazily sync). This step will take a while the first time as it downloads and processes every version of the distribution.

(clpm-client:sync :sources "quicklisp")

Now you can try loading a system, such as alexandria:

(asdf:load-system :alexandria)

CLPM will see that alexandria is not present locally and ask you if you would like to install it automatically.

Project Goals

CLPM is far from the only project manager available for Common Lisp[1], but it makes very different assumptions and design choices than the other available solutions. In this section we describe our high level goals and how they affect our design decisions.

Use existing libraries where possible

If you look at Quicklisp, you'll quickly notice that the client does not rely on any third party code and all functionality, from an HTTP client to untaring, are implemented completely as part of the Quicklisp client project. The upsides of this strategy are that only required functionality is loaded into the development image, it prevents version conflicts between code the Quicklisp client depends on and the code you're developing depends on, and it helps ensure that Quicklisp works on a broad set of Common Lisp implementations. However, it has a major downside: the maintenance and development costs of the project manager are high, potentially making it difficult to implement new features and the project manager does not really drive the quality of code in the community at large higher.

The desire to use existing libraries drives the decision for the CLPM core and client to be separated. The client has no external dependencies outside of ASDF/UIOP, which allows it to share all the benefits of Quicklisp's model, and the core never needs to be loaded into a development image, so it can leverage any dependency that makes sense without impacting the development image in the slightest.

Support HTTPS

HTTPS is becoming more and more ubiquitous. Some websites (such as Github and Gitlab) are only served over HTTPS and some people insist on HTTPS everywhere possible. This trend is not going away, therefore CLPM should natively support HTTPS.

As LispWorks is the only Common Lisp implementation I am aware of that has native support for TLS, this means that CLPM has to use third party tools to achieve this support. This further drives the separation of the core and client, as CLPM can use foreign libraries to provide TLS support and this is not something that should be brought into a development image that does not otherwise need it.

Additionally, Quicklisp packages are served over HTTPS. While the Quicklisp client cannot take advantage of that (without https://github.com/snmsts/quicklisp-https), CLPM can, providing a little more of a guarantee that packages have not been tampered with.

Minimize footprint in development images and deployments

Many Common Lisp implementations allow you to deliver programs by dumping an image to file. For most programs generated this way there is no need to have a bundled project manager. Therefore, there should be an easy way to use a project manager without leaving traces if it in your deployed system.

To this end, CLPM can be used without ever loading the client (for example, clpm bundle exec configures ASDF entirely through environment variables) and if you do choose to use the client for better interactive development, the client is able to remove itself from the image when it is dumped.

Support CI/CD workflows

CI and CD are nice to have and nicer still when the jobs can run quickly and give fast feedback. To that end, CLPM is distributed in both binary and source form. Source for hackers or people who want to use a different feature set and binary for quick and easy installation in other cases.

Support installing multiple package versions

Ideally, updates to packages would never introduce regressions. However, we live in reality and this happens frequently (just look at the packages that get removed from Quicklisp in any given release for failing to build). Additionally, sometimes you just really need to use an old version of a dependency for some project while using the latest version of the same dependency in another project.

To this end, CLPM supports installing multiple versions of the same package simultaneously. This is additionally an enabling feature for managing project specific contexts as well as global contexts.

Support and encourage explicitly versioned systems

ASDF allows developers to provide version numbers for their systems and associate version requirements with dependencies. IMHO this is a criminally underutilized feature of ASDF and it should be required that any release of a package in any package index should declare a version number. However, it is a lot of work to convince others to believe the same way and even if a critical mass did buy in, things wouldn't change overnight.

Therefore, CLPM supports both the status quo (a Quicklisp package index versioned by date) as well as a work in progress package index (working name of CLPI). This new Common Lisp Project Index includes both extra metadata about projects (such as the upstream repo which can be used to check out development versions) as well as information on the ASDF system version numbers.

In-depth Documentation

For more documentation on CLPM, you may find the following files useful:

client
Summary of CLPM's client.

config
Summary of all of CLPM's configuration options.

sources
Summary of all supported software repositories.

bundle
Information on how to use CLPM to manage and repeatably install dependencies for a single project.

storage
Information on where CLPM writes data to your hard drive.

Footnotes

[1] See, for example: Quicklisp, Qlot, and Qi.