Thursday, February 11, 2010

REQUIRE now works with ASDF systems

The ANSI *MODULES*/REQUIRE/PROVIDE interface is usually implemented in a fairly simple manner. When a piece of code wishes to depend on functionality of a given module, a statement of the form

(require :module)

first looks for whether the modules is currently in the list referenced by *MODULES*. If this isn't the case, a Lisp implementation simply looks in a few standard places for a file to load based on the STRING form of ":module", like "module.lisp", or possibly by a FASL. That file is then loaded, during which the form

(provide :module)

is evaluated which pushes the keyword form of its argument to the *MODULES* list.

This very simple dependency mechanism really doesn't provide a mechanism for searching file hierarchies, being more appropriate to manage dependencies in a large collection of files located in a single directory as is the case in the ABCL system directory. Presumably this simplicity and lack of extensibility led to the facility as being noted as deprecated in the HyperSpec. But it is part of the ANSI standard so until the New Implementation of Lisp is released, it forms the only mechanism that portable code can rely on.

In the contemporary Lisp era, ASDF forms the most common method to specify system dependencies. Since ANSI doesn't specify how REQUIRE searches for the dependencies, the clever folks behind SBCL decided to make REQUIRE invoke ASDF if the dependency cannot be resolved by the simple version of the mechanism.

Since implementation is the sincerest form of flattery, as of ABCL svn r12247 we have adopted this extension to REQUIRE. To use this mechanism, you first have to ensure that ASDF itself is loaded via

(require 'asdf)

Afterwards, a command like

(require :hunchentoot)

will search for an ASDF loadable system named "hunchentoot".

But how does ASDF find these systems? By successively searching the contents of ASDF:*CENTRAL-REGISTRY* which is a list of locations to search for files or symbolic links named "hunchentoot.asd". Since maintaining this list for a large number of ASDF definitions would be onerous, the usual practice is to have a single directory which contains symbolic links to the system definitions. It makes maintaining various versions a bit easier as one can simply manipulate the symbolic links.

I use the following piece of code in my "~/.abclrc" to collect all my ASDF pointers in the directory "~/.sbcl/systems/":

(require 'asdf)
(pushnew '(merge-pathnames ".sbcl/systems/" (user-homedir-pathname))

How does one install ASDF systems? One downloads the code and creates a link to its "*.asd" file in "~/.sbcl/systems/". But that can get onerous to do manually especially if one ASDF system depends on other systems. The ASDF-INSTALL package provides a semi-interactive automated mechanism to download, compile, and install ASDF definitions. For now, using SBCL to install ASDF systems that ABCL can share works pretty well, but we're currently working on incorporating a port of ASDF-INSTALL into the base ABCL system. Stay tuned!


  1. Excellent article! Should this be put into the wiki?

  2. Are you going to ship with the latest ASDF?

    In ASDF 1.607, *central-registry* is deprecated in favor of a new system that you can configure with files. See README.source-registry.

    Also, it would be nice if you could contribute ABCL support to the test suite, and any ABCL-specific fixes to ASDF itself or its tests.

  3. ABCL lacks the long form DEFINE-METHOD-COMBINATION needed for any more modern ASDF. Once we get that implemented, we intend to incorporate the latest version of ASDF. And when we join the modern community, we will contribute to the test suite.

  4. Do you have plans to add support for this any time soon?

    Otherwise, I could refactor ASDF to no longer require this DEFINE-METHOD-COMBINATION.

  5. I removed the DEFINE-METHOD-COMBINATION requirement from ASDF 1.636. Does it now work with ABCL?

  6. (Please follow up on the ASDF-devel mailing list.)

  7. We folloed up on the related mailing lists to the point that ABCL-0.19.1 works with ASDF2.