Building SBCL with ABCL

In revisiting our compiler for various improvements, it is helpful to have challenging compilations to perform so that we have some confidence we don't break things along the way. One of the more complicated tasks we've found is using ABCL as a build host to compile SBCL which has seemingly has lots of twisty little macros, all different. A while ago, maybe perhaps a year ago, we broke this behavior, closing off this avenue of testing.

But due to recent work, as of ABCL trunk r13538 we've restored this capability, once again making ABCL capable of bootstrapping an SBCL build. This may prove handy for individuals who wish to port SBCL to new platforms where there is a JVM available.

When we mentioned this on #lisp, we were asked how long it takes ABCL to build SBCL vs. SBCL building itself.

ABCL building SBCL

//build started:  Thu Aug 25 15:12:30 CEST 2011

//build finished: Thu Aug 25 16:49:30 CEST 2011

real	96m59.819s

user	37m35.702s

sys	32m44.641s

SBCL building SBCL

//build started:  Thu Aug 25 16:51:34 CEST 2011

//build finished: Thu Aug 25 17:07:47 CEST 2011

real	16m12.855s

user	4m40.804s

sys	3m28.145s

So, ABCL is roughly a factor of six slower building SBCL than SBCL itself.

Integrating ABCL with Ant

While working with ABCL embedded in Java applications, you typically have java and lisp source files. Whenever I'm in that situaition, I find that I want to be able to build all sources from the same build mechanism.

Last week, Alex Mizrahi and I started to work on reviving ABCL-web (a project to create Java Servlets backed by Common Lisp code running in ABCL). That's when I found myself in need of a solution to this problem again: the project would greatly benefit from having a way to build lisp files for inclusion in the resulting JAR file.

Looking around, I found there are three options to achieve this goal:

1. Create a custom Ant task
2. Implement a command line switch to create fasls from the ABCL command line
3. Use our JSR-223 support to write in-line Lisp in build files

While option (1) looked very attractive at first, it means one of two things: (1) adding an additional dependency for creation of our jar for distribution, or (2) creation of a separate jar file with only this task in it. In all cases, it adds a dependency to our release build. Because our current release build doesn't require Ant (you can build our release using the lisp-based build), I wasn't ready to accept this solution yet.

Option (2) looked very attractive, because it would not only integrate with Ant, but with build systems in general.  This option became less attractive when I found that the standard Ant project doesn't have any helpers for iterating sets. The thing that I had been thinking about was to implement the basics first, ie working the way gcc is used in most projects: 1 invocation per source file. So, in order to support the scheme that Ant does allow, the parameters ABCL would have to support became a lot more complex, needing to support file-sets to be compiled.

Then, I found the 'script' tag, which serves to integrate JSR-223 (Java scripting languages) into Ant build files. As we support JSR-223, I thought I'd give it a try! (So far I hadn't used it yet, I embed ABCL directly in my applications.) This solution -if it works out- is really great: no additional dependencies and no hacks [in our own build system, we feed the lisp that we want evaluated to ABCL through standard input.]

So, I gave it a try and within minutes, I was in business! I'm using a NetBeans generated project, so there's a "-post-compile" target which gets invoked after the "compile" target completes succesfully. 

<target name="-post-compile" description="lisp file compilation">
<script language="ABCL" classpath="lib/abcl.jar">
<![CDATA[
(let* ((the-project (jcall "getProject" self))
       (the-fileset (jcall "getReference" the-project "lisp-files"))
       (files-iterator
        (jcall (jmethod "org.apache.tools.ant.types.Path"
                        "iterator")
               the-fileset)))
   (loop while (jcall "hasNext" files-iterator)
        do (print (jcall "next" files-iterator))))
]]>
</script>
</target>

The snippet basically retrieves a path-set with the ID "lisp-files" defined elsewhere in the project file. Then, it continues to print all the (Java) objects it retrieves from the iterator. The next step is to replace the (print ...) form with a (compile-file ) form, retrieving the path string from the Path object and making a lisp pathname of it. Want to signal failed build? Just do what a real Task implementation would have done: throw a BuildException.

Concluding: while adding command line options may still be a desirable path to pursue, Ant integration is here for anybody embedding ABCL in a broader Java context.