Pitfalls of shared structure, or: fixing XPATH compilation

In recent blog posts, Hans Hübner mentions how ABCL starts to be usable by the average Common Lisp programmer. Yet he also ran into the longer standing issue that the XPATH library doesn't want to be compiled by ABCL 1.1.1. (This is where his remark how the CXML-STP library doesn't work comes from.)

Last weekend I finally found out what the underlying problem was that XPATH triggered: shared structure in forms being compiled. Even though it's quite easy to generate code with shared structure, apparently few projects use it in a way which triggered ABCL into incorrect behaviour.

Case in point was a compiler macro expansion containing a literal list. After expansion, the literal became part of the code being compiled and the compiler modified the literal in place, resulting in problems on all subsequent expansions.

The cause of the issue dates all the way back to even before I started working on ABCL. Back then the compiler used to modify the CARs and CDRs of the forms it was compiling. Much of this behaviour was already replaced before last week as my intuition told me this is undesirable, with the replacement being nice side-effect-free functional code. Even though the reason for this behaviour isn't documented, I'll assume it was to reduce consing and thereby reduce pressure on the JVM's garbage collector. In today's world with new garbage collectors and much improved JIT compilers in the JVM, this issue isn't an issue anymore.

To cut a long story short: XPATH compilation fixed, compiler changed to functional style and libraries depending on XPATH (like CXML-STP) also fixed.

On to the next cl-test-grid failure...

Building SBCL with ABCL

In revisiting our compiler for various improvements, it is helpful to have challenging compilations to perform so that we have some confidence we don't break things along the way. One of the more complicated tasks we've found is using ABCL as a build host to compile SBCL which has seemingly has lots of twisty little macros, all different. A while ago, maybe perhaps a year ago, we broke this behavior, closing off this avenue of testing.

But due to recent work, as of ABCL trunk r13538 we've restored this capability, once again making ABCL capable of bootstrapping an SBCL build. This may prove handy for individuals who wish to port SBCL to new platforms where there is a JVM available.

When we mentioned this on #lisp, we were asked how long it takes ABCL to build SBCL vs. SBCL building itself.

ABCL building SBCL

//build started:  Thu Aug 25 15:12:30 CEST 2011

//build finished: Thu Aug 25 16:49:30 CEST 2011

real	96m59.819s

user	37m35.702s

sys	32m44.641s

SBCL building SBCL

//build started:  Thu Aug 25 16:51:34 CEST 2011

//build finished: Thu Aug 25 17:07:47 CEST 2011

real	16m12.855s

user	4m40.804s

sys	3m28.145s

So, ABCL is roughly a factor of six slower building SBCL than SBCL itself.

Fixed: cl-unicode compilation

cl-unicode contains a few files which contain literal values which - when serialized to a string - result in strings larger than 64kB. Until today, this was a problem for ABCL because the limit for a constant string in a java class file is 64kB and all serialized objects used to be stored that way.

Today, this issue was fixed by storing strings over 64kB as resource files in the JARs being generated. The result is that cl-unicode and the libraries it is a dependency for - such as cl-interpol - can now successfully be compiled using ABCL.

Integrating ABCL with Ant - the second step

While working to extend the ABCL integration into the ABCL-web Ant-based build - the project that I started in my previous post - I spent a lot of time trying to find a way to calculate the output paths, based on the names of the input paths.

Given the lisp way of 1-to-1 correspondence between a lisp file and a fasl, I expected this to be a trivial task to be done at the build system level. So, I kept searching for a solution in Ant (using standard components only). Surely I found the pathconvert and mapper building blocks, however, neither really seemed to work as the latter turned out not to be a generic mechanism but to be supported only by specific tasks (such as apply) and the former converts its output to a string instead of a set of paths.

Realizing that the issue of mapping source paths to output paths was left to the individual tasks, I turned to abcl to solve the problem as part of the compilation script. The final script between the CDATA tags as presented in the previous post looks like this:

(let* ((the-project (jcall "getProject" self))
       (src-iterator (jcall (jmethod "org.apache.tools.ant.types.Path" "iterator")
                            (jcall "getReference" the-project "lisp-files")))
       (src-wildcard (pathname (format nil "~A**/*.lisp" (jcall "getProperty" the-project "build.src.dir"))))
       (dst-wildcard (pathname (format nil "~A**/*.abcl" (jcall "getProperty" the-project "build.dst.dir")))))
   (when (and (jcall "hasNext" src-iterator))
     (loop
        for src-path = (pathname (jcall (jmethod "java.lang.Object" "toString")
                                 (jcall "next" src-iterator)))
        for dst-path = (translate-pathname src-path src-wildcard dst-wildcard)
        do (compile-file src-path :output-file dst-path)
        while (and (jcall "hasNext" src-iterator)))))

It's based on 2 properties having been defined: "build.src.dir" and "build.dst.dir". In addition, it wants a set of paths (fileset or path set) to be defined called "lisp-files". Iterating over the set of source files, it calculates the output file by replacing a path prefix and the file extension.

In my NetBeans project settings, I had to exclude "**/*.lisp" from packaging in the destination JAR file.

Now the JAR contains the fasl instead of the source file!

Do you have useful or resourceful ABCL based scripts plugged into Ant? Let me know and post a reaction.