ABCL

Saturday, January 22, 2011

ABCL 0.24.0 released

Saturday, December 18, 2010

Integrating ABCL with Ant - the second step

While working to extend the ABCL integration into the ABCL-web Ant-based build - the project that I started in my previous post - I spent a lot of time trying to find a way to calculate the output paths, based on the names of the input paths.

Given the lisp way of 1-to-1 correspondence between a lisp file and a fasl, I expected this to be a trivial task to be done at the build system level. So, I kept searching for a solution in Ant (using standard components only). Surely I found the pathconvert and mapper building blocks, however, neither really seemed to work as the latter turned out not to be a generic mechanism but to be supported only by specific tasks (such as apply) and the former converts its output to a string instead of a set of paths.

Realizing that the issue of mapping source paths to output paths was left to the individual tasks, I turned to abcl to solve the problem as part of the compilation script. The final script between the CDATA tags as presented in the previous post looks like this:

(let* ((the-project (jcall "getProject" self))
       (src-iterator (jcall (jmethod "org.apache.tools.ant.types.Path" "iterator")
                            (jcall "getReference" the-project "lisp-files")))
       (src-wildcard (pathname (format nil "~A**/*.lisp" (jcall "getProperty" the-project "build.src.dir"))))
       (dst-wildcard (pathname (format nil "~A**/*.abcl" (jcall "getProperty" the-project "build.dst.dir")))))
   (when (and (jcall "hasNext" src-iterator))
     (loop
        for src-path = (pathname (jcall (jmethod "java.lang.Object" "toString")
                                 (jcall "next" src-iterator)))
        for dst-path = (translate-pathname src-path src-wildcard dst-wildcard)
        do (compile-file src-path :output-file dst-path)
        while (and (jcall "hasNext" src-iterator)))))

It's based on 2 properties having been defined: "build.src.dir" and "build.dst.dir". In addition, it wants a set of paths (fileset or path set) to be defined called "lisp-files". Iterating over the set of source files, it calculates the output file by replacing a path prefix and the file extension.

In my NetBeans project settings, I had to exclude "**/*.lisp" from packaging in the destination JAR file.

Now the JAR contains the fasl instead of the source file!

Do you have useful or resourceful ABCL based scripts plugged into Ant? Let me know and post a reaction.

Sunday, December 12, 2010

Integrating ABCL with Ant

While working with ABCL embedded in Java applications, you typically have java and lisp source files. Whenever I'm in that situaition, I find that I want to be able to build all sources from the same build mechanism.

Last week, Alex Mizrahi and I started to work on reviving ABCL-web (a project to create Java Servlets backed by Common Lisp code running in ABCL). That's when I found myself in need of a solution to this problem again: the project would greatly benefit from having a way to build lisp files for inclusion in the resulting JAR file.

Looking around, I found there are three options to achieve this goal:

Create a custom Ant task
Implement a command line switch to create fasls from the ABCL command line
Use our JSR-223 support to write in-line Lisp in build files

While option (1) looked very attractive at first, it means one of two things: (1) adding an additional dependency for creation of our jar for distribution, or (2) creation of a separate jar file with only this task in it. In all cases, it adds a dependency to our release build. Because our current release build doesn't require Ant (you can build our release using the lisp-based build), I wasn't ready to accept this solution yet.

Option (2) looked very attractive, because it would not only integrate with Ant, but with build systems in general. This option became less attractive when I found that the standard Ant project doesn't have any helpers for iterating sets. The thing that I had been thinking about was to implement the basics first, ie working the way gcc is used in most projects: 1 invocation per source file. So, in order to support the scheme that Ant does allow, the parameters ABCL would have to support became a lot more complex, needing to support file-sets to be compiled.

Then, I found the 'script' tag, which serves to integrate JSR-223 (Java scripting languages) into Ant build files. As we support JSR-223, I thought I'd give it a try! (So far I hadn't used it yet, I embed ABCL directly in my applications.) This solution -if it works out- is really great: no additional dependencies and no hacks [in our own build system, we feed the lisp that we want evaluated to ABCL through standard input.]

So, I gave it a try and within minutes, I was in business! I'm using a NetBeans generated project, so there's a "-post-compile" target which gets invoked after the "compile" target completes succesfully.

<target name="-post-compile" description="lisp file compilation">
<script language="ABCL" classpath="lib/abcl.jar">
<![CDATA[
(let* ((the-project (jcall "getProject" self))
       (the-fileset (jcall "getReference" the-project "lisp-files"))
       (files-iterator
        (jcall (jmethod "org.apache.tools.ant.types.Path"
                        "iterator")
               the-fileset)))
   (loop while (jcall "hasNext" files-iterator)
        do (print (jcall "next" files-iterator))))
]]>
</script>
</target>

The snippet basically retrieves a path-set with the ID "lisp-files" defined elsewhere in the project file. Then, it continues to print all the (Java) objects it retrieves from the iterator. The next step is to replace the (print ...) form with a (compile-file ) form, retrieving the path string from the Path object and making a lisp pathname of it. Want to signal failed build? Just do what a real Task implementation would have done: throw a BuildException.

Concluding: while adding command line options may still be a desirable path to pursue, Ant integration is here for anybody embedding ABCL in a broader Java context.

Thursday, December 2, 2010

ABCL 0.23.1 released: fix for loading from path with spaces

Thursday, November 25, 2010

ABCL 0.23.0 released

On behalf of the developers of ABCL (Armed Bear Common Lisp) I'm glad to be able to announce the 0.23.0 release.

ABCL is a Common Lisp implementation implemented in Java and running on the JVM, featuring both an interpreter and a compiler. The compiler targets the JVM directly meaning that its output is runnable JVM bytecode. The fact that ABCL is written in Java allows for relatively easy embedding in larger applications. For integration with existing applications ABCL implements Java Specification Request (JSR) 223: Java scripting API.

This release features - among lots of other things - CLOS thread safety fixes and concurrency improvements, support for Oracle's JRockit JVM, and a new (generic) class file writer. In addition, this is the first release to complete the Maxima test suite without failures, thanks to close cooperation between our projects. You can find the full release notes at:

http://common-lisp.net/project/armedbear/release-notes-0.23.shtml

and the list of changes at:

http://trac.common-lisp.net/armedbear/browser/trunk/abcl/CHANGES

If you have questions regarding use or licensing, or you find issues, please report back to the development list:

armedbear-devel at common-lisp dot net

Source distribution archives can be downloaded in ZIP or gzipped tar form:

http://common-lisp.net/project/armedbear/releases/0.23.0/abcl-src-0.23.0.tar.gz
http://common-lisp.net/project/armedbear/releases/0.23.0/abcl-src-0.23.0.zip

Signatures are available under:
http://common-lisp.net/project/armedbear/releases/0.23.0/abcl-src-0.23.0.tar.gz.asc
http://common-lisp.net/project/armedbear/releases/0.23.0/abcl-src-0.23.0.zip.asc

In addition, binaries are also available:

http://common-lisp.net/project/armedbear/releases/0.23.0/abcl-bin-0.23.0.tar.gz
http://common-lisp.net/project/armedbear/releases/0.23.0/abcl-bin-0.23.0.zip

With associated signatures:
http://common-lisp.net/project/armedbear/releases/0.23.0/abcl-bin-0.23.0.tar.gz.asc
http://common-lisp.net/project/armedbear/releases/0.23.0/abcl-bin-0.23.0.zip.asc

Sunday, October 10, 2010

ABCL hash tables: threading and CLOS

More people are starting to use ABCL in threaded environments - I have been doing so myself since spring 2008. My own use revealed some threading issues in the compiler which have been long solved now.

Last April, David Kirkman mailed the list with some threading problems in our CLOS implementation. Because of other work - including implementation of METACLASS support - there wasn't much time to do much about the issue.

This week, he mailed having found the problem - accompanied with a patch with a solution. Because of our need to support the four Common Lisp equality operators, ABCL implements its own hash tables -four of them - with a shared ancestor class which requires implementation of a few primitive operations. Hash access from the Lisp world was being synchronized by the common ancestor. However, in some locations on the Java side the - unsynchronized - primitives were being called directly.

The solution was of course to add synchronization to the primitives as well. Evaluating the result, the new situation was rather unsatisfying: only a single thread could be reading or writing at the same time, meaning only a single CLOS effective method dispatch could be happening at the same time. You probably understand my reluctance to accept the status quo.

The quick solution to the CLOS dispatch problem was to use the java.util.concurrent.ConcurrentHashMap type: we were keying on symbols, which have EQ equility in the Java world (ie in terms of Object.equals()).

That still left the lisp world with rather unsatisfactory threading performance of our own hash tables and since those were repeating large chunks of code in their specialized primitives, I decided to refactor them completely.

The result is - admittedly by looking very closely at ConcurrentHashTable - a single hash table implementation with 4 Comparator classes, which have nearly-lockless reads from all threads. Inserts into the hash table are still synchronized from all threads, but that's expected to be a lesser problem: you'd primarily expect many reads per write in a hash table.

Concluding, the only (known) CLOS threading issue has been fixed - there are presumably many more, please report if you find them - and as a bonus we have a much more efficient hash table implementation.

Sunday, October 3, 2010

Maxima on ABCL: full pass on the test suite

Over the past two weeks Raymond Toy helped out to fix the last 9 failures in the Maxima test suite running: ABCL now passes the full 8.666 Maxima tests! We achieved a goal set for ABCL more than two years ago!

Thinking back to the days when Peter Graves had just handed over the project shows just how far we have come: back in September 2008 lots of tests (423, according to Hunter Monroe) would plainly terminate the test suite. That number had come down from over 1000 tests crashing the test suite in January of that year.

After manually disabling the crashing tests, Hunter Monroe writes on September 27 (2008) that he's been able to complete the tests with a total number of failures of 612 (out of 4454 tests) .

Many of the problems were caused by incorrect handling of special variables which was one of the first things to be addressed in ABCL - stabilizing over the first half of 2009. Other improvements which contributed to the stability of Maxima were fixes to the code generating non-local returns (THROW, GOTO stepping out of a closure, RETURN-FROM stepping out of a closure). Next to its reduction in specials, the Maxima team had to change lots of number comparisons from EQ to EQL, because in ABCL integers aren't (guaranteed) to be EQ - which is allowed by the spec, but not very customary in CL implementations.

At the same time evaluation time of the test suite has gone down considerably too - not due to increased processing power, but due to improvements on both sides. One example of an improvement which has greatly benefitted ABCL's performance is the gradual elimination of specials is Maxima - an on-going effort in their team. On the other side, binding and unwinding specials has become much faster in ABCL too, reducing the impact of remaining excessive specials.

The last few tests to be fixed required adjustments to the Maxima test suite as well as fixes to ABCL. Thanks to everybody who helped achieve this result, most notably Robert Dodier, Hunter Monroe and Raymond Toy from the Maxima team as well as Ville Voutilainen and Mark Evenson from the ABCL team.