Even better handling of low memory conditions

In previous research, ABCL came out nicely when tested for handling of Stack Overflow (SO) and Out Of Memory (OOM) conditions. See the blog item "out-of-memory: a sad case"; ABCL was classified as one of the few lisps that handled OOM conditions at all, by ending in the Lisp debugger even.

By looking more closely - which we did before the release of 0.18 - it turned out that handling of these conditions was only part of forms executed from REPL. If the same conditions occurred during program execution, the program was simply terminated, just like the other Lisps in the survey.

Of course, when you have a program running into low memory conditions regularly, it makes sense to make more memory available to it at startup. There are a few startup switches for java which allow to set the initial memory amount, most notably:

• -Xmx: which sets the maximum amount of heap available
  examples: 512m, 1024k, 13m
• -Xms: like -Xmx but specifies the initial amount
• -Xss: specifies the thread stack size

These options are only available at startup though. These options can be added to ABCL's startup wrapper script, either after building, or by letting the build add them automatically. If you opt for the latter, you need to add a line like the following to your abcl.properties file (before abcl-0.19.1 this was named build.properties):

java.options=-Xmx512m -Xss6m

If your program keeps large caches, it may be important to be able to detect OOM conditions, clear some or all of the caches and retry the operation. Other situations where it's good to be able to handle OOM conditions can be thought of. That's why it's important to have your programming environment help you detect these situations.

One of the advantages of running in the JVM is that ABCL itself doesn't doesn't need heuristics to detect these situations: the JVM throws a StackOverflowError or an OutOfMemoryError as appropiate. The only thing ABCL needs to do is handle these errors in appropriate situations.

Starting with 0.18, ABCL has well defined points where it traps the SO and OOM conditions in any Lisp program: HANDLER-BIND (and as a consequence HANDLER-CASE) establishes a try/catch block around the form. This try/catch block makes sure OOM and SO get converted to Lisp errors, for HANDLER-BIND to handle.

The situation isn't perfect: wouldn't it be nice to have a 'low memory' signal which provided an environment with enough memory to allow user-lisp code to run, for example to free up memory held by caches. Even though it's not perfect, ABCL now provides the functionalities which the "sad case" article would lead you to believe it already did.

An option to pursue low memory condition notifications was pointed out by Douglas Miles, one of the people frequenting the #abcl irc channel on freenode.net, would be to install handlers for the JMX (java management extensions) low memory notifications.

ABCL 0.19.1 released

On behalf of the developers of ABCL (Armed Bear Common Lisp) I'm glad to be able to announce the 0.19.1 release. Version 0.19.0 was never released.

ABCL is a Common Lisp implementation implemented in Java and running on the JVM, featuring both an interpreter and a compiler. The compiler targets the JVM directly meaning that its output is runnable JVM bytecode. The fact that ABCL is written in Java allows for relatively easy embedding in larger applications. For integration with existing applications ABCL implements Java Specification Request (JSR) 223: Java scripting API.

This release features - among lots of other things - a fix for unbinding PROGV bound variables upon exiting the PROGV scope and a much improved integration of access to filenames specified by URLs. You can find the full release notes at:

http://common-lisp.net/project/armedbear/release-notes-0.19.shtml

and the list of changes at:

http://trac.common-lisp.net/armedbear/browser/trunk/abcl/CHANGES

If you have questions regarding use or licensing, or you find issues, please report back to the development list:

armedbear-devel at common-lisp dot net

Source distribution archives can be downloaded in ZIP or gzipped tar form:

http://common-lisp.net/project/armedbear/releases/abcl-src-0.19.1.tar.gz

http://common-lisp.net/project/armedbear/releases/abcl-src-0.19.1.zip

Signatures are available under:

http://common-lisp.net/project/armedbear/releases/abcl-src-0.19.1.tar.gz.asc

http://common-lisp.net/project/armedbear/releases/abcl-src-0.19.1.zip.asc

In addition, binaries are also available:

http://common-lisp.net/project/armedbear/releases/abcl-bin-0.19.1.tar.gz

http://common-lisp.net/project/armedbear/releases/abcl-bin-0.19.1.zip

With associated signatures:

http://common-lisp.net/project/armedbear/releases/abcl-bin-0.19.1.tar.gz.asc

http://common-lisp.net/project/armedbear/releases/abcl-bin-0.19.1.zip.asc

REQUIRE now works with ASDF systems

The ANSI *MODULES*/REQUIRE/PROVIDE interface is usually implemented in a fairly simple manner. When a piece of code wishes to depend on functionality of a given module, a statement of the form

(require :module)

first looks for whether the modules is currently in the list referenced by *MODULES*. If this isn't the case, a Lisp implementation simply looks in a few standard places for a file to load based on the STRING form of ":module", like "module.lisp", or possibly by a FASL. That file is then loaded, during which the form

(provide :module)

is evaluated which pushes the keyword form of its argument to the *MODULES* list.

This very simple dependency mechanism really doesn't provide a mechanism for searching file hierarchies, being more appropriate to manage dependencies in a large collection of files located in a single directory as is the case in the ABCL system directory. Presumably this simplicity and lack of extensibility led to the facility as being noted as deprecated in the HyperSpec. But it is part of the ANSI standard so until the New Implementation of Lisp is released, it forms the only mechanism that portable code can rely on.

In the contemporary Lisp era, ASDF forms the most common method to specify system dependencies. Since ANSI doesn't specify how REQUIRE searches for the dependencies, the clever folks behind SBCL decided to make REQUIRE invoke ASDF if the dependency cannot be resolved by the simple version of the mechanism.

Since implementation is the sincerest form of flattery, as of ABCL svn r12247 we have adopted this extension to REQUIRE. To use this mechanism, you first have to ensure that ASDF itself is loaded via

(require 'asdf)

Afterwards, a command like

(require :hunchentoot)

will search for an ASDF loadable system named "hunchentoot".

But how does ASDF find these systems? By successively searching the contents of ASDF:*CENTRAL-REGISTRY* which is a list of locations to search for files or symbolic links named "hunchentoot.asd". Since maintaining this list for a large number of ASDF definitions would be onerous, the usual practice is to have a single directory which contains symbolic links to the system definitions. It makes maintaining various versions a bit easier as one can simply manipulate the symbolic links.

I use the following piece of code in my "~/.abclrc" to collect all my ASDF pointers in the directory "~/.sbcl/systems/":

(require 'asdf)

(pushnew '(merge-pathnames ".sbcl/systems/" (user-homedir-pathname))

asdf:*central-registry*)

How does one install ASDF systems? One downloads the code and creates a link to its "*.asd" file in "~/.sbcl/systems/". But that can get onerous to do manually especially if one ASDF system depends on other systems. The ASDF-INSTALL package provides a semi-interactive automated mechanism to download, compile, and install ASDF definitions. For now, using SBCL to install ASDF systems that ABCL can share works pretty well, but we're currently working on incorporating a port of ASDF-INSTALL into the base ABCL system. Stay tuned!

ABCL 0.18.1 released

Due to some unfortunate regressions in 0.18.0, the ABCL project had to release an 0.18.1 release. It's readily available for download and is strongly advised for everybody who's now running 0.18.0: the .0 release can behave seemingly randomly after loading compiled code.

ABCL 0.18.0 released

On behalf of the developers of ABCL (Armed Bear Common Lisp) I'm glad to be able to announce the 0.18.0 release.

This release features - among lots of other things - faster initial startup, faster special variable lookup and portable fasl files. Please refer to the release notes for the full list.

If you have questions regarding use or licensing, or you find issues, please report back to the development list: armedbear-devel at common-lisp dot net

Source distribution archives can be downloaded in ZIP (zip signature file) or gzipped tar (tar signature file) format.

In addition, a ZIP binary (bin-zip sig file) and gzipped tar binary (bin-tar sig file) are available.

ABCL on Google App Engine (2)

Over the past month further work was done to improve ABCL startup times. This effort is especially directed at supporting ABCL on Google App Engine (GAE).

The startup time of the trivial GAE example application in ABCL's source tree takes 19 seconds to startup, as mentioned in an earlier blog item. Although this is only a "Google theoretical time", because the page is served in only 12 seconds, this is clearly a lot. I heard many GAE applications have startup times between 5 and 10 seconds. It surely would be nice if our trivial application could get closer to that.

A number of different solutions have been evaluated:

1. Reduction of the number of classes loaded at startup; making better use of ABCL's auto-loader facility
2. Supporting binary fasls
3. Create a system for finer-grained auto-loading support
   

The reason exactly these scenarios were evaluated is the fact that I looked for help regarding this performance issue on irc (irc://irc.freenode.net/#...); the answer was "you're doing too much during setup or at the first request". My first reaction was that we didn't have any options: that's how ABCL is designed. After some discussion these scenarios came up though.

The first and third scenarios are the result from many profiling sessions of "ABCL startup" time. The conclusion was that 35 to 45% of ABCL startup time is spent in Java reflection: when loading function classes ABCL needs to look up the class constructors to instantiate an object of the given class.

Scenario 1 is about delaying loading of FASLs until a function in them is required. Scenario 3 goes into more detail about the use of a function: even when a FASL is loaded, not all functions in it will be used (immediately or ever). The idea behind scenario 3 is to delay reflection API access until a function is actually used.

Using scenario (1) startup times could be reduced somewhat, especially in the case of our minimal servlet application: it uses relatively few Lisp functions and the ones it does use are related to printing and streams. Those are concentrated in a limited number of fasls.

In order to implement scenario (3), a quite bit more effort was required. The basic idea - as explained above - is that many functions in a FASL won't be used until a later stage in the application. In order to be able to delay resolution of the bytecode of the function, we introduced an object which - like the auto-loader - acts as a proxy for the unresolved function. This proxy class doesn't exhibit the same overhead, because it is resolved only once.

Upon the first call to the function, its bytecode gets resolved and the proxy in the function slot gets replaced with the actual function. After that, the first call is forwarded to the real function, as if it had been called directly. Although the actual implementation is a bit more complex to account for the loading of nested functions, that's basically it.

With scenario (3) applied to function definitions only, we were able to reduce startup time of the first request on GAE from 19 seconds to 11 seconds (roughly 40%). Today, we started to apply the same strategy to macro functions too. The result is - measured on my local PC, not GAE - a savings of another roughly 13%. Assuming that the same applies to GAE (as it did with the other 40%), we've realized a saving of 50% startup time!

Binary fasls - scenario 2 - were an attempt to reduce the amount of work that needed to be done at startup: because the normal fasl loading process is driven by a text file containing Lisp code, that could have been one of the causes. We didn't remove support for them, but they didn't turn out to be a big saver; that can be explained because the binary fasls are just another ABCL function object which needs to be loaded using reflection.

All in all did we save 50% start up time. Let this be an invitation to start experimenting with ABCL on GAE.

Maxima(l) performance

Last spring, the Maxima challenge for ABCL was to get it to complete its test suite. We've mastered that bit of Maxima for some months now. However, as turned out soon after that, Maxima runs rather slow on ABCL. To some extent - being limited by the JVM - that's to be expected. The performance observed was way off base though: much too slow.

Through analysis, the cause was established to be the fact that Maxima declares lots of symbols to be special [and that ABCL doesn't offer a way to remove that specialness].

Last summer we found that allowing Maxima to undeclare specials increases ABCL performance immensely (roughly 35%). However, the final goal for Maxima is to make more sparingly use of specials and declaring unspecial or undeclaring special isn't defined in the spec. Because of the two reasons it felt not right to implement the solution at the time: it would have been a very Maxima specific one to a problem Maxima intends to fix in the long run.

Peter Graves noted that he converted the special bindings storage in XCL to use the same scheme as in SBCL/CCL, using an array with active bindings instead of a linked list of bindings. He observed a performance gain of 10% in his tests.

Last weekend, I implemented the same scheme in ABCL and although the general speed up doesn't show 10% in our tests [which may very well differ from Peter's], we observed roughly 40% performance gain on Maxima's test suite!