ABCL

Closing in on CLOSER-MOP in abcl-1.1.0-dev

2012-01-24T14:32:00.000-08:00

In response to inquiries recently on #abcl (from @loke among others), we'd like to point out that recent work on the Armed Bear trunk--what will be abcl-1.1--by Rudi has started to converge on a plausible AMOP strategy. Stay tuned.

ABCL 1.0.1 Released

2012-01-10T15:29:00.001-08:00

All users of abcl-1.0.0 are recommended to migrate to abcl-1.0.1 as it fixes the abcl-contrib mechanism, an unfortunate oversight in our giddiness at being able to release at ECLM 2011.

Now the following incantation

(require 'abcl-contrib)

(require 'jss)

should result in a working JSS.

In addition to restoring the loading of ASDF definitions from jar archives, we've randomized some of our String hashes to prevent a purported JVM exploit spotted recently and shipped with ASDF-2.019.

You can find more in the release notes at:

http://common-lisp.net/project/armedbear/release-notes-1.0.1.shtml

and the list of changes at:

http://trac.common-lisp.net/armedbear/browser/tags/1.0.1/abcl/CHANGES

Source and binary distribution archives with associated cryptographic
signatures can be downloaded in ZIP or gzipped tar from:

http://common-lisp.net/project/armedbear/releases/1.0.1

If you have questions regarding use or licensing, or you find issues, please
report back to the development list:

armedbear-devel at common-lisp dot net

abcl-servlet unleashed to unsuspecting public...

2011-11-26T06:17:00.000-08:00

In the scratching the itch department, one of the current problems I face is debugging vast quantities of code in various Java application servers to which I cannot easily attach a debugger. Surprising, I found that I could attach SLIME to a resident ABCL process in such application servers. Currently, I am in the process of extending that slim SLIME wedge to be able to use its interactive Lisp REPL as a poor man's debugger by dynamically specifying aspects of the program execution which I can use to access trace information.

The DSL I am working towards would allow the following statement, executed interactively in a running JVM to record the arguments and return value of the normalizeURI() function:


(defmethod :around org.example.normalizeURI (url url)
  (push args)
  (call-next-method)
  (push result)
  (return result))

A very early draft of code which runs against abcl-1.1.0-dev has been released for forks and comments which only gets as far as establishing the SLIME REPL inside a Tomcat container can be found at http://bitbucket.org/easye/abcl-servlet. At its base, this code is an incorporation of the code from abcl-web pioneered by Alex Mizrahi and then restored by Erik Hülsman.

ABCL 1.0.0 released

2011-10-23T08:59:00.000-07:00

On behalf of the developers of ABCL (Armed Bear Common Lisp) we are happy to be able to announce the 1.0.0 release on the premises of the European Common Lisp Meeting 2011.

ABCL is a Common Lisp implementation implemented in Java and running on the Java Virtual Machine (JVM) featuring both an interpreter and a compiler. The compiler targets the JVM directly meaning that its output is runnable JVM bytecode. The fact that ABCL is implemented in Java allows for relatively easy embedding in larger applications. For integration with existing applications ABCL completely implements the "JSR 223: Java scripting API" specification.

This release delivers the last requirement for ABCL's 1.0 release milestone, namely a conforming implementation of the long form of DEFINE-METHOD-COMBINATION. Additionally, the abcl-contrib mechanism now supports a robust collection of contributions to ABCL beyond ANSI concerned with dynamically linking JVM artifacts at runtime, packaging ASDF system construction within jar archives, and an optional higher-level domain specific syntax called JSS (Java Syntax Sucks) for manipulating JVM libraries. Extensive testing and debugging the implementation with popular Quicklisp systems has resulted in a much more rugged and battle-tested Lisp. A draft version of the user manual is now available.

The list of changes can be viewed at

http://trac.common-lisp.net/armedbear/browser/trunk/abcl/CHANGES

If you have questions regarding use or licensing, or you find issues, please use the development email list for questions:

armedbear-devel at common-lisp dot net

Source distribution archives can be downloaded in ZIP or gzipped tar form:

http://common-lisp.net/project/armedbear/releases/1.0.0/abcl-src-1.0.0.tar.gz
http://common-lisp.net/project/armedbear/releases/1.0.0/abcl-src-1.0.0.zip

Cryptographic signatures are available under:

http://common-lisp.net/project/armedbear/1.0.0/releases/abcl-src-1.0.0.tar.gz.asc
http://common-lisp.net/project/armedbear/1.0.0/releases/abcl-src-1.0.0.zip.asc

In addition, binaries are also available:

http://common-lisp.net/project/armedbear/releases/1.0.0/abcl-bin-1.0.0.tar.gz
http://common-lisp.net/project/armedbear/releases/1.0.0/abcl-bin-1.0.0.zip

With the associated cryptographic signatures:

http://common-lisp.net/project/armedbear/releases/abcl-bin-1.0.0.tar.gz.asc
http://common-lisp.net/project/armedbear/releases/abcl-bin-1.0.0.zip.asc

ABCL User Manual

2011-10-20T01:35:00.000-07:00

In preparation for the upcoming ABCL 1.0.1 maintenance release, we are releasing drafts of the ABCL User Manual for feedback.

** Drafts

Rudi patches the documentation process from the version released withabcl-1.0.1 fresh from the sources via 'pdflatex' on January 28, 2012 abcl-20120124a.pdf.

Compilation fixes for CLOSURE-HTML landed on trunk

2011-09-20T13:43:00.000-07:00

As part of the continued effort to increase ABCL's practical usability (as opposed to its theoretical compliance to the CLHS), we've fixed compilation problems where ABCL would crash with a stack overflow when compiling CLOSURE-HTML.

The problem turned out to be that ABCL's fasl format didn't support serialization of circular structures. Apparently, very few libraries actually use the ability to serialize circular literals, but CLOSURE-HTML does.

The fix required changing the FASL version number. If you're working off trunk, you'll be required to do a full recompile of your work.

DEFINE-METHOD-COMBINATION (long form) no longer experimental

2011-09-11T12:50:00.000-07:00

Last year around the same time ABCL 0.23 was released. This release featured experimental support for the long form of DEFINE-METHOD-COMBINATION. Last February I started testing our implementation using tests that I found in SBCL's test suite. Unfortunately, I quickly found the implementation wasn't labeled experimental without reason and started fixing the tests one by one.

After other priorities interrupted my work in March, I picked up the remaining work two weeks ago. Today, I committed the last test cases ported from SBCL's test suite. They all work. Particularly hard to get right were the ones involving the (:ARGUMENTS . arguments-list) cases.

With the status that we have now DEFINE-METHOD-COMBINATION (long form) is no longer considered experimental! We're another step closer to full CLHS compliance!

ABCL 0.27.0 released

2011-09-03T12:11:00.000-07:00

On behalf of the developers of ABCL (Armed Bear Common Lisp) I'm glad to
be able to announce the 0.27.0 release.

This release features - among other things - wider compatibility for Quicklisp encapsulated systems basedon reports for problems with ABCL: fixes for PARENSCRIPT, NAMED-READTABLES and CL-INTERPOL (CL-UNICODE fix).
Additionally, ABCL has regained the ability to build SBCL. The release notes can be found at:

http://common-lisp.net/project/armedbear/release-notes-0.27.shtml

and the list of changes at:

http://trac.common-lisp.net/armedbear/browser/trunk/abcl/CHANGES

If you have questions regarding use or licensing, or you find issues, please
report back to the development list:

armedbear-devel at common-lisp dot net

Source distribution archives can be downloaded in ZIP or gzipped tar form:

abcl-src-0.27.0.tar.gz
abcl-src-0.27.0.zip

Signatures are available under:
abcl-src-0.27.0.tar.gz.asc
abcl-src-0.27.0.zip.asc

In addition, binaries are also available:

abcl-bin-0.27.0.tar.gz
abcl-bin-0.27.0.zip

With associated signatures:
abcl-bin-0.27.0.tar.gz.asc
abcl-bin-0.27.0.zip.asc

JVM classpath integration (Maven et. al.)

2011-09-03T00:46:00.000-07:00

Disclaimer: I loathe Maven. But I love users, so by unpopular (to me) demand as part of the upcoming abcl-0.27.0 release we've bowed to the inevitable by adding integration for Armedb Bear Common Lisp with Maven in svn r13550.

We've added the ASDF:MVN subclass ASDF:COMPONENT (by way of of ASDF:IRI) that refers to a purely virtual artifact, namely a given jar specified as a mvn dependency whose classes will be loaded in the hosting JVM (i.e. added to the running *classpath*).

The following 'log4j.asd' placed in a file on your local filesystem and then added the the ASDF:*CENTRAL-REGISTRY*:

     ;;;; -*- Mode: LISP -*-

   (in-package :asdf)

   (defsystem :log4j

     :components ((:mvn "log4j/log4j"

                   :version "1.4.9")))

then when the abcl-contrib.jar is loaded via

CL-USER> (require 'abcl-asdf)

an invocation of

CL-USER> (asdf:load-system 'log4j)

with ensure that log4j-1.4.9.jar and all its transitive dependencies as defined by the distributed Maven POM descriptions will be downloaded to the local filesystem if not already present. It will then dynamically add all of these Java classes to the currently hosting JVM.

Further details about additional ASDF extensions for both individual jar files and contents of directories are documented within the contrib/abcl-asdf/README.markdown file from SVN trunk.

Thanks to Alan Ruttenburg for great inspiration (and code) from invoke.lisp which was the first extension to ASDF that could actuall dynamically add jar files to the hosting JVM process.

Building SBCL with ABCL

2011-08-25T07:43:00.001-07:00

In revisiting our compiler for various improvements, it is helpful to have challenging compilations to perform so that we have some confidence we don't break things along the way. One of the more complicated tasks we've found is using ABCL as a build host to compile SBCL which has seemingly has lots of twisty little macros, all different. A while ago, maybe perhaps a year ago, we broke this behavior, closing off this avenue of testing.

But due to recent work, as of ABCL trunk r13538 we've restored this capability, once again making ABCL capable of bootstrapping an SBCL build. This may prove handy for individuals who wish to port SBCL to new platforms where there is a JVM available.

When we mentioned this on #lisp, we were asked how long it takes ABCL to build SBCL vs. SBCL building itself.

ABCL building SBCL

//build started:  Thu Aug 25 15:12:30 CEST 2011

//build finished: Thu Aug 25 16:49:30 CEST 2011

real	96m59.819s

user	37m35.702s

sys	32m44.641s

SBCL building SBCL

//build started:  Thu Aug 25 16:51:34 CEST 2011

//build finished: Thu Aug 25 17:07:47 CEST 2011

real	16m12.855s

user	4m40.804s

sys	3m28.145s

So, ABCL is roughly a factor of six slower building SBCL than SBCL itself.

Fixed: cl-unicode compilation

2011-08-19T14:53:00.000-07:00

cl-unicode contains a few files which contain literal values which - when serialized to a string - result in strings larger than 64kB. Until today, this was a problem for ABCL because the limit for a constant string in a java class file is 64kB and all serialized objects used to be stored that way.

Today, this issue was fixed by storing strings over 64kB as resource files in the JARs being generated. The result is that cl-unicode and the libraries it is a dependency for - such as cl-interpol - can now successfully be compiled using ABCL.

Log tickets and edit the wiki using your Gmail account

2011-08-15T14:18:00.000-07:00

Today, I installed the module AuthOpenID for Trac on ABCL's hosting environment. You can now log in and contribute to the wiki or log your tickets without owning a common-lisp.net login!

Happy contributing!

ABCL 0.26.2 released

2011-08-15T01:06:00.000-07:00

A new patch release is available for ABCL: 0.26.2. This release fixes an important regression on windows where ABCL couldn't load fasls with spaces in their filenames.

Additionally, the SETF expander for APPLY is fixed to allow use with non-CL functions to support CL-CONTAINERS - another small step to support the most popular libraries in Quicklisp.

ABCL runs on newly released Java 7

2011-07-31T23:13:00.000-07:00

Co-developer Mark Evenson tried to compile ABCL on the recently released Java 7 today. The results are very good: compilation succeeds with the minor addition of a patch to specify the Java source conformance, while running the ANSI tests shows we'll need to look at only a very small number of ANSI test cases (5 or 6, I believe) which have regressed since Java 6.

Conclusion: ABCL works with Java 7.

ABCL 0.26.1 released

2011-07-27T07:15:00.000-07:00

On behalf of the developers of ABCL (Armed Bear Common Lisp) I'm glad to be able to announce the 0.26.1 release.

ABCL is a Common Lisp implementation implemented in Java and running on the JVM, featuring both an interpreter and a compiler. The compiler targets the JVM directly meaning that its output is runnable JVM bytecode. The fact that ABCL is written in Java allows for relatively easy embedding in larger
applications. For integration with existing applications ABCL implements JSR 223: Java scripting API.

This release features - among lots of other things - a major reduction in the number of failures in the ANSI test suite: dropping from 26 to 16 failures on some systems (of 21699 tests in total). Additionally, ABCL now runs Hunchentoot and several other packages from the top downloads from Quicklisp. Please read the release notes and the CHANGES file for more. Version 0.26.0 has not been released due to regressions fixed in this version.

If you have questions regarding use or licensing, or you find issues, please report back to the development list:

armedbear-devel at common-lisp dot net

Source distribution archives can be downloaded in ZIP or gzipped tar form:
abcl-src-0.26.1.tar.gz
abcl-src-0.26.1.zip

Signatures are available under:
abcl-src-0.26.1.tar.gz.asc
abcl-src-0.26.1.zip.asc

In addition, binaries are also available:

abcl-bin-0.26.1.tar.gz
abcl-bin-0.26.1.zip

With associated signatures:
abcl-bin-0.26.1.tar.gz.asc
abcl-bin-0.26.1.zip.asc

Upcoming release (0.26)

2011-07-03T14:13:00.000-07:00

This time, we've deviated from the bi-monthly release cycle. Two months ago, it was time to release 0.26 and it would have been time to release 0.27 now.

However, with some of the development team heavily involved in the migration of common-lisp.net, we decided to skip a release. It's release time now, so, it's time to release 0.26 sometime this week.

If you have time to test the current trunk version of ABCL with your favorite application, please do and submit your findings: hopefully we can fix them before the actual release!

This release contains the -very often wished for- improvement for deployment: compilation of ASDF systems into JAR files! And much more...

Repository available over HTTP for anonymous checkout

2011-06-05T15:34:00.000-07:00

It's been quiet over the past months in this forum. Most of that time was spent upgrading running box common-lisp.net and the services it offers. I'm glad to be able to announce to all of you behind corporate proxies that ABCL can now be checked out over HTTP!
To do so:

svn co http://svn.common-lisp.net/armedbear/trunk abcl

Enjoy!

Quicklisp helps focus ABCL development

2011-03-17T14:49:00.000-07:00

One of the great side effects from Quicklisp to us as the developers of an implementation, is the fact that for the first time since my last 10 years of lisp hacking, statistics have been published regarding system downloads.

Within the ABCL community, this list has been picked up as a signal from users telling us what they find important: Starting from the top, we've started compiling quicklisp provided systems and fixing the issues we found. Mark Evenson is working to fix bordeaux-threads on ABCL. Ville Voutilainen implemented an implementation-specific extension to DIRECTORY in order not to resolve symlinks required by Quicklisp itself. While Erik Huelsmann has been working to implement trivial-garbage required functionalities weak references, object finalizers and weak hash tables.

ABCL 0.25.0 released

2011-03-10T13:11:00.000-08:00

On behalf of the developers of ABCL (Armed Bear Common Lisp) I'm glad to be able to announce the 0.25.0 release.

ABCL is a Common Lisp implementation implemented in Java and running on the JVM, featuring both an interpreter and a compiler. The compiler targets the JVM directly meaning that its output is runnable JVM bytecode. The fact that ABCL is written in Java allows for relatively easy embedding in larger
applications. For integration with existing applications ABCL implements JSR 223: Java scripting API.

This release features - among lots of other things - a major refactoring of the compiler to be more robust in light of Java class file verification and object instantiation (MAKE-INSTANCE) performace improvements.
You can find the full release notes at:

http://common-lisp.net/project/armedbear/release-notes-0.25.shtml

and the list of changes at:

http://trac.common-lisp.net/armedbear/browser/trunk/abcl/CHANGES

If you have questions regarding use or licensing, or you find issues, please report back to the development list:

armedbear-devel at common-lisp dot net

Source distribution archives can be downloaded in ZIP or gzipped tar form:

http://common-lisp.net/project/armedbear/releases/0.25.0/abcl-src-0.25.0.tar.gz
http://common-lisp.net/project/armedbear/releases/0.25.0/abcl-src-0.25.0.zip

Signatures are available under:
http://common-lisp.net/project/armedbear/0.25.0/releases/abcl-src-0.25.0.tar.gz.asc
http://common-lisp.net/project/armedbear/0.25.0/releases/abcl-src-0.25.0.zip.asc

In addition, binaries are also available:

http://common-lisp.net/project/armedbear/releases/0.25.0/abcl-bin-0.25.0.tar.gz
http://common-lisp.net/project/armedbear/releases/0.25.0/abcl-bin-0.25.0.zip

With associated signatures:
http://common-lisp.net/project/armedbear/releases/abcl-bin-0.25.0.tar.gz.asc
http://common-lisp.net/project/armedbear/releases/abcl-bin-0.25.0.zip.asc

Finally, this is the first release to be also officially hosted on the Maven Central repository, and thus usable without hassle in Maven projects. Refer to the relevant wiki page for details: http://trac.common-lisp.net/armedbear/wiki/MavenSupport

Bootstrapping Quicklisp on ABCL

2011-03-01T14:51:00.000-08:00

The other day, I decided that - as many others have done before me - I'd give Quicklisp a go. Being on Windows (and ABCL), I was expecting a bumpy ride.

Turns out I was completely wrong: all I needed was a simple
CL-USER> (load "http://beta.quicklisp.org/quicklisp.lisp") [!!]

Quicklisp downloaded all its data nicely into my home directory (which - on ABCL/Win - is C:\Users\, C:\Users\Erik in my case). When I need a new ABCL session, for whatever reason, the only thing I need to type to get quicklisp loaded from my local install is:
CL-USER> (load "c:/users/erik/quicklisp/setup.lisp")

That's it. That's all. No bumpy ride. No ABCL issues. It Just Works!

Thanks Zach!

[!!] Warning: you may not want to do it that way, since it will execute the code from the web UNSEEN by you; you won't have a chance to see any security attacks on your PC by anybody slipping in a file which contains the lisp equivalent of "FORMAT C: /Y"

Summer of the Bear

2011-02-16T23:03:00.000-08:00

We're starting to coordinate [discussion of project that ABCL developers wish to mentor][1] for the Summer of the Bear 2011.

Drop by on IRC via [#summeroflisp][] or [#abcl][] for ideas or feedback.

[1]: http://trac.common-lisp.net/armedbear/wiki/GSoC2011
[#summeroflisp]: http://webchat.freenode.net/
[#abcl]: http://webchat.freenode.net/

CLOS work on trunk

2011-02-15T12:43:00.000-08:00

Over the past few days, work has been going on with respect to our CLOS implementation again. There were a few outstanding issues - some of them rather long already.

One of such longer standing issues was the performance of object instantiation. In the CLHS there's a specific section about the validity of initialization arguments (initargs) and which ones are to be considered valid. A number of improvements have been committed to trunk:
1. The check has been generalized to support all four cases of that specific section of the CLHS.
2. The checks done during object instantiation have been a performance bottleneck ever since they were implemented; as an improvement, a cache has been implemented, speeding up object instantiation significantly: the performance has been doubled (ie the same amount of time now allows creation of twice as many objects).

On a completely different subject (within CLOS), we had an issue where instantiation of an object depending on a forward referenced class caused evaluation to be aborted, without useful feedback. While fixing this issue could have been done quickly and completely unrelated to other changes, I decided to change the behaviour of some of the functions involved according to the AMOP class finalization specification.

So we have taken some small steps on the long (and hard) road toward a performant CLOS/AMOP implementation on the JVM again!

ABCL stand alone (nearly) executables

2011-01-23T13:33:00.000-08:00

Didier Verna describes his efforts to port CLON to ABCL to create stand alone (nearly) executables in his latest blog entry here: http://www.didierverna.com/sciblog/index.php?post/2011/01/22/Towards-ABCL-Standalone-Executables

ABCL 0.24.0 released

2011-01-22T12:08:00.000-08:00

Integrating ABCL with Ant - the second step

2010-12-18T15:38:00.000-08:00

While working to extend the ABCL integration into the ABCL-web Ant-based build - the project that I started in my previous post - I spent a lot of time trying to find a way to calculate the output paths, based on the names of the input paths.

Given the lisp way of 1-to-1 correspondence between a lisp file and a fasl, I expected this to be a trivial task to be done at the build system level. So, I kept searching for a solution in Ant (using standard components only). Surely I found the pathconvert and mapper building blocks, however, neither really seemed to work as the latter turned out not to be a generic mechanism but to be supported only by specific tasks (such as apply) and the former converts its output to a string instead of a set of paths.

Realizing that the issue of mapping source paths to output paths was left to the individual tasks, I turned to abcl to solve the problem as part of the compilation script. The final script between the CDATA tags as presented in the previous post looks like this:

(let* ((the-project (jcall "getProject" self))
       (src-iterator (jcall (jmethod "org.apache.tools.ant.types.Path" "iterator")
                            (jcall "getReference" the-project "lisp-files")))
       (src-wildcard (pathname (format nil "~A**/*.lisp" (jcall "getProperty" the-project "build.src.dir"))))
       (dst-wildcard (pathname (format nil "~A**/*.abcl" (jcall "getProperty" the-project "build.dst.dir")))))
   (when (and (jcall "hasNext" src-iterator))
     (loop
        for src-path = (pathname (jcall (jmethod "java.lang.Object" "toString")
                                 (jcall "next" src-iterator)))
        for dst-path = (translate-pathname src-path src-wildcard dst-wildcard)
        do (compile-file src-path :output-file dst-path)
        while (and (jcall "hasNext" src-iterator)))))

It's based on 2 properties having been defined: "build.src.dir" and "build.dst.dir". In addition, it wants a set of paths (fileset or path set) to be defined called "lisp-files". Iterating over the set of source files, it calculates the output file by replacing a path prefix and the file extension.

In my NetBeans project settings, I had to exclude "**/*.lisp" from packaging in the destination JAR file.

Now the JAR contains the fasl instead of the source file!

Do you have useful or resourceful ABCL based scripts plugged into Ant? Let me know and post a reaction.

Integrating ABCL with Ant

2010-12-12T13:47:00.000-08:00

While working with ABCL embedded in Java applications, you typically have java and lisp source files. Whenever I'm in that situaition, I find that I want to be able to build all sources from the same build mechanism.

Last week, Alex Mizrahi and I started to work on reviving ABCL-web (a project to create Java Servlets backed by Common Lisp code running in ABCL). That's when I found myself in need of a solution to this problem again: the project would greatly benefit from having a way to build lisp files for inclusion in the resulting JAR file.

Looking around, I found there are three options to achieve this goal:

Create a custom Ant task
Implement a command line switch to create fasls from the ABCL command line
Use our JSR-223 support to write in-line Lisp in build files

While option (1) looked very attractive at first, it means one of two things: (1) adding an additional dependency for creation of our jar for distribution, or (2) creation of a separate jar file with only this task in it. In all cases, it adds a dependency to our release build. Because our current release build doesn't require Ant (you can build our release using the lisp-based build), I wasn't ready to accept this solution yet.

Option (2) looked very attractive, because it would not only integrate with Ant, but with build systems in general. This option became less attractive when I found that the standard Ant project doesn't have any helpers for iterating sets. The thing that I had been thinking about was to implement the basics first, ie working the way gcc is used in most projects: 1 invocation per source file. So, in order to support the scheme that Ant does allow, the parameters ABCL would have to support became a lot more complex, needing to support file-sets to be compiled.

Then, I found the 'script' tag, which serves to integrate JSR-223 (Java scripting languages) into Ant build files. As we support JSR-223, I thought I'd give it a try! (So far I hadn't used it yet, I embed ABCL directly in my applications.) This solution -if it works out- is really great: no additional dependencies and no hacks [in our own build system, we feed the lisp that we want evaluated to ABCL through standard input.]

So, I gave it a try and within minutes, I was in business! I'm using a NetBeans generated project, so there's a "-post-compile" target which gets invoked after the "compile" target completes succesfully.

<target name="-post-compile" description="lisp file compilation">
<script language="ABCL" classpath="lib/abcl.jar">
<![CDATA[
(let* ((the-project (jcall "getProject" self))
       (the-fileset (jcall "getReference" the-project "lisp-files"))
       (files-iterator
        (jcall (jmethod "org.apache.tools.ant.types.Path"
                        "iterator")
               the-fileset)))
   (loop while (jcall "hasNext" files-iterator)
        do (print (jcall "next" files-iterator))))
]]>
</script>
</target>

The snippet basically retrieves a path-set with the ID "lisp-files" defined elsewhere in the project file. Then, it continues to print all the (Java) objects it retrieves from the iterator. The next step is to replace the (print ...) form with a (compile-file ) form, retrieving the path string from the Path object and making a lisp pathname of it. Want to signal failed build? Just do what a real Task implementation would have done: throw a BuildException.

Concluding: while adding command line options may still be a desirable path to pursue, Ant integration is here for anybody embedding ABCL in a broader Java context.

ABCL 0.23.1 released: fix for loading from path with spaces

2010-12-02T04:18:00.000-08:00

ABCL 0.23.0 released

2010-11-25T07:35:00.000-08:00

On behalf of the developers of ABCL (Armed Bear Common Lisp) I'm glad to be able to announce the 0.23.0 release.

ABCL is a Common Lisp implementation implemented in Java and running on the JVM, featuring both an interpreter and a compiler. The compiler targets the JVM directly meaning that its output is runnable JVM bytecode. The fact that ABCL is written in Java allows for relatively easy embedding in larger applications. For integration with existing applications ABCL implements Java Specification Request (JSR) 223: Java scripting API.

This release features - among lots of other things - CLOS thread safety fixes and concurrency improvements, support for Oracle's JRockit JVM, and a new (generic) class file writer. In addition, this is the first release to complete the Maxima test suite without failures, thanks to close cooperation between our projects. You can find the full release notes at:

http://common-lisp.net/project/armedbear/release-notes-0.23.shtml

and the list of changes at:

http://trac.common-lisp.net/armedbear/browser/trunk/abcl/CHANGES

If you have questions regarding use or licensing, or you find issues, please report back to the development list:

armedbear-devel at common-lisp dot net

Source distribution archives can be downloaded in ZIP or gzipped tar form:

http://common-lisp.net/project/armedbear/releases/0.23.0/abcl-src-0.23.0.tar.gz
http://common-lisp.net/project/armedbear/releases/0.23.0/abcl-src-0.23.0.zip

Signatures are available under:
http://common-lisp.net/project/armedbear/releases/0.23.0/abcl-src-0.23.0.tar.gz.asc
http://common-lisp.net/project/armedbear/releases/0.23.0/abcl-src-0.23.0.zip.asc

In addition, binaries are also available:

http://common-lisp.net/project/armedbear/releases/0.23.0/abcl-bin-0.23.0.tar.gz
http://common-lisp.net/project/armedbear/releases/0.23.0/abcl-bin-0.23.0.zip

With associated signatures:
http://common-lisp.net/project/armedbear/releases/0.23.0/abcl-bin-0.23.0.tar.gz.asc
http://common-lisp.net/project/armedbear/releases/0.23.0/abcl-bin-0.23.0.zip.asc

ABCL hash tables: threading and CLOS

2010-10-10T02:20:00.000-07:00

More people are starting to use ABCL in threaded environments - I have been doing so myself since spring 2008. My own use revealed some threading issues in the compiler which have been long solved now.

Last April, David Kirkman mailed the list with some threading problems in our CLOS implementation. Because of other work - including implementation of METACLASS support - there wasn't much time to do much about the issue.

This week, he mailed having found the problem - accompanied with a patch with a solution. Because of our need to support the four Common Lisp equality operators, ABCL implements its own hash tables -four of them - with a shared ancestor class which requires implementation of a few primitive operations. Hash access from the Lisp world was being synchronized by the common ancestor. However, in some locations on the Java side the - unsynchronized - primitives were being called directly.

The solution was of course to add synchronization to the primitives as well. Evaluating the result, the new situation was rather unsatisfying: only a single thread could be reading or writing at the same time, meaning only a single CLOS effective method dispatch could be happening at the same time. You probably understand my reluctance to accept the status quo.

The quick solution to the CLOS dispatch problem was to use the java.util.concurrent.ConcurrentHashMap type: we were keying on symbols, which have EQ equility in the Java world (ie in terms of Object.equals()).

That still left the lisp world with rather unsatisfactory threading performance of our own hash tables and since those were repeating large chunks of code in their specialized primitives, I decided to refactor them completely.

The result is - admittedly by looking very closely at ConcurrentHashTable - a single hash table implementation with 4 Comparator classes, which have nearly-lockless reads from all threads. Inserts into the hash table are still synchronized from all threads, but that's expected to be a lesser problem: you'd primarily expect many reads per write in a hash table.

Concluding, the only (known) CLOS threading issue has been fixed - there are presumably many more, please report if you find them - and as a bonus we have a much more efficient hash table implementation.

Maxima on ABCL: full pass on the test suite

2010-10-03T00:25:00.000-07:00

Over the past two weeks Raymond Toy helped out to fix the last 9 failures in the Maxima test suite running: ABCL now passes the full 8.666 Maxima tests! We achieved a goal set for ABCL more than two years ago!

Thinking back to the days when Peter Graves had just handed over the project shows just how far we have come: back in September 2008 lots of tests (423, according to Hunter Monroe) would plainly terminate the test suite. That number had come down from over 1000 tests crashing the test suite in January of that year.

After manually disabling the crashing tests, Hunter Monroe writes on September 27 (2008) that he's been able to complete the tests with a total number of failures of 612 (out of 4454 tests) .

Many of the problems were caused by incorrect handling of special variables which was one of the first things to be addressed in ABCL - stabilizing over the first half of 2009. Other improvements which contributed to the stability of Maxima were fixes to the code generating non-local returns (THROW, GOTO stepping out of a closure, RETURN-FROM stepping out of a closure). Next to its reduction in specials, the Maxima team had to change lots of number comparisons from EQ to EQL, because in ABCL integers aren't (guaranteed) to be EQ - which is allowed by the spec, but not very customary in CL implementations.

At the same time evaluation time of the test suite has gone down considerably too - not due to increased processing power, but due to improvements on both sides. One example of an improvement which has greatly benefitted ABCL's performance is the gradual elimination of specials is Maxima - an on-going effort in their team. On the other side, binding and unwinding specials has become much faster in ABCL too, reducing the impact of remaining excessive specials.

The last few tests to be fixed required adjustments to the Maxima test suite as well as fixes to ABCL. Thanks to everybody who helped achieve this result, most notably Robert Dodier, Hunter Monroe and Raymond Toy from the Maxima team as well as Ville Voutilainen and Mark Evenson from the ABCL team.

ABCL 0.22.0 released

2010-09-26T11:03:00.000-07:00

On behalf of the developers of ABCL (Armed Bear Common Lisp) I'm glad to be able to announce the 0.22.0 release.

ABCL is a Common Lisp implementation implemented in Java and running on the JVM, featuring both an interpreter and a compiler. The compiler targets the JVM directly meaning that its output is runnable JVM bytecode. The fact that ABCL is written in Java allows for relatively easy embedding in larger applications. For integration with existing applications ABCL implements Java Specification Request (JSR) 223: Java scripting API.

The release is a small maintenance release: most efforts were focussed on work happening on branches some of which has already been merged to trunk for 0.23.

The release notes can be viewed at http://common-lisp.net/project/armedbear/release-notes-0.22.shtml.

and the list of changes at http://trac.common-lisp.net/armedbear/browser/trunk/abcl/CHANGES.

Latest and older binary and source distributions can be downloaded from http://common-lisp.net/project/armedbear/releases/

Parsing latin1 data with an utf-8 stream

2010-09-19T14:07:00.001-07:00

We recently noticed that ABCL isn't capable of reading files containing certain latin1 characters, like the author comments in albert. ABCL's DecodingReader uses java.nio.charset.CharsetDecoder, which defaults to throwing exceptions when it finds unmappable character or input it otherwise considers malformed. Therefore, reading albert files, which contain Norwegian/Danish o-slashes resulted in an exception being thrown. Same result occurred with files that contain o-umlauts.

The remedy that seemed obvious was changing the CharsetDecoder's error handling strategy to replace malformed input and unmappable characters. Alas, this didn't work, to much surprise, as it resulted in the DecodingReader to loop endlessly. Despite this document, which seems to suggest that an utf-8 byte sequence can only start with f0-f4, it seemed that o-umlaut (f6) and o-slash (f8) threw the decoder off-balance.

After debugging the problem, we noticed that in the case of these characters, the DecodingReader reported an overflow but didn't advance the buffers it uses. So, after some consideration, we introduced a hack where, in case of an overflow that didn't advance the buffers at all, we advance the buffers manually by one byte and insert a ? character into the resulting stream. This reeks of a hack, but so far we haven't been able to find a more elegant solution. This solution does seem to enable us to parse files with such latin1 chars like o-umlaut and o-slash properly, and the fix was committed into ABCL trunk as commit r12902, and will be part of ABCL 0.22.

ABCL-0.21.0 released....

2010-08-01T23:09:00.000-07:00

.... so long ago that it's time to release 0.22 in the week(s) to come.

URI Pathnames

2010-07-11T13:32:00.000-07:00

Among other goodies, ABCL-0.20 includes good support for using URI resources as arguments to about any function you would specify a filepath. Any builtin scheme like "http" or "file" or "jar" works as a full-on Lisp Pathname, with the caveat that one may not write to an associated Stream. We should be generic enough in implementation that the JVM extension mechanism for extending the classloader works for us for the most common use case, OSGi.

We pick up the ability to refer to URI, associating an input Stream with its representation as a network (or local system) bytes easily as the following one-liner:


CL-USER> (with-open-file (stream #p"http://google.com/index.html")
           (format nil "~A" (read-line stream nil)))

Underneath we have implemented a new subtype of PATHNAME, namely the type >URL-PATHNAME. We name this URL as opposed to URI as the underlying Java object is a java.net.URL with associated java.net.URLConnection that we'll be relying on for input sources.

Looking through an APROPOS for URL-PATHNAME shows the following related symbols:


CL-USER> (apropos (type-of #p"http://google.com/index.html"))
EXTENSIONS::SET-URL-PATHNAME-SCHEME (fbound)
EXTENSIONS::SET-URL-PATHNAME-AUTHORITY (fbound)
EXTENSIONS::SET-URL-PATHNAME-QUERY (fbound)
EXTENSIONS::SET-URL-PATHNAME-FRAGMENT (fbound)
URL-PATHNAME
URL-PATHNAME-AUTHORITY (fbound)
URL-PATHNAME-SCHEME (fbound)
URL-PATHNAME-FRAGMENT (fbound)
URL-PATHNAME-QUERY (fbound)
; No value
CL-USER>

which we see the defined functions to set the AUTHORITY, SCHEME, FRAGMENT, and QUERY portions of a URL-PATHNAME.

After these modifications, the ABCL PATHNAME class now has two "beyond ANSI" subtypes, namely URL-PATHNAME and JAR-PATHNAME. A short document outlining the design of URL-PATHNAME can be found in our documentation directory.

In implementing URL-PATHNAME, we actually had the ability to analyze what another CL did for their implementation. We analyzed the CLForJava technical paper, but decided not to use their abstractions as detailed on a lengthy armedbear-devel post.

ABCL 0.20.0 released, including first funded feature

2010-05-27T12:53:00.000-07:00

On behalf of the developers of ABCL (Armed Bear Common Lisp) I'm glad to be able to announce the 0.20.0 release.

This release is the first to include a funded feature: funds were provided to implement the CLOS METACLASS feature and tests. Next to that, this release contains a large number of fixes and improvements, such

as the ability to use JARs and URLs as pathnames and the a new ASDF version (ASDF2). You can find the full release notes at:

http://common-lisp.net/project/armedbear/release-notes-0.20.shtml

and the list of changes at:

http://trac.common-lisp.net/armedbear/browser/trunk/abcl/CHANGES

Latest and older binary and source distributions can be downloaded from

http://common-lisp.net/project/armedbear/releases/

Running SLIME under cygwin with ABCL

2010-05-06T04:04:00.000-07:00

If ABCL may be said to have an IDE, it has one in the combination of Emacs using the SLIME to connect to running Lisp instance(s). In the parlance of SLIME, Emacs is said to be the superior process with the running Lisp instance(s) known as the inferior Lisp process. With SLIME, Emacs provides symbol completion in a REPL with edit history, Lisp syntax highlighting, finding the source location for a given form, the ability to run multiple instances of the inferior Lisp (or even different versions of Lisps), the ability to interactively inspect Lisp values, a nice wrapper for controlling ASDF, and much more.

When confronted with a Microsoft Windows environment that I need to productively produce code in, the first thing I do is install the cygwin package to get to a minimal UNIX-like environment. Unlike many Windows enhancements, the installation of cygwin does not require Administrator privileges so it is possible even in a somewhat restricted corporate environment. cygwin provides a UNIX compatibility layer that has enabled the porting of many standard GNU utilities to run under Windows, including an X11 server.

One of the first packages that I install for use under cygwin is the Emacs X11 package, which provides The One True editor with a lot of tools that make simple programmatic tasks easy, and the construction of complicated system possible. Historically there have been other efforts to bring Emacs to Windows such as NTEmacs, but now that GNU Emacs 23.1 runs fine in an X11 server with cygwin, I think the ease by which cygwin installs and updates outweighs putative advantages of Windows-specific Emacs ports. In truth, over time, GNU Emacs has absorbed most of the useful features pioneered by other, more native, versions.

People have reported that they have been able to get SLIME working with Emacs under Windows, but I never had much success, maybe because cygwin imposes a UNIX-like pathname structure that are naturally quite different from Windows pathnames. Not only is the '\' <--> '/' convention switch, Windows drive letters (like 'C:') are mapped to mounts unto the UNIX-like root filesystem, so 'C:\' becomes '/cygdrive/c'. Since ABCL under Windows runs on the JVM it expects to be dealing with Windows pathnames, it never worked for me. Until today. Or rather, yesterday. So, without further ado about pathnames, here's …

HOWTO install SLIME under Windows for ABCL

0. Install cygwin, selecting the 'emacs-x11' and 'cvs' packages in the interactive chooser.

1. Make a directory called 'c:\work' to contain ABCL and SLIME. You can of course use whatever directory you wish here, but the adjust the rest of these instructions to whatever you choose. Under cygwin, this directory will appear as '/cygdrive/c/work'.

2. Install ABCL under 'c:\work\abcl', so that 'c:\work\abcl\abcl.bat' invokes ABCL.

3. Use CVS to retrieve a copy of SLIME HEAD later than 2010-05-05. SLIME doesn't do releases which put me off for a bit at first, so even though there is a slime-2.0 floating around somewhere, it's ancient by this time, and a release process probably won't be repeated by the SLIME community in the foreseeable future.

From the bash prompt you installed in step 0, from the /cygdrive/c/work directory issue the following command:

cmd$ cvs -d  :pserver:anonymous:anonymous@common-lisp.net:/project/slime/cvsroot checkout slime

This will checkout a copy of SLIME including documentation. Alternatively one may download a CVS snapshot but then it will be harder to update SLIME later if you wish.

4. Use Emacs to edit '~/.emacs' to contain the following code

;;; .emacs for SLIME working with cygwin emacs

(add-to-list 'load-path "/cygdrive/c/work/slime")
(setq slime-lisp-implementations
 '((abcl ("/cygdrive/c/work/abcl/abcl.bat"))))
(require 'slime)
(slime-setup '(slime-repl slime-asdf slime-fancy slime-banner))

(defun slime-to-lisp-translation (filename)
(replace-regexp-in-string
"\n" "" (shell-command-to-string
  (format "cygpath.exe --windows %s" filename))))

(defun lisp-to-slime-translation (filename)
(replace-regexp-in-string
"\n" "" (shell-command-to-string
  (format "cygpath.exe --unix %s filename"))))

(setq slime-to-lisp-filename-function #'slime-to-lisp-translation)
(setq lisp-to-slime-filename-function #'lisp-to-slime-translation)

This code sets up SLIME to be autoloaded by Emacs, using the 'cygpath.exe' to translate pathnames between the cygwin UNIX conventions (which Emacs expects), and the native conventions (which ABCL running on the JVM expects).

5. Upon evaluating the elisp code which you just used to configure SLIME (which can be affected by restarting Emacs), SLIME can be invoked via "M-x slime". ABCL will be invoked as a separate process, it will compile the elisp server known as swank that communicates between Emacs and the inferior Lisp process, it will load that code to initiate a connection, you'll see some "flying letters" coalesce into a REPL with a message in the mini-buffer welcoming you to SLIME

Happy hacking!

ABCL on N900

2010-04-26T14:11:00.000-07:00

I decided to play a bit with my new mobile toy. The Nokia N900 should be capable of all kinds of things, including running a Java VM. After some googling, I found this page:

http://wiki.maemo.org/OpenJDK_6.0_0_%28Cambridge_Software_Labs%29_on_N900

The instructions there seemed simple and straightforward, so I did as advised, downloaded the jdk package and extracted it onto the device, and then just copied abcl.jar over and ran it with the incantation that's usually found in the abcl wrapper script. The results are pretty good:

~ $ /opt/OpenJDK-camswl/bin/java -cp /home/user/abcl.jar org.armedbear.lisp.Main
Armed Bear Common Lisp 0.20.0-dev
Java 1.6.0_0 Sun Microsystems Inc.
OpenJDK Zero VM
Low-level initialization completed in 4.436 seconds.
Startup completed in 16.765 seconds.
Type ":help" for a list of available commands.
CL-USER(1): (format t "abcl rules!")
abcl rules!
NIL
CL-USER(2):

It runs obviously much slower than on any desktop, the device is not as powerful, and its i/o is slower which hurts autoloading. But it runs, and it is much easier to setup than java ever was on N800 and such, and it also runs much faster than the non-zero VMs on N800 did.

CGOL on ABCL

2010-04-24T07:55:00.000-07:00

The other day I received an e-mail from an ABCL user, containing the following:

I took a brief look at cgol. Neat! Interestingly, I tried it in
clisp, sbcl, ecl, cmulisp, and abcl, and it only works in abcl!

For those wanting to play with CGOL too, it's available from the CMU Artificial Intelligence Repository.

Seems like ABCL is Common Lisp conformant enough to be able to run some of the older lisp code available: this source stems from 1993 and is unchanged since - judging by the timestamps in the download archive.

Even better handling of low memory conditions

2010-04-03T03:36:00.000-07:00

In previous research, ABCL came out nicely when tested for handling of Stack Overflow (SO) and Out Of Memory (OOM) conditions. See the blog item "out-of-memory: a sad case"; ABCL was classified as one of the few lisps that handled OOM conditions at all, by ending in the Lisp debugger even.

By looking more closely - which we did before the release of 0.18 - it turned out that handling of these conditions was only part of forms executed from REPL. If the same conditions occurred during program execution, the program was simply terminated, just like the other Lisps in the survey.

Of course, when you have a program running into low memory conditions regularly, it makes sense to make more memory available to it at startup. There are a few startup switches for java which allow to set the initial memory amount, most notably:

-Xmx: which sets the maximum amount of heap available
examples: 512m, 1024k, 13m
-Xms: like -Xmx but specifies the initial amount
-Xss: specifies the thread stack size

These options are only available at startup though. These options can be added to ABCL's startup wrapper script, either after building, or by letting the build add them automatically. If you opt for the latter, you need to add a line like the following to your abcl.properties file (before abcl-0.19.1 this was named build.properties):

java.options=-Xmx512m -Xss6m

If your program keeps large caches, it may be important to be able to detect OOM conditions, clear some or all of the caches and retry the operation. Other situations where it's good to be able to handle OOM conditions can be thought of. That's why it's important to have your programming environment help you detect these situations.

One of the advantages of running in the JVM is that ABCL itself doesn't doesn't need heuristics to detect these situations: the JVM throws a StackOverflowError or an OutOfMemoryError as appropiate. The only thing ABCL needs to do is handle these errors in appropriate situations.

Starting with 0.18, ABCL has well defined points where it traps the SO and OOM conditions in any Lisp program: HANDLER-BIND (and as a consequence HANDLER-CASE) establishes a try/catch block around the form. This try/catch block makes sure OOM and SO get converted to Lisp errors, for HANDLER-BIND to handle.

The situation isn't perfect: wouldn't it be nice to have a 'low memory' signal which provided an environment with enough memory to allow user-lisp code to run, for example to free up memory held by caches. Even though it's not perfect, ABCL now provides the functionalities which the "sad case" article would lead you to believe it already did.

An option to pursue low memory condition notifications was pointed out by Douglas Miles, one of the people frequenting the #abcl irc channel on freenode.net, would be to install handlers for the JMX (java management extensions) low memory notifications.

ABCL 0.19.1 released

2010-03-27T12:29:00.000-07:00

On behalf of the developers of ABCL (Armed Bear Common Lisp) I'm glad to be able to announce the 0.19.1 release. Version 0.19.0 was never released.

This release features - among lots of other things - a fix for unbinding PROGV bound variables upon exiting the PROGV scope and a much improved integration of access to filenames specified by URLs. You can find the full release notes at:

http://common-lisp.net/project/armedbear/release-notes-0.19.shtml

and the list of changes at:

http://trac.common-lisp.net/armedbear/browser/trunk/abcl/CHANGES

If you have questions regarding use or licensing, or you find issues, please report back to the development list:

armedbear-devel at common-lisp dot net

Source distribution archives can be downloaded in ZIP or gzipped tar form:

http://common-lisp.net/project/armedbear/releases/abcl-src-0.19.1.tar.gz

http://common-lisp.net/project/armedbear/releases/abcl-src-0.19.1.zip

Signatures are available under:

http://common-lisp.net/project/armedbear/releases/abcl-src-0.19.1.tar.gz.asc

http://common-lisp.net/project/armedbear/releases/abcl-src-0.19.1.zip.asc

In addition, binaries are also available:

http://common-lisp.net/project/armedbear/releases/abcl-bin-0.19.1.tar.gz

http://common-lisp.net/project/armedbear/releases/abcl-bin-0.19.1.zip

With associated signatures:

http://common-lisp.net/project/armedbear/releases/abcl-bin-0.19.1.tar.gz.asc

http://common-lisp.net/project/armedbear/releases/abcl-bin-0.19.1.zip.asc

REQUIRE now works with ASDF systems

2010-02-11T04:36:00.000-08:00

The ANSI *MODULES*/REQUIRE/PROVIDE interface is usually implemented in a fairly simple manner. When a piece of code wishes to depend on functionality of a given module, a statement of the form

(require :module)

first looks for whether the modules is currently in the list referenced by *MODULES*. If this isn't the case, a Lisp implementation simply looks in a few standard places for a file to load based on the STRING form of ":module", like "module.lisp", or possibly by a FASL. That file is then loaded, during which the form

(provide :module)

is evaluated which pushes the keyword form of its argument to the *MODULES* list.

This very simple dependency mechanism really doesn't provide a mechanism for searching file hierarchies, being more appropriate to manage dependencies in a large collection of files located in a single directory as is the case in the ABCL system directory. Presumably this simplicity and lack of extensibility led to the facility as being noted as deprecated in the HyperSpec. But it is part of the ANSI standard so until the New Implementation of Lisp is released, it forms the only mechanism that portable code can rely on.

In the contemporary Lisp era, ASDF forms the most common method to specify system dependencies. Since ANSI doesn't specify how REQUIRE searches for the dependencies, the clever folks behind SBCL decided to make REQUIRE invoke ASDF if the dependency cannot be resolved by the simple version of the mechanism.

Since implementation is the sincerest form of flattery, as of ABCL svn r12247 we have adopted this extension to REQUIRE. To use this mechanism, you first have to ensure that ASDF itself is loaded via

(require 'asdf)

Afterwards, a command like

(require :hunchentoot)

will search for an ASDF loadable system named "hunchentoot".

But how does ASDF find these systems? By successively searching the contents of ASDF:*CENTRAL-REGISTRY* which is a list of locations to search for files or symbolic links named "hunchentoot.asd". Since maintaining this list for a large number of ASDF definitions would be onerous, the usual practice is to have a single directory which contains symbolic links to the system definitions. It makes maintaining various versions a bit easier as one can simply manipulate the symbolic links.

I use the following piece of code in my "~/.abclrc" to collect all my ASDF pointers in the directory "~/.sbcl/systems/":

(require 'asdf)

(pushnew '(merge-pathnames ".sbcl/systems/" (user-homedir-pathname))

asdf:*central-registry*)

How does one install ASDF systems? One downloads the code and creates a link to its "*.asd" file in "~/.sbcl/systems/". But that can get onerous to do manually especially if one ASDF system depends on other systems. The ASDF-INSTALL package provides a semi-interactive automated mechanism to download, compile, and install ASDF definitions. For now, using SBCL to install ASDF systems that ABCL can share works pretty well, but we're currently working on incorporating a port of ASDF-INSTALL into the base ABCL system. Stay tuned!

ABCL 0.18.1 released

2010-01-18T11:44:00.000-08:00

Due to some unfortunate regressions in 0.18.0, the ABCL project had to release an 0.18.1 release. It's readily available for download and is strongly advised for everybody who's now running 0.18.0: the .0 release can behave seemingly randomly after loading compiled code.

ABCL 0.18.0 released

2010-01-13T11:21:00.000-08:00

On behalf of the developers of ABCL (Armed Bear Common Lisp) I'm glad to be able to announce the 0.18.0 release.

This release features - among lots of other things - faster initial startup, faster special variable lookup and portable fasl files. Please refer to the release notes for the full list.

If you have questions regarding use or licensing, or you find issues, please report back to the development list: armedbear-devel at common-lisp dot net

Source distribution archives can be downloaded in ZIP (zip signature file) or gzipped tar (tar signature file) format.

In addition, a ZIP binary (bin-zip sig file) and gzipped tar binary (bin-tar sig file) are available.

ABCL on Google App Engine (2)

2009-12-28T05:36:00.000-08:00

Over the past month further work was done to improve ABCL startup times. This effort is especially directed at supporting ABCL on Google App Engine (GAE).

The startup time of the trivial GAE example application in ABCL's source tree takes 19 seconds to startup, as mentioned in an earlier blog item. Although this is only a "Google theoretical time", because the page is served in only 12 seconds, this is clearly a lot. I heard many GAE applications have startup times between 5 and 10 seconds. It surely would be nice if our trivial application could get closer to that.

A number of different solutions have been evaluated:

Reduction of the number of classes loaded at startup; making better use of ABCL's auto-loader facility
Supporting binary fasls
Create a system for finer-grained auto-loading support

The reason exactly these scenarios were evaluated is the fact that I looked for help regarding this performance issue on irc (irc://irc.freenode.net/#...); the answer was "you're doing too much during setup or at the first request". My first reaction was that we didn't have any options: that's how ABCL is designed. After some discussion these scenarios came up though.

The first and third scenarios are the result from many profiling sessions of "ABCL startup" time. The conclusion was that 35 to 45% of ABCL startup time is spent in Java reflection: when loading function classes ABCL needs to look up the class constructors to instantiate an object of the given class.

Scenario 1 is about delaying loading of FASLs until a function in them is required. Scenario 3 goes into more detail about the use of a function: even when a FASL is loaded, not all functions in it will be used (immediately or ever). The idea behind scenario 3 is to delay reflection API access until a function is actually used.

Using scenario (1) startup times could be reduced somewhat, especially in the case of our minimal servlet application: it uses relatively few Lisp functions and the ones it does use are related to printing and streams. Those are concentrated in a limited number of fasls.

In order to implement scenario (3), a quite bit more effort was required. The basic idea - as explained above - is that many functions in a FASL won't be used until a later stage in the application. In order to be able to delay resolution of the bytecode of the function, we introduced an object which - like the auto-loader - acts as a proxy for the unresolved function. This proxy class doesn't exhibit the same overhead, because it is resolved only once.

Upon the first call to the function, its bytecode gets resolved and the proxy in the function slot gets replaced with the actual function. After that, the first call is forwarded to the real function, as if it had been called directly. Although the actual implementation is a bit more complex to account for the loading of nested functions, that's basically it.

With scenario (3) applied to function definitions only, we were able to reduce startup time of the first request on GAE from 19 seconds to 11 seconds (roughly 40%). Today, we started to apply the same strategy to macro functions too. The result is - measured on my local PC, not GAE - a savings of another roughly 13%. Assuming that the same applies to GAE (as it did with the other 40%), we've realized a saving of 50% startup time!

Binary fasls - scenario 2 - were an attempt to reduce the amount of work that needed to be done at startup: because the normal fasl loading process is driven by a text file containing Lisp code, that could have been one of the causes. We didn't remove support for them, but they didn't turn out to be a big saver; that can be explained because the binary fasls are just another ABCL function object which needs to be loaded using reflection.

All in all did we save 50% start up time. Let this be an invitation to start experimenting with ABCL on GAE.

Maxima(l) performance

2009-11-10T12:05:00.000-08:00

Last spring, the Maxima challenge for ABCL was to get it to complete its test suite. We've mastered that bit of Maxima for some months now. However, as turned out soon after that, Maxima runs rather slow on ABCL. To some extent - being limited by the JVM - that's to be expected. The performance observed was way off base though: much too slow.

Through analysis, the cause was established to be the fact that Maxima declares lots of symbols to be special [and that ABCL doesn't offer a way to remove that specialness].

Last summer we found that allowing Maxima to undeclare specials increases ABCL performance immensely (roughly 35%). However, the final goal for Maxima is to make more sparingly use of specials and declaring unspecial or undeclaring special isn't defined in the spec. Because of the two reasons it felt not right to implement the solution at the time: it would have been a very Maxima specific one to a problem Maxima intends to fix in the long run.

Peter Graves noted that he converted the special bindings storage in XCL to use the same scheme as in SBCL/CCL, using an array with active bindings instead of a linked list of bindings. He observed a performance gain of 10% in his tests.

Last weekend, I implemented the same scheme in ABCL and although the general speed up doesn't show 10% in our tests [which may very well differ from Peter's], we observed roughly 40% performance gain on Maxima's test suite!

ABCL 0.17.0 released

2009-11-07T12:02:00.000-08:00

On behalf of the developers of ABCL (Armed Bear Common Lisp) I'm glad to be able to announce the 0.17.0 release.

This release features - among lots of other things - performance improvements, a fix for unexpected thread termination due to uncaught exceptions and example code for running ABCL on Google App Engine. Please refer to the release notes for the full list.

If you have questions regarding use or licensing, or you find issues, please report back to the development list:

armedbear-devel at common-lisp dot net

Source distribution archives can be downloaded in ZIP (zip signature file) or gzipped tar (tar signature file) format. In addition, a ZIP binary (bin-zip sig file) and gzipped tar binary (bin-tar sig file) are available.

ABCL on Google App Engine

2009-10-26T15:25:00.000-07:00

Triggered by the interest of one of our users, last week was mostly dedicated to finding out how (how well) ABCL runs on Google App Engine (GAE). This is what we found out:

To those readers who don't know: GAE is an environment for hosting web applications, backed by Google's storage and server clouds. It supports running applications written in Python and Java.

GAE's Java environment turns out to be a servlet environment. This means there is a single servlet instance per JVM which gets a chance to initialize itself in an init() method. The first request waits for this method to complete before it's being processed. Google makes no guarantees regarding the number of JVMs your application might be running on concurrently, or the lifetime of a single JVM: when GAE needs memory to run other apps, your JVM might get torn down when not serving any requests.

Knowing the above, getting ABCL to run on GAE involved several steps:

Getting the Java SDK for GAE (don't forget to get Java's SDK too!)
Implementing Java classes wrapping ABCL
Writing a minimal servlet in ABCL

Step (1) turns out to be rather easy: just get it from the GAE website, unzip it and -if you want your local paths to match the examples on their site- rename it to remove the version number at the end (appengine-java-sdk-1.2.6 becomes appengine-java-sdk).

Step (2) turned out to be a bit more involved, but after some twiddling, we found that we needed a minimum of 2 classes: at least one servlet class and a singleton class which loads a single ABCL into the JVM. [Note: a web application may contain any number of servlets with a minimum of 1.] The resulting application classes were committed to the ABCL repository.

From there, a minimal "Hello world" web app was easy, making step (3) a quick one. The end result was entirely committed to the examples directory in ABCL's repository.

GAE offers a performance dashboard to monitor your application through an administration web interface. From there, you can check the application logs, response times (called latency in the dashboard) and see how much CPU your application is using. For the latter, they use an indicative measure: the time which it would have taken to handle the request on an unloaded Core2 1.2Ghz processor. This compensates for many of the variances in the Google infrastructure which influence how long it actually takes to handle the request.

With a working application in place, the next step was performance. Most notably that of the first request: all subsequent requests are handled within miliseconds (7 to 15 miliseconds), so there's no issue there. This is the part that dominated last week: It turned out that although the latency was around 12 seconds, the CPU consumption was around 19secs [1], both very high and said to be close to some upper limit which remains unspecified.

We're striving to get these figures down: even though they would not really impact operation of a servlet in a regular hosting situation, GAE's regular servlet restarting makes these times more important. The best way to reduce figures like these is to get the figures the application scores on your local system down first. One of the first things which comes to mind is ABCL's "long" startup time: on my local Core2 - 1GB - 1.6Ghz machine it takes 1.7 seconds.

More on the steps we took to optimize this startup time in a next blog post.

Conclusion: ABCL - if you accept the long initial request response - is definitely an option for writing your web applications in Common Lisp on a Java/JVM based infrastructure. It'll even run on Google App Engine. We'll keep you posted on how we fare on supporting that even better!

[1] In comparison: On the Clojure mailing list, 5.5 seconds is mentioned for Clojure and on #appengine (on irc.freenode.net), 7 to 10 seconds are said to be normal for JVM based apps.

ABCL 0.16.1 released

2009-10-17T12:50:00.000-07:00

On behalf of the ABCL development team, I would like to announce the new ABCL release: 0.16.1

This release is a maintenance release on the 0.16 version line. In total four issues have been fixed:

In case of unsuccessful completion of a COMPILE call, ABCL would leak (= not correctly clean up behind itself) temporary files
In some cases, an exception thrown by a GO or RETURN-FROM form would not be correctly caught, causing the thread in which it occurs to terminate
Honor appearance of &allow-other-keys in CLOS MAKE-INSTANCE argument checking
More careful checking of 'null' arguments when processing stack frames

Should you have any comments, remarks or bugs to be found in this release, please contact the developers through the mailing list mentioned on the ABCL project home page.

CLOS performance

2009-10-12T06:03:00.000-07:00

Traditionally, ABCL's CLOS implementation has been depicted as one of its weekest spots. The two main reasons being on one hand its speed and on the other the absense of a good MOP.

I'm proud to see that some of the improvements over the past year turn out to be real performance improvers. Ofcourse, there's more to wish for and improvements will keep coming in. However, another improvement of 90% as in the linked post... ?

As shown for a fact, ABCL's CLOS is improving and while possibly still it's weekest spot, it's definitely becoming useable. If you find performance issues, preferably with example code to show the issue, please report to the Armed Bear mailing list (address information on the project front page).

[Note added: Though 90% improvement is hard to achieve, another 30% (24 seconds to 16 seconds) were realised last weekend.]

Struggling with and surviving TAGBODY and BLOCK compilation

2009-10-06T01:11:00.000-07:00

At first glance, compilation of TAGBODY and BLOCK forms seems simple: the tags in the TAGBODY are all static and the exit point for the BLOCK is also well defined. Summarizing: no complexities with the dynamic environment of any kind.

However, taking a closer look and adding closures to the picture, things get a little more complicated: GO or RETURN-FROM can mean transfers of control to exit points outside the current function. Consider the snippet below:

(block NIL
  (funcall (lambda () (return-from NIL 3)))
  4)

Apart from the fact that the example is too obvious: the lambda form can be inlined, it demonstrates what I mean by "RETURN-FROM will cause a transfer of control to a non-local exit point": The exit point to jump to is located outside the lambda.

So far, so good: the above existed for a long time in ABCL and is achieved by raising Java exceptions. Now things get more contrived: we'll use a closure created inside a BLOCK form which is part of a recursively called function.

(defun foo (fun)
  (block B
      (if fun
         (funcall fun)
         (foo (lambda () (return-from B :good)))
      :bad))

Now, evaluating (foo nil) will cause the following function calls:

 1: (foo nil)
2:   (foo (lambda () ...)
3:     (funcall (lambda () ...)

To which exit point would you expect the lambda to return? You should expect it to return to the exit point associated with B block from the first FOO call. Now it's becoming clear what's so hairy about compiling BLOCK (and TAGBODY which has the same issue): there are 2 B blocks on the stack, so, which one to return to?

The way this last issue - recursive BLOCK and TAGBODY forms - was solved in ABCL just last weekend was to create a (hidden) variable which gets set to a certain unique value upon entry of the block at run time. Then, any non-local transfer of control uses that block identifier value to find the right block to jump to.

Actually, this solution has a slight advantage for TAGBODY over the pre-existing solution: the old solution checked all symbols in a TAGBODY before concluding the GO wasn't meant for the given tagbody. With the new approach, a single variable test (equality of object pointers) allows checking if the GO is meant for any given TAGBODY or that stack unwinding should continue.

But then there's another issue: closures can be assigned to variables which outlive the extent of the originating block or tagbody. Like the snippet below:

(progn
   (block B
      (setq a (lambda () (return-from B 3))))
   (funcall a))

As indicated above, ABCL uses Java exceptions for non-local transfers of control. Now, if the (funcall a) form would throw a Go exception, regardless of the fact that there's no matching try/catch block, the exception would remain unhandled and the processing thread would exit. This is the situation in ABCL as it existed before last weekend.

Now, the solution to the problem with the recursive function calls has a nice additional benefit. Since there's storage shared between the closure and the BLOCK - they now share a variable - the variable can be used to let the block communicate to the closure that its extent has ended by setting it to a specific value. The GO form can then check for that condition before it throws the actual Go exception, making sure there's always a matching try/catch block. If there's no such block ABCL now generates a call to ERROR, allowing interactive error handling and selection of restarts where it used to unwind the stack to some location which happened to catch the exception - or terminate the thread if that didn't happen. Quite an improvement I'd say.

As a consequence of the changes described above, the code presented in the lisp paste at http://paste.lisp.org/display/88240 - which tests the full requirements of the CL spec - does succeed with today's ABCL (whereas it didn't last week!)...

Loading FASLs from JAR files

2009-09-15T03:38:00.001-07:00

Currently ABCL is a pretty decent decent Common Lisp that runs on the JVM, but we have really not started to add the necessary features which cater to making that "special" relationship easier. One of this rough sports involves how one packages ABCL applications for distribution. Since JAR files are currently the natural base unit for the distribution of JVM packages, it would make sense if one could load ABCL FASLs from JAR files. Indeed, this is an often requested feature on the armedbear-develop mailing list. Right after we published abcl-0.16.0, one of the first features to hit the trunk was the ability to load FASLs from JAR files, which we would like to explain a little bit about here.

N.B. you'll need to build ABCL from SVN trunk to use this feature.

ABCL has long had an extension to the semantics of Common Lisp PATHNAME that allowed one to specify entries of JAR files. Typing the following at an ABCL REPL:

CL-USER> (defvar *jar-entry* #p"jar:file:/home/evenson/foo.jar!/bar.abcl")

would create a reference in *JAR-ENTRY* to a PATHNAME that contained a reference in the path of the JAR in the DEVICE field, with the reminder of the PATHNAME referring to the actual JAR entry. I think these semantics were used from J to load extension functions, but they were currently unused in the ABCL codebase.

Now, *JAR* has the following parts:

CL-USER> (pathname-device *jar-entry*)

#P"/home/evenson/foo.jar"

CL-USER> (pathname-name *jar-entry*)

"foo"

Note that DEVICE is actually a reference to another PATHNAME at this point.

What we added in commit 12141 was the ability to load the FASL

CL-USER> (load *jar-entry*)

which would load the code compiled into the 'bar.abcl' FASL that was packed into the JAR. The ".abcl" extension is not strictly necessary: we check for entries ending in "*.abcl" and "*.lisp" for you in the same way LOAD works on the filesystem

What doesn't work yet is MERGE-PATHNAME with these special PATHNAMES. And looking for the JAR file on the current CLASSPATH. With these sort of conveniences it will be possible to include ASDF packaged systems in the JAR easily. Stay tuned!

ABCL 0.16.0 released

2009-09-06T03:20:00.000-07:00

On behalf of the developers of ABCL (Armed Bear Common Lisp) - a lisp implementation running on the JVM - I'm glad to be able to announce the 0.16.0 release.

This release features - among lots of other things - performance improvements,
better type checking for the THE form and ANSI tests fixes. Starting this release
JSR-223 support is delivered in the sources (this corrects an error in the procedure for earlier releases). You can find the release notes at:

http://common-lisp.net/project/armedbear/release-notes-0.16.shtml

If you have questions regarding its license or use, or you find issues, please report back to the development list:

armedbear-devel at common-lisp dot net

The sources can be downloaded in ZIP or .tar.gz form common-lisp.net:

http://common-lisp.net/project/armedbear/releases/abcl-src-0.16.0.tar.gz
http://common-lisp.net/project/armedbear/releases/abcl-src-0.16.0.zip

Signatures are available under:

http://common-lisp.net/project/armedbear/releases/abcl-src-0.16.0.tar.gz.asc
http://common-lisp.net/project/armedbear/releases/abcl-src-0.16.0.zip.asc

Type checking (or: safety belt on with compiler code)

2009-08-23T10:45:00.000-07:00

ABCL's compiler stores properties of blocks of code in a structure. Although it recognizes a number of different types of blocks (BLOCK, TAGBODY, etc.), there's only one structure type - called BLOCK-NODE.

While cleaning up this situation, separating the different block-node uses into different structures, I found ABCL didn't verify that the argument passed to the accessor functions for structure slots.

Cutting a long story short, we had to implement:

Structure type verification in the accessor functions
THE special operator type verification in the interpreter
THE special operator type verification for other policies than a *safety* value of 3 in the compiler

The first point being an issue with the accessor functions generated by ABCL: they didn't generate code to verify the argument passed. The effect being that a different structure with the same (or larger) number of slots could be passed in without an error occuring.

The second point being an issue that - even if there was a THE form - the interpreter would never verify the type specified as if it wasn't there. Talking to Peter Graves, I found that he had never intended the interpreter to be a full Common Lisp interpreter, meaning for ABCL to be a compiler-only system. The interpreter was merely there as a bootstrapping mechanism.
With all the energy spent last year to get it to the same level of CL conformance as the compiler, this point just had to be ironed out.

The third point being the issue that the compiler would treat THE as TRUELY-THE for any other *safety* value than 3. This is clearly not strict enough: it means no type verification takes place at all at any of these levels, while the user may expect some level of type verification for any level of *safety* other than zero.

Now, I can continue the reorganization of the compiler code with a safety-belt on: with the right *safety* setting, I know my structure types (and their changes) are being verified!

As a general benefit: this applies to all code running in ABCL, of course. Should you want to prevent type-verification (for example for speed reasons) in your code, just use a *safety* value of zero. In this case, the compiler simply assumes the type fits.

Efforts paying off

2009-08-10T07:46:00.001-07:00

Well, we've had a nice break on our self-set applicability target. As explained before, a Common Lisp implementation isn't of much use if it's not able to run much of the already-existing software available for the language. As an indicator we use ABCL's ability to run a number of software packages. Maxima is one of them.

Last autumn, ABCL wasn't able to complete the test suite delivered with Maxima. This spring ABCL would run it, but with many errors. Due to continued efforts - from both sides - and mainly the fact that Maxima changed their number comparisons to be CL-compliant (EQL), there are only 3 failures remaining - out of over 4500.

The remaining 3 failure clearly ABCL issues, where the outcome returned is of a lesser precision than expected by Maxima's test suite. So, even though we're not completely there yet, I'd say we're definitely usable with Maxima.

The note to add would be that our performance is less than acceptable with Maxima: a lot slower than any other Lisp implementation. The performance issue will also be addressed from both sides: we'll research how to improve performance (generally) on our side. One thing we know to be a performance issue with Maxima is their over-use of special variables. This is - according to Robert Dodier - something being addressed on their side.

With the specials over-use fixed, ABCL can be over 40% faster, as proven with a local hack to compensate for part of the over-use. Unfortunately, the hack can't make it into the ABCL repository as a general solution.

Work is under way though to improve repeated special binding access. Such repetitions occur if a loop uses a special variable as a looping variable, or if a special variable is used to collect results. The former is a use-case in Maxima's code; the latter is a pattern regularly seen in ABCL's compiler.

In ABCL's current code, each time a special variable is accessed, be it for reading or writing, the binding is looked up. The work focuses on reusing a binding after it has been looked up once. While in most cases this won't make a performance difference: the binding will be used only once, in some applications it may help improve performance quite a bit.

Beyond correctness and conformance: applicability and performance

2009-07-12T13:03:00.000-07:00

Last time was about how the team is working to advance ABCL's correctness and conformance; these two are mostly measured by the ANSI tests. Next to those characteristics, we also introduced a measure called 'applicability'. It's a qualitative measure which we say increases when (more) existing Lisp code is made to (or is proven to) work with ABCL.

So far, we've been using the following software to test (and improve) the applicability of ABCL:

Maxima
AP5
cl-bench
SLIME/swank

Next to the above we know a number of people are using ABCL to write their own applications. See the testimonials page. If you have software to complete the list of known-working software, please contact our developers list. If you have software which is almost working, please help us improve our applicability by at least reporting; patches to resolve the situation are even more appreciated.

Another aspect of our user experience is performance. Being an implementation on the JVM, of course we're restricted in many ways on which code we can generate and how we store our data. As a result we'll be in a disadvantageous position to start with.

That doesn't mean there's no room for improvement in ABCL at the moment. The development team has gradually moved into the area of profiling and improving the implementation over the last 2 or 3 months. As it turns out, there's still a lot which could be improved, with the following examples:

Support for unboxed single-floats and double-floats in the compiler
Careful creation of new objects - focussing on re-use - in the support library
Prevent unnecessary re-opening of system files (fasls and others)
Prevent unnecessary loading of compiled files (which clear out the HotSpot caches)

As you can see, more parts and more aspects of the system are receiving our attention.

ABCL actively developed

2009-07-12T00:28:00.001-07:00

Last year around August, Peter Graves asked me (Erik Huelsmann) to take over ABCL (Armedbear Common Lisp) development. I gladly accepted.

Since then we - a small team of lisp hackers - have been hard working to improve ABCL. Yesterday it was pointed out to me that the Lisp community was probably unaware of the renewed energy being put into ABCL's development.

Our efforts have concentrated on 3 areas of improvement:

Correctness - doing right what is already implemented
Conformance - implementing what the spec says should be
Features - things the spec doesn't require, but which are too handy not to provide standard

With respect to correctness and conformance we are glad to be able to say that the ANSI test failures have been drastically reduced to below 40 (coming from hundreds).

With respect to the features, we're proud of our achievements, which are two-fold: first of all we have added some Lisp features most implementations provide (

compiler improvements such as unboxed local variables
MACROEXPAND-ALL
COMPILER-LET and
locking primitives for threading
lots of other stuff;

second we added some features which matter for a Lisp in a Java environment:

JSR-223 support: being a scripting engine for any Java application
Threading primitives (different ones) which inter operate with the Java world
Improved Ant-based build system
more...

But we did more: ABCL was separated out of the source tree of the J editor (and moved to common-lisp.net). A wiki, mailing lists, new project pages, a repository and defect tracker have been set up. A bi-monthly release schedule has been put into place - source releases only for now and we're working toward binary releases too.

On this blog, I will discuss the advances and difficulties that make up the ABCL development experience.