On behalf of the developers of ABCL (Armed Bear Common Lisp) I'm glad to be able to announce the 0.17.0 release.
This release features - among lots of other things - performance improvements, a fix for unexpected thread termination due to uncaught exceptions and example code for running ABCL on Google App Engine. Please refer to the release notes for the full list.
If you have questions regarding use or licensing, or you find issues, please report back to the development list:
armedbear-devel at common-lisp dot net
Source distribution archives can be downloaded in ZIP (zip signature file) or gzipped tar (tar signature file) format. In addition, a ZIP binary (bin-zip sig file) and gzipped tar binary (bin-tar sig file) are available.
Saturday, November 7, 2009
Monday, October 26, 2009
ABCL on Google App Engine
Triggered by the interest of one of our users, last week was mostly dedicated to finding out how (how well) ABCL runs on Google App Engine (GAE). This is what we found out:
To those readers who don't know: GAE is an environment for hosting web applications, backed by Google's storage and server clouds. It supports running applications written in Python and Java.
GAE's Java environment turns out to be a servlet environment. This means there is a single servlet instance per JVM which gets a chance to initialize itself in an init() method. The first request waits for this method to complete before it's being processed. Google makes no guarantees regarding the number of JVMs your application might be running on concurrently, or the lifetime of a single JVM: when GAE needs memory to run other apps, your JVM might get torn down when not serving any requests.
Knowing the above, getting ABCL to run on GAE involved several steps:
Step (2) turned out to be a bit more involved, but after some twiddling, we found that we needed a minimum of 2 classes: at least one servlet class and a singleton class which loads a single ABCL into the JVM. [Note: a web application may contain any number of servlets with a minimum of 1.] The resulting application classes were committed to the ABCL repository.
From there, a minimal "Hello world" web app was easy, making step (3) a quick one. The end result was entirely committed to the examples directory in ABCL's repository.
GAE offers a performance dashboard to monitor your application through an administration web interface. From there, you can check the application logs, response times (called latency in the dashboard) and see how much CPU your application is using. For the latter, they use an indicative measure: the time which it would have taken to handle the request on an unloaded Core2 1.2Ghz processor. This compensates for many of the variances in the Google infrastructure which influence how long it actually takes to handle the request.
With a working application in place, the next step was performance. Most notably that of the first request: all subsequent requests are handled within miliseconds (7 to 15 miliseconds), so there's no issue there. This is the part that dominated last week: It turned out that although the latency was around 12 seconds, the CPU consumption was around 19secs [1], both very high and said to be close to some upper limit which remains unspecified.
We're striving to get these figures down: even though they would not really impact operation of a servlet in a regular hosting situation, GAE's regular servlet restarting makes these times more important. The best way to reduce figures like these is to get the figures the application scores on your local system down first. One of the first things which comes to mind is ABCL's "long" startup time: on my local Core2 - 1GB - 1.6Ghz machine it takes 1.7 seconds.
More on the steps we took to optimize this startup time in a next blog post.
Conclusion: ABCL - if you accept the long initial request response - is definitely an option for writing your web applications in Common Lisp on a Java/JVM based infrastructure. It'll even run on Google App Engine. We'll keep you posted on how we fare on supporting that even better!
[1] In comparison: On the Clojure mailing list, 5.5 seconds is mentioned for Clojure and on #appengine (on irc.freenode.net), 7 to 10 seconds are said to be normal for JVM based apps.
To those readers who don't know: GAE is an environment for hosting web applications, backed by Google's storage and server clouds. It supports running applications written in Python and Java.
GAE's Java environment turns out to be a servlet environment. This means there is a single servlet instance per JVM which gets a chance to initialize itself in an init() method. The first request waits for this method to complete before it's being processed. Google makes no guarantees regarding the number of JVMs your application might be running on concurrently, or the lifetime of a single JVM: when GAE needs memory to run other apps, your JVM might get torn down when not serving any requests.
Knowing the above, getting ABCL to run on GAE involved several steps:
- Getting the Java SDK for GAE (don't forget to get Java's SDK too!)
- Implementing Java classes wrapping ABCL
- Writing a minimal servlet in ABCL
Step (2) turned out to be a bit more involved, but after some twiddling, we found that we needed a minimum of 2 classes: at least one servlet class and a singleton class which loads a single ABCL into the JVM. [Note: a web application may contain any number of servlets with a minimum of 1.] The resulting application classes were committed to the ABCL repository.
From there, a minimal "Hello world" web app was easy, making step (3) a quick one. The end result was entirely committed to the examples directory in ABCL's repository.
GAE offers a performance dashboard to monitor your application through an administration web interface. From there, you can check the application logs, response times (called latency in the dashboard) and see how much CPU your application is using. For the latter, they use an indicative measure: the time which it would have taken to handle the request on an unloaded Core2 1.2Ghz processor. This compensates for many of the variances in the Google infrastructure which influence how long it actually takes to handle the request.
With a working application in place, the next step was performance. Most notably that of the first request: all subsequent requests are handled within miliseconds (7 to 15 miliseconds), so there's no issue there. This is the part that dominated last week: It turned out that although the latency was around 12 seconds, the CPU consumption was around 19secs [1], both very high and said to be close to some upper limit which remains unspecified.
We're striving to get these figures down: even though they would not really impact operation of a servlet in a regular hosting situation, GAE's regular servlet restarting makes these times more important. The best way to reduce figures like these is to get the figures the application scores on your local system down first. One of the first things which comes to mind is ABCL's "long" startup time: on my local Core2 - 1GB - 1.6Ghz machine it takes 1.7 seconds.
More on the steps we took to optimize this startup time in a next blog post.
Conclusion: ABCL - if you accept the long initial request response - is definitely an option for writing your web applications in Common Lisp on a Java/JVM based infrastructure. It'll even run on Google App Engine. We'll keep you posted on how we fare on supporting that even better!
[1] In comparison: On the Clojure mailing list, 5.5 seconds is mentioned for Clojure and on #appengine (on irc.freenode.net), 7 to 10 seconds are said to be normal for JVM based apps.
Saturday, October 17, 2009
ABCL 0.16.1 released
On behalf of the ABCL development team, I would like to announce the new ABCL release: 0.16.1
This release is a maintenance release on the 0.16 version line. In total four issues have been fixed:
Should you have any comments, remarks or bugs to be found in this release, please contact the developers through the mailing list mentioned on the ABCL project home page.
This release is a maintenance release on the 0.16 version line. In total four issues have been fixed:
- In case of unsuccessful completion of a COMPILE call, ABCL would leak (= not correctly clean up behind itself) temporary files
- In some cases, an exception thrown by a GO or RETURN-FROM form would not be correctly caught, causing the thread in which it occurs to terminate
- Honor appearance of &allow-other-keys in CLOS MAKE-INSTANCE argument checking
- More careful checking of 'null' arguments when processing stack frames
Should you have any comments, remarks or bugs to be found in this release, please contact the developers through the mailing list mentioned on the ABCL project home page.
Monday, October 12, 2009
CLOS performance
Traditionally, ABCL's CLOS implementation has been depicted as one of its weekest spots. The two main reasons being on one hand its speed and on the other the absense of a good MOP.
I'm proud to see that some of the improvements over the past year turn out to be real performance improvers. Ofcourse, there's more to wish for and improvements will keep coming in. However, another improvement of 90% as in the linked post... ?
As shown for a fact, ABCL's CLOS is improving and while possibly still it's weekest spot, it's definitely becoming useable. If you find performance issues, preferably with example code to show the issue, please report to the Armed Bear mailing list (address information on the project front page).
[Note added: Though 90% improvement is hard to achieve, another 30% (24 seconds to 16 seconds) were realised last weekend.]
I'm proud to see that some of the improvements over the past year turn out to be real performance improvers. Ofcourse, there's more to wish for and improvements will keep coming in. However, another improvement of 90% as in the linked post... ?
As shown for a fact, ABCL's CLOS is improving and while possibly still it's weekest spot, it's definitely becoming useable. If you find performance issues, preferably with example code to show the issue, please report to the Armed Bear mailing list (address information on the project front page).
[Note added: Though 90% improvement is hard to achieve, another 30% (24 seconds to 16 seconds) were realised last weekend.]
Tuesday, October 6, 2009
Struggling with and surviving TAGBODY and BLOCK compilation
At first glance, compilation of TAGBODY and BLOCK forms seems simple: the tags in the TAGBODY are all static and the exit point for the BLOCK is also well defined. Summarizing: no complexities with the dynamic environment of any kind.
(block NIL
(funcall (lambda () (return-from NIL 3)))
4)
Apart from the fact that the example is too obvious: the lambda form can be inlined, it demonstrates what I mean by "RETURN-FROM will cause a transfer of control to a non-local exit point": The exit point to jump to is located outside the lambda.
So far, so good: the above existed for a long time in ABCL and is achieved by raising Java exceptions. Now things get more contrived: we'll use a closure created inside a BLOCK form which is part of a recursively called function.
(defun foo (fun)
(block B
(if fun
(funcall fun)
(foo (lambda () (return-from B :good)))
:bad))
Now, evaluating (foo nil) will cause the following function calls:
1: (foo nil)
2: (foo (lambda () ...)
3: (funcall (lambda () ...)
To which exit point would you expect the lambda to return? You should expect it to return to the exit point associated with B block from the first FOO call. Now it's becoming clear what's so hairy about compiling BLOCK (and TAGBODY which has the same issue): there are 2 B blocks on the stack, so, which one to return to?
The way this last issue - recursive BLOCK and TAGBODY forms - was solved in ABCL just last weekend was to create a (hidden) variable which gets set to a certain unique value upon entry of the block at run time. Then, any non-local transfer of control uses that block identifier value to find the right block to jump to.
Actually, this solution has a slight advantage for TAGBODY over the pre-existing solution: the old solution checked all symbols in a TAGBODY before concluding the GO wasn't meant for the given tagbody. With the new approach, a single variable test (equality of object pointers) allows checking if the GO is meant for any given TAGBODY or that stack unwinding should continue.
But then there's another issue: closures can be assigned to variables which outlive the extent of the originating block or tagbody. Like the snippet below:
(progn
(block B
(setq a (lambda () (return-from B 3))))
(funcall a))
As indicated above, ABCL uses Java exceptions for non-local transfers of control. Now, if the (funcall a) form would throw a Go exception, regardless of the fact that there's no matching try/catch block, the exception would remain unhandled and the processing thread would exit. This is the situation in ABCL as it existed before last weekend.
Now, the solution to the problem with the recursive function calls has a nice additional benefit. Since there's storage shared between the closure and the BLOCK - they now share a variable - the variable can be used to let the block communicate to the closure that its extent has ended by setting it to a specific value. The GO form can then check for that condition before it throws the actual Go exception, making sure there's always a matching try/catch block. If there's no such block ABCL now generates a call to ERROR, allowing interactive error handling and selection of restarts where it used to unwind the stack to some location which happened to catch the exception - or terminate the thread if that didn't happen. Quite an improvement I'd say.
As a consequence of the changes described above, the code presented in the lisp paste at http://paste.lisp.org/display/
Tuesday, September 15, 2009
Loading FASLs from JAR files
Currently ABCL is a pretty decent decent Common Lisp that runs on the JVM, but we have really not started to add the necessary features which cater to making that "special" relationship easier. One of this rough sports involves how one packages ABCL applications for distribution. Since JAR files are currently the natural base unit for the distribution of JVM packages, it would make sense if one could load ABCL FASLs from JAR files. Indeed, this is an often requested feature on the armedbear-develop mailing list. Right after we published abcl-0.16.0, one of the first features to hit the trunk was the ability to load FASLs from JAR files, which we would like to explain a little bit about here.
N.B. you'll need to build ABCL from SVN trunk to use this feature.
ABCL has long had an extension to the semantics of Common Lisp PATHNAME that allowed one to specify entries of JAR files. Typing the following at an ABCL REPL:
CL-USER> (defvar *jar-entry* #p"jar:file:/home/evenson/foo.jar!/bar.abcl")
would create a reference in *JAR-ENTRY* to a PATHNAME that contained a reference in the path of the JAR in the DEVICE field, with the reminder of the PATHNAME referring to the actual JAR entry. I think these semantics were used from J to load extension functions, but they were currently unused in the ABCL codebase.
Now, *JAR* has the following parts:
CL-USER> (pathname-device *jar-entry*)
#P"/home/evenson/foo.jar"
CL-USER> (pathname-name *jar-entry*)
"foo"
Note that DEVICE is actually a reference to another PATHNAME at this point.
What we added in commit 12141 was the ability to load the FASL
CL-USER> (load *jar-entry*)
which would load the code compiled into the 'bar.abcl' FASL that was packed into the JAR. The ".abcl" extension is not strictly necessary: we check for entries ending in "*.abcl" and "*.lisp" for you in the same way LOAD works on the filesystem
What doesn't work yet is MERGE-PATHNAME with these special PATHNAMES. And looking for the JAR file on the current CLASSPATH. With these sort of conveniences it will be possible to include ASDF packaged systems in the JAR easily. Stay tuned!
N.B. you'll need to build ABCL from SVN trunk to use this feature.
ABCL has long had an extension to the semantics of Common Lisp PATHNAME that allowed one to specify entries of JAR files. Typing the following at an ABCL REPL:
CL-USER> (defvar *jar-entry* #p"jar:file:/home/evenson/foo.jar!/bar.abcl")
would create a reference in *JAR-ENTRY* to a PATHNAME that contained a reference in the path of the JAR in the DEVICE field, with the reminder of the PATHNAME referring to the actual JAR entry. I think these semantics were used from J to load extension functions, but they were currently unused in the ABCL codebase.
Now, *JAR* has the following parts:
CL-USER> (pathname-device *jar-entry*)
#P"/home/evenson/foo.jar"
CL-USER> (pathname-name *jar-entry*)
"foo"
Note that DEVICE is actually a reference to another PATHNAME at this point.
What we added in commit 12141 was the ability to load the FASL
CL-USER> (load *jar-entry*)
which would load the code compiled into the 'bar.abcl' FASL that was packed into the JAR. The ".abcl" extension is not strictly necessary: we check for entries ending in "*.abcl" and "*.lisp" for you in the same way LOAD works on the filesystem
What doesn't work yet is MERGE-PATHNAME with these special PATHNAMES. And looking for the JAR file on the current CLASSPATH. With these sort of conveniences it will be possible to include ASDF packaged systems in the JAR easily. Stay tuned!
Sunday, September 6, 2009
ABCL 0.16.0 released
On behalf of the developers of ABCL (Armed Bear Common Lisp) - a lisp implementation running on the JVM - I'm glad to be able to announce the 0.16.0 release.
This release features - among lots of other things - performance improvements,
better type checking for the THE form and ANSI tests fixes. Starting this release
JSR-223 support is delivered in the sources (this corrects an error in the procedure for earlier releases). You can find the release notes at:
http://common-lisp.net/project/armedbear/release-notes-0.16.shtml
If you have questions regarding its license or use, or you find issues, please report back to the development list:
armedbear-devel at common-lisp dot net
The sources can be downloaded in ZIP or .tar.gz form common-lisp.net:
http://common-lisp.net/project/armedbear/releases/abcl-src-0.16.0.tar.gz
http://common-lisp.net/project/armedbear/releases/abcl-src-0.16.0.zip
Signatures are available under:
http://common-lisp.net/project/armedbear/releases/abcl-src-0.16.0.tar.gz.asc
http://common-lisp.net/project/armedbear/releases/abcl-src-0.16.0.zip.asc
This release features - among lots of other things - performance improvements,
better type checking for the THE form and ANSI tests fixes. Starting this release
JSR-223 support is delivered in the sources (this corrects an error in the procedure for earlier releases). You can find the release notes at:
http://common-lisp.net/
If you have questions regarding its license or use, or you find issues, please report back to the development list:
armedbear-devel at common-lisp dot net
The sources can be downloaded in ZIP or .tar.gz form common-lisp.net:
http://common-lisp.net/
http://common-lisp.net/
Signatures are available under:
http://common-lisp.net/
http://common-lisp.net/
Subscribe to:
Posts (Atom)