A Future History of Arming Bears

As part of the ongoing Online Lisp Meeting series, a recording of a talk about ABCL with the following precis is now available online:

With the recent releases of Armed Bear Common Lisp over the past six months, the future of extending the implementation has come into sharper focus.  The majority of this work has occurred within the head of one individual with little chance for public review and reflection. We believe that our externalized exposition of the reasoning behind these efforts will be of interest to those interested in the future history of Common Lisp implementations.

In the past few months, we released abcl-1.6.0 in which we extended the set of underlying Java Virtual Machines (JVM) that the implementation runs on to include openjdk11 and openjdk14 while maintaining compatibilty with openjdk6.  With the internal overhaul or arrays specialized on unsigned bytes in abcl-1.7.0, we made it possible to share such byte vectors with memory allocated outside of the hosting JVM via system interfaces such as malloc().  

We first present a brief prehistory on the Armed Bear Common Lisp Implementation.  Then, we first present the goals and challenges in affecting these changes within the ABCL codebase by showing examples from recent experience.  Then, we use this initial exposition to serve as a springboard to discuss outstanding needed changes in the ABCL 1 branch, and to outline some of the features intended to be present in ABCL 2, due to be released in the Fall of 2020.

A revised version of the notes for the presentation is available for deeper, asynchronous reflection. 

The Bear Arms for Sharing Byte Vectors with 1.7.0

We are pleased to announce the immediate availability of the ABCL
1.7.0 release.

After consuming a steady diet of java.nio.ByteBuffer objects over the
past month, the Bear has managed to incorporate the use of these
abstractions for arrays specialized on the commonly used unsigned-byte
types (or (unsigned-byte 8) (unsigned-byte 16) (unsigned-byte 32)).
This replacement of the use arrays of primitive bytes is denoted by
the presence of the :NIO keyword in CL:*FEATURES*.

With this :NIO overhaul, we have extended our implementation of ANSI
Common Lisp CL:MAKE-ARRAY with two additional keywords,
viz. :NIO-BUFFER and :NIO-DIRECT.

Now, the :NIO-BUFFER keyword argument to CL:MAKE-ARRAY allows one to construct a vector directly utilizing the contents of an already allocated
java.nio.ByteBuffer object.  When combined with the ability of JNA to
allocate memory on the heap via a malloc() system call, we implemented
shareable byte vectors in CFFI-SYS:MAKE-SHAREABLE-BYTE-VECTOR.

    (let* ((length 16)
           (byte-buffer (java:jstatic "allocate"
                                      "java.nio.ByteBuffer" length)))
      (make-array length :element-type ’(unsigned-byte 8) 
                         :nio-buffer byte-buffer))

When the :NIO-DIRECT keyword argument is called with a non-NIL value,
the implementation creates a byte vector with a "directly allocated"
java.nio.ByteBuffer object.  Such direct buffers typically have
somewhat higher allocation and deallocation costs than non-direct
buffers.  The contents of direct buffers may reside outside of the
normal garbage-collected heap, and so their impact upon the memory
footprint of an application might not be obvious. It is therefore
recommended that direct buffers be allocated primarily for large,
long-lived buffers that are subject to the underlying system’s native
I/O operations.  In general it is best to allocate direct buffers only
when they yield a measurable gain in program performance. In the near
future, we intend to explore the performance gains available CL:LOAD
by accessing direct buffers memory mapped to our on-disk fasl
representation.  Our fasls, as zipped archives, currently require a
new seek() from the beginning for each component they
contain.  With a memory mapped direct buffer we should be able to
simply read from the appropriate byte offset for each component.

A complete overview of the accumulated fixes and changes since the
previous release may be viewed in the file describing our CHANGES.