ABCL hash tables: threading and CLOS

More people are starting to use ABCL in threaded environments - I have been doing so myself since spring 2008. My own use revealed some threading issues in the compiler which have been long solved now.

Last April, David Kirkman mailed the list with some threading problems in our CLOS implementation. Because of other work - including implementation of METACLASS support - there wasn't much time to do much about the issue.

This week, he mailed having found the problem - accompanied with a patch with a solution. Because of our need to support the four Common Lisp equality operators, ABCL implements its own hash tables -four of them - with a shared ancestor class which requires implementation of a few primitive operations. Hash access from the Lisp world was being synchronized by the common ancestor. However, in some locations on the Java side the - unsynchronized - primitives were being called directly.

The solution was of course to add synchronization to the primitives as well. Evaluating the result, the new situation was rather unsatisfying: only a single thread could be reading or writing at the same time, meaning only a single CLOS effective method dispatch could be happening at the same time. You probably understand my reluctance to accept the status quo.

The quick solution to the CLOS dispatch problem was to use the java.util.concurrent.ConcurrentHashMap type: we were keying on symbols, which have EQ equility in the Java world (ie in terms of Object.equals()).

That still left the lisp world with rather unsatisfactory threading performance of our own hash tables and since those were repeating large chunks of code in their specialized primitives, I decided to refactor them completely.

The result is - admittedly by looking very closely at ConcurrentHashTable - a single hash table implementation with 4 Comparator classes, which have nearly-lockless reads from all threads. Inserts into the hash table are still synchronized from all threads, but that's expected to be a lesser problem: you'd primarily expect many reads per write in a hash table.

Concluding, the only (known) CLOS threading issue has been fixed - there are presumably many more, please report if you find them - and as a bonus we have a much more efficient hash table implementation.