Common Lisp's library problem
People complain a lot about Common Lisp libraries. In my opinion, if we want to get more people using Common Lisp, the place where we can make the most difference, and have the highest value for the lowest cost, is by improving the problems with libraries.
Less-sophisticated people or beginners tend to say "Oh, there aren't any libraries". Of course, this is not actually true. But there are more sophisticated criticisms. I'll list below the ones I know about. I will use somewhat exaggerated versions, just for brevity, rather than qualify all my staements to get better accuracy, at least for now. But I think these are all valid complaints.
- I don't know where to find them: The free Common Lisp implementations are not, in the current phase, "batteries included". You get a language runtime, a compiler, some documentation, ASDF and some other good things. But you don't get a comprehensive set of libraries, and you usually don't get told where to find them.
- They're all over: Despite the fact that several groups have put together sites with collections of useful libraries, there are too many such sites, and it takes too long to find them. Also, they aren't indexed well enough, so it's hard to find what you want, even if it's there.
- They aren't portable: Some libraries work on some implementations. Most do not run on all implementations. Few have been tested on all (or even a majority) of implementations. There are 11 maintained implementations to try. Many of those behave differently on different operating systems (particularly as regards threads), so sometimes you have to check even more environments, if you want your program to port between implementations.
- They aren't documented well: Of course this varies widely, but it's often true.
- Dependencies are complicated: Many libraries depend on other libraries, and so in order to use one, you must find and install the others. Not all are stored on hosts that are highly-available. It's not always clear what versions of the dependent libraries you need.
- They aren't comprehensive: Many libraries don't implement enough features. For example, Araneida, a good HTTP server, is missing important abilities. The other leading candidate, Hunchentoot, is much better this way. But how is a beginner to know that?
- You can't tell which are the good ones: There are often multiple libraries for the same purpose, and it's hard to know which one to pick. Which is better tested? Which is being actively maintained? If you're just trying to get a project done, you don't have time to do a lot of testing and research.
- There are too few libraries: Despite that there are lot more libraries than most people think, it's true that it's hard to keep up with more popular languages. There are certainly a whole lot more libraries available for Java (for the JVM, actually), with more being created every day by a large community, many of whom are paid to do this. Can the Common Lisp community ever keep up?
Now, here are some answers to these points. However, none of these answers is so powerful that we can say there's no problem at all.
- (Where to find them.) My survey paper points to pretty much every major collection of libraries. If we can manage to publicize it to new users, that would help. This is one of the things We should do with the lisp.org site.
- (They're all over.)The Cliki repository has the best index I've seen, althogh I think it could be made even better. However, the Common-Lisp.net repository just has an alphabetical list of names of libraries, with no explanation. We need a central repository that would be, for Common Lisp, what CPAN is for Perl. I will discuss this in a separate topic.
- (Portablilty.)Not everybody needs portability. But it does come up. One of the reasons that the well-known Reddit site converted from Common Lisp to Python was that the main developer used different platforms for development and deployment, and had trouble running under both. That was in 2005; nowadays there are a lot of good portability libraries, such as usocket. But new Lisp people don't know that nor have an easy way to find out. (A "with batteries" implementation would do well to include usocket, in my opinion.)
- (Documention.) Documentation is hard to write. Some excellent developers are also excellent technical writers, but not all of them. I think many Common Lisp programmers don't take the trouble, perhaps figuring that the only people interested are probably such good Common Lisp developers that they can just figure it out from the source. That's not a good assumption if we want to bring more Common Lisp programmers into the fold.
- (Dependencies) The dependency problem arises with a lot of free/open software, not just Common Lisp. I've run into exactly the same problem with Java libraries (a security package that depended on version 3 of a crypto package, but only version 4 was available!). Still, saying "everybody else is just as bad" is hardly an excuse. Getting this right is not that easy, but not that difficult.
- (Comprehensive) Why aren't they? I think some people write a library for their own uses, and don't spend a lot of time on the features that they don't care about.
- (Which are good?) There needs to be a way to learn about the quality of libraries and whether they are being actively maintained. There's no consistent way to do that; there's often no feasbile way at all.
- (Too few) There is now a portable, high-quality common FFI (foreign function interface), allowing Lisp to take advantage of some libraries written in other languages. However, libraries written in Java or C# are not so easy to use from the FFI.
There are some underlying causes behind all of these problems. Here are some that I know about and think are significant:
- There are few (perhaps none) developers who are paid to produce free/open-source Common Lisp libraries. Many other languages have corporate support for open-source development.
- Common Lisp does not seem to have a robust open source community of people developing libraries on their own time, without being paid. Of course some people shine in this area, but there are too few of them, and I think there isn't a "culture" of doing this, in some sense.
- There's the vicious-cycle, bootstrapping problem, even after so many years of Lisp and Common Lisp: Users shy away from a language with too few libraries, and developers are less motivated to develop libraries for a smaller user community.
Improving these things is relatively easy, because we don't need to get all the implementors to agree, or all the implementations to change, which solving some other problems requires. And most of the work can be done gradually and many people can help.
What else can we do about these issues? I would love to hear your ideas and suggestions!