Common Lisp's library problem

People complain a lot about Common Lisp libraries. In my opinion, if we want to get more people using Common Lisp, the place where we can make the most difference, and have the highest value for the lowest cost, is by improving the problems with libraries.

Less-sophisticated people or beginners tend to say "Oh, there aren't any libraries". Of course, this is not actually true. But there are more sophisticated criticisms. I'll list below the ones I know about. I will use somewhat exaggerated versions, just for brevity, rather than qualify all my staements to get better accuracy, at least for now. But I think these are all valid complaints.

  1. I don't know where to find them: The free Common Lisp implementations are not, in the current phase, "batteries included". You get a language runtime, a compiler, some documentation, ASDF and some other good things. But you don't get a comprehensive set of libraries, and you usually don't get told where to find them.
  2. They're all over: Despite the fact that several groups have put together sites with collections of useful libraries, there are too many such sites, and it takes too long to find them. Also, they aren't indexed well enough, so it's hard to find what you want, even if it's there.
  3. They aren't portable: Some libraries work on some implementations. Most do not run on all implementations. Few have been tested on all (or even a majority) of implementations. There are 11 maintained implementations to try. Many of those behave differently on different operating systems (particularly as regards threads), so sometimes you have to check even more environments, if you want your program to port between implementations.
  4. They aren't documented well: Of course this varies widely, but it's often true.
  5. Dependencies are complicated: Many libraries depend on other libraries, and so in order to use one, you must find and install the others. Not all are stored on hosts that are highly-available. It's not always clear what versions of the dependent libraries you need.
  6. They aren't comprehensive: Many libraries don't implement enough features. For example, Araneida, a good HTTP server, is missing important abilities. The other leading candidate, Hunchentoot, is much better this way. But how is a beginner to know that?
  7. You can't tell which are the good ones: There are often multiple libraries for the same purpose, and it's hard to know which one to pick. Which is better tested? Which is being actively maintained? If you're just trying to get a project done, you don't have time to do a lot of testing and research.
  8. There are too few libraries: Despite that there are lot more libraries than most people think, it's true that it's hard to keep up with more popular languages. There are certainly a whole lot more libraries available for Java (for the JVM, actually), with more being created every day by a large community, many of whom are paid to do this. Can the Common Lisp community ever keep up?

Now, here are some answers to these points. However, none of these answers is so powerful that we can say there's no problem at all.

  1. (Where to find them.) My survey paper points to pretty much every major collection of libraries. If we can manage to publicize it to new users, that would help. This is one of the things We should do with the lisp.org site.
  2. (They're all over.)The Cliki repository has the best index I've seen, althogh I think it could be made even better. However, the Common-Lisp.net repository just has an alphabetical list of names of libraries, with no explanation. We need a central repository that would be, for Common Lisp, what CPAN is for Perl. I will discuss this in a separate topic.
  3. (Portablilty.)Not everybody needs portability. But it does come up. One of the reasons that the well-known Reddit site converted from Common Lisp to Python was that the main developer used different platforms for development and deployment, and had trouble running under both. That was in 2005; nowadays there are a lot of good portability libraries, such as usocket. But new Lisp people don't know that nor have an easy way to find out. (A "with batteries" implementation would do well to include usocket, in my opinion.)
  4. (Documention.) Documentation is hard to write. Some excellent developers are also excellent technical writers, but not all of them. I think many Common Lisp programmers don't take the trouble, perhaps figuring that the only people interested are probably such good Common Lisp developers that they can just figure it out from the source. That's not a good assumption if we want to bring more Common Lisp programmers into the fold.
  5. (Dependencies) The dependency problem arises with a lot of free/open software, not just Common Lisp. I've run into exactly the same problem with Java libraries (a security package that depended on version 3 of a crypto package, but only version 4 was available!). Still, saying "everybody else is just as bad" is hardly an excuse. Getting this right is not that easy, but not that difficult.
  6. (Comprehensive) Why aren't they? I think some people write a library for their own uses, and don't spend a lot of time on the features that they don't care about.
  7. (Which are good?) There needs to be a way to learn about the quality of libraries and whether they are being actively maintained. There's no consistent way to do that; there's often no feasbile way at all.
  8. (Too few) There is now a portable, high-quality common FFI (foreign function interface), allowing Lisp to take advantage of some libraries written in other languages. However, libraries written in Java or C# are not so easy to use from the FFI.

There are some underlying causes behind all of these problems. Here are some that I know about and think are significant:

  • There are few (perhaps none) developers who are paid to produce free/open-source Common Lisp libraries. Many other languages have corporate support for open-source development.
  • Common Lisp does not seem to have a robust open source community of people developing libraries on their own time, without being paid. Of course some people shine in this area, but there are too few of them, and I think there isn't a "culture" of doing this, in some sense.
  • There's the vicious-cycle, bootstrapping problem, even after so many years of Lisp and Common Lisp: Users shy away from a language with too few libraries, and developers are less motivated to develop libraries for a smaller user community.

Improving these things is relatively easy, because we don't need to get all the implementors to agree, or all the implementations to change, which solving some other problems requires. And most of the work can be done gradually and many people can help.

What else can we do about these issues? I would love to hear your ideas and suggestions!

Quicklisp

You can take a lot of the hassle out of getting Common Lisp libraries with Quicklisp. I think it's safe to say it's the de facto library management system for CL at the moment.

Reverse approach: drop CL, adopt PLT

Instead of trying to fix Common Lisp so it becomes easier to layer libraries on top of it, and then still have to create and document all the missing libraries, why not take the opposite stance?

Adopt PLT Scheme and its superb module system and large set of libraries, and fix whatever problems there may be left with it.

Really, the only feature I've seen boasted by CL proponents for which PLT doesn't have a better solution is an optimizing compiler with static-type-directed optimizations that can rival with SBCL and decent multithreading support. A new backend could be added to PLT.

If you love CL, you might even create a front-end that accepts full Common Lisp syntax and/or semantics to leverage legacy code. And because PLT is so flexible, you might even simultaneously have many language levels and variants, that do away with various CL inadequacies to leverage code that doesn't rely on them while providing better alternatives.

Then you'll have the greatest Lisp system ever made.

I don't think this addresses the whole problem

Obviously, not everybody is going to abandon Common Lisp. In just about all of the issues that I bring up, though, there are two categories of answer: what to do about Common Lisp as it exists, versus what to do in the greater frame of Lisp-family languages. So it's fine to discuss this.

You mention first the module system. I didn't list that as one of the problems, in my original post. Although I understand the drawbacks of the CL package system and have suffered with them for years, I don't actually think that this is as important a problem as the eight that I listed.

You say that PLT Scheme has a large set of libraries. I don't know how to judge that against the set of libraries available in Common Lisp. But even if there are, that only addresses point 8 in my original mail.

I wonder if there might be opportunities to take existing PLT Scheme libraries, and translate them into Common Lisp?

Saying that we should adopt PLT Scheme and then "fix whatever problems there may be left" and write a new back-end that generates good code and add threading may or may not be easier than fixing the eight problems I've listed.

Yeah, but...

Adopt PLT Scheme and its superb module system and large set of libraries, and fix whatever problems there may be left with it.

This proposal raises the obvious question: What are those problems? Dan's points refer to specific and known issues. This idea seems to be raising many unknowns.

Also, having a large set of libraries doesn't imply solving all the library problems Dan mentioned. (As someone who has to use Java for employment reasons, I can attest to this problem.)

A new backend could be added to PLT.

How much work would this be? Perhaps "if you build it, they will come", but I'd be hard pressed to jump to using PLT Scheme until this was actually written and heavily tested.

As much as I like PLT Scheme, the proposed idea sounds like "if Lispers would only take up Scheme", which has never really panned out before. Nevertheless, it's given me something to think about.