Fundamental problems with the Common Lisp language
Common Lisp has some fundamental problems. (There are loads of little problems, but I think it's more interesting to think about the systemic and big issues.)
Here are some that I know about or have heard discussed by knowledgable people:
- Too many concepts; irregular: A language should be intellectually concise, with regularity, and a disciplined approach to new language constructs, in order to make it feasible for a programmer to "get his or her head around it". Common Lisp has too many fundamental concepts that aren't really needed (property lists versus alists is a simple example). In general, it was a design-by-committee that was a superset of all the recent ideas that had been implemented in the parent dialects, plus compatibility with everything that had gone before.
- Hard to compile efficiently: It's too hard to write a high-quality Common Lisp compiler that generates very efficient code. Certainly experience to date has validated this: not that it's impossible, but that it's very hard. Truth be told, SBCL and Clozure CL, both considered excellent implementations, do not generate very good code. To be sure, partly this is because generating excellent code has not been one of the things most Common Lisp users need as much as other language improvements, and so this has not been given the highest priority. Nevertheless, I think the maintainers of those implementations would agree that writing a compiler that generates excellent code is very hard. And ideally, the programmer should not have to write a lot of type declarations to get this efficient code, Lisp claiming to be a dynamically-typed language, although it is hard to produce competitive code for, say, addition, when runtime type dispatching is needed.
Furthermore, at the time Common Lisp was designed, it was assumed that many common use cases and patterns could and would be optimized by a "sufficiently clever compiler". (I'm talking here about higher-level optimizations than peephole optimizers. For example, a "reduce" can be compiled more efficiently in certain patterns, and many sequence functions can be compiled to better code if you know at compile time that certain keyword arguments aren't being used.) Unfortunately, nobody demonstrated such a compiler at the time, and it turned out to be far harder to do than the designers had anticipated. (The phase "sufficiently clever compiler" is used only in an ironic sense, these days!) There was a hope that a common, portable front-end of a compiler would be created and made available to everybody, that would do these optimizations, but that never happened.
-
- Too much memory: Small programs should run in a small amount of physical memory (working set). They should not have poor paging performance, and they should start up very quickly. Ideally, Common Lisp compiled code should be just as viable for writing little scripts as PERL is.
- Unimportant features that are hard to implement: There are some features in Common Lisp that were introduced by the Lisp Machine project at MIT, that are hard to implement efficiently without special support at the hardware/microcode level, and that aren't really that important, such as displaced arrays. It's not impossible to implement them; the software-only version of Genera does it pretty well, and all the Common Lisp implementations can do it, but it can easily slow down array access slightly in the general case. It's not worth having such a little-used feature if it's going to slow down the common case.
- Not portable: The Common Lisp standard is full of cases where the behavior is defined to be "implementation dependent". This makes it harder to write portable code. Compare this with the Java Language Specification, which never says that: every case is defined. As a result, Java code (at least server-side) is extremely portable. (It's really true. I used to work at BEA and nearly zero effort was spent on porting Java code, to a very wide range of platforms.) This was done because existing implementations were hard to change, or because the definers of the language could not reach consensus. Also, there's just carelessness and ambiguity. For example, does "subst" allow partial structure sharing or demand a complete copy? Exactly what code does a backquoted form expand into (sometimes it does matter).
- Archaic naming: Common Lisp is full of archaic names like car, cddr, assq, setq, and progn, making it harder to learn.
- Not uniformly object-oriented: Although Common Lisp has object-oriented programming, it's not built into the basic runtime functions. For example, you cannot define your own kind of sequence that will be accepted by the sequence functions, or you own kind of stream, or your own kind of array. (Yes, I know about libraries for writing streams, but those aren't in the standard, and that's only an easy example of the general point.)
These are issues about the Common Lisp language and its implementations. Discussions of libraries, social issues, etc. will be discussed in separate forum topics, in order to keep things focused.
What am I leaving out?
Is there anything we can do about any of this?
|
How could we implement the necessary changes!
Hello All,
As f_lynx put it, the problem is now to implement the changes. There is 2 solutions (sorry, I will repeat what as been said already). Write something from scratch: Paul Graham and Arc for example. Or redefine the Common Lisp standard to round up the rough edges (Modules, Object model, distributivity, multi-processors, threads, net (TCP, web...), GUI, language bindings (C, SQL, and why not Ada and Prolog)... and more specialized details that Lisp pro are aware of).
As many specified in the thread, the new standard should not leave any ambiguities: the standard should specify explicitly how to do it, so that the code would not be dependent of a specific CL implementation. Think about a student learning Lisp with a GNU-style compiler. After graduation, he finds a job in an enterprise that uses one of the major commercial Lisp. He will be able to use his previous work, libraries... immediately without having to memorize all the little differences between implementations (LispWorks and Franz should participate in this process too). That is a way, I think, to build a solid base.
Now, one way to implement changes would be to follow the example of the BSDs (how to do it) and Erlang or Mozart-Oz, even if the syntax of the latter is more ALGOl-like (good documentation, nice implementation, good GUI-see gs module). I thing the BSD way fits well the Lisp hacking style (a hacker is not a cracker...). A small group of Lisp aficionados charged to implement the changes, while listening to the Lisp community suggestions, but that will have the last word about the implementation (this implies that once the group has received the green light from the Lisp community, it has to commit to their choice). Now the problem is to find these people: enough free time to do it (maybe have a site that offers the possibility to give donations to compensate them for the time they spend on the project), a schedule to implement the changes, enough people in the community willing to test the changes an give polite and constructive feedback.
All the ingredients are here. I thing that the topic of the next meeting in 2011 should be the implementation of such a group.
My background is biology, not programming, So my category will be in the testing part. But I can tell you that after going through some imperative programming languages (C, Pascal, Ada, Eiffel, Oberon...), the discovery of declarative languages (Prolog, Erlang, Lisp, "Mozart-Oz") was an eye opener. Lisp parentheses are not as terrible than many } or "end". These languages are actually easy to learn and more to the point for somebody out of the field like a biologist. My first introduction to programming at the university was through Pascal (simulating how to move a robot around a room), utterly useless. With Prolog and Lisp, a biologist for example can build knowledge base systems to test theoretical systems (Prolog and Lisp) or build more applicative system the way you would do it with C (Lisp). This to say that potentially, the Lisp community can include many type of people besides the lisp pro. The potential is here, NewLisp has a nice programming GUI that is a good model for people not comfortable with emacs.
To finish, there an excellent GUI to build graphic interfaces: XPCE (see http://www.swi-prolog.org/packages/xpce/ ). It has been conceived with declarative language in mind and has been used in the beginning (more like mid 90s) by the major commercial Lisps implementations (Lucid Common Lisp and LispWorks, see this document also: http://www.swi-prolog.org/download/xpce/doc/lisp/). For some reason, it has not been used by other Prolog implementations (they use tk instead for most) and LispWorks stopped using it. I am trying to learn why, and will post the answer if there is any interest. But have a look at it, the people at swi-prolog are open minded and it would be easy to collaborate with them.
hope all this is not redundant,
Hans Meyer.
Two easy(?) partial fixes
It seems non-controversial to say that CL has arcane features and naming. How hard would it be to get the community to agree on a list of such features and just deprecate them? Call the new standard CL 1.1. It would be backwards compatible in that CL 1.1 code could run on CL 1.0 systems. Though, I don't know how hard it would be to have a utility to convert 1.0 code to 1.1. Some obscure feature replacements could require elaborate refactoring.
The other big complaint that could be easy to fix is code readability. How to make CL as easily scanable as Python? I'm inspired in this by the approach outlined by Damian Conway in his book _Perl Best Practices_ and implemented in Perl::Critic. To me, it demonstrates how even a messy language like Perl can be made more congenial to work with. I've experimented a little with alternate formatting of CL code and I think it's possible to make it more readable without going to extreme lengths such as changing over to infix syntax.
How to get agreement, and who has time?
My impression is that everybody has been assuming that it would be infeasible to get agreement from all relevant parties. The parties in question are, I think, the implementers of the Lisp implementations. I am not criticizing them; they have a hard problem here.
First, what mechanism is used to get everyone to come to a consensus? If you have been through a language standardization process, you will know how hard this is. For the original CLTL, we had advantages: (1) a limited number of participants, who knew each other and had a lot of mutual trust and respect, (2) no formal standards organization imposing painful rules, and (3) Guy Steele, with his amazing diplomatic skills, his amazingly high-quality writing, and his willingness to do quite a lot of hard work. The later X3J13 process had none of these, and was much more painful and, I think, less effective.
Second, the implementers do not have a lot of spare resources. Many implementations are done part-time by only a few people. Others are done by bigger teams but they usually have many other demands on their time. And remember that there are eleven of them!
One good thing is that a lot of the changes we're talking about could be implemented in a portable library, which would reduce the amount of actually work that the implementers would need to do.
Problems with optimizing Common Lisp code
http://jng.imagine27.com/articles/2009-04-07-135246_cl_hard_to_compile_e...
the next step...
IMO the problems are identified and discussed enough. what should be done now is an attempt to solve them.
The problem is the problem.
I think the problem statement should be added to that list too. Isn't macros the solution to all problems? *provocative*
Irregularity not considered harmful
A language should be intellectually concise, with regularity, and a disciplined approach to new language constructs, in order to make it feasible for a programmer to "get his or her head around it".
And the intellectual concision behind perl, python and PHP is? Behind the (market-driven) evolution of C# and Java? And yet orders of magnitude more programmers wrap their heads around these than (bother with) Lisp. Perhaps a language "should" have this property, but it can't possibly be for the reason you cite.
Actually it is harmful
Ruby has a significant following, which it has benefited from. However, Ruby has also been hurt by irregularities in its parsing. This has significantly slowed down the development of language tools and virtual machines.
It's not that irregularity is not harmful, it's that a developer ecosystem is such a powerful asset, it can compensate for the harm irregularity brings.
What the past two decades has shown us is that the next hot language should have *both* a disciplined approach to new language constructs and support for market-driven evolution.
not fundamental problems with the Common Lisp language
Too much memory?!? What planet are you living on, Dan?
Archaic naming? True, but I am not convinced this makes it hard to learn. If all the naming was uniform, it would be a maze of twisty little passages all alike, which would not be conducive to learning. By the way, there is no assq in Common Lisp.
Difficulty of learning probably really comes from too many partially redundant concepts, lack of orthogonality, not enough good documentation, not enough really well designed libraries, and to some extent no doubt from the unconventional syntax.
Scripting, archaisms
OK, I should not have put it in terms of memory. The real point is that it should be possible to write programs that start up fast. In fact, I've been having a whole conversation on another blog about this. See
http://www.newartisans.com/2009/03/the-jvm-and-costs-vs-benefits.html
I think you're playing word games with "uniform". I'm sure you see why the naming in Dylan is better than the naming in Common Lisp. I mean that it could be more consistent. I admit that this isn't as important as the other factors.
archaic is not an issue with naming
phalanx, piston, petroleum
All three predate CDR by a fair margin. They are technical terms that people learned to accept. Similarly, CAR and CDR and C{AD}[4]R embody a simple concept and are fairly easily learned (though intro CS courses would do good to only teach CAR and CDR). Likewise with PROG1, PROG2, and PROGN (but why weren't they generalized?).
If I got to choose CL names to pick on, the list would start with MAPC, MAPCAR, MAPCAN, MAPL, MAPLIST, and MAPCON. Did nobody see the problem, the almost-but-not-quite system? Also PROGV for not fitting in with the rest. Talk about twisty passages.
Compare the following
mapc map-car
mapl map-cdr
mapcar map-car-cons
maplist map-cdr-cons
mapcan map-car-nconc
mapcon map-cdr-nconc
The first column contains obscure CL names. The second, equally obscure but much easier to learn names.
(map-x-y f &rest lists): apply F to each X of each list, storing the results using Y...
Anyone want to play (defpackage "CL2k")?
Map being unnecessarily list-specific belongs in another thread.
MAP list-specific?
Re: "Map being unnecessarily list-specific belongs in another thread."
In my book, MAP and MAP-INTO work on sequences. Did I misunderstand you?
Re: MAP lisp-specific?
Ok. MAP and friends aren't purely list-specific. But it seems to me that we should also be able to define ways of mapping over non-sequence types without resorting to heavyweight constructs such as LOOP. Stuff like
(mapc #'print #(1 2 3))
(let ((tab (make-hash-table)))
(setf (gethash 1 tab) 1
(gethash 3 tab) 2
(gethash 3 tab) 3)
(mapc #'print tab))
I'd have been surprised if the hashtable worked, but when learning lisp I was disappointed that map couldn't handle simple vectors.
Maybe defining extensible sequences (see Christophe's ILC07 talk) and/or iterators is the right way to solve this. FWIW, these approaches are well supported in other programming languages (e.g. C++, Java, Clojure).
Using extensible sequences, the above examples would be rewritten something like
(mapc #'print (as-sequence #(1 2 3)))
(let ((tab (make-hash-table)))
(setf (gethash 1 tab) 1
(gethash 3 tab) 2
(gethash 3 tab) 3)
(mapc #'print (as-sequence tab)))
If other implementations would adopt/improve on SBCL's user-extensible sequences, maybe container libraries like FSet wouldn't have to struggle with algorithm name clashes. Its implemented in sbcl/src/pcl/sequence.lisp.
Or I could just define MAP as my own generic functions. Nope; need congruent argument lists...
Root causes of problems with Common Lisp
Most, maybe all, of the important problems with Common Lisp derive from three root causes (just in my opinion, of course):
- the goal of Common Lisp was not to create a great language, rather to assemble something that all the players could agree on. Like legislation, the result of the ugly process contained something for each observer to like and plenty to dislike.
- almost all of Common Lisp was designed before object-oriented programming was added. Thus most of the language does not enjoy the benefits one would expect from the use of objects.
- economics preclude a major investment in improvements to the language. There is no actor with deep pockets who cares about Common Lisp, and has not been for twenty years.
On the other hand, it is useful. We wouldn't want to make the perfect the enemy of the good. So I don't know what to suggest.
Bring them together: Scientific institutions and OSC !
You wrote: "There is no actor with deep pockets who cares about Common Lisp, and has not been for twenty years."
I myself often thought "There is no (Dylan) project that can be send to, for example, the german Klaus Tschira Foundation (KTF) . KTF’s funding practice is funding can only be provided for scientific institutions, not for individuals.
Actually I looking forward to hear that there are several scientific institutions which are interested in a NextLisp (and NextDylan) around the world.
Maybe the ILC 2009 can
Your reputation can be an invaluable start-up capital for any funding for a NextLisp. Either talk to professionals who are working at scientific institutions or write a nice letter to Steve Jobs. Just do it with having fun.
NextLisp and Scientific Computing
In your announce to the ILC you use the phrase „What would make you want to come to such a conference?“. Let me rephrase that to „What would you expect such a conference can deliver to you?“. Before starting to answer let me describe my perspective from which I write about my exspectations.
Back in 1998 an article of the german computer magazine „Objekt Spektrum“ attracted me to Dylan. I’m not a computer scientist, but like to dive into free computer literature in my free time. But now let’s turn to what I like to see most:
A clear statement which language research result
could be integrated into a NextLisp.
I think the ILC could provide a clear vision by stating which language research results are attractive to a NextLisp. I like to provide here some ideas for Lighting Talks related to NextLisp Syntax and Scientific Computing.
NextLisp Syntax
Your “Fundamental problems with the Common Lisp language” mentions, for example, “Archaic naming”. I would interpret this as “time is ripe for a NextLisp Specification”. Related to “Archaic naming” the Dylan project once tried to deliver a solution (imho).
I can envision that the NextLisp development environment supports several “surface syntax types (languages)”. As possible surface language I can imagine: ARC, Dylan prefix syntax and Dylan infix syntax, and -- as the systems nowaday support Unicode -- a more mathematical syntax is also thinkable.
So I looking forward to a Light Talk video where people discuss
( The Ligthing Talk section after David Moon’s contribution “Genuine, full-power, hygienic macro system for a language with syntax” and before the “Panel Session: The Future of Lisp” on Monday might be a good point in time).
Personally I would also appreciate a statement of the LISP community whether the old Dylan Prefix Specification can somehow be a base for a NextLisp Specification. I know that many Lisp users dislike the infix syntax of the current Dylan Specification, but do all Dylan detractors know the old - prefix syntax based – specification.
Many former Dylan Language Design and Implementation Team Members work at one of the sponsors of the ILC. It would be nice to hear what they mention about using the Dylan Spec as base for a NextLisp Specification. (Sometimes I feel that they got really deeply frustrated about the unhappy shut down of the Dylan project. But that is IMHO no reason to dishonour Dylan project results by not discussing their value for a NextLisp.)
Scientific Computing
I expect that the ILC delivers a video of the “SciCL: Lisp Extensions for Scientific Computing” presentation.
Also I would appreciate a Lighting Talk Video where John Amuedo comments on
(see the The MGS home )
(In the case that John Amuedo does not already know the French MGS project, I would appreciate, if you forward him the link to the MGS home ( http://mgs.ibisc.univ-evry.fr/ ). Otherwise he won't be able to compare his approach with this one. )
As you see I vote for videos as I have no sponsor (budget) to travel to Boston.
... but yes I know the next change is in 2011.
Peter Robisch (pet-ro)
Too much state at the base- and meta- level
Because of its reliance on side-effecting a global state of the Lisp world, CL makes the modular cooperation between several Lisp threads or processes particularly difficult. Isolating code from various modules into conceptual or enforced sandboxes is nearly impossible.
While global side-effects in defining functions and classes are a bad thing, the worst offender is the package system itself. Its flat symbol space with per-world aliasing, is useless for the modular cooperation of multiple processes or versions of software. The only symbols that can be guaranteed to be sharable between two divergent Lisp images are keywords (i.e. immutable strings). You can never rely on using a symbol to communicate across versions of the same software. And so the whole symbol infrastructure is essentially useless for programming in the large today (with persistent data and/or concurrent processes using divergent versions of the software).
Common Lisp is defined in terms of programs being meta-level side-effects of the One True Lisp Image. This model is wholly inadequate. Its completely dynamic nature is a nuisance for program analysis and therefore efficient compilation inside a given Lisp image. But nothing in CL addresses the dynamic nature of the world beyond the current Lisp image: versioning of code and data, URIs or UUIDs as global anchors, module systems to divide problems into differently moving parts, support for concurrency and persistence, continuations and mobile agents, etc.
CL adds plenty of extrinsic stateful interactions to your software systems, and doesn't help you with their intrinsic needs for state.
Underspecified semantics
The semantics of CL is largely left unspecified which means that it is a big minefield for purportedly portable programs. Programmers of "portable" code must be on the lookout for a wide range of "anti-patterns" that they have to memorize, and that cripple the language. Moreover, because the semantics is so complex and leaky, detecting these patterns is itself a hard task, not computable in the general case, and expensive even in the common case.
This is the natural result of Lisp being a backwards-compatible compromise between now long-forgotten implementations: "Backwards compatible -- If it's not backwards it's not compatible" -- Greg Newton.
Of course, with a module system that allows for arbitrary language frontends, one could easily provide much better backwards compatibility all the while allowing progress. PLT has that, CL doesn't. Oops.
No modularity
CL is missing big time regarding modularity: the ability to affordably divide work so that some (human or machine) agents may be responsible for some functionality whereas other agents will be responsible for other functionality, with a small interface between the two.
CL has no notion of module or interface. Neither its "packages" nor its "systems" provide a way to factor software into parts without a lot of leaky abstractions and costly interactions.
Even the standard itself is non-modular, and parts that SHOULD have been delegated haven't been (see my rant: http://fare.livejournal.com/139755.html ).
Yeah, I'd like...
I like the approach to modularity followed by Modula-2, Erlang, and Limbo. But I expect it would require a major revision or major kludge to incorporate such a system into CL.
But not like Dylan
But not like Dylan's package/module system, which made "Hello, World" excessively hard to write! Dylan had goals involving libraries that could be packaged and distributed, and could be "sealed" so that the compiler of the library can know that users of the library won't be doing stuff like extending classes or otherwise messing with the internals of the library that aren't exported. As far as I can tell (I am hardly a Dylan expert!), the ideas are fine but the execution left something to be desired.
It's not easy to do these things! Yes, I'd love to see it done well. I have not yet looked at what approach Clojure takes.