Lisp's reputation is so bad that many people don't even take a look at Lisp

Lisp is rarely chosen as the language to use for anything, except by people with a lot of Lisp experience, or who have been adopted into the community. It's almost a "cult" language in the sense of "cult movie": there's a fan base that's small but wildly enthusiastic.

When people hear Lisp success stories, they are usually surprised. When they hear that Lisp is suitable for a transaction processing system (ITA) or a shrink-wrapped education tool with a beautiful user interface (InspireData) or interactive Web commerce (Yahoo! Stores) or on the Deep Space 1 spacecraft (just to mention a few), they're often astonished if not incredulous.

Why won't more people consider using Lisp? Here's how I see the problem; I'd be interested to hear what you've heard.

These are some of the things I hear often from people who don't know Lisp but have heard a few things about it, or who have only had a quick exposure to Lisp. They have become "factoids" (inaccurate statement or statistics believed to be true because of broad repetition.) These factoids apply to the whole Lisp family, not just Common Lisp.

  • Lisp is old and moldy. It must be primitive by today's standards.
  • The exciting languages to learn now are Python, Ruby, Groovy, etc.
  • Lisp is only for artificial intelligence.
  • Lisp is interpreted and slow.
  • Lisp programs page heavily, so they're even slower.
  • The parentheses look weird and hard to manage.
  • Lisp uses prefix notation for things that every other popular language does with infix notation, which is just very weird. (And it leads to more parentheses.)
  • Lisp is too big.
  • Lisp is too hard to learn.
  • Lisp is all about lists and recursion; that's all you do with it.
  • I don't know which implementation to use.
  • It isn't "batteries included" (that is, it's not easy to get set up with all the tools and libraries you need).
  • I don't like Emacs, and you can't use Lisp without Emacs.
  • Lisp isn't useful for my career; few potential employers want it.

(At least the standard old complaints about garbage collection have mostly vanished, now that Java, C#, Python, and Ruby all do it.)

There are, of course, many possible replies to such concerns, and I think most of us are familiar with the replies. The problem is, someone with these preconceptions often doesn't even look for our replies!

The issue I'm interested in here is how we can go about rehabilitating Lisp's reputation, so that these misconceptions no longer steer people away from finding out our replies, and learning what's especially good about Lisp. (In other topics on this forum, I will discuss more sophisticated criticism of Lisp.)

What can we do?

For me

I first used lisp when I took a programming languages class in college. It was kind of interesting so I looked in the university's library nothing published after I was born, dead language maybe. Get on yea olde google. What do I find? One guy(Paul Graham ) going on about how great it is, and every other lisp hacker in the universe whining about how bad lisp is, why it isn't noob friendly and why I shouldn't bother wasting my time. LISP is so little used that its rep has died for everyone not in the community the only thing talking about it does is warn away the curious.

I figured out what it is that bothers me

It took a long time for me to figure out what was giving me fits about s-expressions. Especially since I think they are such an elegant solution to the whole code-as-data approach. At first, I thought it was because I was used to the other function notation due to other programming languages and math notation. But that wasn't it, because I've been able to adapt to other types of notations without a problem (such as the many ways to represent a conditional or a loop).

First, in normal English writing, what appears in parentheses is supplemental material to a main subject. A sentence would read just as good without the content in parentheses, but that content just gives you a bit more information. Whereas in Lisp, the main subject (the function name) is also included in the parentheses, so it doesn't stand out. And to me, the function name is more important than the parameters (they are just details), at least when trying to get the general gist of the code. That is, first you need to understand the functions that are used, then you try to understand the detail that is fed into those functions.

A second problem is that unless you already know the full structure of that function, you don't know which parameters are passed as is, or which ones are expanded first. There is no distinction between special forms, regular functions, and macros -- so you just don't know what is going on until you dig to the bottom of the program.

Now I don't have an easy answer for how to solve these two issues and still keep the ability to treat code as data. The second issue might be solvable by not having special forms, but instead requiring a prefix character to be prepended to elements that aren't to be expanded before getting passed to the target function. But that would make for uglier code.

Reading Lisp

Regarding the use of parens by comparison with English: I'm surprised that the way parens are used in English make it harder for you to read Lisp. Punctuation in any major computer language has very little to do with the way English is punctuated.

Just for fun, here's some Lisp code, followed by the way it would look if all functions (but not special forms nor macros) were written as the name of the function followed by the open-paren instead of preceded by it:

(defmethod socket-accept ((socket socket))
  (with-sockaddr-for (socket sockaddr)
    (let ((fd (sockint::accept (socket-file-descriptor socket)
                               sockaddr
                               (size-of-sockaddr socket))))
      (cond
        ((and (= fd -1)
              (member (sb-unix::get-errno)
                      (list sockint::EAGAIN sockint::EINTR)))
         nil)
        ((= fd -1) (socket-error "accept"))
        (t (apply #'values
                  (let ((s (make-instance (class-of socket)
                              :type (socket-type socket)
                              :protocol (socket-protocol socket)
                              :descriptor fd)))
                    (sb-ext:finalize s (lambda () (sockint::close fd))))
                  (multiple-value-list (bits-of-sockaddr socket sockaddr))))))))

(defmethod socket-accept ((socket socket))
  (with-sockaddr-for (socket sockaddr)
    (let ((fd sockint::accept (socket-file-descriptor (socket)
                               sockaddr
                               size-of-sockaddr (socket))))
      (cond
        ((and = (fd -1)
              member (sb-unix::get-errno ()
                      list (sockint::EAGAIN sockint::EINTR)))
         nil)
        ((= fd -1) socket-error ("accept"))
        (t apply( #'values
                  (let ((s make-instance (class-of (socket)
                              :type socket-type (socket)
                              :protocol socket-protocol (socket)
                              :descriptor fd)))
                    sb-ext:finalize (s (lambda () sockint::close (fd))))
                  (multiple-value-list bits-of-sockaddr (socket sockaddr))))))))

I think it would be harder to explain "code as data", and explain how to do Lisp macros, were this syntax used.

[Argh, sorry, I don't know how to make the indentation appear.]

As for not being able to tell which names are functions and which are special forms or macros, one way to handle that is in the IDE. Emacs Lisp mode colors the names of special forms such as "defmethod", "let", "cond", and so on. It does not work perfectly (there are things that should be colored and are not) and should be improved, at least in XEmacs 21.4, Dec 2006; I should upgrade.

When people search for Common Lisp...

They see testimonials like this and move on...

http://briancarper.net/blog/clojure-1-common-lisp-0

Ignoring the fact that Clojure is also a Lisp - the negative experience with using CL for a website (which I've also experienced, but slogged through anyway) is telling. Perhaps Clojure is the future, but I'm still quite fond of Common Lisp!

A Coherent Portal into the Lisp World

My two (hundred) cents on this issue...

I believe that the rising generation of programmers don't care very much about most of the issues Dan raises as objections to Lisp. What the 'cool kids' are really accustomed to is a coherent community, extensive Google-indexed support for specific issues, documentation and good design. Specifically, I think expectations for visual and conceptual packaging have changed dramatically. Those who grew up with PDPs, VAXes, and even on the Lisp Machines don't appreciate how critical this is to making a language accessible to the rising generation of hackers.

Peter Siebel showed that even small packaging steps can have big impact. Design, documentation and a common starting place to get into lisp would go a long way towards bringing a new generation of users to Lisp.

Edi commented to me that the real value of PCL was that it presented lisp cleanly and from a modern perspective. I think this is a critical observation. It doesn't hurt at all that Peter is also a great writer.

There is tremendous elegance in the lisp language, but none at all in the presentation of the language to the world.

Of course there are many ugly corners in the language and we can debate this ad-infinitum. However, if you look at Perl, Python, Ruby and even Java, they all are elegant for certain tasks and yet have many ugly corners that people learn to work around. I think people are happy to work around these issues, if they have proper incentives. The first steps should be easy, have immediate value, and the ultimate rewards need to be clearly articulated.

Hackers love to experiment, explore, and learn by doing - but are accustomed to a low cost of entry. We've got many of the pieces in the CL community, but the conceptual packaging is awful and the websites are ugly (even the commercial ones). This is especially true for Common Lisp. We have to realize that almost all new-comers will get into lisp with no peer groups to work with or learn from.

What to do?

My suggestions below are focused on Common Lisp because it is what I use and know best. I also went through the bootstrapping effort myself only a few years ago and these are things that would have helped me.

1) A Common Lisp Portal

There are about 20 'psuedo-portal' pages on the nets that provide pointers to the half-dozen or so major repositories of libraries, partial information, etc. I think that we need a coherent starting point and it should be sexy. There should be some clear short term wins for using lisp(e.g. http://web.telia.com/~u43518104/articles/lispweb.htm) and motivational examples about long term benefits (e.g. stories from ITA, ViaWeb, others).

It should support and link into...

2) Lisp Tutorials and Guides

There are quite a few partial or dispersed tutorials introducing lisp. Between Siebel's book, Pascal's 'Highly Opinionated' introduction and a multitude of other writeups and blogs there is some great content out there. Integration of that content, with author permissions, into something like the Lisp Handbook suggestion is a good start. Don't start over, emphasize packaging the good stuff that is there.

In my humble opinion, PCL and OnLisp are the two best introductions to the language and since both are available online - perhaps we could get author support to integrate these more directly. It would also motivate book sales!

Examples of beautiful, elegant code to learn from would also be valuable. LISA, for example, is really quite elegant and a tutorial that walked through the internals would be a great resource. So how to get these golden nuggets....?

3) Library Documentation and Bounties

Documentation is the least rewarding and most annoying part of building libraries after testing. A single place to go (see Portal above) that contained a coherent view of a 'core library set' and all the relevant documentation would be awesome.

Where documentation is poor or missing the ALU, for example, could provide incentives for people to take these on as hobby projects. I suspect that even a $100-200 bounty for a good tutorial or for documenting libraries, the quality of which will be judged by respected members of the community and acceptance of a final version releases the bounty. The numbers should certainly change based on the size of the project.

Clear recognition and social acknowledgement rewards are also important. The younger set which are more likely to be motivated by moderate sums will also learn alot from documenting a library.

4) An evolutionary path for the core language

The core of common lisp has hundreds of commands that are not very well organized. The hyperspec is awful as an educational tool as is CLTL2. Organizing these commands, and integrating common extensions that are useful for solving specific problems would be extremely helpful. Currently you download a lisp, get the hyperspec and some tutorials and the root around looking for small functions that don't fit neatly into a library like cl-ppcre but are effectively language extensions that fill in holes in CL or provide new facilities.

Example commonly-used utilities include:
- strcat
- iterate
- let-bind
- matchers
- control macros: aif, awhen, etc.
- memoization

There must be 30 copies of awhen in my current source tree. Think packaging. These extensions should look like a clean, integrated and easy to access extension to the language. This is very low hanging fruit; no new language specs, no new compilers, very little new software, etc.

We need a way to organize all this. I have two proposals that I'm willing to put some effort into that I'll describe later and elsewhere.

a. A web-based community site for editing, annotating and searching such an extended namespace of core functions. I'll release a prototype shortly.

b. A repackaging of the core package namespaces, for example...

- std.cl

This includes the cl namespace + cl-ppcre + usocket + bordeaux-threads, trivial-gray-streams, etc. This would be an extensible, cross-platform, unified API for an extended core language tied back to documentation.

- std.util (anaphors, expanded string facilities, etc.)

This may even be included in std.cl - but I think there is room for much discussion and debate here. Of course to make practical progress there needs to be one or a few people who make the final calls.

- std.contrib.me

Leave room for people to include project-specific or personal variations that nevertheless can remain in this common namespace. People can propose that a utility be migrated into a common namespace like std.cl, std.util, etc.

- std.? (.net, .xml, .json)

Rename common, existing libraries into a coherent package namespace and index on the web so users can find common functions and libs easily and via one portal. With the cooperation of the authors, the underlying libraries can be fully renamed, or we can simply provide packages that re-export these names in the interim.

I made a stab at this earlier and at least 3 or 4 of the primary popular utility library developers expressed a willingness to move over to such a model and release their code for use here.

5) Alternative editors

SLIME provides a wonderful bridge to Lisp and it has been used in other editors (CUSP for Eclipse) and for other dialects. I think that investing in expanding support for CUSP will help new users make the transition faster. Emacs can continue to be a power-user tool and perhaps an aspiration but not a requirement for new programmers.

Another project might be to take another popular multi-platform editor and integrate SLIME support.

6) Simple things...

Posting without joining on cl.net

It would be great to allow users to post on 'project'-dev on common-lisp.net or the equivalent so they can ask questions without going through the overhead of signing up to receive e-mail.

Real time Q&A -> FAQs

Common questions are often dealt with on IRC and other discussing forums, but then are not indexed or easily findable. Making a concerted effort to post those to an indexed FAQ service might be helpful. Perhaps a lisp-paste extension to make it easy to post and index discussions and code with some tags.

Technical needs discussed elsewhere on this site:
- A coherent distribution model. LispBox+
- To clean up the library packaging model and provide a standard with a primary emphasis on testing and documentation.
- Testing models/platforms and other support for distributions

On the potential role of the ALU:

I was encouraged by the discussions related to expanding the ALUs charter to help promote growth and adoption of the wider lisp community. My personal opinion is that the above emphasis on packaging will catalyze activity and providing a model and support for project longevity. Until we have a larger community of open-source users, there simply aren't enough people using any specific package to have high confidence that good projects will be maintained when the creator or maintainer moves on.

I also think that we should emphasize supporting language dialect communities; not a one-sized fits all approach to libraries, etc. I wouldn't try to make common-lisp into something it isn't; I'd put that energy into supporting and improving new dialects of Lisp like Clojure, Qi, Arc, etc.

Learning Lisp

I've been working to learn Lisp since October. I read constantly on everything I can find Lisp related and hack around on scripts, or read lots of code duplicating it in my REPL and breaking, etc. I've read PCL, OnLisp, ANSI Lisp and am working on LOL (which has been a pleasant read so far).

So far the "experience" has been a lot of work. Choosing an implementation was the initial problem as SBCL has some nice features, but CLISP is small and "script-able", etc. Diversity in the different implementations seems to me to be the biggest issue. When I want Ruby I simply go to ruby-lang.org and download Ruby, it compiles on everything and then I start scripting. The view from 50,000 feet is that this is impossible in Lisp (of course I know you can dive right in, but the propaganda on the web suggests otherwise).

I also think, that while Emacs is awesome and SLIME makes life easier, there needs to be less focus on using Emacs to hack Lisp. Vi(m) is a perfectly good tool for hacking Lisp. But people get in their heads that they can't live without Emacs, etc. Believe me my first 2 months were switching from 8 years of VIM to Emacs so that I could get familiar with the environment to be a "Lisp guru" and use the "right" tools. It was daunting, and while I found it rewarding (although a lot of work) it wasn't exactly a nice introduction to hacking on Lisp. Granted, did I need to go through all of that to use Lisp? No. However, do a search on Google, it's almost sacrilegious to use Lisp without Emacs/Slime.

So rather than claim Emacs is the only tool to hack Lisp, more emphasis should be on the fact that it is written in Lisp, "see what Lisp can do??" those types of things.

Another issue is the Newsgroup. While I find it entertaining and often times very rewarding with regards to the "ninja's" that exist on there posting their code, most of the time it's a waste of bits. So much garbage needs to be filtered through to get to anything resembling an intelligent (or even on-topic) response that it's a turn off. I am a masochist so I continue to read, but to "noobs" this is an issue.

I've been toying with the idea of a Lisp Handbook. Essentially replicating FreeBSD's Handbook but all about Lisp. How to get started, how to do this, that and the other thing, etc. I am still trying to figure out the right way to go about this, but I think it would be useful, especially if people worked to keep it updated. It could also pull from all the implementations, etc. It's a lot of work, but might be useful for people wanted to get started.

My last comment is sort of a pie in the sky type of comment that can be ignored...that's a warning...

I think Paul Graham is onto something with Arc and it's syntax. Granted he hasn't really been pushing it nor is it a viable replacement for anything at this point. But syntactically, structurally and ideally it's exactly what the Lisp community needs. Just look at some of the ideas here: http://catdancer.github.com/

If Lisp "looked" and "was" this easy to get programming with (granted Arc easy to get started with) and was concise and elegant like a lot of that code, Python and Ruby would be dying communities. That's simply my opinion. Note, I don't want to give up the power of Lisp, nor give away anything to achieve this, but I think overall there could be a refactoring of ideas, syntax (I am not talking about parantheses but more symbol names, etc) and in general the community. I love the standard, as I prefer them, but in some cases this has contributed to the stagnation of Lisp and it's ability to truely be agile. In some ways it's ironic that a "programmable programming language" is sort of "stuck".

Lisp is gorgeous, it's powerful and so far is pretty awe-inspiring to me, and I want others to experience this, but the entry fee is pricey and the rewards are long term. That's a turn off for most people. Is that a motivation to change things? It shouldn't be, but it doesn't have to not be either.

Anyways, this is long, those are my thoughts, hopefully they contribute something. I wish I could attend this conference.

Lisp's reputation is so bad...

The reputation or popularity of a programming language (or most anything else) is rarely due to any inherent qualities. For over 35 years working with and on Lisp systems and applications, the popularity question has always been a constant and annoying background hum.

If a programming language is complete, then it is equivalent to any other complete programming language. If it is incomplete then it is pretty much useless. Eventually, any programming language compiles down to machine instructions that shift bits around registers. At best, a good programming language hides some of the grubby details while still giving access to the underlying complexity that makes the computer useful.

Lisp is very likely unpopular because it exposes most effectively all the complexity possible in a computing system. The most common complaint is about the parentheses, but the parentheses are the very essence of the power and elegance of Lisp. There are atomic constructs and there are composite constructs. The composites are denoted by parentheses. Any additional notation is purely to create the illusion of greater expressiveness.

Lisp will never be popular, but it will never be replaced. Since Lisp has a built-in concept of extensibility, it can and will stay at the frontier of exploratory computation. But like any powerful tool. it will and should remain in the hands of skilled practitioners.

We should continue to improve Lisp, it can always be even more perfect, but it is unlikely that we can make it popular and still retain its power.

Easy to learn, hard to use

I have thought that compared to say C++, lisp is very easy to learn. At least the syntax. But as is pointed out above in the 3rd paragraph, the programmer is exposed to computing complexity and subtle thought almost immediately. So there is a small investment of time and effort, and the new user is rewarded with problems that require hard thinking.

A language like C++ or Java on the other hand take a lot longer to master the syntax, and then the problems tackled are usually not so hard, at least at first. So there is a big investment and the appearance of payoff.

LISP isn't bad, but...

It's not enough for a language to be good, it's got to be better than the alternatives -- it's got be sufficiently better than the alternatives to justify the cost from switching from something that already works.

Ideas from LISP have migrated into almost all of the mainstream programming languages that we use today, and in the research languages.

Today I do a lot of work in PHP and C# because I can get the work done that I need to get done in them. If I were looking for a more advanced language, the list of languages that I'd consider would include Erlang, Scala and F#. It's nothing personal about Lisp, but a lot of great languages have been made in the 50 years since Lisp was invented.

Not actually about the reputation issue

Paul, I agree with just about everything you're saying here, but I don't think these particular points are directly germane to what the original post is about. I'm mostly talking about the subjective impression that people get when they hear the name "Lisp".

One possibility for a new Lisp would be to avoid calling it "Lisp". Saying somewhere that it's a dialect of Lisp, or heavily influenced by Lisp, or something, is fine, as long as it has its own "brand" name that isn't "Lisp".

This doesn't make me feel particularly good, but one has to face the facts, and address the reputation problem somehow. That's one possible way. I think there will be some pain no matter what we do.

Name

Well, many people do not know that Scheme is another Lisp, and does it help it much? I would not think so.

What's in a name?

It's not quite the same situation with Scheme. Scheme is seen largely as a language for education and research. As far as I know, there are a lot fewer production applications written in Scheme than in Common Lisp.

What I'd like to see is a world in which Lisp (whether Common Lisp or a new dialect) is popular as a language for people to do ordinary tasks, such as writing web sites, application servers, games, and so on. I want something that can compete with Python and Ruby. It should have the reputation of being solid, reliable, and fast, "cool", and offering unique advantages over the competing languages. This isn't a goal of Scheme.

For many people those are

For many people those are precisely the goals of Scheme.

The story about Reddit and why they stopped using Lisp

The well-known web site reddit.com stopped using Lisp, and switched to Python. I have even heard this event cited as evidence that there's something wrong with Common Lisp.

Reddit co-founder Steve Huffman wrote an explanation, which can be found at http://blog.reddit.com/2005/12/on-lisp.html. It says many good things about Lisp. His reasons for giving up on Lisp were:

  • He develops on a Mac, and runs Reddit on FreeBSD. He needs proper multi-threading and low-level socket access. On the Mac, he could only use OpenMCL, and on FreeBSD he could only use CMUCL. Threading and low-level socket access were not portable between these two implementations, so he was force to do all development on the FreeBSD development server, which was a pain.
  • He found that there was a lack of widely-used and tested libraries. Although the libraries exist, he found that they were not widely used and not well-documented.
  • In general, he is building a site largely standing on the shoulders of others, and there just aren't as many shoulders on which to stand.

He wrote this blog posting in December of 2005. Let's assume that he wants to only use free, open-source implementations.

There are now four free, open-source implementations that support real threading on both MacOS and FreeBSD: Armed Bear CL, Clozure CL (formerly OpenMCL), Embedded CL, and Steel Bank CL. The names of the functions for threading are not the same, but there's an easy-to-use portability library that hides that.

As for low-level socket support, there is now the usocket library, which works on all of those implementations.

The problems about libraries and the community, however, are still real ones. I will address each of these in separate topics.