February 27, 2007

What is the point of macros?

One of the distinguishing features of Lisp and Scheme is the ability to define macros that allow you to extend the base language with new language constructs. Here are two examples what this actually means and why this is a good idea.

Let's take a common idiom from Java: Whenever you want to open a file, process its contents, and close it again afterwards, you have to write code roughly like this:

FileInputStream in = new FileInputStream(filename);
try {
} finally {

Note how the orange code is the code that you are actually interested in: That's the part that states which file you want to open, that you want to open it for reading, and that you want to do something with the file. The rest is just idiomatic code that you have to write just to make the machinery work: You declare a variable outside of the try block (this is important for the visibility of the variable!), and you have to put the call of the close method in a finally block to ensure that the file is always closed even when an exception occurs.

The problem here is that you have to write this code over and over and over again, and everytime you write this code, you have to get the details right. For example, when I was still a Java hacker, I always have put the variable declaration inside the try block, just to get the error message from the compiler that this is wrong. This is annoying - it would be nice if I didn't have to think about this. (I have actually gotten it wrong again in the first version of this posting. Go figure.)

Here is the same code in Common Lisp:

(let ((in (open filename)))
(do-something in)
(close in)))

This is not yet an improvement: The orange part of the code is still deep down in other code that just ensures that the machinery for opening and closing the file works. I still have to declare the variable first (here with a let), I still have to use the Lisp-equivalent of try-finally (here it's called unwind-protect), and I still have to ensure that the closing of the file happens as part of the clean-up step of an unwind-protect.

However, Lisp gives us a way to get rid of the idiomatic code. What we would actually like to write is this:

(with-input-file (in filename)
(do-something in))

This code states exactly what we want and nothing else: We want to execute some code with an open file bound to some variable, and we implicitly want to ensure that it is closed again at the end of this code fragment.

Here is a macro that defines this construct:

(defmacro with-input-file ((variable filename) &body body)
`(let ((,variable (open ,filename))
(progn ,@body)
(close ,variable))))

Here is an explanation of the code:

  • The macro is defined with defmacro and we name it like the language construct we want to build. It takes two variables: a variable name and a piece of code that will give us the filename in the resulting code. It also takes a body of code for the with-input-file construct.

  • The macro creates a piece of code that the Lisp compiler should use instead of the original code that uses the with-input-file construct: We basically define a code template in which the variable, filename and body spots are filled in with the parameters that the macro receives.

  • Some minor details: The progn groups a number of expressions into one - this is similar to grouping code in a pair of curly braces in Java. The backquote, comma and comma-at notations are there to clarify how the parameters should be inserted - that's something that you can better learn in good Lisp tutorials.

Although writing macros takes a little practice, it is very straightforward once you are used to doing this. Macros help you to avoid typing a lot of idiomatic code: whenever you notice some coding pattern, you can put it in some macro and then use it instead.

Syntactic Abstractions

Let's get a little closer to the heart of what macros actually are: They provide you with a way to define syntactic abstractions. Abstractions are used for hiding away implementation details that client code should not be interested in. For example, in classes you can define private fields and methods for internal use and define a public interface that client code has to go through to ensure that the internal details are always used properly. Likewise, with functional abstractions - closures - you can pass around a function that can refer to variables in its lexical environment without giving anyone direct access to these lexical variables.

Many implementation details can be hidden very well with such abstraction mechanisms. However, others cannot be hidden well, and this is when macros become interesting.

For example, assume that your language doesn't provide any iteration constructs but requires you to use recursion instead. You can relatively easily build your own iteration constructs - for example a while function - like this in Lisp:

(defun while-function (predicate block)
(if (not (funcall predicate)) (return))
(funcall block)
(while-function predicate block))

This function works as follows: It takes a predicate function and a block of code, also provided as a function. When calling the predicate (with funcall) returns false, the while-function returns. Otherwise, the block is executed (again with funcall) and then the while-function calls itself again.

Here is an example of using the while-function:

(let ((i 0))
(while-function (lambda () (< i 10))
(lambda ()
(print i)
(incf i))))

This code fragment increments i from 0 up to 10 and prints i at each step. This piece of code doesn't look very nice. One reason is that Lisp's lambda construct is somewhat lengthy. Smalltalk, Ruby, and other languages have nicer syntax for lambda expressions, and would make this code shorter. However, even then there is a problem here: We still have to write idiomatic code to write a while loop although we are actually not interested in the idiom. Here, the idiomatic element is the use of a lambda expression to ensure that some code is not immediatily executed, but only later under the control of the while-function. However, what we actually want to say is this, which doesn't contain any idiomatic code at all:

(let ((i 0))
(while (< i 10)
(print i)
(incf i)))

And, as you might have guessed, here is a macro to implement this while construct:

(defmacro while (expression &body body)
`(while-function (lambda () ,expression)
(lambda () ,@body)))

This macro uses the while-function in the code that it creates. This is actually one of the typical ways to write Lisp code: We first define the functional (or object-oriented, or imperative, or whatever) abstractions, and then we add some syntactic layer on top to make the code look nicer, and especially to hide away implementation details that we cannot hide otherwise.

Why is it so interesting to hide away implementation details, like the use of lambda expressions to delay evaluation? Well, we could actually decide not to use lambda expressions at all in our expansion. Here is an alternative implementation of the while macro:

(defmacro while (expression &body body)
start (if (not ,expression) (go end))
(go start)

Yes, Common Lisp provides the tagbody construct inside which you can use go, i.e., a goto statement! Gotos are not a very good idea to use in your own code, but gotos are very useful for generating efficient code in macros. In this second implementation of the while macro, there is for example no need anymore to allocate space for the closures that the two lambda expressions create, because we can simply jump around the code to delay its execution.

Of course, this is a toy example, so the little gain in efficiency here is probably not worth the effort. However, what is important is that the "API" for the while macro hasn't changed at all compared to the first version. You still write this:

(let ((i 0))
(while (< i 10)
(print i)
(incf i)))

That's one of the important advantages of abstractions: You can change internal implementation details while all the client code can be left unchanged!

And this is the whole point of macros: You can abstract away code idioms that you cannot astract away in any other way, and you effictively have a lot more options to change implementation details without affecting any clients.

Macros are also one of the fundamental reasons for why Lispers like Lisp's strange syntax so much: The syntax is a direct representation of what is elsewhere called the abstract syntax tree, and the fact that the AST is directly available for manipulation as a very simple list data structure makes it so straightforward and convenient to implement and use macros.

If you now have gotten the taste of this, you may want to read some more introductory material about Lisp, like my own opinionated guide, Peter Seibel's excellent book about Common Lisp, or any of the other available tutorials.

February 21, 2007

Call for Participation: ILC'07


Clare College, Cambridge, England - April 1-4, 2007

In cooperation with ACM SIGPLAN

Sponsored by The Association of Lisp Users

The Association of Lisp Users is pleased to announce the 2007 International Lisp Conference will be held in Cambridge, England at Clare College from April 1st to 4th, 2007. This year's program consists of tutorials at beginners' and advanced levels, prominent invited speakers from the Lisp and Scheme communities, an excellent technical session, tours of Central Cambridge, Anglesey Abbey and Ely, and a quintessential English experience: a traditional dinner served in the college's Great Hall. The advance registration deadline is March 11th. The ILC'07 programming contest is also still running until March 3rd.


  • Saturday, March 31st

  • Sunday, April 1st

    • Tutorials and workshops

      • Ernst van Waning, Extended Tutorial: Common Lisp in One Day

      • Pascal Costanza, Context-oriented Programming in Common Lisp

      • Richard Brooksby, Improve your Lisp using the Memory Pool System

      • Duane Rettig, Optimizing and Debugging Programs in Allegro CL

  • Monday, April 2nd

    • Invited presentations

      • Christian Queinnec, Teaching CS to undergraduates at UPMC

      • Michael Sperber, It's All about Being Right: Lessons from the R6RS Process

      • Herbert Stoyan, Lisp: Themes and History

    • Presentations of accepted papers

  • Tuesday, April 3rd

    • Invited presentations

      • Jans Aasman, Scalable Lisp Applications

      • Ralf Moeller, Building a Commercial OWL Reasoner with Lisp

      • Manuel Serrano, HOP: An Environment for Developing Web 2.0 Applications

    • Presentations of accepted papers

    • Annual meeting of the Association of Lisp Users

    • Conference banquet

  • Wednesday, April 4th

    • Invited presentations

      • Richard Jones, Dynamic Memory Management

      • John Mallery, Lisp/CL-HTTP

    • Presentations of accepted papers

  • Thursday, April 5th

Conference Registration

Conference registration is now open. The advance registration deadline is March 11th. You can get further discounts as an ACM/SIGPLAN and/or ALU member. Registration includes: access to all events, morning and afternoon teas / coffees, self-service lunch, banquet (Tuesday April 3rd), proceedings and hopefully a conference t-shirt. Accomodation is available in Clare College's "Memorial Court". Credit cards and PayPal are accepted, as are cheques (sterling or US dollars) and international bank transfers.

Organizing Committee

  • Co-Chairs: Carl Shapiro (SRI International), Pascal Costanza (Vrije Universiteit Brussel)

  • Members: Rusty Johnson (ALU), Peter Lindahl (ALU)

  • Program Chair: JonL White (The Ginger Ice Cream Factory / ALU)

  • Local chair: Nick Levine (Ravenbrook / ALU)

  • General correspondence: ilc07-organizing-committee at alu.org

Mailing Lists

General conference announcements are made on a very occasional basis to the low-volume mailing list ilc07-announce. If you're thinking of participating in ILC 2007, you should either join this list or take an occasional look at the archives.

February 05, 2007

Let's Lisp again!

There are several news about this year's International Lisp Conference to be held in Cambridge, UK in the first week of April.

  1. The Register has published a press release about the conference. Nick Levine told me that it's a good idea to "digg" this article (whatever that means ;). Apparently, you can do this directly via this entry at digg.

  2. There is a conference poster available for download. Please print it off and distribute it everywhere.

  3. Finally, you can show off your Lisp programming skills by providing implementations for the conference's programming contest.