Why did I write the Clasp compiler?

Imagine that you have a collection of highly optimized C++ libraries that do some amazing stuff in a very time and memory efficient way and you want to write experimental code to do some complicated things by calling the API of those C++ libraries – what do you do?

You could write everything in C++ – but C++ is a tedious language to write complex, experimental code within because there is a lot of boilerplate that you need to write. I write a lot of C++ code, a couple of hours a day, and much of my time is consumed by figuring out how to express what I want to do in C++ and then when I rewrite the code when I realize that it fell short of my desires.

You could write a wrapper library or foreign function interface to expose your C++ libraries in some dynamic language like Python, Lua and then write your experimental code in that. However, you will very quickly run into the problem that you are spending much of your time writing and maintaining the foreign function interface. You will also encounter nasty problems involving the differences in memory management, exception handling and myriad other differences between C++ and the dynamic language you choose.

I’ve spent a lot of time doing both of these things and the problems that I’ve encountered have invariably derailed my ambitions and my ideas.

So I decided to do something different.

I decided to write a language from the ground up that would smoothly interoperate with C++.

I chose the most expressive, well-defined language in existence – Common Lisp.  I’ll say more about why later.

When I started the LLVM library (http://llvm.org) was just coming into maturity and I decided to use it as the back-end for my language.  I could leverage all of the work done by really smart hobbyists and people at Google, Apple and many other companies to perform much of the back-end code generation.  Choosing LLVM would also give me access to the entire LLVM ecology of tools and I would be able to incorporate the Clang C++/C/Objective-C compiler as a C++ library to more closely weave together Common Lisp and C++.

So now we can write complex code in Common Lisp that calls C++ APIs and have C++ code call Common Lisp code while spending very little time worrying about the interface between them.

Two C++ libraries that are already exposed within Clasp are LLVM and Clang.  The LLVM library exposes C++ API for LLVM-IR generation and is used by the Clasp compiler written in Common Lisp. It’s available within Clasp and I think it’s a great playground for exploring LLVM-IR.

I also exposed the Clang C++ compiler front end library including the entire Clang AST (Abstract Syntax Tree) and the ASTMatcher library. This enables a programmer to write C++ static analyzers and C++ refactoring tools in Common Lisp!   I used it to analyze the entire Clasp source code (165 C++ source files) and automatically build ~10,000 lines of C++ code that interfaces Clasp with the Memory Pool System copying garbage collector by Ravenbrook Systems (http://www.ravenbrook.com/project/mps).



4 thoughts on “Why did I write the Clasp compiler?

  1. I enjoyed watching your google presentation.
    You’ve re-inspired me to “do” things in life without waiting for “permission” from life.

    Also, this could be what brings me back to lisp from scheme.

    Are you also aware of the XML lisp bindings tricks of https://code.google.com/p/xmlisp/, a key point being expressed here:

    “XML is used to easily serialize/de-serialize scenes into files. XML content is usually much simpler to integrate with other tools. For most simple cases you will not have to worry about XML. You simply define your classes. The Common Lisp MOP will automagically turn your CLOS objects into XML expression and vice versa.”

    I mention this in case it is new to you, I want your project to have all the goodness it can!

    • Sam, thanks – I am aware of XML – it was kind of a gateway that led me to Common Lisp and S-expressions . I used to use XML to serialize data in my chemistry code but now I’ve switched to the Common Lisp printer/reader using S-expressions. They are more compact and fundamental than XML.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s