Release 0.2 of clasp is available

I uploaded a new release of Clasp today (January 25th, 2015) that brings a lot of stability improvements, improved performance, improved compatibility with the Common Lisp standard and ASDF, SLIME and Quicklisp support.

It requires a reinstallation of externals-clasp.  We are working on eliminating the need for externals-clasp.

Features include:

  1. ASDF support has been added – it can be accessed using (require :asdf)
  2. SLIME support has been added and the changes to SLIME have been uploaded to the standard SLIME repository.  You should be able to access it the standard way you install and upgrade slime.
  3. Quicklisp support has been added.  I’ve submitted the changes to quicklisp to Zach Bean (the developer of quicklisp) and they should be available soon.
  4. Improvements in performance of the compiled code (Linux code is at least 2x faster).
  5. Improvements in stability.
  6. Almost all Common Lisp symbols have been implemented (18 left, you can see them using (core:calculate-missing-common-lisp-symbols).
  7. Example code for Clasp/C++ interoperation is available.
  8. Weak-key-hash-tables, weak-pointer, weak-key-mapping types have been implemented in both garbage collectors.

This release focused on getting SLIME working because I want to incorporate a new compiler front end for Clasp and I want to do it in this extremely powerful programming environment.

Debugging with the Clasp debugger

Clasp provides two ways of debugging code. In interactive sessions Clasp invokes a built in Common Lisp debugger when errors or other exceptional situations arise. The Clasp compiler also generates DWARF debugging information that can be used by the GDB debugger (and hopefully soon the LLDB debugger) to display Clasp Common Lisp source information interleaved with C++ source information.

To see this start up clasp and type what follows the > prompt:

Top level.
> (defun c () (break "In c"))

C
> (defun b () (c))

B
> (defun a () (b))

A
> (a)

Condition of type: SIMPLE-CONDITION
In c

Available restarts:
(use :r1 to invoke restart 1)

1. (CONTINUE) Return from BREAK.
2. (RESTART-TOPLEVEL) Go back to Top-Level REPL.

Broken at frame[14] CORE::REP.
 File: #<CORE:SOURCE-FILE-INFO #P"/Users/meister/Development/clasp/src/lisp/kernel/lsp/top.lsp"> (Position #573)
>> 

The double prompt >> indicates that we are now in the Clasp Common Lisp debugger. This debugger is inherited from ECL (because Clasp uses the excellent Common Lisp source code from ECL). To get a list of commands that are available in the debugger type:


>> :h

Top level commands:
:cf		Compile file.
:exit or ^D	Exit Lisp.
:ld		Load file.
:step		Single step form.
:tr(ace)	Trace function.
:untr(ace)	Untrace function.
:pwd	Print the current value of *default-pathname-defaults*.
:cd	Change the current value of *default-pathname-defaults*.

Help commands:
:apropos	Apropos.
:doc(ument)	Document.
:h(elp) or ?	Help.  Type ":help help" for more information.

Break commands:
:q(uit)		Return to some previous break level.
:pop		Pop to previous break level.
:c(ontinue)	Continue execution.
:b(acktrace)	Print backtrace.
:f(unction)	Show current function.
:p(revious)	Go to previous function.
:d(own)         Alias to :previous.
:n(ext)		Go to next function.
:u(p)           Alias to :next.
:g(o)		Go to next function.
:fs             Search forward for function.
:bs             Search backward for function.
:disassemble	Disassemble current function.
:l(ambda-)e(expression)	Show lisp code for current function.
:v(ariables)	Show local variables, functions, blocks, and tags.
:hide		Hide function.
:unhide		Unhide function.
:hp		Hide package.
:unhp		Unhide package.
:unhide-all     Unhide all variables and packages.
:bds            Show binding stack.
:frs            Show frame stack.
:m(essage)      Show error message.
:hs		Help stack.
:i(nspect)      Inspect value of local variable.

Restart commands:
:r1             Return from BREAK. (CONTINUE).
:r2             Go back to Top-Level REPL. (RESTART-TOPLEVEL).
>> 


Clasp/ECL use Common Lisp keywords to activate debugger functionality.

To generate a backtrace type:


>> :b

--------STACK TRACE--------
   frame#  0toplevel         epilogueForm     0/0   REPL
   frame#  1/c              top.lsp   419/2   CORE::TOP-LEVEL
   frame#  2/c              top.lsp   615/21  CORE::TPL
   frame#  3/c              top.lsp   605/32  CORE::REP
   frame#  4/b         evaluator.cc  2353/0   CORE:TOP-LEVEL-EVAL-WITH-ENV
   frame#  5/b         evaluator.cc  2351/0   CORE:COMPILE-FORM-AND-EVAL-WITH-ENV
   frame#  6/c            -no-file-     1/0   nil
   frame#  7/c            -no-file-     1/0   A
   frame#  8/c            -no-file-     1/0   B
   frame#  9/c            -no-file-     1/0   C
   frame# 10/c       conditions.lsp   457/8   COMMON-LISP:BREAK
   frame# 11/c              top.lsp  1507/9   COMMON-LISP:INVOKE-DEBUGGER
   frame# 12/c              top.lsp  1489/5   CORE::DEFAULT-DEBUGGER
   frame# 13/c              top.lsp   618/7   CORE::TPL
-->frame# 14/c              top.lsp   605/32  CORE::REP
   frame# 15/b         evaluator.cc  2353/0   CORE:TOP-LEVEL-EVAL-WITH-ENV
   frame# 16/b         evaluator.cc  2351/0   CORE:COMPILE-FORM-AND-EVAL-WITH-ENV
   frame# 17/c            -no-file-     0/0   nil
   frame# 18/c              top.lsp  1088/3   CORE::TPL-BACKTRACE
   frame# 19/b            stacks.cc   712/0   CORE:IHS-BACKTRACE

NIL
>> 

The —-> indicates the current frame that the debugger has stopped on. Since the error handling code and the debugger functions are all written in Common Lisp, those functions also appear on the backtrace. The functions we entered are in frames 7, 8, and 9.

At this point we could go to a specific frame using :g and view the environment of that frame using :v or we can print variables by just typing their names.

For now we will just leave the debugger and return to the top level REPL by invoking a restart.


>> :r2

> 

Now we are back in the top level REPL and can continue working.

Next I’ll show you how to use the DWARF generated debugging information embedded in compiled Common Lisp code to debug Clasp using GDB or LLDB.

Things you can do with Clasp #1

Now that people are starting to build Clasp I thought I would say a few things about what you can do with it.

Clasp is an implementation of Common Lisp (80-90% complete) and I plan to make it an ANSI compliant Common Lisp as soon as possible. I also plan to add a faster compiler and expose some powerful C++ libraries to extend its capabilities.

But here are a few things you can do right away to get a sense of what Clasp does and to test some of its capabilities. Code that you type will (look-like-this). Don’t worry about how what you see isn’t colored or highlighted as shown here, I’m exploring different ways of rendering program output in this blog.

(defun add (x y) (+ x y))

ADD

(add 3 5)

8

(disassemble 'add)

Disassembling function: #<COMMON-LISP:COMPILED-FUNCTION CORE:ADD>

"#<LLVM-SYS::FUNCTION 
define internal void @ADD({ {}*, i32 }* %result-ptr, { {}* }* %closed-af-ptr, i32 %num-varargs, {}* %farg0, {}* %farg1, {}* %farg2, i8* %va-list) {
\"(TRY-0).entry\":
  %lambda-args-42- = alloca { {}* }
  call void @newAFsp({ {}* }* %lambda-args-42-)
  %temp = alloca { {}* }
  call void @newTsp({ {}* }* %temp)
  %0 = alloca { {}* }
  call void @newTsp({ {}* }* %0)
  call void @makeValueFrame({ {}* }* %lambda-args-42-, i32 2, i32 2000058)
  call void @setParentOfActivationFrame({ {}* }* %lambda-args-42-, { {}* }* %closed-af-ptr)
  %correct-num-args = icmp eq i32 %num-varargs, 2
  br i1 %correct-num-args, label %\"(TRY-0).continue3\", label %\"(TRY-0).error\"

\"(TRY-0).error\":                                  ; preds = %\"(TRY-0).entry\"
  %enough-args = icmp slt i32 %num-varargs, 2
  br i1 %enough-args, label %\"(TRY-0).error1\", label %\"(TRY-0).continue\"
...

That gobbledy-gook is pure LLVM-IR code generated by the Clasp Common Lisp compiler and using the fantastic LLVM library. It is exactly the same intermediate language that the Clang compiler converts C++/C/Objective-C into before it lowers it to machine code. Clasp shares the same LLVM backend library with the Clang compiler and once Clasp generates LLVM-IR, that is as efficient as that generated by Clang, then Clasp will generate code that runs as fast as C++ and C!

For some more information on Common Lisp you can check out the following links or use Google. I haven’t tested the tutorial against Clasp but almost everything should work.

For a tutorial:
http://en.wikibooks.org/wiki/Common_Lisp/First_steps/Beginner_tutorial

For beginners:
http://www.amazon.com/Land-Lisp-Learn-Program-Game/dp/1593272812/ref=sr_1_1?s=books&ie=UTF8&qid=1412084665&sr=1-1&keywords=land+of+lisp

I’ll post more later.

How Clasp compiles itself

This is the process that Clasp uses to compile itself from the repository. I wrote it stream-of-consciousness in response to a question on IRC and I thought I should document this for later.

Clasp starts up running with a slow S-expression walking interpreter.
It loads clasp/src/lisp/kernel/init.lsp. Within init.lsp is a list of modules that it loads to build everything. The list is in the variable *init-files* which contains a list of symbols that are translated to file pathnames by the system. The list also includes keywords like :base and :cmp – these denote way-points in the build process. Clasp then loads files from :start to :cmp into the interpreter. That loads in the Common Lisp code for the Clasp compiler. Then Clasp COMPILE-FILE(s) each source file from :start to :cmp and after compiling each file it loads the new bitcode file that is generated.
It’s slow to start but once the compiler starts compiling the files between :pre-cmp and :cmp it gets faster and faster as interpreted functions are replaced by compiled functions. Then it loads cmp/cmprepl (these files are all in src/lisp/kernel/lsp/* and src/lisp/kernel/cmp/*)
cmp/cmprepl installs a function that automatically compiles every form passed to EVAL before it evaluates it. Then it loads the files from :cmp to :min and then COMPILE-FILE(s) them. Then Clasp links all the bitcode files together with a bitcode file generated from src/llvmo/intrinsics.cc. So C++ code gets inlined into the compiled Lisp code. Then it generates min-boehm:image.bundle this is a minimal common lisp without CLOS. Then Clasp starts compiling the full-boehm version. It boots with the min-boehm:image.bundle and LOADs everything from :base to :all. Then it COMPILE-FILE(s) everything from :base to :all.
Then it links all of that together with intrinsics.cc and some epilogue code to start up the REPL.

Whew – that’s the build system.

Building Clasp and externals-clasp

Hey folks,

— Update 18:00 EST Sept 29, 2014 —

Things are starting to stabilize and multiple people are getting Clasp built on OS X and Linux. The precise systems that Clasp has been built on are listed in the README.md file on github.

We have a chatroom on IRC on freenode in #clasp. Drop in if you have some time.

On linux you need either (a) gcc 4.8 -or- (b) gcc 4.9 and clang 3.5. In the (a) case, the clang 3.6 compiler that comes with externals-clasp can be used. (Sorry, it’s a bit complicated).

These linux requirements are so specific because of changes that have been made to gcc 4.9 and llvm/clang that are outside of my control.

Thank you for being so brave and installing a new package like clasp in the first days after it was released.

You need to install externals-clasp first before installing clasp.

People are starting to post notifications about bugs in Clasp and that’s fantastic! Thank you. I’ll get to them as quickly as possible.

I love to hear peoples comments, feedback and suggestions regarding how to improve clasp.

Best,

.Chris.

Binding C++ to Clasp

This document borrows heavily from the luabind documentation. http://luabind.sourceforge.net/docs.html#intro

Clbind is a C++ template library that helps you create bindings between C++ and Clasp Common Lisp. It is very different from a typical FFI (foreign function interfaces) libraries. It allows you to expose functions, classes, methods, enums and other C++ features to Clasp. Clbind also provides the functionality to create classes in Common Lisp that subclass C++ classes and add CLOS slots to those derived classes. Common Lisp functions can override C++ methods.

Clbind is implemented using C++ template meta programming. It is heavily inspired by boost::python and luabind, two other C++ template libraries that provide interoperation between C++ and Python and Lua respectively. Clbind doesn’t require you to run extra preprocessing passes to compile your project (all of the work is done by the C++ compiler). Clbind doesn’t require you to know the exact signature of each function that you register because the C++ template library will determine this information at compile time.

Clbind closely follows the API used by luabind. I will more fully describe the Clbind API in the coming weeks.

For examples of how Clbind is used see:  clasp/src/asttooling/astExpose.cc::initialize_astExpose()  or clasp/src/asttooling/clangTooling.cc::initialize_clangTooling()

in the Clasp source code.

Announcing Clasp

Hello, everyone!

Click here for up to date build instructions

Today I am happy to make the first release of the Common Lisp implementation “Clasp”. Clasp uses LLVM as its back-end and generates native code. Clasp is a super-set of Common Lisp that interoperates smoothly with C++. The goal is to integrate these two very different languages together as seamlessly as possible to provide the best of both worlds. The C++ interoperation allows Common Lisp programmers to easily expose powerful C++ libraries to Common Lisp and solve complex programming challenges using the expressive power of Common Lisp.  Clasp is licensed under the LGPL.

Common Lisp is considered by many to be one of the most expressive programming languages in existence. Individuals and small teams of programmers have created fantastic applications and operating systems within Common Lisp that require much larger effort when written in other languages. Common Lisp has many language features that have not yet made it into the C++ standard. Common Lisp has first-class functions, dynamic variables, true macros for meta-programming, generic functions, multiple return values, first-class symbols, exact arithmetic, conditions and restarts, optional type declarations, a programmable reader, a programmable printer and a configurable compiler. Common Lisp is the ultimate programmable programming language.

Clasp has several features that make it unique and of interest to the wider programming community.

  • Clasp uses the LLVM library and has a built-in “just-in-time” compiler that generates native code that takes advantage of the fantastic LLVM toolchain.
  • Clasp exposes the LLVM C++ library within Common Lisp providing a rich, dynamic, programming environment for exploring the LLVM library and for writing new compilers that generate LLVM-IR.
  • Clasp exposes the Clang AST library and the Clang ASTMatcher library. This allows programmers to write tools in Common Lisp that automatically analyze and refactor C++ programs. Clasp can be used to automatically clean-up and refactor large C++ codebases!
  • Clasp has a built in C++ template library that makes it easy to expose other C++ (and C) libraries and make them fully available to programs written in Clasp-Common Lisp. Programmers can expose C++ classes as well as functions and class methods with a single line of code that provides the Clasp name and a pointer to the function/method. The “clbind” C++ template library works like “boost::python” and “luabind” and builds efficient wrapper functions at compile time to convert arguments and return types between Clasp and C++/C.
  • Clasp uses the Memory Pool System (MPS) a fast, compacting garbage collector by Ravenbrook. The MPS garbage collector guarantees efficient reuse of memory and provides near pause-less operation. Clasp also supports the Boehm garbage collector which is used for bootstrapping.
  • Clasp contains a static analyzer written in Clasp-CL that uses the Clang AST/ASTMatcher library to automatically write its garbage collector interface with the MPS (~15,000 LOC). This static analyzer serves as a prototype and a demonstration of the power of Clasp to analyze C++ code using the Clang C++ compiler front end.
  • Because Clasp is written in C++ from the ground up, common problems of interfacing C++ with other languages that arise from C++ name mangling, and C++ exception handling disappear.
  • Clasp builds on the excellent Common Lisp implementation: ECL “Embeddable Common Lisp”. Clasp borrows 31,500 lines of Common Lisp source code from ECL.

This is a “very alpha” release — it’s not fully tested. Clasp compiles and runs itself and its garbage collection static-analyzer. Clasp is not yet a complete Common Lisp – about 10% of the standard 978 Common Lisp symbols are not yet implemented. A faster Clasp compiler is coming soon – the current Clasp compiler generates slow native code that is about 100x slower than highly tuned Common Lisp compilers like Steel Bank Common Lisp. Currently Clasp builds on OS X 10.9 and Linux.

The Clasp project is actively looking for developers who want to contribute to this exciting open-source project. Future enhancements include – achieving full Common Lisp standard compliance, direct incorporation of C++ code within Common Lisp, a port to Windows, multi-threading (Clasp is single threaded at the moment) and incorporation of a new compiler (under development) that makes many language level optimizations. Our goal is to make Clasp the fastest and most capable dynamic scripting language for C++ libraries.

with best regards, Christian Schafmeister.

Links:

Link to Clasp github repository

Link to llvm.org

Link to ECL – Embedded Common Lisp

Link to Ravenbrook (MPS compacting garbage collector)

A Youtube video that I made that shows of Clasp C++ AST searching/refactoring

Why did I write the Clasp compiler?

Imagine that you have a collection of highly optimized C++ libraries that do some amazing stuff in a very time and memory efficient way and you want to write experimental code to do some complicated things by calling the API of those C++ libraries – what do you do?

You could write everything in C++ – but C++ is a tedious language to write complex, experimental code within because there is a lot of boilerplate that you need to write. I write a lot of C++ code, a couple of hours a day, and much of my time is consumed by figuring out how to express what I want to do in C++ and then when I rewrite the code when I realize that it fell short of my desires.

You could write a wrapper library or foreign function interface to expose your C++ libraries in some dynamic language like Python, Lua and then write your experimental code in that. However, you will very quickly run into the problem that you are spending much of your time writing and maintaining the foreign function interface. You will also encounter nasty problems involving the differences in memory management, exception handling and myriad other differences between C++ and the dynamic language you choose.

I’ve spent a lot of time doing both of these things and the problems that I’ve encountered have invariably derailed my ambitions and my ideas.

So I decided to do something different.

I decided to write a language from the ground up that would smoothly interoperate with C++.

I chose the most expressive, well-defined language in existence – Common Lisp.  I’ll say more about why later.

When I started the LLVM library (http://llvm.org) was just coming into maturity and I decided to use it as the back-end for my language.  I could leverage all of the work done by really smart hobbyists and people at Google, Apple and many other companies to perform much of the back-end code generation.  Choosing LLVM would also give me access to the entire LLVM ecology of tools and I would be able to incorporate the Clang C++/C/Objective-C compiler as a C++ library to more closely weave together Common Lisp and C++.

So now we can write complex code in Common Lisp that calls C++ APIs and have C++ code call Common Lisp code while spending very little time worrying about the interface between them.

Two C++ libraries that are already exposed within Clasp are LLVM and Clang.  The LLVM library exposes C++ API for LLVM-IR generation and is used by the Clasp compiler written in Common Lisp. It’s available within Clasp and I think it’s a great playground for exploring LLVM-IR.

I also exposed the Clang C++ compiler front end library including the entire Clang AST (Abstract Syntax Tree) and the ASTMatcher library. This enables a programmer to write C++ static analyzers and C++ refactoring tools in Common Lisp!   I used it to analyze the entire Clasp source code (165 C++ source files) and automatically build ~10,000 lines of C++ code that interfaces Clasp with the Memory Pool System copying garbage collector by Ravenbrook Systems (http://www.ravenbrook.com/project/mps).

 

About me

I’m a chemistry professor who writes software to design molecules that I hope will make the world a better place. I’ve been programming since I was 12, I have written code in a lot of programming languages including Basic, X86 assembler, Pascal, Prolog, Fortran, Smalltalk, TCL, PHP, Python, C, C++ and Common Lisp. More on this later.

I run a research group in the Chemistry Department at Temple University in Philadelphia, PA. We have developed a way to make the largest, most complex and most “programmable” molecules outside of biology. We call them “spiroligomers”, they are large, shape-programmable and functional group programmable molecules that let us construct molecules that bind proteins as therapeutics and accelerate chemical reactions the way that enzymes do. The goal is to create molecules that can do everything that proteins can do in nature but be designable and evolvable by human beings. More on this later.