Things you can do with Clasp #1

Now that people are starting to build Clasp I thought I would say a few things about what you can do with it.

Clasp is an implementation of Common Lisp (80-90% complete) and I plan to make it an ANSI compliant Common Lisp as soon as possible. I also plan to add a faster compiler and expose some powerful C++ libraries to extend its capabilities.

But here are a few things you can do right away to get a sense of what Clasp does and to test some of its capabilities. Code that you type will (look-like-this). Don’t worry about how what you see isn’t colored or highlighted as shown here, I’m exploring different ways of rendering program output in this blog.

(defun add (x y) (+ x y))

ADD

(add 3 5)

8

(disassemble 'add)

Disassembling function: #<COMMON-LISP:COMPILED-FUNCTION CORE:ADD>

"#<LLVM-SYS::FUNCTION 
define internal void @ADD({ {}*, i32 }* %result-ptr, { {}* }* %closed-af-ptr, i32 %num-varargs, {}* %farg0, {}* %farg1, {}* %farg2, i8* %va-list) {
\"(TRY-0).entry\":
  %lambda-args-42- = alloca { {}* }
  call void @newAFsp({ {}* }* %lambda-args-42-)
  %temp = alloca { {}* }
  call void @newTsp({ {}* }* %temp)
  %0 = alloca { {}* }
  call void @newTsp({ {}* }* %0)
  call void @makeValueFrame({ {}* }* %lambda-args-42-, i32 2, i32 2000058)
  call void @setParentOfActivationFrame({ {}* }* %lambda-args-42-, { {}* }* %closed-af-ptr)
  %correct-num-args = icmp eq i32 %num-varargs, 2
  br i1 %correct-num-args, label %\"(TRY-0).continue3\", label %\"(TRY-0).error\"

\"(TRY-0).error\":                                  ; preds = %\"(TRY-0).entry\"
  %enough-args = icmp slt i32 %num-varargs, 2
  br i1 %enough-args, label %\"(TRY-0).error1\", label %\"(TRY-0).continue\"
...

That gobbledy-gook is pure LLVM-IR code generated by the Clasp Common Lisp compiler and using the fantastic LLVM library. It is exactly the same intermediate language that the Clang compiler converts C++/C/Objective-C into before it lowers it to machine code. Clasp shares the same LLVM backend library with the Clang compiler and once Clasp generates LLVM-IR, that is as efficient as that generated by Clang, then Clasp will generate code that runs as fast as C++ and C!

For some more information on Common Lisp you can check out the following links or use Google. I haven’t tested the tutorial against Clasp but almost everything should work.

For a tutorial:
http://en.wikibooks.org/wiki/Common_Lisp/First_steps/Beginner_tutorial

For beginners:
http://www.amazon.com/Land-Lisp-Learn-Program-Game/dp/1593272812/ref=sr_1_1?s=books&ie=UTF8&qid=1412084665&sr=1-1&keywords=land+of+lisp

I’ll post more later.

How Clasp compiles itself

This is the process that Clasp uses to compile itself from the repository. I wrote it stream-of-consciousness in response to a question on IRC and I thought I should document this for later.

Clasp starts up running with a slow S-expression walking interpreter.
It loads clasp/src/lisp/kernel/init.lsp. Within init.lsp is a list of modules that it loads to build everything. The list is in the variable *init-files* which contains a list of symbols that are translated to file pathnames by the system. The list also includes keywords like :base and :cmp – these denote way-points in the build process. Clasp then loads files from :start to :cmp into the interpreter. That loads in the Common Lisp code for the Clasp compiler. Then Clasp COMPILE-FILE(s) each source file from :start to :cmp and after compiling each file it loads the new bitcode file that is generated.
It’s slow to start but once the compiler starts compiling the files between :pre-cmp and :cmp it gets faster and faster as interpreted functions are replaced by compiled functions. Then it loads cmp/cmprepl (these files are all in src/lisp/kernel/lsp/* and src/lisp/kernel/cmp/*)
cmp/cmprepl installs a function that automatically compiles every form passed to EVAL before it evaluates it. Then it loads the files from :cmp to :min and then COMPILE-FILE(s) them. Then Clasp links all the bitcode files together with a bitcode file generated from src/llvmo/intrinsics.cc. So C++ code gets inlined into the compiled Lisp code. Then it generates min-boehm:image.bundle this is a minimal common lisp without CLOS. Then Clasp starts compiling the full-boehm version. It boots with the min-boehm:image.bundle and LOADs everything from :base to :all. Then it COMPILE-FILE(s) everything from :base to :all.
Then it links all of that together with intrinsics.cc and some epilogue code to start up the REPL.

Whew – that’s the build system.

Building Clasp and externals-clasp

Hey folks,

— Update 18:00 EST Sept 29, 2014 —

Things are starting to stabilize and multiple people are getting Clasp built on OS X and Linux. The precise systems that Clasp has been built on are listed in the README.md file on github.

We have a chatroom on IRC on freenode in #clasp. Drop in if you have some time.

On linux you need either (a) gcc 4.8 -or- (b) gcc 4.9 and clang 3.5. In the (a) case, the clang 3.6 compiler that comes with externals-clasp can be used. (Sorry, it’s a bit complicated).

These linux requirements are so specific because of changes that have been made to gcc 4.9 and llvm/clang that are outside of my control.

Thank you for being so brave and installing a new package like clasp in the first days after it was released.

You need to install externals-clasp first before installing clasp.

People are starting to post notifications about bugs in Clasp and that’s fantastic! Thank you. I’ll get to them as quickly as possible.

I love to hear peoples comments, feedback and suggestions regarding how to improve clasp.

Best,

.Chris.

Binding C++ to Clasp

This document borrows heavily from the luabind documentation. http://luabind.sourceforge.net/docs.html#intro

Clbind is a C++ template library that helps you create bindings between C++ and Clasp Common Lisp. It is very different from a typical FFI (foreign function interfaces) libraries. It allows you to expose functions, classes, methods, enums and other C++ features to Clasp. Clbind also provides the functionality to create classes in Common Lisp that subclass C++ classes and add CLOS slots to those derived classes. Common Lisp functions can override C++ methods.

Clbind is implemented using C++ template meta programming. It is heavily inspired by boost::python and luabind, two other C++ template libraries that provide interoperation between C++ and Python and Lua respectively. Clbind doesn’t require you to run extra preprocessing passes to compile your project (all of the work is done by the C++ compiler). Clbind doesn’t require you to know the exact signature of each function that you register because the C++ template library will determine this information at compile time.

Clbind closely follows the API used by luabind. I will more fully describe the Clbind API in the coming weeks.

For examples of how Clbind is used see:  clasp/src/asttooling/astExpose.cc::initialize_astExpose()  or clasp/src/asttooling/clangTooling.cc::initialize_clangTooling()

in the Clasp source code.

Announcing Clasp

Hello, everyone!

Click here for up to date build instructions

Today I am happy to make the first release of the Common Lisp implementation “Clasp”. Clasp uses LLVM as its back-end and generates native code. Clasp is a super-set of Common Lisp that interoperates smoothly with C++. The goal is to integrate these two very different languages together as seamlessly as possible to provide the best of both worlds. The C++ interoperation allows Common Lisp programmers to easily expose powerful C++ libraries to Common Lisp and solve complex programming challenges using the expressive power of Common Lisp.  Clasp is licensed under the LGPL.

Common Lisp is considered by many to be one of the most expressive programming languages in existence. Individuals and small teams of programmers have created fantastic applications and operating systems within Common Lisp that require much larger effort when written in other languages. Common Lisp has many language features that have not yet made it into the C++ standard. Common Lisp has first-class functions, dynamic variables, true macros for meta-programming, generic functions, multiple return values, first-class symbols, exact arithmetic, conditions and restarts, optional type declarations, a programmable reader, a programmable printer and a configurable compiler. Common Lisp is the ultimate programmable programming language.

Clasp has several features that make it unique and of interest to the wider programming community.

  • Clasp uses the LLVM library and has a built-in “just-in-time” compiler that generates native code that takes advantage of the fantastic LLVM toolchain.
  • Clasp exposes the LLVM C++ library within Common Lisp providing a rich, dynamic, programming environment for exploring the LLVM library and for writing new compilers that generate LLVM-IR.
  • Clasp exposes the Clang AST library and the Clang ASTMatcher library. This allows programmers to write tools in Common Lisp that automatically analyze and refactor C++ programs. Clasp can be used to automatically clean-up and refactor large C++ codebases!
  • Clasp has a built in C++ template library that makes it easy to expose other C++ (and C) libraries and make them fully available to programs written in Clasp-Common Lisp. Programmers can expose C++ classes as well as functions and class methods with a single line of code that provides the Clasp name and a pointer to the function/method. The “clbind” C++ template library works like “boost::python” and “luabind” and builds efficient wrapper functions at compile time to convert arguments and return types between Clasp and C++/C.
  • Clasp uses the Memory Pool System (MPS) a fast, compacting garbage collector by Ravenbrook. The MPS garbage collector guarantees efficient reuse of memory and provides near pause-less operation. Clasp also supports the Boehm garbage collector which is used for bootstrapping.
  • Clasp contains a static analyzer written in Clasp-CL that uses the Clang AST/ASTMatcher library to automatically write its garbage collector interface with the MPS (~15,000 LOC). This static analyzer serves as a prototype and a demonstration of the power of Clasp to analyze C++ code using the Clang C++ compiler front end.
  • Because Clasp is written in C++ from the ground up, common problems of interfacing C++ with other languages that arise from C++ name mangling, and C++ exception handling disappear.
  • Clasp builds on the excellent Common Lisp implementation: ECL “Embeddable Common Lisp”. Clasp borrows 31,500 lines of Common Lisp source code from ECL.

This is a “very alpha” release — it’s not fully tested. Clasp compiles and runs itself and its garbage collection static-analyzer. Clasp is not yet a complete Common Lisp – about 10% of the standard 978 Common Lisp symbols are not yet implemented. A faster Clasp compiler is coming soon – the current Clasp compiler generates slow native code that is about 100x slower than highly tuned Common Lisp compilers like Steel Bank Common Lisp. Currently Clasp builds on OS X 10.9 and Linux.

The Clasp project is actively looking for developers who want to contribute to this exciting open-source project. Future enhancements include – achieving full Common Lisp standard compliance, direct incorporation of C++ code within Common Lisp, a port to Windows, multi-threading (Clasp is single threaded at the moment) and incorporation of a new compiler (under development) that makes many language level optimizations. Our goal is to make Clasp the fastest and most capable dynamic scripting language for C++ libraries.

with best regards, Christian Schafmeister.

Links:

Link to Clasp github repository

Link to llvm.org

Link to ECL – Embedded Common Lisp

Link to Ravenbrook (MPS compacting garbage collector)

A Youtube video that I made that shows of Clasp C++ AST searching/refactoring