Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/makuto/cakelisp

Metaprogrammable, hot-reloadable, no-GC language for high perf programs (especially games), with seamless C/C++ interop
https://github.com/makuto/cakelisp

c-code-generator code-generation game-development lisp lisp-dialect macros

Last synced: about 1 month ago
JSON representation

Metaprogrammable, hot-reloadable, no-GC language for high perf programs (especially games), with seamless C/C++ interop

Awesome Lists containing this project

README 

        
#+title: Cakelisp



[[file:images/CakeLisp_gradient_128.png]]



* Cakelisp has moved!

Cakelisp has moved off of GitHub. *The GitHub repository will no longer be updated.*



Please visit [[https://macoy.me/code/macoy/cakelisp][Cakelisp's main host]] instead. Development will continue normally. Email [email protected] with any questions.



* Introduction



This is a programming language where I [[https://en.wikipedia.org/wiki/You_can%27t_have_your_cake_and_eat_it][can have my cake and eat it too]].



Cakelisp is a compiled language ideal for high performance, iteratively-developed applications. There is an emphasis on compile-time code generation, modification, and execution. These features are used to acquire and build 3rd party dependencies, automatically instrument the code, generate introspection metadata, and do various other things that are normally impossible with most compiled languages.



It is implemented as a transpiler which generates C or C++ from an S-expression syntax.



You can see the [[https://macoy.me/blog/programming/CakelispIntro][introduction to Cakelisp]] and check out the [[https://news.ycombinator.com/item?id=25491568][Hacker News announcement thread]]. Cakelisp is also a part of [[https://cakelisp.handmade.network/][Handmade Network]].



Note that while Cakelisp has "Lisp" in the name, Cakelisp takes some inspiration from Lisp, but is not compatible and does not aspire to become "a Lisp". I was inspired by Naughty Dog's use of GOAL, GOOL, and Racket/Scheme (on their modern titles). I've also taken several ideas from Jonathan Blow's talks on Jai.



* Features

- *The metaprogramming capabilities of Lisp:* True full-power macro support. Macros can use compile-time code execution to conditionally change what is output based on the context of the invocation

- *The performance of C:* No heavyweight runtime, boxing/unboxing overhead, etc.

- *"Real" types:* Types are identical to C types, e.g. ~int~ is 32 bits with no sign bit or anything like other Lisp implementations do

- *No garbage collection:* I primarily work on games, which make garbage collection pauses unacceptable. I also think garbage collectors add /more/ complexity than manual management

- *Hot reloading:* It should be possible to make modifications to functions /and structures/ at runtime to quickly iterate

- *Truly seamless C and C++ interoperability:* No bindings, no wrappers: C/C++ types and functions are as easy to declare and call as they are in C/C++. In order to support this, I've decided to ignore type deduction when possible and instead rely on the C compiler/linker to relay typing errors. Cakelisp will blindly generate what look like C/C++ function calls without knowing if that function actually exists, because the C/C++ compiler will tell us what the answer is

- *Compile-time code modification:* After all macros are expanded, the programmer can specify compile-time functions which can do arbitrary modification of the expanded code. This makes it possible to validate functions, automatically insert profiling instrumentation (similar to this [[https://www.youtube.com/watch?v=59lKAlb6cRg][Jai demonstration]]), and other tasks which would be cumbersome or impossible to do with macros alone

- *Output human-readable C/C++ source and header files:* This makes it possible to use Cakelisp in a subset of your project. It also means Cakelisp will work on any platform C/C++ works on. Generated code closely resembles the source Cakelisp code whenever possible

- *Build system:* Simple projects will automatically be built and linked into an executable. Complex projects can use compile-time code execution to override stages of the build process. The code essentially knows how to build itself!



For more advantages, see [[file:doc/NeatUses.org][doc/NeatUses.org]].



Some of these features come naturally from using C as the backend. Eventually it would be cool to not have to generate C (e.g. generate LLVM bytecode instead), but that can a project for another time.

* Terms

Cakelisp is copyright (C) 2021 Macoy Madson ~~.



Licensed under ~GPL-3.0-or-later~, with added [[https://www.gnu.org/licenses/gpl-faq.en.html#LinkingOverControlledInterface][Linking Over Controlled Interface]] exception.

~runtime/~ is licensed under MIT License.



Please see [[file:doc/Legal.org][doc/Legal.org]] for a detailed explanation.



Contact [email protected]~ if you would like to negotiate an exception for your use-case.

* Building Cakelisp itself

~doc/Tutorial_ProjectSetup.org~ provides a step-by-step tutorial on how to set up a Cakelisp project. More brief instructions follow.

** Linux

Run ~Build.sh~ in ~cakelisp/~:

#+BEGIN_SRC sh

cd cakelisp

./Build.sh

#+END_SRC



This script first builds cakelisp explicitly, then uses the ~cakelisp_bootstrap_v2~ executable to build Cakelisp using Cakelisp. Subsequent executions of ~Build.sh~ will also build using Cakelisp, which means all the caching features will be in effect (making builds much faster).



You can then run ~./bin/cakelisp~. Have fun! A simple test:

#+BEGIN_SRC sh

./bin/cakelisp --execute test/Hello.cake

#+END_SRC



It shouldn't be hard to build Cakelisp using your favorite build system. Simply build all the ~.cpp~ files in ~src~ (with ~UNIX~ defined), then link them into an executable which also links ~-ldl~. Leave out ~Main.cpp~ and you can embed Cakelisp in a static or dynamic library!

** MacOS

Do the same as Linux, only call the ~Build_Mac.sh~ script instead. (etc.)

** Windows

Windows does not come with a compiler installed by default. In order to provide a solid "native" experience, Cakelisp supports MSVC. It should be possible to compile on Windows using MinGW as well (in which case, you should refer to the Linux instructions, or open an Issue requesting Windows-native MinGW bootstrap building).



Once you do have a compiler, I aspire to make Cakelisp the /easiest/ way to make C/C++-based projects on Windows. You shouldn't need to touch Visual Studio project files ever again!



The following instructions will assume you are using MSVC.



1. Install Visual Studio. You can download the Community edition for free [[https://visualstudio.microsoft.com/downloads/][here]].



   Ensure that you pick the C++ track, and make sure the C++ Developer Tools box is checked (if you clicked C++, you shouldn't need to make any other changes).



   *Note* that you need to upgrade depending on what kinds of projects you use, e.g. if you are in a team making proprietary software, you must pay for Professional or Enterprise

2. Download Cakelisp. If you have git or any git client installed, clone this repository using the URL provided on this page. If you don't have Git, download the ~.zip~ file from the repository home page

3. In ~cakelisp/~, double-click ~Build.bat~.



   This should automatically find your Visual Studio and set the proper environment variables. If it doesn't, see the [[https://docs.microsoft.com/en-us/cpp/build/building-on-the-command-line?view=msvc-160#developer_command_file_locations][MSVC documentation]] on running from the command line. You may have to update the path to ~vcvars64.bat~ set in ~Build.bat~ to whatever Visual Studio version you installed. You should be able to find the script via searching for ~vcvars~ in your ~C:\Program Files (x86)~ folder.



4. Check that the build succeeded. Cakelisp is first built manually, then bootstrap builds itself using ~Bootstrap_MSVC.cake~. You may need to modify ~Build.bat~ or ~Bootstrap_MSVC.cake~ to change compiler settings. Subsequent executions of ~Build.bat~ will use cached versions to only build changed files



If you have ~cakelisp.exe~ in ~bin/~, you are ready to use Cakelisp! Due to the use of environment variables to select the compiler, you can only build programs if you've set the variables. To do so, refer to ~Build.bat~, or create a ~.bat~ file based on this template:

#+BEGIN_SRC bat

echo off

rem Set environment variables for compiler/linker selection

rem Include help message in case this file isn't present

if exist "C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Auxiliary\Build\vcvars64.bat" (

call "C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Auxiliary\Build\vcvars64.bat"

) else (

echo This script builds using MSVC.

echo You must download and install MSVC before it will work. Download it here:

echo https://visualstudio.microsoft.com/downloads/

echo Select workloads for C++ projects. Ensure you install the C++ developer tools.

echo If you're still seeing this, you may need to edit Build.bat to your vcvars path

echo Please see the following link:

echo https://docs.microsoft.com/en-us/cpp/build/building-on-the-command-line?view=msvc-160

goto fail

)



rem EDIT ME! If you keep this build script in the same directory as your project, update

rem "bin\cakelisp.exe" to wherever you have a built version of Cakelisp

rem Add --execute before the .cake files to run your project after building

"bin\cakelisp.exe" YourProgram.cake



rem Make the result clear

@if %ERRORLEVEL% == 0 (

  echo Success!

  goto success

) else (

  echo Error while building

  goto fail

)



:fail

goto end



:success

goto end



:end

rem Give the user a chance to read the input (not required)

pause

#+END_SRC

*** Building from Visual Studio

It is also possible to create a Visual Studio project to build Cakelisp. This is especially useful if you are debugging Cakelisp itself.



You can refer to an existing project in ~cakelisp/VisualStudio~. The steps to create a new project are as follows:



- Open Visual Studio

- ~File -> New -> New Project~

- ~Visual C++ -> Empty Project~

- Set the name, location, and solution as desired, then click OK

- ~Project -> Add Existing Item~

- Browse to ~cakelisp/src~, then select all the files (via ~Ctrl-A~, etc.) and click Add. This makes Cakelisp itself built by Visual Studio (i.e., not bootstrap built)

- ~Build -> Configuration Manager~

- Select ~x64~ for ~Active solution platform~. Cakelisp works in ~Win32~, but you should probably be using ~x64~ anyways, because it's the dominant architecture, and the hard-coded compile/link commands assume ~x64~

- ~Project ->  Properties...~

- In the ~Debugging~ section, set ~Working Directory~ to wherever your project's working directory is, e.g. where your ~.cake~ files are

- Set ~Command Arguments~ to the ~.cake~ files you want to build. You can leave this empty to see the Cakelisp help page

- Set the ~Environment~ to include the necessary environment variables. This is required so that Cakelisp knows how to find MSVC when launching it as a sub-process. The easiest way to find these is to open a Command Prompt, then run a ~vcvars~ script. For example, I run the following:

  #+BEGIN_SRC bat

  "C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Auxiliary\Build\vcvars64.bat"

  #+END_SRC

  Your path may be different based on your Visual Studio version and installation. See [[https://docs.microsoft.com/en-us/cpp/build/building-on-the-command-line?view=msvc-160][documentation]] on how to find this file.



  Next, run ~set~ in that same Command Prompt. Select all of the text output by that command and hit Enter to copy it. Finally, return to the ~Environment~ setting in Visual Studio, click the down arrow on the field, then ~~. Paste into the top text field, then hit OK.

- Expand ~Configuration Properties -> C/C++ -> Preprocessor~

- Double click or ~~ the ~Preprocessor Definitions~ field and add the following to the beginning:

  #+BEGIN_SRC sh

  CAKELISP_EXPORTING;_CRT_SECURE_NO_WARNINGS;WINDOWS;

  #+END_SRC

  ~CAKELISP_EXPORTING~ indicates Cakelisp should export its symbols to DLLs. The ~CRT~ definition is going to be removed eventually; it makes MSVC more lenient with some errors Cakelisp has. The ~WINDOWS~ definition ensures you build with Cakelisp's Windows-specific code enabled

- Hit ~F5~ or go to ~Debug -> Start Debugging~. Visual Studio will build Cakelisp, and if it succeeds, launch Cakelisp. If you have no ~Command Arguments~ set, you should see the Cakelisp help output in a command window.



This project is for building Cakelisp itself; you don't need to make any new projects for your projects written in Cakelisp (in ~.cake~ files). Change the ~Debugging -> Command Arguments~ setting to build different Cakelisp files, or change the ~Working Directory~ to build a different Cakelisp project (e.g. one in a separate repository).

** Dependencies

Currently, Cakelisp has no dependencies other than:

- C++ STL and runtime: These are normally included in your toolset

- Child-process creation: On *nix, ~unistd.h~. On Windows, ~windows.h~

- Dynamic loading: On *nix, ~libdl~. On Windows, ~windows.h~

- File modification times: On *nix, ~sys/stat.h~

- C++ compiler toolchain: Cakelisp needs a C++ compiler and linker to support compile-time code execution, which is used for macros and generators



I'm going to try to keep it very lightweight. It should make it straightforward to port Cakelisp to other platforms.



Note that your /project/ does not have to include or link any of these unless you use hot-reloading, which requires dynamic loading. This means projects using Cakelisp are just as portable as any C/C++ project - there's no runtime to port (except hot-reloading, which is optional).

* Building a project using Cakelisp

Cakelisp will automatically figure out how to build simple projects into executables.



For more complex projects, many hooks and variables are provided for overriding the build process. Your code is defined in Cakelisp, and so are all build commands. This gives the code the ability to know how to build itself.



For example, you could have a ~.cake~ module which includes a 3rd party graphics library. By importing that module, the module's compile-time hooks are added to the build process, which can do things like add the 3rd party graphics library's ~lib~ files to the link stage.



The build hooks are all regular Cakelisp code, which means you could do something as advanced as cloning a repository from the internet, launching a subprocess to ~cmake~ and ~make~ that project, then let Cakelisp finish the build by linking the output libraries.



One huge advantage to defining your build process in a "real" programming language (as opposed to a domain-specific language interpreted by a build system) is that you can attach a debugger and single step through the build process when things go wrong.

* Learning Cakelisp

Check out [[file:doc/Cakelisp.org][doc/Cakelisp.org]] for a detailed explanation of the Cakelisp language and build system.



Explore ~test/~ and ~runtime/~ for examples of Cakelisp code. [[https://macoy.me/code/macoy/gamelib][GameLib]] is a collection of modules built for making games in Cakelisp. Check both ~src/~ and ~test/~ in GameLib for more extensive code examples.

* Tooling support

See [[file:doc/ToolsIntegration.org][doc/ToolsIntegration.org]] for e.g. editor integrations.



** Build systems

A build system may work with Cakelisp, because Cakelisp outputs C/C++ source/header files. Cakelisp must run before your regular build system runs, or in a stage where Cakelisp can create and add files to the build. This is because Cakelisp handles its own modules such that adding support to an existing build system would be challenging.



Ideally, you should be able to rely on Cakelisp's built-in build system. This allows Cakelisp files to know how to build themselves.



If you do want to use your build system instead of Cakelisp, pass ~--skip-build~ to ~cakelisp~ so that Cakelisp generates the files but doesn't build them.

** Debugging

See [[file:doc/Debugging.org][doc/Debugging.org]]. Cakelisp doesn't really have an interpreter. Cakelisp always generates C/C++ code to do meaningful work. This means the Cakelisp transpiler, macros, generators, and final code output can be debugged using a regular C/C++ debugger like GDB, LLDB, or Visual Studio Debugger.



Mapping files will make it possible to step through code in the Cakelisp language (i.e. not in the generated language). This is similar to how debuggers allow you to step through code in C files, when under the hood it's actually stepping through machine code. It will require building support into your editor in order to properly jump to the right Cakelisp file and line (among other things).

* Similar applications/languages

See [[file:doc/VsOtherLanguages.org][doc/VsOtherLanguages.org]] for projects similar to Cakelisp.