Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/fosskers/vend
Manage your Common Lisp project dependencies.
https://github.com/fosskers/vend
common-lisp
Last synced: 2 days ago
JSON representation
Manage your Common Lisp project dependencies.
- Host: GitHub
- URL: https://github.com/fosskers/vend
- Owner: fosskers
- License: mpl-2.0
- Created: 2025-01-04T09:53:39.000Z (23 days ago)
- Default Branch: master
- Last Pushed: 2025-01-20T21:52:32.000Z (6 days ago)
- Last Synced: 2025-01-20T22:51:15.151Z (6 days ago)
- Topics: common-lisp
- Language: Common Lisp
- Homepage:
- Size: 200 KB
- Stars: 24
- Watchers: 4
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.org
- Changelog: CHANGELOG.md
- License: LICENSE
Awesome Lists containing this project
- trackawesomelist - vend (⭐25) - Just vendor your dependencies! MPL-2.0. (Recently Updated / [Who Wants to Be a Millionare](https://www.boardgamecapital.com/who-wants-to-be-a-millionaire-rules.htm))
README
#+title: Vend
=vend= is a dependency management tool for Common Lisp. The concept is simple:
#+begin_quote
Just vendor your dependencies!
#+end_quote
=vend='s focus is first and foremost on *simplicity*.
- =vend get= to fetch dependencies directly into your project.
- =vend repl= to open an isolated Lisp session.
- =vend test= to run all your test suites.
- =vend= only has 3 dependencies itself and compiles to a 300kb binary.
- Neither Quicklisp (the tool) nor Quicklisp (the repository) are used.
- Trivial integration with Sly / Slime / Lem / Slimv.
It's time for some peace of mind.
* Table of Contents :TOC_5_gh:noexport:
- [[#why-vendor-dependencies][Why vendor dependencies?]]
- [[#why-vend][Why vend?]]
- [[#best-practices][Best Practices]]
- [[#installation][Installation]]
- [[#from-source][From Source]]
- [[#linux][Linux]]
- [[#arch][Arch]]
- [[#editor-integration][Editor Integration]]
- [[#emacs][Emacs]]
- [[#doom-emacs][Doom Emacs]]
- [[#lem][Lem]]
- [[#vim][Vim]]
- [[#usage][Usage]]
- [[#vend-get][vend get]]
- [[#vend-repl][vend repl]]
- [[#vend-graph][vend graph]]
- [[#vend-check][vend check]]
- [[#vend-search][vend search]]
- [[#vend-test][vend test]]
- [[#integrations][Integrations]]
- [[#coverage][Coverage]]
- [[#compiler-compatibility][Compiler Compatibility]]
- [[#faq][FAQ]]
- [[#how-do-i-update-dependencies][How do I update dependencies?]]
- [[#how-can-i-build-executables-of-my-application][How can I build executables of my application?]]
- [[#how-do-i-refer-to-local-dependencies][How do I refer to local dependencies?]]
- [[#can-i-install-new-dependencies-while-vend-repl-is-running][Can I install new dependencies while =vend repl= is running?]]
- [[#does-this-use-git-submodules][Does this use git submodules?]]
- [[#why-ecl][Why ECL?]]
* Why vendor dependencies?
Fast internet connections and centralised repositories have gotten us used to
the idea that dependencies are free; things we can pluck off a shelf and employ
with no risk. In languages like Javascript and Rust, it's not uncommon to have
projects with several hundred dependencies.
But are these really free? Have you inspected each one? Do you know and trust
the authors, as well as the pipeline that feeds you updates? Is your project
still guaranteed to compile in 5-10 years with no extra intervention? Can you
write code on a plane? Can users reliably build your project after little more
than a =git clone=?
The truth is that your dependencies are your code. And quite literally so - they
might make up the majority of your final compiled artifact. By vendoring your
dependencies directly into your project, you're taking responsibility for its
true form.
** Why vend?
Dependency management in Common Lisp has traditionally centred around [[https://www.quicklisp.org/beta/][Quicklisp]].
A desire for access to more rapid package updates spawned [[https://ultralisp.org/][Ultralisp]]. The need
for version pinning and isolation birthed [[https://github.com/fukamachi/qlot][Qlot]]. The want for a better
distribution system brought us [[https://github.com/ocicl/ocicl][OCICL]].
But, could there be a simpler paradigm than just /downloading the code and
putting it right there/?
With =vend=:
- We need not perform bespoke installation scripts to get started.
- We need not wait for Quicklisp to update.
- We need not relegate all our systems to =~/common-lisp/=.
- We need not worry about where ASDF is looking for systems.
- We need not fret over tools performing strange internal overrides.
- We need not manage extra config files or lockfiles.
Plus, =vend= is actually an external tool with extra commands to help you inspect
and manage your dependencies.
** Best Practices
For library development, you are encouraged to:
1. =vend get= to fetch dependencies.
2. Add =vendored/*= to your =.gitignore=.
As this allows downstream users the most freedom when consuming your library.
For application development, you are encouraged to:
1. =vend get= to fetch dependencies.
2. =rm -rf vendored/**/.git/=
3. Actively commit =vendored/= to git.
By committing these dependencies directly, it is never a mystery to your users
how your software should be provisioned.
* Installation
In all cases, =vend= requires [[https://ecl.common-lisp.dev/][ECL]] to build and run.
** From Source
=vend='s dependencies are all vendored, so it is enough to run:
#+begin_example
ecl --load build.lisp
mv vend ~/.local/bin/
#+end_example
** Linux
*** Arch
=vend= is [[https://aur.archlinux.org/packages/vend][available on the AUR]] and can be installed with tools like [[https://github.com/fosskers/aura][Aura]]:
#+begin_example
aura -A vend
#+end_example
* Editor Integration
If rough order of support / integration quality.
#+begin_quote
⚠ In all cases below, when starting a Lisp session, you must do so from the
top-level of the repository. Doing this from the =.asd= file makes it easy.
#+end_quote
** Emacs
[[https://github.com/joaotavora/sly][Sly]] and [[https://github.com/slime/slime][Slime]] have variables for setting how Lisp REPLs should be launched:
#+begin_src emacs-lisp
(setq sly-default-lisp 'sbcl
sly-lisp-implementations '((sbcl ("vend" "repl" "sbcl") :coding-system utf-8-unix)
(ecl ("vend" "repl" "ecl") :coding-system utf-8-unix)
(abcl ("vend" "repl" "abcl") :coding-system utf-8-unix)
(clasp ("vend" "repl" "clasp") :coding-system utf-8-unix)))
#+end_src
Adjust as necessary for Slime.
Note that adding ="--dynamic-space-size" "4GB"= to the =sbcl= list is useful for
hungry projects like [[https://github.com/Shirakumo/trial][Trial]].
*** Doom Emacs
As of 2025 January, you also need to manually disable =sly-stepper= and
=sly-quicklisp= or they will interfere with the REPL starting:
#+begin_src emacs-lisp
(package! sly-stepper :disable t)
(package! sly-quicklisp :disable t)
#+end_src
** Lem
[[https://lem-project.github.io/][Lem]] is built and configured in Common Lisp and so offers excellent support for
it. To start a REPL with =vend=:
#+begin_example
C-u M-x slime vend repl
#+end_example
And all your local systems will be available for loading.
** Vim
[[https://github.com/kovisoft/slimv][Slimv]] is a port of Slime from Emacs that utilises Slime's Swank backend server
for a very similar experience to Emacs. However, unlike Emacs which supports
multiple running Lisps, Slimv requires one standalone server that persists
through Vim restarts.
If we want our dependencies in =vendored/= to be visible to Slimv, we must start
its server manually from our project directory:
#+begin_example
> cd project/
> vend repl ecl --load /home/YOU/.vim/pack/common-lisp/start/slimv/slime/start-swank.lisp
#+end_example
Now, =,c= (REPL Connect) within Vim will automatically find the running server,
and you can load any system available in your project and in =vendored/=.
If you want to switch projects, you would need to quit the REPL server manually
and restart it as above. You may also wish to set a shell alias or create a
wrapper script for the long invocation shown above.
* Usage
** vend get
From the top-level directory of your project, simply =vend get= to fetch all
dependencies. They will be stored in =vendored/=. From here, they are yours. You
are free to inspect, edit, and remove them as you please.
#+begin_example
> vend get
[vend] Downloading dependencies.
[vend] Fetching FN-MACRO
[vend] Fetching ARROW-MACROS
[vend] Fetching TRANSDUCERS
...
[vend] Done.
#+end_example
If during your usage of =vend= you discover a project that fails to resolve,
please [[https://github.com/fosskers/vend/issues][open an Issue]].
** vend repl
From the top-level directory of your project, =vend repl= opens a Lisp REPL while
instructing ASDF to only look within this directory tree for =.asd= files.
#+begin_example
> vend repl
This is SBCL 2.4.9, an implementation of ANSI Common Lisp.
> (asdf:load-system :transducers)
; Lots of compilation here.
T
>
#+end_example
By default, =vend repl= starts SBCL. You can easily override this:
#+begin_example
> vend repl ecl
ECL (Embeddable Common-Lisp) 24.5.10
> (+ 1 1)
#+end_example
=vend repl= actually accepts any number of arguments, which is useful for adding
additional settings for hungry projects like [[https://github.com/Shirakumo/trial][Trial]]:
#+begin_example
> vend repl sbcl --dynamic-space-size 4GB
#+end_example
** vend graph
After running =vend get=, you can inspect your full dependency graph via =vend graph=:
#+begin_example
> vend graph
#+end_example
This produces a =deps.dot= file, which can be viewed directly with =xdot=:
#+begin_example
> xdot deps.dot
#+end_example
Or you can render it into a static PNG to send around to your friends to brag
about how few dependencies you're using:
#+begin_example
> cat deps.dot | dot -Tpng -o deps.png
#+end_example
In the case of =vend=, this produces an image like:
[[file:deps.png]]
If the graph is too messy, you can "focus" it with an extra argument to =vend graph=:
#+begin_example
vend graph lem
#+end_example
In the case of the large [[https://github.com/lem-project/lem][Lem]] project, this would display a graph of only the
core application and not its test suites, etc.
** vend check
Since your dependencies are your code, you should care about what's in there.
#+begin_example
> vend check
DYNAMIC-CLASSES is deprecated.
PGLOADER -> CL-MARKDOWN -> DYNAMIC-CLASSES
TRIVIAL-BACKTRACE is deprecated.
PGLOADER -> TRIVIAL-BACKTRACE
#+end_example
Woops! And while Common Lisp has a culture of "done means done, not dead", it's
still important to know what you're getting yourself into.
** vend search
Search the known systems via some term.
#+begin_example
> vend search woo
woo https://github.com/fukamachi/woo.git
wookie https://github.com/orthecreedence/wookie.git
#+end_example
** vend test
Detect and run testable systems. Yields a proper error code to the terminal if
failures are detected (good for CI!).
#+begin_example
> vend test
[vend] Running tests.
...
;; Summary:
Passed: 68
Failed: 0
Skipped: 0
#+end_example
Pass an additional arg to switch compilers:
#+begin_example
> vend test ecl
#+end_example
*** Integrations
| Library | Compatibility | Notes |
|-----------+---------------+--------------------------------------------|
| [[https://github.com/Shinmera/parachute][Parachute]] | ✅ | |
| [[https://github.com/lispci/fiveam][FiveAM]] | ✅* | Test system must export =all-tests= function |
| [[https://github.com/fukamachi/rove][Rove]] | ❌ | Usage of =package-inferred-system= |
If no specific (or an unknown) testing library is used, =vend= will fall back to a
naive =(asdf:test-system :foo)= call. However, this will not yield the correct
error code to the terminal in the event of test failures.
If you desire integration with libraries not listed here, please [[https://github.com/fosskers/vend/issues][open an Issue]].
* Coverage
=vend= does not cover all of what's available on Quicklisp, but it does have
significant enough coverage to resolve and compile a number of large, modern
projects:
- Resolves: Does =vend get= complete?
- Compiles: Does =(asdf:load-system :foo)= within =vend repl= complete?
- Tests: Does =(asdf:test-system :foo)= (or similar) within =vend repl= pass?
| Project | Resolves? | Compiles? | Tests? | Category | Notes |
|------------+-----------+-----------+--------+----------+-------------------------------------|
| [[https://github.com/Shirakumo/alloy][Alloy]] | ✅ | ✅ | ✅ | UI | |
| [[https://github.com/phantomics/april][April]] | ✅ | ✅ | - | Language | |
| [[https://github.com/rabbibotton/clog][Clog]] | ✅ | ✅ | - | GUI | |
| [[https://github.com/coalton-lang/coalton][Coalton]] | ✅ | ✅ | ✅ | Language | |
| [[https://github.com/bohonghuang/cl-gtk4][GTK4]] | ✅ | ✅ | - | GUI | |
| [[https://github.com/Shirakumo/kandria][Kandria]] | ✅ | ✅ | - | Game | |
| [[https://github.com/lem-project/lem][Lem]] | ✅ | ❌ | - | Editor | Usage of =package-inferred-system= |
| [[https://github.com/Lisp-Stat/lisp-stat][Lisp-stat]] | ✅ | ✅ | - | Math | |
| [[https://codeberg.org/McCLIM/McCLIM][McCLIM]] | ✅ | ✅ | ✅ | GUI | |
| [[https://github.com/fukamachi/mito][Mito]] | ✅ | ✅ | ✅ | Database | |
| [[https://codeberg.org/cage/nodgui][Nodgui]] | ✅ | ✅ | ✅ | GUI | |
| [[https://github.com/atlas-engineer/nyxt][Nyxt]] | ✅ | ✅ | ✅ | Browser | |
| [[https://github.com/ocicl/ocicl][OCICL]] | ✅ | ✅ | - | Dev tool | |
| [[https://github.com/marijnh/Postmodern][Postmodern]] | ✅ | ✅ | ✅ | Database | |
| [[https://github.com/fukamachi/qlot][Qlot]] | ❌ | - | - | Dev tool | Usage of =package-inferred-system= |
| [[https://github.com/quicklisp/quicklisp-client][Quicklisp]] | ✅ | ❌ | - | Dev tool | System is unloadable: [[https://github.com/quicklisp/quicklisp-client/issues/125][(1)]] [[https://github.com/quicklisp/quicklisp-client/issues/140][(2)]] |
| [[https://github.com/Shirakumo/radiance][Radiance]] | ✅ | ✅ | ✅ | Web | |
| [[https://github.com/roswell/roswell][Roswell]] | ✅ | ❌ | - | Dev tool | Requires =quicklisp= internally |
| [[https://github.com/Shirakumo/trial][Trial]] | ✅ | ✅ | ✅ | Gamedev | [[https://github.com/Shirakumo/trial-assets/][trial-assets]] manual setup for demos |
| [[https://github.com/fukamachi/woo][Woo]] | ✅ | ✅ | ❌ | Web | Usage of =package-inferred-system= |
If during your usage of =vend= you discover a project that fails to resolve,
please [[https://github.com/fosskers/vend/issues][open an Issue]].
* Compiler Compatibility
=vend repl= works with the following compilers:
| Compiler | Status | Notes |
|-----------+--------+-------|
| SBCL | ✅ | |
| ECL | ✅ | |
| ABCL | ✅ | |
| Clasp | ✅ | |
| CCL | ✅ | |
| Allegro | ✅ | |
| LispWorks | ❓ | |
[[https://wiki.archlinux.org/title/Common_Lisp#Historical][Historical implementations]] are not considered.
* FAQ
** How do I update dependencies?
The intent is that by vendoring, you're taking responsibility for the "true
shape" of your program. So, upgrading dependencies should always be a conscious
choice, done for a specific reason. Therefore there is no "bulk update" button.
To update a single dependency, you can =git pull= it specifically. If you've
already committed that dependency to your repo (as in the application case),
you're still able to:
#+begin_example
rm -rf vendored/old-dep
vend get
rm -rf vendored/old-dep/.git/
#+end_example
But you're discouraged from doing this habitually.
** How can I build executables of my application?
See [[file:build.lisp][build.lisp]] for how =vend= is built, which uses ECL. For SBCL, consider adding:
#+begin_src lisp
(sb-ext:save-lisp-and-die #p"foobar"
:toplevel #'foobar:launch
:executable t
:compression t)
#+end_src
** How do I refer to local dependencies?
=vend get= fetches dependencies it knows about via =git=, but sometimes you want to
refer to a dependency that already exists somewhere else on your local machine.
To trick =vend=, you can either:
- Make a copy of the local project within =vendored/=.
- Create a symlink inside =vendored/= that refers to the local project.
Then, when running =vend get= it will see the folder you added and assume it had
already fetched it via a previous call to =vend get=. Likewise, =vend repl= should
"just work".
** Can I install new dependencies while =vend repl= is running?
Probably not. At least, =vend= assumes that Quicklisp doesn't exist, and it tells
ASDF to only look for systems in the current directory tree. It's not clear what
a call to =(ql:quickload ...)= would do in that case.
If you want new packages available to =vend repl=, you can:
- Manually clone them into =vendored/= (discouraged).
- Add them to your =.asd= explicitly and call =vend get=.
Either way, it's expected that you ensure that when a user freshly clones your
repository, runs =vend get=, and then loads your system, everything should work.
** Does this use git submodules?
No. Submodules need to be recloned by your users, which is a weaker long-term
guarantee than true vendoring. Submodules are also a pain in the neck to manage.
With =vend=, if you want to change and make new commits to vendored dependencies,
you're still free to do so.
** Why ECL?
=vend= itself is built with [[https://ecl.common-lisp.dev/main.html][ECL]] but you're free to use it to manage Lisp projects
of any (modern) compiler. ECL typically produces very small binaries; in the
case of =vend= it's only a few hundred kilobytes, which eases distribution.