Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/cmdruid/regtest-workbench

A containerized stack of Bitcoin and Lightning, plus a full suite of development tools. Prototype and deploy your next project with lightning speed!
https://github.com/cmdruid/regtest-workbench

bitcoin core-lightning developer-tools lightning-network regtest

Last synced: about 1 month ago
JSON representation

A containerized stack of Bitcoin and Lightning, plus a full suite of development tools. Prototype and deploy your next project with lightning speed!

Awesome Lists containing this project

README 

        
# Regtest Workbench



A robust, featured-rich development environment for building on-top of Bitcoin and Lightning.



Launch multiple Bitcoin / Lightning nodes, and watch them form an automated network of miners, validators, and lightning channels. Each node contains an interactive terminal, web wallet, and suite of development tools.



Also included in each node:

 - Multiple interfaces to connect with: Spark, REST, LightningRPC, and more.

 - Full Tor (V3) services support for all interfaces (plus peering!).

 - Support for latest version of Zeus wallet via REST and sparko connect.

 - Fully automated opening and balancing of channels between nodes.

 - Automated loading and hot-loading of plugins for Core Lightning.

 - Multiple launch configurations: safe mode, dev mode, and headless mode.

 - Plugins for auto-pay and invoice traffic generation.



Everything is configured out of the box. Simply open up a node port and connect your application. Prototype and deploy your next project with lightning speed!



## How to Use



> *Note: Make sure that docker is installed, and your user is part of the docker group.*



To spin up a basic network of three nodes:



```shell

## Clone this repository, and make it your working directory.



git clone "https://github.com/cmdruid/regtest-workbench.git"

cd regtest-workbench



## Compiles all the binaries that we will need. May take a while!

./workbench.sh build



## Your first (and main) node. Seeds the blockchain and mines blocks.

./workbench.sh start master --miner



## Meet Alice, who connects to master and requests funding.

./workbench.sh start alice --faucet master



## Meet Bob, also funded by master, who peers and opens a channel with Alice.

./workbench.sh start bob --faucet master --channels alice



## Meet Carol, who likes to use shorter commands, and start in headless mode.

./workbench.sh start carol -f master -c alice,bob --headless



## ... repeat for as many nodes as you like!



## You can also launch many nodes at once.

./workbench.sh spawn



## A detailed guide is built into the help screen.

./workbench.sh --help

```

The `start` keyword designates your node with a name tag. Each node will automatically connect to its designated peers, request funds from a faucet, open payment channels, and collectively balance those channels.



The `--peers` flag will instruct nodes on whom to peer with, and share transactions / blocks.

The `--faucet` flag will instruct your node to peer and request funds from another node.

The `--channels` flag will instruct your node to peer and open channels with another node.

The `--peers` and `--channels` flags accept a comma-separated list of nametags (*e.x alice,bob,carol*).



By default, nodes will request a balance of 10 BTC, and open channels with a balance of 5 million sats (split 50/50 between parties). You can modify these settings (and much more) by editing the `.env.sample` file at the root of this repo.



Use the `--miner` flag with your first node in order to initiate the chain. Your miner will detect an empty chain, then auto-generate blocks up to a certain height. Block rewards require at least 100 blocks to mature, so the default starting height is 150.



By default, Your miner is configured to watch the mempool, then auto-generate blocks when it sees an unconfirmed transaction. The default poll time is 2 seconds, so transactions should confirm quickly. If you wish to deploy multiple miners, use a different configuration that avoids chain splits.



The format for configuring each miner is `--miner=poll-time,interval-time,fuzz-amount` in seconds. For example, the configuration `0,10,20` will disable polling, schedule a block every 10 seconds, plus a random value between 1 and 20 seconds.



> Tip: *Use your initial miner node as your main faucet, since the block rewards will have made him filthy rich! Miners may also generate more blocks in order to procure funds if their wallet balance is low.*



Nodes with the `--rest` flag will provide a REST interface on port 3001 (provided by CL-REST). Include `--ports 3001 --ports 4001` if you wish to connect to it locally.



Nodes with the `--tor` flag will auto-magically use onion routing when peered with other tor-enabled nodes, no extra configuration required. This flag will also configure a node's endpoints as (v3) hidden services. Tor nodes can still communicate with non-tor nodes, but will default to using tor when available (unless given the `--local` flag).



All pertinent settings, keys and credentials for a node is namespaced and stored in the `/share` folder. Nodes will use this folder in order to peer and communicate with other nodes. The files for each node are refreshed when you restart that node, so configuration data will always be current.



> Note: *Nodes are designed to work completely over tor, using the `/share` folder. You can copy / distribute a node's shared files onto other machines and build a regtest network across the web!*



Each node launches with a simple web-wallet for managing invoices and test payments (*provided by sparko*). Nodes will print their local address and login credentials to the console upon startup. You can also manage payments easily using `bitcoin-cli` and `lightning-cli`.



> Tip: *The short-hand `bcli` and `lcli` are also available. Check out the `config/.bash_aliases` file for more handy aliases!*



Use the `--devmode` flag to enter a node's console before the startup script is run. This flag will also mount the /run folder directly, allowing you to hack / modify the source code for this project in real-time. Don't forget to use version control. ;-)



> Tip: *If you have a node suffering from issues, run the `node-start` command within the node's terminal. It is the main start script, and designed to help refresh configurations, restart services, or diagnose / resolve issues that crop up during the startup process. If all else fails, restart the node using the`--wipe` flag to erase its persistent storage, and give your node a fresh start!*



All nodes ship with Flask and Nodejs included, plus a core library of development tools. Check out the example projects located in `contrib/examples` if you want to jump into web/app development right away!



> *FYI contrib/examples is sparse at the moment. More demo projects will be added mid-June. Please contact me if you would like to help contribute!*



## Repository Overview



### \#\# ./build



This path contains the build script, related dockerfiles, and compiled binaries. When you run the `workbench.sh` script, it will fist scan the `build/dockerfiles` and `build/out` path in order to compare files. If a binary appears to be missing, the start script will then call the build script (with the related dockerfile), and request to build the missing binary from source. Compiled binaries are then copied `build/out`.



If you have just cloned this repositry, it's best to run `./workbench.sh build` as a first step, so that launching your first node doesn't force you to compile everything at once.



All files located in `build/out` are copied over to the main docker image and installed at build time, so feel free to include any binaries you wish! The script recognizes tar.gz compression, and will strip the first folder before unpacking into `/usr`, so make sure to pack your files accordingly.



You can also add your own `build/dockerfiles`, or modify the existing ones in order to try different builds and versions. If you add a custom dockerfile, make sure it also names the binary with a matching substring, so the start script can correctly determine if your binary is present / missing.



### \#\# ./config



These are the configuration files used by the main services in the stack. The entire config folder is copied at build time, to `/root/config` in the image. Feel free to modify these files or add your own, then use commands `build` or `rebuild` to refresh the image.



The `.bash_aliases` file is also loaded upon startup, feel free to use it to customize your environment!



### \#\# ./contrib



*Work in progress. Will contain example templates and demos for building bitcoin / lightining connected apps.*



### \#\# ./run



This folder contains the main `entrypoint.sh` script, libraries and tools, plus all other scripts used to manage services in the stack.



- Most of the source code for this project is located in `lib`.

- Files in `lib\bin` are available in the container's PATH.

- Files in `lib\pylib` are available in the container's PYPATH.

- Files in `lib\nodelib` are available in the container's NODEPATH.

- Files in `plugins` are loaded by lightningd at startup (*main script must match folder name*).

- Scripts in `startup` are executed in alpha-numeric order when `node-start` is run.



The entire run folder is copied at build time, located at `/root/run` in the image. Feel free to modify these files or add your own, then use `--build` or `--rebuild` flags to refresh the image when restarting a node. When a node is started in `--devmode`, the `run` folder is mounted directly, and you can modify the source files in real-time.



### \#\# ./share



Each node will mount this folder on startup, then use it as a shared repository for providing their own configuration data. Each folder is namespaced after a node's hostname, and used by other nodes in order to peer and connect with each other. These files are constantly created and destroyed by their respective nodes, so that the data remains fresh and accurate.



If tor is enabled for a given node, its share data can also be copied to other machines for more complex configurations.



## CL-REST



Starting workbench with `--ports 3001` maps the `3001` port to the host's `3001` port. `3001` is CL-RESTS's data port. Likewise with `--ports 4001`, giving you access to the documentation port `https://localhost:4001/api-docs`.



### Copying the cert and macaroon



You'll want to copy the cert and macaroon out of the container:



```

docker cp bob.regtest.node:/data/certs/certificate.pem .

docker cp bob.regtest.node:/data/certs/access.macaroon .

```



Here's an sample Python call to the '/v1/getinfo` endpoint:



```python

import base64, codecs, json, requests



url = "https://localhost:3001/v1/getinfo"

cert_path = "certificate.pem"

macaroon = codecs.encode(open("access.macaroon", "rb").read(), "hex")

headers = {"macaroon": macaroon, "encodingtype": "hex"}

r = requests.get(url, headers=headers, verify=cert_path)

print(r.json())

```



## Development



*Work in progress. Feel free to hack the project on your own!*



There are three main modes to choose from when launching a container: **safe mode**, **development mode**, and **headless mode**.



By default, a node will launch in safe mode. A copy of the `/run` folder is made at build time, and code changes to `/run` will not affect the node (unless you rebuild and re-launch). The node will continue to run in the background once you exit the node terminal. The node is also configured to self-restart in the event of a crash.



When launching a node with `--devmode` enabled, a few things change. The `node-start` script will not start automatically; you will have to call it yourself. The `/run` folder will be mounted directly into the container, so you can modify the source code in real-time.



Any changes to the source code will apply to *all* nodes. Re-run the start script to apply changes. When you exit the terminal, the container will be instantly destroyed, however the internal `/data` store will persist.



If you end up borking a node, use the `--wipe` flag at launch to erase the node's persistent storage. The start scripts are designed to be robust, and nodes are highly disposable. Feel free to crash, wipe, and re-launch nodes as often as you like!



Both safe-mode and dev-mode can be augmented with `--headless` mode, which launches a node without connecting to it. You can still monitor nodes through a management service like **portainer**, or simply login to the node using `./workbench login *nametag*`.



To mount folders into a node's environment, use the format `--mount local/path:/mount/path` for each folder you wish to mount. Paths can be relative or absolute.



To open and forward ports from a node's environment, use the format `--ports int1:ext1,int2:ext2, ...`, with a comma to separate port declarations, and colon to separate internal:external ports.



The `--passthru` flag will allow you to pass a quoted string of flags directly to the internal `docker run` script. With great power comes great responsibility! :-)



## Contribution



All suggestions and contributions are welcome! If you have any questions about this project, feel free to submit an issue up above, or contact me on social media.



## Tools



**Bitcoin Core Config Generator**  

Covers the myriad of options available for bitcoin.conf.  

https://jlopp.github.io/bitcoin-core-config-generator



**LNURL Decoder**

Decode lnurl strings.    

https://lnurl.fiatjaf.com/codec



## Resources



**Bitcoin Developer Reference**  

A great resource for documentation on Bitcoin's RPC interface, and other technical details.  

https://developer.bitcoin.org/reference/index.html



**Bitcoin Github Docs**  

Another great resource for all things related to Bitcoin Core.  

https://github.com/bitcoin/bitcoin/tree/master/doc



**Core Lightning Documentation**  

The go-to resource for Core Lightning's RPC interface.  

https://lightning.readthedocs.io



**Core Lighting REST API Docs**  

Documentation for the REST interface that is provided by the RTL team.  

https://github.com/Ride-The-Lightning/c-lightning-REST#apis



**LNURL Specification**  

Documentation that details the LNURL specification.  

https://github.com/fiatjaf/lnurl-rfc



**Bolt 12 Landing Page**  

A nice landing page for info regarding the Bolt 12 specification.  

https://bolt12.org



**Sparko Client**  

Incredibly useful web-rpc interface and web-client for Core Lightning.  

https://github.com/fiatjaf/sparko-client



**Pyln-client**  

The main library for interfacing with Core Lightning over RPC. Very powerful.  

https://github.com/ElementsProject/lightning/tree/master/contrib/pyln-client



**Python Documentation**  

Official resource for the python language.  

https://docs.python.org/3



**Flask Documentation**  

The go-to resource for documentation on using Flask.  

https://flask.palletsprojects.com/en/2.1.x



**Node.js Documentation**  

The go-to resource for documentation on using Nodejs.  

https://nodejs.org/api