{"id":20272304,"url":"https://github.com/steemit/tinman","last_synced_at":"2025-08-17T10:04:11.355Z","repository":{"id":30458916,"uuid":"109043032","full_name":"steemit/tinman","owner":"steemit","description":"Testnet management scripts","archived":false,"fork":false,"pushed_at":"2021-12-04T01:21:13.000Z","size":233,"stargazers_count":8,"open_issues_count":24,"forks_count":20,"subscribers_count":20,"default_branch":"develop","last_synced_at":"2025-04-11T04:41:55.471Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/steemit.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2017-10-31T19:28:02.000Z","updated_at":"2021-12-04T01:17:47.000Z","dependencies_parsed_at":"2022-08-07T15:15:50.679Z","dependency_job_id":null,"html_url":"https://github.com/steemit/tinman","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/steemit/tinman","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/steemit%2Ftinman","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/steemit%2Ftinman/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/steemit%2Ftinman/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/steemit%2Ftinman/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/steemit","download_url":"https://codeload.github.com/steemit/tinman/tar.gz/refs/heads/develop","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/steemit%2Ftinman/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":270831792,"owners_count":24653414,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-17T02:00:09.016Z","response_time":129,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-14T12:42:47.905Z","updated_at":"2025-08-17T10:04:11.291Z","avatar_url":"https://github.com/steemit.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\n# Overview\n\nThe `tinman` set of utilities is a set of scripts to create a testnet.\nA `tinman` testnet allows all, or some subset of, user accounts to\neasily be *ported* from the main network.\n\n# Tinman commands\n\nThis repository contains utilities to create a testnet.\n\n`tinman snapshot` : Gets account data and other data from the blockchain as necessary to do offline initialization of testnet\n`tinman txgen` : Translate the output of `snapshot.py` to a set of actions to perform offline initialization of testnet\n`tinman keysub` : Substitute secret keys into a list of actions\n`tinman submit` : Submit the output of `txgen.py` to testnet node\n\n# Installation\n\n## Linux\n\n```bash\n$ sudo apt-get install virtualenv python3 libyajl-dev git\n```\n\n## macOS\n\n```bash\n$ brew install python3 yajl\n$ pip3 install virtualenv\n```\n\n## Creating a virtualenv\n\nIn this step we create a virtualenv to isolate our project from the\nsystem-wide Python installation.  The virtualenv is *activated*,\nmodifying the `PATH` and the prompt of the current shell,\nby sourcing the `activate` script:\n\n```bash\n$ virtualenv -p $(which python3) ~/ve/tinman\n$ source ~/ve/tinman/bin/activate\n```\n\n## Dependency Notes\n\nTinman should work right out of the box, but on some more delicately configured machines, some users report `ijson` errors.  Running `pip install ijson` or `pip3 install ijson` should take care of that.\n\nThe `ijson` requirement also uses `yajl` for performance improvements.  But `yajl` is optional and if it cannot be installed, there will be a warning that can be ignored.\n\n## Using tinman\n\nThe `tinman` source can be checked out with `git`.  This documentation\nassumes the source code lives in `~/src/tinman`:\n\n**Note:**`tinman`'s default branch is develop. `master` is condsidered stablish.\n\n```bash\n$ mkdir -p ~/src\n$ cd ~/src\n$ git clone --branch master https://github.com/steemit/tinman.git\n$ cd tinman\n$ pip install pipenv\n$ pipenv install\n$ pip install .\n```\n\nIf everything is set up correctly, you should be able to run commands\nsuch as `tinman --help` as follows:\n\n```bash\n# Execute inside tinman virtualenv\n$ tinman --help\n```\n\nNote, the `tinman` script in `~/ve/tinman/bin/tinman` may be symlinked\nelsewhere (for example, `ln -s ~/ve/tinman/bin/tinman ~/bin/tinman`)\nto allow `tinman` to run without the `virtualenv` being active.\n\n# Example Usage\n\n```bash\n# First, take a snapshot of all accounts on mainnet using your own local mainnet\n# node on port 8090.\n$ tinman snapshot -s http://127.0.0.1:8090 -o snapshot.json\n```\n\nOnce the `snapshot.json` file has been created, copy `txgen.conf.example` to\n`txgen.conf`:\n\n* `snapshot_file` - make sure this is the same name as your new `snaptshot.json`\n\n```bash\n# Next, create actions.\n$ tinman txgen -c txgen.conf -o txgen.actions\n```\n\nCreate a bash script, call it `bootstrap.sh`, make sure the `--get-dev-key` and `--signer` arguments point to the correct binaries:\n\n```bash\n# Port the actions over to your local bootstrap node on port 9990, with a secret\n# set to \"xyz-\"\n( \\\n  echo '[\"set_secret\", {\"secret\":\"xyz-\"}]' ; \\\n  cat txgen.actions \\\n) | \\\ntinman keysub --get-dev-key /path/to/steem/programs/util/get_dev_key | \\\ntinman submit --realtime -t http://127.0.0.1:9990 \\\n  --signer /path/to/steem/programs/util/sign_transaction \\\n  -f fail.json \\\n  -t 600\n```\n\nAfter allowing this script to run, you have now bootstrapped your testnet and you can point your witnesses at this node to start seeding and signing blocks.\n\nTo check the number of accounts created on the bootstrap node:\n\n```bash\n$ curl -s --data '{\"jsonrpc\":\"2.0\", \"method\":\"condenser_api.get_account_count\", \"params\":[], \"id\":1}' http://localhost:9990\n```\n\n# Detailed Usage\n\nMore detail about general usage:\n\n* [Mainnet steemd](#mainnet-steemde)\n* [Taking a snapshot](#taking-a-snapshot)\n* [Generating actions](#generating-actions)\n* [Keys substitution](#keys-substitution)\n  * [Deriving secret keys](#deriving-secret-keys)\n  * [Command-line key generator](#command-line-key-generator)\n* [Running testnet fastgen node](#running-testnet-fastgen-node)\n  * [Pipelining transactions to testnet](#pipelining-transactions-to-testnet)\n* [Durables](#durables)\n* [Warden](#warden)\n* [Gatling transactions from mainnet](#gatling-transactions-from-mainnet)\n* [Running testnet witness node(s)](#running-testnet-witness-node-s-)\n* [Tests](#tests)\n\n## Mainnet steemd\n\nFirst, we set up a `steemd` for the main network.  This `steemd` must be the following characteristics:\n\n- The `steemd` must be `appbase` version\n- The `chain`, `webserver` and `database_api` plugins must be enabled\n- The `webserver-http-endpoint` assumed by the following examples is `127.0.0.1:8090`\n- If a snapshot at a well-defined single point in time is desired, no seed nodes should be used, so it does not connect to the p2p network\n\n## Taking a snapshot\n\n```bash\n$ tinman snapshot -s http://127.0.0.1:8090 -o snapshot.json\n```\n\nAs of this writing, the above command takes approximately 5 minutes, writing an approximately 2 GB JSON file with 1,000,000 lines.\nIf you're running `tinman snapshot` interactively and you would like a visual progress indicator, you can install the `pv` program\n(`apt-get install pv`) and use it to display the output line count in real time:\n\n```bash\n$ tinman snapshot -s http://127.0.0.1:8090 | pv -l \u003e snapshot.json\n```\n\n## Generating actions\n\nNow you can use `tinman txgen` to create a list of *actions*.  Actions include\ntransactions which create and fund the accounts, and wait-for-block instructions\nwhich control the rate at which transactions occur:\n\n```bash\n# As of this writing, this command takes ~10 minutes to start writing actions,\n# consumes ~200MB of RAM, with all actions created in about two hours\n$ tinman txgen -c txgen.conf -o tn.txlist\n```\n\nSome notes about `tinman txgen`:\n\n- All accounts have `porter` as an additional authority, allowing the testnet creator to act as any account on the testnet\n- The private keys for `porter` and other accounts are deterministically created based on the `secret` option in the config file\n- Balances are created by dividing `total_port_balance` proportionally among the live STEEM and vesting, subject to `min_vesting_per_account`.\n- Therefore, testnet balance is not equal to mainnet balance.  Rather, it is proportional to mainnet balance.\n- Accounts listed in `txgen.conf` are considered system accounts, any identically named account in the snapshot will not be ported\n\n## Keys substitution\n\nTo maintain separation of concerns in `tinman` tools, the `tinman txgen` tool\ndoes not directly generate transactions containing private keys (except\nthe `STEEM_INIT_PRIVATE_KEY` WIF, `5JNHfZYKGaomSFvd4NUdQ9qMcEAC43kujbfjueTHpVapX1Kzq2n`).\nInstead *keystrings* such as `publickey:active-porter` are outputted in place\nof the actual public key of the `porter` account.\n\nSince transactions can contain arbitrary user-specified data, a fixed\nescape sequence cannot be used to delimit keystrings, since it might appear\nin user-specified data.  Instead, a variable escape sequence (a short value\nthat does not appear in the data) is introduced and stored in the `\"esc\"`\nvariable.\n\nSo a program that knows the keys must substitute them before the transactions\ncan be submitted to the network.  This is the role of the key substitution tool\n`tinman keysub`.  The `tinman keysub` tool takes as input a list of actions,\ngenerates the specified keys, and substitutes them into each action.\n\n### Deriving secret keys\n\nBy default, the private keys generated by `tinman keysub` have\nknown (i.e. insecure) seed strings.  However, a secret may be\nadded to each seed string by prepending a `set_secret` action\nto `tinman keysub`.\n\n### Command-line key generator\n\nThe `get_dev_key` program provided with `steemd` derives\nkeys using the same algorithm as `tinman keysub`.\n\n## Running testnet fastgen node\n\nNow that the transactions have been created, let's use them to initialize a testnet.\nSince many blocks worth of transactions are created, `tinman submit` will\nimplement the block wait using `debug_node_plugin` to generate blocks in the\npast as rapidly as possible.  So we will run a special node, let's call it the\n\"fastgen node\", with the debug plugin enabled.  The fastgen node is only used to\ninitialize the network.  (It is called \"fastgen\" because it generates blocks\nas fast as possible, rather than waiting 3 seconds of real time between each\nblock.)  Later, one or more normal witness nodes will connect\nto the fastgen node over p2p, get blocks, and begin normal block production.\n\nThe fastgen node needs the following:\n\n- The `steemd` must be `appbase` version\n- The testnet `blockchain` directory should be empty (try `rm -Rf testnet_datadir/blockchain`)\n- The following plugins should be enabled:  `chain p2p webserver debug_node database_api network_broadcast_api debug_node_api block_api`\n- The `webserver-http-endpoint` assumed by the following examples is `127.0.0.1:9990`\n- It must contain functionality from PR's #1722 #1723\n- It should listen for p2p, the following examples assume it is listening on `0.0.0.0:12001`\n\nOn the testnet, some serializations are different from the main network, and\n[they are not handled properly by steem_python](https://github.com/steemit/steem-python/issues/89).\nTherefore, `tinman submit` outsources signing of those transactions to the\n`sign_transaction` binary included with `steemd`.\n\n### Pipelining transactions to testnet\n\n```bash\n( \\\n  echo '[\"set_secret\", {\"secret\":\"xyz-\"}]' ; \\\n  tinman txgen -c txgen.conf \\\n) | \\\ntinman keysub | \\\ntinman submit -t http://127.0.0.1:9990 --signer steem/programs/util/sign_transaction -f fail.json\n```\n\n# Other Modules\n\nOnce the testnet has been bootstrapped, other modules can be used to facilitate deeper orchestration.\n\n## Durables\n\nFor consistency across testnet deployments, fixture-like object that must exist for external testing are recreated by the `durables` module.\n\nCopy `durables.conf.example` to `durables.conf`, add any desired objects, and run (typically after initial bootstrap and before `gatling`):\n\n```bash\n( \\\n  echo '[\"set_secret\", {\"secret\":\"xyz-\"}]' ; \\\n  tinman durables -c durables.conf \\\n) | \\\ntinman keysub | \\\ntinman submit -t http://127.0.0.1:9990 --signer steem/programs/util/sign_transaction -f die\n```\n\n## Warden\n\nUse `warden` to check the current condition of a given chain.  It does some\nbasic checks to make sure the chain is up and running, then returns error codes.\n\nReturning error code zero (0) means everything looks good.  Non-zero means\nsomething is amiss.\n\n```bash\n$ tinman warden -s http://127.0.0.1:8090 \u0026\u0026 echo LGTM || echo Bummer.\n```\n\nAs an example, you can add `warden` to your deployment script to delay the next step until your seed node has synchronized with the initial bootstrap node.\n\n```bash\nwhile [[ $all_clear -ne 0 ]]\ndo\n    tinman warden -s http://my-seed-node:8080\n    all_clear=$?\n    echo Waiting for warden to sound the all-clear.\n    sleep 60\ndone\n\necho Ready to proceed.\n```\n\n## Gatling transactions from mainnet\n\nPopulating the test network with transactions from the main network.\n\nTo stream from genesis:\n\n```bash\n$ tinman gatling -f 1 -o -\n```\n\nTo stream from block 25066272 to 25066292:\n\n```bash\n$ tinman gatling -f 25066272 -t 25066292 -o -\n```\n\nTo stream starting from block 25066272:\n\n```bash\n$ tinman gatling -f 25066272 -o -\n```\n\n## Running testnet witness node(s)\n\nAt the end of the transactions to be submitted, `tinman txgen` creates witnesses `init-0` through `init-20`\nand votes for them with large amount of `TESTS`.  The keys of these witnesses are generated by a deterministic\nalgorithm compatible with the `get_dev_key` utility program, so the keys of the witnesses may be obtained\nas follows:\n\n```bash\n$ programs/util/get_dev_key xxx- block-init-0:21\n```\n\nwhere `xxx` is the `\"secret\"` string in `txgen.conf`.\n\nSo in order to transition block production duties away from the initial node, all that is required\nis to connect witness nodes with the correct block production settings.  Each witness node should\nspecify the fastgen node using the `p2p-seed-node` option in the config file.\n\nTherefore we may add the witness definitions and private keys to the witness config file:\n\n```bash\ni=0 ; while [ $i -lt 21 ] ; do echo witness = '\"'init-$i'\"' \u003e\u003e testnet_datadir/config.ini ; let i=i+1 ; done\nsteem/programs/util/get_dev_key xxx- block-init-0:21 | cut -d '\"' -f 4 | sed 's/^/private-key = /' \u003e\u003e testnet_datadir/config.ini\n```\n\nWitness duties may of course be split among multiple nodes if desired, simply put the\n`witness` and `private-key` definitions in the datadir of each node, and have each\nwitness node connect to all the others with the `p2p-seed-node` option.\n\nAdditionally, because there is a large gap in block timestamps between the end of the\ninitialization blocks and the beginning of normal production, the witness will need\nto specify `--enable-stale-production` and `--required-participation=0` flags.  As\nlong as a sufficient number of other witness nodes are timely producing blocks, it\nis not necessary to use these flags once 128 blocks have been produced after the\ntransition.\n\n## Tests\n\nTo test `tinman`:\n\n```bash\n$ cd test\n$ pip install .. \u0026\u0026 python -m unittest *_test.py\n```\n\n\u003cimg src=\"https://i.imgur.com/h57pDVE.png\" width=\"25%\" height=\"25%\" /\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsteemit%2Ftinman","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsteemit%2Ftinman","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsteemit%2Ftinman/lists"}