{"id":16300517,"url":"https://github.com/melezhik/sparky","last_synced_at":"2025-12-24T21:39:01.794Z","repository":{"id":48531187,"uuid":"184821756","full_name":"melezhik/sparky","owner":"melezhik","description":"Lightweight CI server  and task runner ","archived":false,"fork":false,"pushed_at":"2024-05-28T13:49:03.000Z","size":789,"stargazers_count":14,"open_issues_count":30,"forks_count":0,"subscribers_count":3,"default_branch":"master","last_synced_at":"2024-05-29T02:26:56.373Z","etag":null,"topics":["ci","devops","rakulang"],"latest_commit_sha":null,"homepage":null,"language":"Raku","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/melezhik.png","metadata":{"files":{"readme":"README.md","changelog":"Changes","contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-05-03T21:21:56.000Z","updated_at":"2024-05-30T10:16:08.589Z","dependencies_parsed_at":"2024-05-20T21:04:02.151Z","dependency_job_id":"8ce26910-aac0-4ff5-b93c-eb116e2d659d","html_url":"https://github.com/melezhik/sparky","commit_stats":null,"previous_names":[],"tags_count":6,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/melezhik%2Fsparky","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/melezhik%2Fsparky/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/melezhik%2Fsparky/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/melezhik%2Fsparky/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/melezhik","download_url":"https://codeload.github.com/melezhik/sparky/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243818171,"owners_count":20352629,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ci","devops","rakulang"],"created_at":"2024-10-10T20:51:29.528Z","updated_at":"2025-12-24T21:39:01.785Z","avatar_url":"https://github.com/melezhik.png","language":"Raku","funding_links":[],"categories":[],"sub_categories":[],"readme":"# SYNOPSIS\n\nSparky is a flexible and minimalist continuous integration server and distribute tasks runner written in Raku.\n\n![Sparky Logo](https://raw.githubusercontent.com/melezhik/sparky/master/logos/sparky.small.png)\n\nSparky features:\n\n* Defining jobs scheduling times in crontab style\n* Triggering jobs using external APIs and custom logic\n* Jobs scenarios are pure Raku code with additional support of [Sparrow6](https://github.com/melezhik/Sparrow6/blob/master/documentation/dsl.md) automation framework\n* Use of plugins on different programming languages\n* Everything is kept in SCM repository - easy to port, maintain and track changes\n* Jobs get run in one of 3 flavors - 1) on localhost 2) on remote machines via ssh 3) on docker instances\n* Nice web UI to run jobs and read reports\n* Could be runs in a peer-to-peer network fashion with distributed tasks support\n\n# Build status\n\n![Github actions](https://github.com/melezhik/sparky/actions/workflows/main.yml/badge.svg)\n![SparrowCI](https://ci.sparrowhub.io/project/gh-melezhik-sparky/badge)\n\n# Sparky workflow in 4 lines:\n\n```bash\n$ sparkyd # run Sparky daemon to trigger jobs\n$ cro run # run Sparky CI UI to see job statuses and reports\n$ nano ~/.sparky/projects/my-project/sparrowfile  # write a job scenario\n$ firefox 127.0.0.1:4000 # run jobs and get reports\n```\n\n# Installation\n\n```bash\n$ sudo apt-get install sqlite3\n$ git clone https://github.com/melezhik/sparky.git\n$ cd sparky \u0026\u0026 zef install .\n```\n\n## Database initialization\n\nSparky requires a database to operate.\n\nRun database initialization script to populate database schema:\n\n```bash\n$ raku db-init.raku\n```\n\n# Sparky components\n\nSparky comprises of several components:\n\n* Jobs scheduler\n\n* Jobs Definitions\n\n* Jobs workers (including remote jobs)\n\n* Jobs UI\n\n* CLI\n\n## Job scheduler\n\nTo run Sparky jobs scheduler (aka daemon) runs in console:\n\n```bash\n$ sparkyd\n```\n\nScheduler logic:\n\n* Sparky daemon traverses sub directories found at the project root directory.\n\n* For every directory found initiate job run process invoking sparky worker ( `sparky-runner` ).\n\n* Sparky root directory default location is `~/.sparky/projects`.\n\n* Once all the sub directories are passed, sparky daemon sleeps for $timeout seconds.\n\n* A `timeout` option allows to balance a load on your system.\n\n* You can change a timeout by applying `--timeout` parameter when running sparky daemon:\n\n```bash\n$ sparkyd --timeout=600 # sleep 10 minutes\n```\n\n* You can also set a timeout by using `SPARKY_TIMEOUT` environment variable:\n\n```bash\n$ SPARKY_TIMEOUT=30 sparkyd ...\n```\n## Sparky Jobs UI\n\nSparky has a simple web UI to allow trigger jobs and get reports.\n\nTo run Sparky UI web application:\n\n```bash\n$ cro run\n```\n\nBy default Sparky UI application listens on host `0.0.0.0`, port `4000`, \nto override these settings set  `SPARKY_HOST`, `SPARKY_TCP_PORT` \nin `~/sparky.yaml` configuration file:\n\n```yaml\nSPARKY_HOST: 127.0.0.1\nSPARKY_TCP_PORT: 5000 \n```\n\n## Sparky jobs definitions\n\nSparky job needs a directory located at the sparky root directory:\n\n```bash\n$ mkdir ~/.sparky/projects/teddy-bear-app\n```\n\nTo create a job scenario, create file named `sparrowfile` located in job directory.\n\nSparky uses pure [Raku](https://raku.org) for job syntax, for example:\n\n```bash\n$ nano ~/.sparky/projects/hello-world/sparrowfile\n```\n\n```raku\n#!raku\nsay \"hello Sparky!\";\n```\n\nTo allow job to be executed by scheduler one need to create `sparky.yaml` - yaml based\njob definition, minimal form would be:\n\n```bash\n$ nano ~/.sparky/projects/hello-world/sparky.yaml\n```\n\n```yaml\nallow_manual_run: true\n```\n\n## Extending scenarios with Sparrow automation framework\n\nTo extend core functions, Sparky is fully integrated with [Sparrow](https://github.com/melezhik/Sparrow6) automation framework.\n\nHere in example of job that uses Sparrow plugins, to build typical Raku project:\n\n```bash\n$ nano ~/.sparky/projects/raku-build/sparrowfile\n```\n\n```raku\ndirectory \"project\";\n\ngit-scm 'https://github.com/melezhik/rakudist-teddy-bear.git', %(\n  to =\u003e \"project\",\n);\n\nzef \"{%*ENV\u003cPWD\u003e}/project\", %( \n  depsonly =\u003e True \n);\n\nzef 'TAP::Harness App::Prove6';\n\nbash 'prove6 -l', %(\n  debug =\u003e True,\n  cwd =\u003e \"{%*ENV\u003cPWD\u003e}/project/\"\n);\n```\n\nRepository of Sparrow plugins is available at [https://sparrowhub.io](https://sparrowhub.io)\n\n\n## Systemd units\n\nTo install sparky UI, sparkyd systemd units:\n\n```bash\n$ sparrowdo --sparrowfile=utils/install-sparky-systemd.raku --localhost\n```\n\n## Sparky workers\n\nSparky uses [Sparrowdo](https://github.com/melezhik/sparrowdo) to launch jobs in three fashions:\n\n* on localhost ( the same machine where Sparky is installed, default)\n* on remote host with ssh\n* docker container on localhost / remote machine \n\n```\n/--------------------\\                                             [ localhost ]\n| Sparky on localhost| --\u003e sparrowdo client --\u003e job (sparrow) --\u003e  [ container ]\n\\--------------------/                                             [ ssh host  ]\n```\n\nBy default job scenarios get executed _on the same machine you run Sparky at_, \nto run jobs on _remote host_ set sparrowdo section in `sparky.yaml` file:\n\n```bash\n$ nano ~/.sparky/projects/teddy-bear-app/sparky.yaml\n```\n\n```yaml\nsparrowdo:\n  host: '192.168.0.1'\n  ssh_private_key: /path/to/ssh_private/key.pem\n  ssh_user: sparky\n  no_index_update: true\n  sync: /tmp/repo\n```\n\nFollow [sparrowdo cli](https://github.com/melezhik/sparrowdo#sparrowdo-cli) documentation for `sparrowdo` configuration section explanation.\n\n### Skip bootstrap\n\nSparrowdo client bootstrap might take some time. \n\nTo disable bootstrap use  `bootstrap: false` option. \n\nUseful if sparrowdo client is already installed on target host.\n\n```yaml\nsparrowdo:\n  bootstrap: false\n```\n\n### Purging old builds\n\nTo remove old job builds set `keep_builds` parameter in `sparky.yaml`:\n\n```bash\n$ nano ~/.sparky/projects/teddy-bear-app/sparky.yaml\n```\n\nPut number of builds to keep:\n\n```yaml\nkeep_builds: 10\n```\n\nThat makes Sparky remove old builds and only keep last `keep_builds` builds.\n\n### Run jobs by cron\n\nTo run Sparky jobs periodically, set `crontab` entry in sparky.yaml file.\n\nFor example, to run a job every hour at 30,50 or 55 minutes:\n\n```bash\n$ nano ~/.sparky/projects/teddy-bear-app/sparky.yaml\n```\n\n```cron\ncrontab: \"30,50,55 * * * *\"\n```\n\nFollow [Time::Crontab](https://github.com/ufobat/p6-time-crontab) documentation on crontab entries format.\n\n### Manual run\n\nTo trigger job manually from web UI, use `allow_manual_run`:\n\n```bash\n$ nano ~/.sparky/projects/teddy-bear-app/sparky.yaml\n```\n\n```yaml\nallow_manual_run: true\n```\n\n### Trigger job by SCM changes\n\nTo trigger Sparky jobs on SCM changes, define `scm` section in `sparky.yaml` file:\n\n```yaml\nscm:\n  url: $SCM_URL\n  branch: $SCM_BRANCH\n```\n\nWhere:\n\n* `url` - git URL\n* `branch` - git branch, optional, default value is `master`\n\nFor example:\n\n```yaml\nscm:\n  url: https://github.com/melezhik/rakudist-teddy-bear.git\n  branch: master\n```\n\nOnce a job is triggered respected SCM data is available via `tags()\u003cSCM_*\u003e` function:\n\n```raku\ndirectory \"scm\";\n\nsay \"current commit is: {tags()\u003cSCM_SHA\u003e}\";\n\ngit-scm tags()\u003cSCM_URL\u003e, %(\n  to =\u003e \"scm\",\n  branch =\u003e tags\u003cSCM_BRANCH\u003e\n);\n\nbash \"ls -l {%*ENV\u003cPWD\u003e}/scm\";\n```\n\nTo set default values for SCM_URL and SCM_BRANCH, use sparrowdo `tags`:\n\n`sparky.yaml`:\n\n```yaml\n  sparrowdo:\n    tags: SCM_URL=https://github.com/melezhik/rakudist-teddy-bear.git,SCM_BRANCH=master\n```\n\nThese is useful when trigger job manually.\n\n### Flappers protection mechanism \n\nFlapper protection mechanism kicks out SCM urls that are timeouted (certain amount of time) during git connection, from scheduling, this mechanism protects sparkyd worker from stalling.\n\nTo disable flappers protection mechanism, set `SPARKY_FLAPPERS_OFF` environment variable\nor adjust `~/sparky.yaml` configuration file:\n\n```yaml\nworker:\n  flappers_off: true\n```\n\n### Disable jobs\n\nTo prevent Sparky job from execution use `disable` option:\n\n```bash\n$ nano ~/.sparky/projects/teddy-bear-app/sparky.yaml\n\ndisabled: true\n```\n\n# Advanced topics\n\nFollowing are advanced topics covering some cool Sparky features.\n\n## Job UIs\n\nSparky UI DSL allows to grammatically describe UI for Sparky jobs\nand pass user input into a scenario as variables.\n\nRead more at [docs/ui.md](https://github.com/melezhik/sparky/blob/master/docs/ui.md)\n\n## Downstream jobs\n\nDownstream jobs get run after some _main_ job has finished.\n\nRead more at [docs/downstream.md](https://github.com/melezhik/sparky/blob/master/docs/downstream.md)\n\n## Sparky triggering protocol (STP)\n\nSparky triggering protocol allows to trigger jobs automatically by creating files in special format.\n\nRead more at [docs/stp.md](https://github.com/melezhik/sparky/blob/master/docs/stp.md)\n\n## Job API\n\nJob API allows to orchestrate multiple Sparky jobs.\n\nRead more at [docs/job_api.md](https://github.com/melezhik/sparky/blob/master/docs/job_api.md)\n\n## Sparky plugins\n\nSparky plugins is way to extend Sparky jobs by writing reusable plugins as Raku modules.\n\nRead more at [docs/plugins.md](https://github.com/melezhik/sparky/blob/master/docs/plugins.md)\n\n## HTTP API\n\nSparky HTTP API allows execute Sparky jobs remotely over HTTP.\n\nRead more at [docs/api.md](https://github.com/melezhik/sparky/blob/master/docs/api.md)\n\n## Security\n\n### Authentication\n\nSparky web server _comes with_ two authentication protocols,\nchoose proper one depending on your requirements.\n\nRead more at [docs/auth.md](https://github.com/melezhik/sparky/blob/master/docs/auth.md)\n\n### ACL\n\nSparky ACL allows to create access control lists to manage role based access to Sparky resources.\n\nRead more at [docs/acl.md](https://github.com/melezhik/sparky/blob/master/docs/acl.md)\n\n## Databases support\n\nSparky keeps it's data in database, by default it uses sqlite,\nfollowing databases are supported:\n\n* SQLite\n* MySQL/MariaDB\n* PostgreSQL\n\nRead more at [docs/database.md](https://github.com/melezhik/sparky/blob/master/docs/database.md)\n\n## TLS Support\n\nSparky web server may run on TLS. To enable this add a couple of parameters to `~/sparky.yaml`\n\nconfiguration file:\n\n```\nSPARKY_USE_TLS: true\ntls:\n private-key-file: '/home/user/.sparky/certs/www.example.com.key'\n certificate-file: '/home/user/.sparky/certs/www.example.com.cert'\n```\n\n`SPARKY_USE_TLS` enables SSL mode and `tls` section has paths to ssl certificate ( key and certificate parts ).\n\n# Additional topics\n\n## Sparky on docker\n\nHow to run Sparky via docker container. See [docs/sparky_on_docker.md](docs/sparky_on_docker.md) document.\n\n## Sparman\n\nSparman is a cli to ease SparrowCI management. See [docs/sparman.md](docs/sparman.md) document.\n\n## Sparky cli\n\nSparky cli allows to trigger jobs in terminal.\n\nRead more at [docs/cli.md](https://github.com/melezhik/sparky/blob/master/docs/cli.md)\n\n## Sparky Environment variables\n\nUse environment variables to tune Sparky configuration.\n\nRead more at [docs/env.md](https://github.com/melezhik/sparky/blob/master/docs/env.md)\n\n## Glossary\n\nSome useful glossary.\n\nRead more at [docs/glossary.md](https://github.com/melezhik/sparky/blob/master/docs/glossary.md)\n\n## CSS\n\nSparky uses [Bulma](https://bulma.io/) as CSS framework for web UI.\n\n## Sparky job examples\n\nExamples of various Sparky jobs could be found at [examples/](https://github.com/melezhik/sparky/tree/master/examples) folder.\n\n# See also\n\n* [Cro](https://cro.services) - Raku Web Framework\n\n* [Sparky-docker](https://github.com/melezhik/sparky-docker) - Run Sparky as Docker container\n\n# Author\n\nAlexey Melezhik\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmelezhik%2Fsparky","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmelezhik%2Fsparky","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmelezhik%2Fsparky/lists"}