{"id":16726223,"url":"https://github.com/deas/satrn","last_synced_at":"2025-10-08T23:07:00.819Z","repository":{"id":232586608,"uuid":"473513499","full_name":"deas/satrn","owner":"deas","description":"Satrn - Solr Companion 🪐","archived":false,"fork":false,"pushed_at":"2023-04-18T18:22:09.000Z","size":185,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-10-08T23:06:59.115Z","etag":null,"topics":["clojure","clojurescript","gitops","solr","solrcloud"],"latest_commit_sha":null,"homepage":"","language":"Clojure","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/deas.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2022-03-24T08:15:17.000Z","updated_at":"2023-04-18T18:29:29.000Z","dependencies_parsed_at":"2024-04-10T13:24:50.556Z","dependency_job_id":"bc743096-0e14-49ea-a484-bf3ca148e451","html_url":"https://github.com/deas/satrn","commit_stats":null,"previous_names":["deas/satrn"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/deas/satrn","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/deas%2Fsatrn","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/deas%2Fsatrn/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/deas%2Fsatrn/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/deas%2Fsatrn/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/deas","download_url":"https://codeload.github.com/deas/satrn/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/deas%2Fsatrn/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":279000730,"owners_count":26082862,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-08T02:00:06.501Z","response_time":56,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["clojure","clojurescript","gitops","solr","solrcloud"],"created_at":"2024-10-12T22:52:32.203Z","updated_at":"2025-10-08T23:07:00.790Z","avatar_url":"https://github.com/deas.png","language":"Clojure","funding_links":[],"categories":[],"sub_categories":[],"readme":"# satrn 🪐\n[Orbiting around Apache Solr](https://youtu.be/8q8110rRTCA?t=18283).\n\nThis project started out a basic tool synchronizing collection configuration between Solr clusters (`satrn.clj`). From there, it grew along with a project bringing SRE and Continuous Delivery to Solr environments - minimalism and simplicity being key design elements.\n\nCommand line tools:\n\n`satrn.clj`: Reconcile collections, configsets and aliases to desired state - the GitOps game. It may be helpful to know that this tool started out using a Solr server reprensenting desire state - not git. State, such as shard leaders or nodesets (creation) has no natural representation there. Hence, things have been stretched a bit and covered by tool configuration and more recently, metadata in the Solr Blob store.\n\nRemoval and destruction may be missing for some objects, but implementation should be fairly straight forward.\n\n`update_docs.clj`:  Read documents from stdin and update them in collection.\n\n`export_docs.clj` : Export documents from collection to stdout.\n\n`clone.clj` : Export and update - piped shortcut.\n\n`rnb.clj` : Restore and backup collections.\n\n`dff.clj` : Diff json files\n\n`cnt_qdff.clj` : Compare counts for queries\n\n`rpc.clj` : Read RPC style command sequence from stdin and send them to Solr. Safest way to try it out is `./src/rpc.clj \u003ctest/rpc-clusterstatus.edn` (read only call).\n\n`logs.clj` : Analyze/Visualize/Structure logs. Cloud Console/Shell is not enough.\n\nMost of them have a `--help` option.\n\nUnless explicitly set (e.g. using environment variables `SOLR_BASE_URL`/ `SOLR_AUTH`, `LOG_LEVEL`), the library assumes defaults `http://localhost:8983` / `solr:SolrRocks` / `info`. Setting the log level to `debug` will log http call details.\n\nUsing [`babashka`](https://babashka.org/) for portability and minimalism. [`asdf`](https://asdf-vm.com/) can be used to install `babashka` as follows:\n\n```shell\nasdf plugin add babashka\nasdf plugin add k6 # if you want to try JavaScript InterOp Library\nasdf install\n```\n\nThere are various ways to use these tools (Script, Container, Library, Java Interop) - Don't get confused.\n\n\n## Development\nRun NREPL-server `bb nrepl-server` and jack in with your editor. This is the bare minimum. Visual Studio Code with Calva extension is a great editor.\n\nEven though the Open Container Image only uses `babashka` (and not a full JVM), you might want to use `deps.edn` with a JVM for a better dev experience.\n\n## Solr Service\nTo quickly get your feet wet, you can spin up a Solr cluster using `docker-compose`:\n\n```shell\n# Spin up solrcloud single node\n# docker-compose up\n# Spin up solrcloud multi node\ndocker-compose -f docker-compose.yml -f docker-compose-multi-solr.yml up\n# Set default solr:SolrRocks auth\n# docker exec -i satrn_solr-1_1 sh -c 'cat \u003e/tmp/security.json \u0026\u0026 solr zk cp /tmp/security.json zk:security.json -z zoo-1:2181' \u003cassets/security.json\n\n# Create a single shard collection with 3 (requires multi node version)\n./src/rpc.clj \u003ctest/rpc-create-collection.edn\n\n# Create a single shard collection with 1 replica (requires multi node version)\nsed -e s/'replicationFactor.*'/'replicationFactor \"1\"'/g \\\n  \u003ctest/rpc-create-collection.edn | ./src/rpc.clj \n```\n\n## Usage\n\nRun sync application:\n\n```shell\nexport config=config-default.edn\n./src/satrn.clj sync # Run w/o argument to get help\n\n```\n\nRun the project's tests:\n```shell\n./test-runner.clj\n```\n\nBuild uberscripts (bake relatated use cases into single files):\n```shell\nbb build\n```\n\nBuild AOT- uberjar (Java InterOp):\n```shell\nclj -T:build uber\n```\n\nUsing `rnb` from Java `main`:\n```shell\njava -classpath target/satrn-*-standalone.jar satrn.RnB\n```\n\nBuild Node Library (CommonJS InterOp - targeting `k6`):\n```shell\nclj -A:shadow-cljs release :lib\n```\nRunning basic `k6` sample (using InterOp) \n```shell\nk6 run --http-debug=full -i 1 ./sample-k6.js\n```\n\nBuild and run an Open Container Image:\n```shell\nTAG=satrn:0\ndocker build -t ${TAG}  .\ndocker run --rm ${TAG}\n# Run with custom config \n# docker run --rm -v `pwd`/config-sample.edn:/config-default.edn ${TAG}\n```\n\nUse as a library from Clojure\n```shell\nclojure -Sdeps '{:deps {io.github.deas/satrn {:sha \"...\" }}}' \\\n    -e \"(require '[satrn.command :as c])(-\u003e (c/cmd-cluster-status) c/execute c/response-map)\"\n```\n\nRun update on Kubernetes using image from private registry:\n\n```shell\nK8S_CTX=default\nK8S_NS=search\nK8S_IMAGE=ghcr.io/deas/satrn:latest\nzcat collection.json.gz | \\\n  kubectl --context $K8S_CTX run update-docs --rm -i --image $K8S_IMAGE \\\n  --overrides='{\"apiVersion\": \"v1\", \"spec\": {\"template\": {\"spec\": { \"imagePullSecrets\": [{\"name\": \"regcred\"}]}}}}' \\\n  -- bb /update_docs.clj -d '{:destination {:base-url \"http://solr-ingress:8983\", :basic-auth [\"solr\" \"SolrRocks\"]}}' collection\n```\n\n## Hints\n- Solr 9 is expected to ship functionality around replica placement that may be worth checkout out\n- Logging is configurable, but currently optimized for Google Cloud Platform\n- `satrn.clj` exposes Prometheus metrics\n\n## TODO\n- ~~Refactor for use as a library~~\n- ~~Leverage `babashka` tasks for build~~\n- Leverage github actions cache\n- Beware: `config.edn` in classpath still appears overlay `config` environment (-\u003e `test`)\n- Consistently use default at latest possible moment - not in CLIs (-\u003e testing)\n- Remove TODO: tags in code\n- Refactor to use sub-commands (appears cli-matic/spec is fine with `babashka` now? Or `docopt.clj`?)\n- Update credentials in `security.json`\n- ~~Export config to folder/zip?~~\n- ~~Discover Solr server version and support recent (source) versions~~\n- ~~`satrn.clj` Config: Externalize secrets?~~\n- Clean up config maps/merging\n- Improve test coverage\n- Improve jvm interop\n- Improve node-/`k6` library target (Idea to leverage code from `k6` came later so this is a bit of a retrofit)\n- Shim `k6/http` for node repl exploration\n- Improve validation and error handling\n- Leverage nested diagnostic context for collections.\n- ~~`satrn.clj` : Suppport for stored queries~~\n- `logs.clj` Streaming : `gcloud alpha logging tail`/ raw Kubernetes support (likely needs timestamp sorting)\n- Release helm chart to github (pages?), probably https://github.com/helm/chart-releaser-action and/or https://github.com/helm/chart-releaser\n- Migrate `asdf` -\u003e `nix`\n- Try out Solr-9\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdeas%2Fsatrn","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdeas%2Fsatrn","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdeas%2Fsatrn/lists"}