Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/deas/satrn

Satrn - Solr Companion 🪐
https://github.com/deas/satrn

clojure clojurescript gitops solr solrcloud

Last synced: about 1 month ago
JSON representation

Satrn - Solr Companion 🪐

Awesome Lists containing this project

README 

        
# satrn 🪐

[Orbiting around Apache Solr](https://youtu.be/8q8110rRTCA?t=18283).



This project started out a basic tool synchronizing collection configuration between Solr clusters (`satrn.clj`). From there, it grew along with a project bringing SRE and Continuous Delivery to Solr environments - minimalism and simplicity being key design elements.



Command line tools:



`satrn.clj`: Reconcile collections, configsets and aliases to desired state - the GitOps game. It may be helpful to know that this tool started out using a Solr server reprensenting desire state - not git. State, such as shard leaders or nodesets (creation) has no natural representation there. Hence, things have been stretched a bit and covered by tool configuration and more recently, metadata in the Solr Blob store.



Removal and destruction may be missing for some objects, but implementation should be fairly straight forward.



`update_docs.clj`:  Read documents from stdin and update them in collection.



`export_docs.clj` : Export documents from collection to stdout.



`clone.clj` : Export and update - piped shortcut.



`rnb.clj` : Restore and backup collections.



`dff.clj` : Diff json files



`cnt_qdff.clj` : Compare counts for queries



`rpc.clj` : Read RPC style command sequence from stdin and send them to Solr. Safest way to try it out is `./src/rpc.clj /tmp/security.json && solr zk cp /tmp/security.json zk:security.json -z zoo-1:2181'  (c/cmd-cluster-status) c/execute c/response-map)"

```



Run update on Kubernetes using image from private registry:



```shell

K8S_CTX=default

K8S_NS=search

K8S_IMAGE=ghcr.io/deas/satrn:latest

zcat collection.json.gz | \

  kubectl --context $K8S_CTX run update-docs --rm -i --image $K8S_IMAGE \

  --overrides='{"apiVersion": "v1", "spec": {"template": {"spec": { "imagePullSecrets": [{"name": "regcred"}]}}}}' \

  -- bb /update_docs.clj -d '{:destination {:base-url "http://solr-ingress:8983", :basic-auth ["solr" "SolrRocks"]}}' collection

```



## Hints

- Solr 9 is expected to ship functionality around replica placement that may be worth checkout out

- Logging is configurable, but currently optimized for Google Cloud Platform

- `satrn.clj` exposes Prometheus metrics



## TODO

- ~~Refactor for use as a library~~

- ~~Leverage `babashka` tasks for build~~

- Leverage github actions cache

- Beware: `config.edn` in classpath still appears overlay `config` environment (-> `test`)

- Consistently use default at latest possible moment - not in CLIs (-> testing)

- Remove TODO: tags in code

- Refactor to use sub-commands (appears cli-matic/spec is fine with `babashka` now? Or `docopt.clj`?)

- Update credentials in `security.json`

- ~~Export config to folder/zip?~~

- ~~Discover Solr server version and support recent (source) versions~~

- ~~`satrn.clj` Config: Externalize secrets?~~

- Clean up config maps/merging

- Improve test coverage

- Improve jvm interop

- Improve node-/`k6` library target (Idea to leverage code from `k6` came later so this is a bit of a retrofit)

- Shim `k6/http` for node repl exploration

- Improve validation and error handling

- Leverage nested diagnostic context for collections.

- ~~`satrn.clj` : Suppport for stored queries~~

- `logs.clj` Streaming : `gcloud alpha logging tail`/ raw Kubernetes support (likely needs timestamp sorting)

- Release helm chart to github (pages?), probably https://github.com/helm/chart-releaser-action and/or https://github.com/helm/chart-releaser

- Migrate `asdf` -> `nix`

- Try out Solr-9