{"id":37019706,"url":"https://github.com/awagen/kolibri-base","last_synced_at":"2026-01-14T02:12:04.576Z","repository":{"id":45873058,"uuid":"350051741","full_name":"awagen/kolibri-base","owner":"awagen","description":"Kolibri - A concurrent, clusterable, open-source    execution framework based on Akka, written in Scala","archived":true,"fork":false,"pushed_at":"2021-11-30T22:45:51.000Z","size":2502,"stargazers_count":5,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2023-07-03T07:09:37.249Z","etag":null,"topics":["akka","cluster","information-retrieval","relevancy","scala","search"],"latest_commit_sha":null,"homepage":"","language":"Scala","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/awagen.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2021-03-21T16:17:46.000Z","updated_at":"2023-01-28T05:02:58.000Z","dependencies_parsed_at":"2022-09-13T08:41:34.402Z","dependency_job_id":null,"html_url":"https://github.com/awagen/kolibri-base","commit_stats":null,"previous_names":[],"tags_count":9,"template":null,"template_full_name":null,"purl":"pkg:github/awagen/kolibri-base","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/awagen%2Fkolibri-base","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/awagen%2Fkolibri-base/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/awagen%2Fkolibri-base/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/awagen%2Fkolibri-base/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/awagen","download_url":"https://codeload.github.com/awagen/kolibri-base/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/awagen%2Fkolibri-base/sbom","scorecard":{"id":218736,"data":{"date":"2025-08-11","repo":{"name":"github.com/awagen/kolibri-base","commit":"4be1b1798125f71efaa069d17b67c19efe6a021a"},"scorecard":{"version":"v5.2.1-40-gf6ed084d","commit":"f6ed084d17c9236477efd66e5b258b9d4cc7b389"},"score":3.4,"checks":[{"name":"Packaging","score":-1,"reason":"packaging workflow not detected","details":["Warn: no GitHub/GitLab publishing workflow detected."],"documentation":{"short":"Determines if the project is published as a package that others can easily download, install, easily update, and uninstall.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#packaging"}},{"name":"Maintained","score":0,"reason":"project is archived","details":["Warn: Repository is archived."],"documentation":{"short":"Determines if the project is \"actively maintained\".","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#maintained"}},{"name":"Code-Review","score":0,"reason":"Found 0/2 approved changesets -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project requires human code review before pull requests (aka merge requests) are merged.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#code-review"}},{"name":"Dangerous-Workflow","score":10,"reason":"no dangerous workflow patterns detected","details":null,"documentation":{"short":"Determines if the project's GitHub Action workflows avoid dangerous patterns.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#dangerous-workflow"}},{"name":"Token-Permissions","score":0,"reason":"detected GitHub workflow tokens with excessive permissions","details":["Warn: no topLevel permission defined: .github/workflows/scala.yml:1","Info: no jobLevel write permissions found"],"documentation":{"short":"Determines if the project's workflows follow the principle of least privilege.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#token-permissions"}},{"name":"Binary-Artifacts","score":10,"reason":"no binaries found in the repo","details":null,"documentation":{"short":"Determines if the project has generated executable (binary) artifacts in the source repository.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#binary-artifacts"}},{"name":"CII-Best-Practices","score":0,"reason":"no effort to earn an OpenSSF best practices badge detected","details":null,"documentation":{"short":"Determines if the project has an OpenSSF (formerly CII) Best Practices Badge.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#cii-best-practices"}},{"name":"Security-Policy","score":0,"reason":"security policy file not detected","details":["Warn: no security policy file detected","Warn: no security file to analyze","Warn: no security file to analyze","Warn: no security file to analyze"],"documentation":{"short":"Determines if the project has published a security policy.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#security-policy"}},{"name":"Fuzzing","score":0,"reason":"project is not fuzzed","details":["Warn: no fuzzer integrations found"],"documentation":{"short":"Determines if the project uses fuzzing.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#fuzzing"}},{"name":"Vulnerabilities","score":10,"reason":"0 existing vulnerabilities detected","details":null,"documentation":{"short":"Determines if the project has open, known unfixed vulnerabilities.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#vulnerabilities"}},{"name":"Pinned-Dependencies","score":0,"reason":"dependency not pinned by hash detected -- score normalized to 0","details":["Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/scala.yml:15: update your workflow using https://app.stepsecurity.io/secureworkflow/awagen/kolibri-base/scala.yml/main?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/scala.yml:17: update your workflow using https://app.stepsecurity.io/secureworkflow/awagen/kolibri-base/scala.yml/main?enable=pin","Warn: containerImage not pinned by hash: Dockerfile:1: pin your Docker image by updating adoptopenjdk/openjdk13:jdk-13.0.2_8-alpine-slim to adoptopenjdk/openjdk13:jdk-13.0.2_8-alpine-slim@sha256:24d174970fc0eaa9f4b948a409088f9ff886ce534f1c13530ab64dcce596d813","Info:   0 out of   2 GitHub-owned GitHubAction dependencies pinned","Info:   0 out of   1 containerImage dependencies pinned"],"documentation":{"short":"Determines if the project has declared and pinned the dependencies of its build process.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#pinned-dependencies"}},{"name":"License","score":10,"reason":"license file detected","details":["Info: project has a license file: LICENSE:0","Info: FSF or OSI recognized license: Apache License 2.0: LICENSE:0"],"documentation":{"short":"Determines if the project has defined a license.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#license"}},{"name":"Signed-Releases","score":-1,"reason":"no releases found","details":null,"documentation":{"short":"Determines if the project cryptographically signs release artifacts.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#signed-releases"}},{"name":"Branch-Protection","score":0,"reason":"branch protection not enabled on development/release branches","details":["Warn: branch protection not enabled for branch 'main'"],"documentation":{"short":"Determines if the default and release branches are protected with GitHub's branch protection settings.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#branch-protection"}},{"name":"SAST","score":0,"reason":"SAST tool is not run on all commits -- score normalized to 0","details":["Warn: 0 commits out of 29 are checked with a SAST tool"],"documentation":{"short":"Determines if the project uses static code analysis.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#sast"}}]},"last_synced_at":"2025-08-17T02:06:42.052Z","repository_id":45873058,"created_at":"2025-08-17T02:06:42.052Z","updated_at":"2025-08-17T02:06:42.052Z"},"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28408711,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-14T01:52:23.358Z","status":"online","status_checked_at":"2026-01-14T02:00:06.678Z","response_time":107,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["akka","cluster","information-retrieval","relevancy","scala","search"],"created_at":"2026-01-14T02:12:03.813Z","updated_at":"2026-01-14T02:12:04.569Z","avatar_url":"https://github.com/awagen.png","language":"Scala","funding_links":[],"categories":[],"sub_categories":[],"readme":"# DEPRECATED Kolibri Base\n\nNote: the current repository was moved to ```https://github.com/awagen/kolibri``` to have one repo for all\nKolibri projects. Please check the new repo location for updates.\n\nThis project provides the mechanism to execute jobs based on akka, making use of clustering to distribute job batches.\n![Alt text](images/kolibri.svg?raw=true \"Kolibri Base\")\n\nDocumentation link: https://awagen.github.io/kolibri/ \\\nDoc converging towards completion, getting there (check for updates).\n\n## Tests\n### Execution\n- when executing in intellij, make sure to set env variable PROFILE=test\n- when executed via sbt test, as defined in build.sbt, thie PROFILE env var is already set\n\n### Writing tests with / without cluster startup\n\n- the actor system for tests is started on per-test basis via letting the test extending KolibriTestKit (in case a\n  cluster is needed for the test - despite being just 1-node cluster) or KolibriTestKitNoCluster (in case test needs no\n  cluster at all)\n\n## Notes on correct sender\n\n- There are cases when sender() of a sent message is set to the default sender, which would be dead letter channel; one\n  case where this happens is if a tell to some actor is made within an actor by calling an outside method that has no\n  implicit actorRef: ActorRef attribute set. Here solutions are either to declare the implicit value on the external\n  method (and propagating it down to where the tell is executed) or explicitly pass a sender and send it with the\n  message, so that the other actor knows where to respond to.\n\n## Troubleshoot\n\nConnection refused:\n\n- check ```ss -tnlp | grep :8558``` (e.g replace the port with the one used for contact probing (check discovery config/\n  log messages for connection refused))\n- in case firewall blocks something, might wanna check ```tcpdump -n icmp```\n\n## Build jar, build docker image, startup local\n\n- you might temporarily need to clone kolibri-datatypes and publish locally (see kolibri-datatypes README on instructions)\n- build jar (find it in target folder afterwards): ```./scripts/buildJar.sh```\n- build docker image for local usage: ```docker build . -t kolibri-base:0.1.0-rc0```\n- run single-node cluster (compute and httpserver role, access via localhost:\n  8000): ```./scripts/docker_run_single_node.sh```\n    - sets interface of http server to 0.0.0.0 to allow requests from host system to localhost:8000 reach the service\n      within the container\n- start local 3-node cluster (one compute and httpserver node, two 'compute'-only nodes, access via localhost:\n  8000): ```docker-compose up```\n  - NOTE: starting response-juggler as used in the docker-compose.yaml requires cloning ```https://github.com/awagen/response-juggler``` and building\n  the image locally via ```docker build -t response-juggler:0.1.0 .```. \n  This service provides a basic response fake to avoid having to have a real search system running, and the parameters defined in the docker-compose\n  file contain the same product_ids that are used within the test-judgements.txt referenced in the example job definition.\n  The response-juggler will respond to each request with a random\n  sampling of the comma-separated PID_SAMPLEs, each result containing between MIN_PRODUCTS and MAX_PRODUCTS.\n  If another response structure needed, this can easily be adjusted within the response-juggler \n\n## Example clustered job execution: pi-estimation with dart throws\n\nA very simple example of a distributed job is the pi-calculation via dart throws, assuming circle bounded by square and\ncounting the propotion of throws landing in the circle. With increasing number of throws, the proportion should\napproximate pi/4. Steps to execute this simple job example:\n\n- build jar and docker image according to above calls\n- start up cluster by ```docker-compose up``` command in project root\n- example url call: ```http://localhost:8000/pi_calc?jobName=job1\u0026nrThrows=10000000\u0026batchSize=10000\u0026resultDir=/app/data```\n  Note that giving the volume mount ```./kolibri-test:/app/data``` as given in the docker-compose, this will write into\n  the projects root's ```kolibri-test``` folder per default into the file ```dartThrowResult.txt```. Note that the job\n  is set up as a simple example that has no boundary on the nr of throws processed per second, and all flowing into\n  aggregator, thus if the aggregator is asked and the response waited for, this can lead to ask timeouts. This differs\n  between jobs, and most jobs take more time than simple random number pick between 0 and 1, reducing the number of\n  result messages. Yet given the way the aggregating actor is instantiated in the same instance as the\n  RunnableExecutionActor executing the ActorRunnable (which produces the result messages), the message passing doesnt\n  need serialization but is done via ref passing.\n- kill job: ```http://localhost:8000/stopJob?jobId=job1```\n- check status of job: ```http://localhost:8000/getJobStatus?jobId=job1```\n\n## Example clustered job execution: grid evaluation search endpoint\nWithin the scripts subfolder you find a testSearchEval.json which provides the general outline of the job definition.\nThe script start_searcheval.sh fires this json to the correct endpoint, which then starts a grid evaluation on the given\nparameters and writes an overall aggregation as well as per-query results. The results are written within the kolibri-test subfolder\nin project root, and therein in subfolder equal to the job name given in the abovementioned json job definition.\nNote that the json needs the connections to fire requests against to be defined, assuming service with name search-service\nbeing added to docker-compose on port 80. Adjust to your individual endpoint(s).\nFurther, right now assumes responses corresponding to play json selector \n```\"response\" \\ \"docs\" \\\\ \"product_id\"```, meaning structure at least containing the hierarchies: \n```{\"response\": {\"docs\": [{\"product_id\": ...}, {\"product_id\":...}], ...},...}```\nThis is changeable in the json job definition.\n\n## Monitoring\nMetrics are collected utilizing the open source lib Kamon and exposed for prometheus to scrape the metrics.\nIn the grafana subfolder you'll find an example dashboard for the kolibri system.\nAn overview of the represented metrics can be seen in below screenshot:\n![Alt text](images/Kolibri-Dashboard-Grafana.png?raw=true \"Kolibri Grafana Dashboard\")\n\n## Serialization\n\nWithin definitions of the jobs, care has to be taken to define all parts that are part of any message to be\nserializable. The standard way for kolibri is the KolibriSerializable interface. Within the application.conf\nsection ```akka.actor.serializers```\nthe actual serializers are defined, while binding to specific classes is found\nin ```akka.actor.serialization-bindings```. Here you will find that a serializer is bound to KolibriSerializable. Thus\non adding new definitions, make sure to make the parts extend this interface. Also, take into account that Lambda's are\nnot Serializable, so rather use the SerializableSupplier, SerializableFunction1 (function with one argument) and\nSerializableFunction2 (function with two arguments) from the kolibri-datatypes library (or add new ones in case of more\narguments / other needs) or the respective Scala function interface, since scala functions extend Serializable. Avoid\nusing lambda's and java functions.\n\nFor testing purposes, within the test application.conf ```serialize-messages``` can be set true to serialize messages in\nthe tests even though they share the same jvm. ```serialize-creators``` does the same for Props (Props encapsulate the\ninfo needed for an actor to create another one).\n\n## Cluster singleton router\n\nA cluster singleton router can be utilized via the below code snippet in case of config setting\nstartClusterSingletonRouter is set to true, which also causes the ClusterSingletonManager to be started at cluster\nstart.\n\n```\nval singletonProxyConfig: Config = ConfigFactory.load(config.SINGLETON_PROXY_CONFIG_PATH)\ncontext.system.actorOf(\n  ClusterSingletonProxy.props(\n    singletonManagerPath = config.SINGLETON_MANAGER_PATH,\n    settings = ClusterSingletonProxySettings.create(singletonProxyConfig)\n  )\n)\n```\n\n## Local docker-compose setup\nThe docker-compose.yml can be found in the project root. Following setup is provided:\n- prometheus: http://localhost:9000\n- grafana: http://localhost:3000\n- 3 node Kolibri cluster, with metrics endpoint on http://localhost:9095/metrics (exposed by Kamon lib)\n\nAn example grid search evaluation can be performed as given in the ```scripts/start_searcheval.sh```. You will need \nto adjust paths where appropriate. The example fires requests to a search instance on the local machine, which is not provided here.\n\n## Few notes on the use of TypeTaggedMap\nIn case TypeTaggedMap (such as implementation TypedMapStore) is used as type safe map, note that you might see error messages\nindicating a type mismatch of the type that is assumed and the data parsed in case the parsing\nis stored in a variable of generic type Seq[_], e.g where error message will include scala.collection.*.colon::colon as bit unclear description.\nThis refers to the unclear type of Seq[_], so you should parse to specific known types in those cases, e.g two samples where the first will yield\nerror and the second will be fine (the below assumes that the seqSelector selects a Seq[JsValue] from the JsValue input and casts the single\nelements to type String, while seqSelector.classTyped is of type ClassTyped[Seq[String]]):\n```\nval value: Seq[_] = seqSelector.select(jsValue)\ntypedMap.put(seqSelector.classTyped, value)\n```\n\n```\nval value: Seq[String] = seqSelector.select(jsValue).asInstanceOf[Seq[String]]\ntypedMap.put(seqSelector.classTyped, value)\n```\n\n## A few notes on play json lib vs spray json lib\nIn the project you will currently find both spray json lib for json parsing and the play json lib.\nAt the moment incoming requests (e.g job definitions in json format and so on) are handled by the spray lib, \nwhile selectors in the parsed responses when requesting another service (e.g look at the job definitions for the \nsearch optimization use case) are handled with the play lib. This has origin in better out of the box functionality of the \nplay lib when parsing elements from a json, single or recursive. \nFor spray there is an additional library providing this functionality (https://github.com/jrudolph/json-lenses), \nwhich seems to even provide more functionality. For this sake ayou can expect the play json lib will be removed in \nfurther iterations for the sake of only using spray.\n\n## Notes on local execution with access to AWS account\nOne way to enable container access to AWS account (e.g as for writing results into S3 bucket or similar),\nit is enough to mount the local directory where the credentials reside into the root .aws directory in the container,\nby adding to the docker-compose below volume mount (assuming the aws credentials reside in the home folder on the host machine\nwithin the .aws folder, which is the default location when calling ```aws configure``` (official docs: https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-quickstart.html)):\n```\nvolumes:\n  - [path-to-home-on-host-machine]/.aws:/home/kolibri/.aws:ro\n```\nNow configure the AWS_PROFILE env var to any profile name you want to assume (and for which the above mounted folder contains\ncredentials). See example within docker-compose.yaml.\nNote that if the configuration file is not in the default location ```~/.aws/config```, additionally the env variable\n```AWS_CONFIG_FILE``` has to be set to the directory within the container where the config file was mounted to.\n\n## Notes on local execution with access to GCP account\nAnalogue to the description of enabling AWS access, which refers to a credentials folder,\nfor GCP we have to set the env variable ```GOOGLE_APPLICATION_CREDENTIALS``` to the full path\nto the json service account key file (how to create a service account and respective key file\nyou can check in the google cloud documentation, its a matter of a few clicks).\nTo make it accessible within docker, you have to mount the directory containing the key file on your local machine\ninto the container:\n```\nvolumes:\n  - [path-to-dir-containing-key-file-on-local-machine]/:/home/kolibri/gcp:ro\n```\nand then setting env variable\n```\nGOOGLE_APPLICATION_CREDENTIALS: '/home/kolibri/gcp/[sa-key-file-name].json'\n```\nThis enables the access for the app running within the docker container.\nTo utilize google storage for result reading / writing, a few more env variables are needed:\n```\nGCP_GS_BUCKET: [bucket name without gs:// prefix]\nGCP_GS_PATH: [path from bucket root to append to all paths that are requested]\nGCP_GS_PROJECT_ID: [the project id for which the used service account is defined and for which the gs bucket was created]\nPERSISTENCE_MODE: 'GCP'\n```\nWith those settings, execution results are stored and read from google storage bucket.\n\n## Status endpoints example outputs\n- jobStates:\n```\n\"\n[\n  {\n    \"jobId\":\"testJob\",\n    \"jobType\":\"SearchEvaluation\",\n    \"resultSummary\":{\"failedBatches\":[],\"nrOfBatchesSentForProcessing\":2,\"nrOfBatchesTotal\":24,\"nrOfResultsReceived\":0,\"result\":\"RUNNING\"},\n    \"startTime\":\"2021-09-29T01:32:09.292+02:00[CET]\"\n  }\n]\n\"\n```\n\n- /nodeState:\n```\n[\n{\n  \"capacityInfo\":\n    {\n      \"cpuCapactiy\":-1.0,\n      \"heapCapactiy\":0.930558969033882,\n      \"loadCapacity\":0.505,\n      \"mixCapacity\":0.717779484516941},\n      \"cpuInfo\":{\n          \"cpuCombined\":-1.0,\n          \"cpuStolen\":-1.0,\n          \"loadAvg\":3.96,\n          \"nrProcessors\":8\n      },\n     \"heapInfo\":{\n        \"heapCommited\":1073741824,\n        \"heapMax\":4294967296,\n        \"heapUsed\":298246957\n     },\n     \"host\":\"kolibri1\",\n     \"port\":8001\n    },\n{\n  \"capacityInfo\":\n    {\n      \"cpuCapactiy\":-1.0,\n      \"heapCapactiy\":0.9374729155097157,\n      \"loadCapacity\":0.5925,\n      \"mixCapacity\":0.7649864577548579},\n      \"cpuInfo\":{\n        \"cpuCombined\":-1.0,\n        \"cpuStolen\":-1.0,\n        \"loadAvg\":3.26,\n        \"nrProcessors\":8\n      },\n      \"heapInfo\":{\n        \"heapCommited\":1073741824,\n        \"heapMax\":4294967296,\n        \"heapUsed\":268551783\n      },\n      \"host\":\"kolibri2\",\n      \"port\":8001\n    }\n]\n```\n\n\n## Code coverage \nTo calculate code coverage, the sbt-scoverage plugin is used (https://github.com/scoverage/sbt-scoverage).\nThe commands to generate the reports are as follows:\n- run tests with coverage: ```sbt clean coverage test``` (or in case project contains integration tests: ```sbt clean coverage it:test```)\n- generate coverage report: ```sbt coverageReport``` (reports to be found in ```target/scala-\u003cscala-version\u003e/scoverage-report```)\n\nIts also possible to enable coverage for each build via sbt setting ```coverageEnabled := true```.\nFor more settings (such as minimal coverage criteria for build to succeed), see above-referenced project page.\n\n## Adding templates for execution definitions\nKolibri accepts posting job definitions to the server, which then is processed. In order to simplify the process\nof defining such a definition, templates can be referenced from the storage (see config settings to see the distinct \navailable storage types, e.g AWS, GCP, LOCAL,...). \nFor this, relative to the base path configured, config property ```JOB_TEMPLATES_PATH``` defines where to look for folders.\nEach folder corresponds to a distinct type. \nThe routes to retrieve available template types and corresponding json templates are given in ```ResourceRoutes```.\nFurther, endpoint to store a new template of defined type is also defined here.\nOn calling with a specified type (given by the folder name), this name is normalized\n(upper-cased and - replaced with _) and matched against values of ```TemplateTypeValidationAndExecutionInfo```.\nIf there is a match, the enum value's ```isValid``` call tries to match to the specific type the json shall correspond to\nand its requestPath property defines against which endpoint the template can be thrown to actually execute it.\nThis route is used within route ```startExecutionDefinition```, which redirects a passed execution json to the\nexecution route for its particular type.\nNote that within a specific template folder, an ```info.json``` can be placed, containing a json with keys corresponding to the valid\nkeys of the template structure and values giving descriptions of the possible values.\nOn retrieval of a specific template via the ```ResourceRoute's routes```, if available this info is loaded and passed under the key\n```info``` in the response (the actual specific template is provided under key ```template```).\n\nThus, simplified to add a new template, you have to do:\n- add a ```subfolder``` in the path given by ```JOB_TEMPLATES_PATH``` (which is relative to the configured base path)\n- in ```ResourceRoutes.TemplateTypeValidationAndExecutionInfo```, define an enumeration value that equals to the folder's name \nafter normalization (by default uppercase and - replaced with _; so folder search_evaluation corresponds to enum with name\nSEARCH_EVALUATION)\n- place the templates you want (and which should correspond to json representations of the defined template type) in there\nwith suffix ```.json```\n- optionally place an ```info.json``` file in the folder, providing a json with keys being the possible template keys, \nand the values corresponding to descriptions of the values\n- you can request available templates for given types via ```getJobTemplateOverviewForType``` route, and the content of\na template along with (if info.json is placed) some descriptions of the fields and values via the \n```getJobTemplateByTypeAndIdentifier``` route.\n\n## Local execution - Issues and Fixes\n- starting the setup as provided in docker-compose file can be resource intensive. You might experience within the\ncluster heartbeat failures if not enough resources are available within docker. Thus make sure to allow sufficient \nresources (e.g \u003e= 4gb ram) to avoid this. The mentioned heartbeat failures will lead to loss of inter-node connections.\n\n## License\n\nThe kolibri-base code is licensed under APL 2.0.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fawagen%2Fkolibri-base","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fawagen%2Fkolibri-base","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fawagen%2Fkolibri-base/lists"}