{"id":15096890,"url":"https://github.com/apache/incubator-liminal","last_synced_at":"2026-01-14T08:28:43.135Z","repository":{"id":38183772,"uuid":"271182596","full_name":"apache/incubator-liminal","owner":"apache","description":"Apache Liminals goal is to operationalise the machine learning process, allowing data scientists to quickly transition from a successful experiment to an automated pipeline of model training, validation, deployment and inference in production. Liminal provides a Domain Specific Language to build ML workflows on top of Apache Airflow.","archived":true,"fork":false,"pushed_at":"2024-07-19T17:00:32.000Z","size":6680,"stargazers_count":145,"open_issues_count":9,"forks_count":40,"subscribers_count":11,"default_branch":"master","last_synced_at":"2025-12-22T06:48:14.101Z","etag":null,"topics":["ai","airflow","big-data","data-science","machine-learning","ml","workflows"],"latest_commit_sha":null,"homepage":"https://liminal.apache.org","language":"Python","has_issues":false,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/apache.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-06-10T04:57:10.000Z","updated_at":"2025-10-16T04:56:55.000Z","dependencies_parsed_at":"2024-09-19T22:02:17.465Z","dependency_job_id":"40bb69ba-2051-4763-8f30-52c78854204e","html_url":"https://github.com/apache/incubator-liminal","commit_stats":{"total_commits":158,"total_committers":21,"mean_commits":7.523809523809524,"dds":0.6772151898734178,"last_synced_commit":"aee9827f3b3354b60898efd7c1412b673a10bc5d"},"previous_names":[],"tags_count":19,"template":false,"template_full_name":null,"purl":"pkg:github/apache/incubator-liminal","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/apache%2Fincubator-liminal","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/apache%2Fincubator-liminal/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/apache%2Fincubator-liminal/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/apache%2Fincubator-liminal/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/apache","download_url":"https://codeload.github.com/apache/incubator-liminal/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/apache%2Fincubator-liminal/sbom","scorecard":{"id":201433,"data":{"date":"2025-08-11","repo":{"name":"github.com/apache/incubator-liminal","commit":"aee9827f3b3354b60898efd7c1412b673a10bc5d"},"scorecard":{"version":"v5.2.1-40-gf6ed084d","commit":"f6ed084d17c9236477efd66e5b258b9d4cc7b389"},"score":3.9,"checks":[{"name":"Maintained","score":0,"reason":"project is archived","details":["Warn: Repository is archived."],"documentation":{"short":"Determines if the project is \"actively maintained\".","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#maintained"}},{"name":"Dangerous-Workflow","score":10,"reason":"no dangerous workflow patterns detected","details":null,"documentation":{"short":"Determines if the project's GitHub Action workflows avoid dangerous patterns.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#dangerous-workflow"}},{"name":"Code-Review","score":8,"reason":"Found 20/24 approved changesets -- score normalized to 8","details":null,"documentation":{"short":"Determines if the project requires human code review before pull requests (aka merge requests) are merged.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#code-review"}},{"name":"Token-Permissions","score":0,"reason":"detected GitHub workflow tokens with excessive permissions","details":["Warn: no topLevel permission defined: .github/workflows/pre_commits.yml:1","Warn: no topLevel permission defined: .github/workflows/unittests.yml:1","Warn: no topLevel permission defined: .github/workflows/versioning.yml:1","Info: no jobLevel write permissions found"],"documentation":{"short":"Determines if the project's workflows follow the principle of least privilege.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#token-permissions"}},{"name":"Packaging","score":-1,"reason":"packaging workflow not detected","details":["Warn: no GitHub/GitLab publishing workflow detected."],"documentation":{"short":"Determines if the project is published as a package that others can easily download, install, easily update, and uninstall.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#packaging"}},{"name":"Binary-Artifacts","score":10,"reason":"no binaries found in the repo","details":null,"documentation":{"short":"Determines if the project has generated executable (binary) artifacts in the source repository.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#binary-artifacts"}},{"name":"CII-Best-Practices","score":0,"reason":"no effort to earn an OpenSSF best practices badge detected","details":null,"documentation":{"short":"Determines if the project has an OpenSSF (formerly CII) Best Practices Badge.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#cii-best-practices"}},{"name":"Fuzzing","score":0,"reason":"project is not fuzzed","details":["Warn: no fuzzer integrations found"],"documentation":{"short":"Determines if the project uses fuzzing.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#fuzzing"}},{"name":"License","score":10,"reason":"license file detected","details":["Info: project has a license file: LICENSE:0","Info: FSF or OSI recognized license: Apache License 2.0: LICENSE:0"],"documentation":{"short":"Determines if the project has defined a license.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#license"}},{"name":"Signed-Releases","score":-1,"reason":"no releases found","details":null,"documentation":{"short":"Determines if the project cryptographically signs release artifacts.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#signed-releases"}},{"name":"Branch-Protection","score":0,"reason":"branch protection not enabled on development/release branches","details":["Warn: branch protection not enabled for branch 'master'"],"documentation":{"short":"Determines if the default and release branches are protected with GitHub's branch protection settings.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#branch-protection"}},{"name":"Security-Policy","score":10,"reason":"security policy file detected","details":["Info: security policy file detected: github.com/apache/.github/.github/SECURITY.md:1","Info: Found linked content: github.com/apache/.github/.github/SECURITY.md:1","Info: Found disclosure, vulnerability, and/or timelines in security policy: github.com/apache/.github/.github/SECURITY.md:1","Info: Found text in security policy: github.com/apache/.github/.github/SECURITY.md:1"],"documentation":{"short":"Determines if the project has published a security policy.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#security-policy"}},{"name":"Pinned-Dependencies","score":0,"reason":"dependency not pinned by hash detected -- score normalized to 0","details":["Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/pre_commits.yml:21: update your workflow using https://app.stepsecurity.io/secureworkflow/apache/incubator-liminal/pre_commits.yml/master?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/pre_commits.yml:26: update your workflow using https://app.stepsecurity.io/secureworkflow/apache/incubator-liminal/pre_commits.yml/master?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/pre_commits.yml:33: update your workflow using https://app.stepsecurity.io/secureworkflow/apache/incubator-liminal/pre_commits.yml/master?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/pre_commits.yml:48: update your workflow using https://app.stepsecurity.io/secureworkflow/apache/incubator-liminal/pre_commits.yml/master?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/unittests.yml:22: update your workflow using https://app.stepsecurity.io/secureworkflow/apache/incubator-liminal/unittests.yml/master?enable=pin","Warn: third-party GitHubAction not pinned by hash: .github/workflows/unittests.yml:24: update your workflow using https://app.stepsecurity.io/secureworkflow/apache/incubator-liminal/unittests.yml/master?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/unittests.yml:30: update your workflow using https://app.stepsecurity.io/secureworkflow/apache/incubator-liminal/unittests.yml/master?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/versioning.yml:17: update your workflow using https://app.stepsecurity.io/secureworkflow/apache/incubator-liminal/versioning.yml/master?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/versioning.yml:19: update your workflow using https://app.stepsecurity.io/secureworkflow/apache/incubator-liminal/versioning.yml/master?enable=pin","Warn: containerImage not pinned by hash: examples/extensibility/images/custom_image_builder/Dockerfile:18: pin your Docker image by updating alpine:3.4 to alpine:3.4@sha256:b733d4a32c4da6a00a84df2ca32791bb03df95400243648d8c539e7b4cce329c","Warn: containerImage not pinned by hash: liminal/build/image/python/Dockerfile:20","Warn: containerImage not pinned by hash: liminal/build/image/python_server/Dockerfile:20","Warn: containerImage not pinned by hash: liminal/build/image/spark/Dockerfile:18: pin your Docker image by updating bitnami/spark:3.1.2-debian-10-r23 to bitnami/spark:3.1.2-debian-10-r23@sha256:5c02b0a5864e3731432b70e687cfd01c46a47d02a6df6ff4e07d8907fa9542d8","Warn: containerImage not pinned by hash: scripts/Dockerfile-airflow:19: pin your Docker image by updating apache/airflow:2.1.2-python3.8 to apache/airflow:2.1.2-python3.8@sha256:012d883b14348cce60fa1f8ceb6bb99676497d84ae6ac2859944060090bfaa70","Warn: pipCommand not pinned by hash: liminal/build/image/python/Dockerfile:37","Warn: pipCommand not pinned by hash: liminal/build/image/python_server/Dockerfile:32","Warn: pipCommand not pinned by hash: liminal/build/image/python_server/Dockerfile:35","Warn: pipCommand not pinned by hash: .github/workflows/pre_commits.yml:41","Warn: pipCommand not pinned by hash: .github/workflows/pre_commits.yml:42","Warn: pipCommand not pinned by hash: .github/workflows/pre_commits.yml:43","Warn: pipCommand not pinned by hash: .github/workflows/unittests.yml:35","Warn: pipCommand not pinned by hash: .github/workflows/unittests.yml:36","Warn: pipCommand not pinned by hash: .github/workflows/versioning.yml:24","Warn: pipCommand not pinned by hash: .github/workflows/versioning.yml:25","Info:   0 out of   8 GitHub-owned GitHubAction dependencies pinned","Info:   0 out of   1 third-party GitHubAction dependencies pinned","Info:   0 out of   5 containerImage dependencies pinned","Info:   1 out of  11 pipCommand dependencies pinned"],"documentation":{"short":"Determines if the project has declared and pinned the dependencies of its build process.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#pinned-dependencies"}},{"name":"SAST","score":0,"reason":"SAST tool is not run on all commits -- score normalized to 0","details":["Warn: 0 commits out of 29 are checked with a SAST tool"],"documentation":{"short":"Determines if the project uses static code analysis.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#sast"}},{"name":"Vulnerabilities","score":0,"reason":"146 existing vulnerabilities detected","details":["Warn: Project is vulnerable to: PYSEC-2018-66 / GHSA-562c-5r94-xh97","Warn: Project is vulnerable to: PYSEC-2019-179 / GHSA-5wv5-4vpf-pj6m","Warn: Project is vulnerable to: PYSEC-2023-62 / GHSA-m2qf-hxjv-5gpq","Warn: Project is vulnerable to: PYSEC-2024-110 / GHSA-jw8x-6495-233v","Warn: Project is vulnerable to: PYSEC-2020-108 / GHSA-jxfp-4rvq-9h9m","Warn: Project is vulnerable to: GHSA-2g68-c3qc-8985","Warn: Project is vulnerable to: GHSA-f9vj-2wh5-fj8j","Warn: Project is vulnerable to: PYSEC-2023-221 / GHSA-hrfv-mqp8-q5rw","Warn: Project is vulnerable to: PYSEC-2023-57 / GHSA-px8h-6qxv-m22q","Warn: Project is vulnerable to: GHSA-q34m-jh98-gwm2","Warn: Project is vulnerable to: PYSEC-2023-58 / GHSA-xg9f-g7g7-2323","Warn: Project is vulnerable to: PYSEC-2022-203","Warn: Project is vulnerable to: PYSEC-2022-48 / GHSA-77rm-9x9h-xj3g","Warn: Project is vulnerable to: GHSA-8gq9-2x98-w8hf","Warn: Project is vulnerable to: GHSA-8qvm-5x2c-j2w7","Warn: Project is vulnerable to: PYSEC-2017-65 / GHSA-jwvw-v7c5-m82h","Warn: Project is vulnerable to: GHSA-32g6-mg92-ghm2","Warn: Project is vulnerable to: GHSA-7pc3-pr3q-58vg","Warn: Project is vulnerable to: GHSA-wjvx-jhpj-r54r","Warn: Project is vulnerable to: PYSEC-2023-44 / GHSA-329j-jfvr-rhr6","Warn: Project is vulnerable to: PYSEC-2022-42976 / GHSA-43xg-8wmj-cw8h","Warn: Project is vulnerable to: GHSA-4x9r-j582-cgr8","Warn: Project is vulnerable to: PYSEC-2023-72 / GHSA-59hw-j9g6-mfg3","Warn: Project is vulnerable to: PYSEC-2021-142 / GHSA-8q59-q68h-6hv4","Warn: Project is vulnerable to: PYSEC-2018-49 / GHSA-rprw-h62v-c2w7","Warn: Project is vulnerable to: PYSEC-2023-134 / GHSA-269x-pg5c-5xgm","Warn: Project is vulnerable to: PYSEC-2023-119 / GHSA-2h84-3crq-vgfj","Warn: Project is vulnerable to: PYSEC-2023-197 / GHSA-32wr-qqw6-5mfp","Warn: Project is vulnerable to: PYSEC-2023-106 / GHSA-3h4m-m55v-gx4m","Warn: Project is vulnerable to: PYSEC-2022-42983 / GHSA-3q8r-f3pj-3gc4","Warn: Project is vulnerable to: PYSEC-2022-30 / GHSA-3v7g-4pg3-7r6j","Warn: Project is vulnerable to: GHSA-45r6-j3cc-6mxx","Warn: Project is vulnerable to: PYSEC-2022-11 / GHSA-4jh2-3c85-q67h","Warn: Project is vulnerable to: PYSEC-2023-267 / GHSA-5938-79hg-xh3q","Warn: Project is vulnerable to: PYSEC-2023-105 / GHSA-5946-8p38-vffp","Warn: Project is vulnerable to: GHSA-5f35-pq34-c87q","Warn: Project is vulnerable to: PYSEC-2022-29 / GHSA-65xw-pcqw-hjrh","Warn: Project is vulnerable to: GHSA-666g-rfc5-c9jv","Warn: Project is vulnerable to: PYSEC-2022-42982 / GHSA-6pw3-8h9w-32gc","Warn: Project is vulnerable to: PYSEC-2024-245 / GHSA-6v6w-h8m6-7mv2","Warn: Project is vulnerable to: PYSEC-2024-42 / GHSA-6xwf-xvf3-v459","Warn: Project is vulnerable to: GHSA-7wqf-h36w-47mc","Warn: Project is vulnerable to: PYSEC-2023-265 / GHSA-8f57-wcmg-4jmh","Warn: Project is vulnerable to: PYSEC-2024-212 / GHSA-92xg-gmrq-5c3w","Warn: Project is vulnerable to: PYSEC-2024-195 / GHSA-9xpj-62mm-24h2","Warn: Project is vulnerable to: PYSEC-2024-13 / GHSA-c3c6-f2ww-xfr2","Warn: Project is vulnerable to: GHSA-c732-xvv8-g94c","Warn: Project is vulnerable to: PYSEC-2023-202 / GHSA-cgx2-rrmr-jx43","Warn: Project is vulnerable to: PYSEC-2022-42971 / GHSA-f9fq-78ch-4wmj","Warn: Project is vulnerable to: PYSEC-2022-42981 / GHSA-fvw2-2pf7-77vw","Warn: Project is vulnerable to: PYSEC-2023-104 / GHSA-ggwr-4vr8-g7wv","Warn: Project is vulnerable to: PYSEC-2022-42970 / GHSA-h63r-9xxf-f2c7","Warn: Project is vulnerable to: PYSEC-2023-2 / GHSA-h6g5-wqqr-3mw3","Warn: Project is vulnerable to: PYSEC-2021-326 / GHSA-h88f-r7cw-8fv3","Warn: Project is vulnerable to: PYSEC-2023-232 / GHSA-hm9r-7f84-25c9","Warn: Project is vulnerable to: PYSEC-2023-203 / GHSA-j3w8-2p2h-mrr9","Warn: Project is vulnerable to: PYSEC-2024-189 / GHSA-j482-47xf-p25c","Warn: Project is vulnerable to: GHSA-j857-2pwm-jjmm","Warn: Project is vulnerable to: PYSEC-2023-59 / GHSA-jchm-fm4q-c2fp","Warn: Project is vulnerable to: PYSEC-2023-171 / GHSA-mjqh-v5f2-g2mw","Warn: Project is vulnerable to: PYSEC-2023-158 / GHSA-pm87-24wq-r8w9","Warn: Project is vulnerable to: PYSEC-2022-261 / GHSA-q8h9-pqcx-59hw","Warn: Project is vulnerable to: PYSEC-2023-231 / GHSA-r7x6-xfcm-3mxv","Warn: Project is vulnerable to: PYSEC-2022-42984 / GHSA-rg94-84xj-7gq3","Warn: Project is vulnerable to: GHSA-rmf2-pwfq-h75j","Warn: Project is vulnerable to: PYSEC-2023-60 / GHSA-vcf6-3wv2-5vcr","Warn: Project is vulnerable to: PYSEC-2024-14 / GHSA-vm5m-qmrx-fw8w","Warn: Project is vulnerable to: PYSEC-2024-181 / GHSA-w7cp-g8v7-r54m","Warn: Project is vulnerable to: PYSEC-2023-170 / GHSA-wpg8-mf6h-gm92","Warn: Project is vulnerable to: PYSEC-2023-152 / GHSA-x2mh-8fmc-rqgh","Warn: Project is vulnerable to: PYSEC-2023-103 / GHSA-xvw9-3mhm-xjqq","Warn: Project is vulnerable to: PYSEC-2023-136","Warn: Project is vulnerable to: PYSEC-2023-3","Warn: Project is vulnerable to: PYSEC-2024-182","Warn: Project is vulnerable to: GHSA-w695-p3j5-hrj9","Warn: Project is vulnerable to: PYSEC-2024-4 / GHSA-2mqj-m65w-jghx","Warn: Project is vulnerable to: PYSEC-2023-165 / GHSA-cwvm-v4w8-q58c","Warn: Project is vulnerable to: PYSEC-2022-42992 / GHSA-hcpj-qp55-gfph","Warn: Project is vulnerable to: PYSEC-2023-137 / GHSA-pr76-5cm5-w9cj","Warn: Project is vulnerable to: PYSEC-2023-161 / GHSA-wfm5-v35h-vwf4","Warn: Project is vulnerable to: PYSEC-2019-217 / GHSA-462w-v97r-4m45","Warn: Project is vulnerable to: PYSEC-2014-8 / GHSA-8r7q-cvjq-x353","Warn: Project is vulnerable to: GHSA-cpwx-vrp4-4pq7","Warn: Project is vulnerable to: PYSEC-2014-82 / GHSA-fqh9-2qgg-h84h","Warn: Project is vulnerable to: PYSEC-2021-66 / GHSA-g3rq-g295-4j3m","Warn: Project is vulnerable to: GHSA-h5c8-rqwp-cp95","Warn: Project is vulnerable to: GHSA-h75v-3vvj-5mfj","Warn: Project is vulnerable to: PYSEC-2019-220 / GHSA-hj2j-77xm-mc5v","Warn: Project is vulnerable to: GHSA-q2x7-8rv6-6q7h","Warn: Project is vulnerable to: GHSA-9hjg-9r4m-mvj7","Warn: Project is vulnerable to: GHSA-9wx4-h78v-vm56","Warn: Project is vulnerable to: PYSEC-2023-74 / GHSA-j8r2-6x86-q33q","Warn: Project is vulnerable to: PYSEC-2022-43017 / GHSA-qwmp-2cf2-g9g6","Warn: Project is vulnerable to: GHSA-3c5c-7235-994j","Warn: Project is vulnerable to: GHSA-3f63-hfp8-52jq","Warn: Project is vulnerable to: PYSEC-2021-41 / GHSA-3wvg-mj6g-m9cv","Warn: Project is vulnerable to: PYSEC-2020-77 / GHSA-3xv8-3j54-hgrp","Warn: Project is vulnerable to: PYSEC-2020-80 / GHSA-43fq-w8qq-v88h","Warn: Project is vulnerable to: GHSA-44wm-f244-xhp3","Warn: Project is vulnerable to: GHSA-4fx9-vc88-q2xc","Warn: Project is vulnerable to: PYSEC-2021-35 / GHSA-57h3-9rgr-c24m","Warn: Project is vulnerable to: PYSEC-2020-172 / GHSA-5gm3-px64-rw72","Warn: Project is vulnerable to: PYSEC-2021-331 / GHSA-7534-mm45-c74v","Warn: Project is vulnerable to: PYSEC-2021-92 / GHSA-7r7m-5h27-29hp","Warn: Project is vulnerable to: PYSEC-2020-78 / GHSA-8843-m7mw-mxqm","Warn: Project is vulnerable to: PYSEC-2023-227 / GHSA-8ghj-p4vj-mr35","Warn: Project is vulnerable to: PYSEC-2014-87 / GHSA-8m9x-pxwq-j236","Warn: Project is vulnerable to: PYSEC-2022-10 / GHSA-8vj2-vxx3-667w","Warn: Project is vulnerable to: PYSEC-2021-36 / GHSA-8xjq-8fcg-g5hw","Warn: Project is vulnerable to: PYSEC-2016-6 / GHSA-8xjv-v9xq-m5h9","Warn: Project is vulnerable to: PYSEC-2021-42 / GHSA-95q3-8gr9-gm8w","Warn: Project is vulnerable to: PYSEC-2022-168 / GHSA-9j59-75qj-795w","Warn: Project is vulnerable to: PYSEC-2014-10 / GHSA-cfmr-38g9-f2h7","Warn: Project is vulnerable to: PYSEC-2020-76 / GHSA-cqhg-xjhh-p8hf","Warn: Project is vulnerable to: PYSEC-2021-40 / GHSA-f4w8-cv6p-x6r5","Warn: Project is vulnerable to: PYSEC-2021-69 / GHSA-f5g8-5qq7-938w","Warn: Project is vulnerable to: PYSEC-2021-139 / GHSA-g6rj-rv7j-xwp4","Warn: Project is vulnerable to: PYSEC-2015-16 / GHSA-h5rf-vgqx-wjv2","Warn: Project is vulnerable to: PYSEC-2016-5 / GHSA-hggx-3h72-49ww","Warn: Project is vulnerable to: PYSEC-2020-84 / GHSA-hj69-c76v-86wr","Warn: Project is vulnerable to: PYSEC-2016-7 / GHSA-hvr8-466p-75rh","Warn: Project is vulnerable to: PYSEC-2015-15 / GHSA-j6f7-g425-4gmx","Warn: Project is vulnerable to: GHSA-j7hp-h8jx-5ppr","Warn: Project is vulnerable to: PYSEC-2019-110 / GHSA-j7mj-748x-7p78","Warn: Project is vulnerable to: GHSA-jgpv-4h4c-xhw3","Warn: Project is vulnerable to: PYSEC-2022-42979 / GHSA-m2vv-5vj5-2hm7","Warn: Project is vulnerable to: PYSEC-2021-37 / GHSA-mvg9-xffr-p774","Warn: Project is vulnerable to: PYSEC-2020-83 / GHSA-p49h-hjvm-jg3h","Warn: Project is vulnerable to: PYSEC-2022-8 / GHSA-pw3c-h7wp-cvhx","Warn: Project is vulnerable to: PYSEC-2021-93 / GHSA-q5hq-fp76-qmrc","Warn: Project is vulnerable to: PYSEC-2020-82 / GHSA-r7rm-8j6h-r933","Warn: Project is vulnerable to: PYSEC-2014-23 / GHSA-r854-96gq-rfg3","Warn: Project is vulnerable to: PYSEC-2016-8 / GHSA-rwr3-c2q8-gm56","Warn: Project is vulnerable to: PYSEC-2020-81 / GHSA-vcqg-3p29-xw73","Warn: Project is vulnerable to: PYSEC-2020-79 / GHSA-vj42-xq3r-hr3r","Warn: Project is vulnerable to: PYSEC-2021-70 / GHSA-vqcj-wrf2-7v73","Warn: Project is vulnerable to: PYSEC-2016-9 / GHSA-w4vg-rf63-f3j3","Warn: Project is vulnerable to: PYSEC-2014-22 / GHSA-x895-2wrm-hvp7","Warn: Project is vulnerable to: PYSEC-2022-9 / GHSA-xrcv-f9gm-v42c","Warn: Project is vulnerable to: PYSEC-2021-137","Warn: Project is vulnerable to: PYSEC-2021-138","Warn: Project is vulnerable to: PYSEC-2021-317","Warn: Project is vulnerable to: PYSEC-2021-38","Warn: Project is vulnerable to: PYSEC-2021-39","Warn: Project is vulnerable to: PYSEC-2021-94","Warn: Project is vulnerable to: PYSEC-2023-175"],"documentation":{"short":"Determines if the project has open, known unfixed vulnerabilities.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#vulnerabilities"}}]},"last_synced_at":"2025-08-16T22:54:32.693Z","repository_id":38183772,"created_at":"2025-08-16T22:54:32.693Z","updated_at":"2025-08-16T22:54:32.693Z"},"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28414028,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-14T08:16:59.381Z","status":"ssl_error","status_checked_at":"2026-01-14T08:13:45.490Z","response_time":107,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai","airflow","big-data","data-science","machine-learning","ml","workflows"],"created_at":"2024-09-25T16:01:59.653Z","updated_at":"2026-01-14T08:28:43.092Z","avatar_url":"https://github.com/apache.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003c!--\nLicensed to the Apache Software Foundation (ASF) under one\nor more contributor license agreements.  See the NOTICE file\ndistributed with this work for additional information\nregarding copyright ownership.  The ASF licenses this file\nto you under the Apache License, Version 2.0 (the\n\"License\"); you may not use this file except in compliance\nwith the License.  You may obtain a copy of the License at\n\n  http://www.apache.org/licenses/LICENSE-2.0\n\nUnless required by applicable law or agreed to in writing,\nsoftware distributed under the License is distributed on an\n\"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\nKIND, either express or implied.  See the License for the\nspecific language governing permissions and limitations\nunder the License.\n--\u003e\n\n# Apache Liminal\n\nApache Liminal is an end-to-end platform for data engineers \u0026 scientists, allowing them to build,\ntrain and deploy machine learning models in a robust and agile way.\n\nThe platform provides the abstractions and declarative capabilities for\ndata extraction \u0026 feature engineering followed by model training and serving.\nLiminal's goal is to operationalize the machine learning process, allowing data scientists to\nquickly transition from a successful experiment to an automated pipeline of model training,\nvalidation, deployment and inference in production, freeing them from engineering and\nnon-functional tasks, and allowing them to focus on machine learning code and artifacts.\n\n# Basics\n\nUsing simple YAML configuration, create your own schedule data pipelines (a sequence of tasks to\nperform), application servers,  and more.\n\n## Getting Started\n\nA simple getting stated guide for Liminal can be found [here](docs/getting-started/hello_world.md)\n\n## Apache Liminal Documentation\n\nFull documentation of Apache Liminal can be found [here](docs/liminal)\n\n## High Level Architecture\n\nHigh level architecture documentation can be found [here](docs/architecture.md)\n\n## Example YAML config file\n\n```yaml\n---\nname: MyLiminalStack\nowner: Bosco Albert Baracus\nvolumes:\n  - volume: myvol1\n    local:\n      path: /Users/me/myvol1\nimages:\n  - image: my_python_task_img\n    type: python\n    source: write_inputs\n  - image: my_parallelized_python_task_img\n    source: write_outputs\n  - image: my_server_image\n    type: python_server\n    source: myserver\n    endpoints:\n      - endpoint: /myendpoint1\n        module: my_server\n        function: myendpoint1func\npipelines:\n  - pipeline: my_pipeline\n    start_date: 1970-01-01\n    timeout_minutes: 45\n    schedule: 0 * 1 * *\n    metrics:\n      namespace: TestNamespace\n      backends: [ 'cloudwatch' ]\n    tasks:\n      - task: my_python_task\n        type: python\n        description: static input task\n        image: my_python_task_img\n        env_vars:\n          NUM_FILES: 10\n          NUM_SPLITS: 3\n        mounts:\n          - mount: mymount\n            volume: myvol1\n            path: /mnt/vol1\n        cmd: python -u write_inputs.py\n      - task: my_parallelized_python_task\n        type: python\n        description: parallelized python task\n        image: my_parallelized_python_task_img\n        env_vars:\n          FOO: BAR\n        executors: 3\n        mounts:\n          - mount: mymount\n            volume: myvol1\n            path: /mnt/vol1\n        cmd: python -u write_inputs.py\nservices:\n  - service: my_python_server\n    description: my python server\n    image: my_server_image\n```\n\n# Installation\n\n1. Install this repository (HEAD)\n\n```bash\n   pip install git+https://github.com/apache/incubator-liminal.git\n```\n\n2. Optional: set LIMINAL_HOME to path of your choice (if not set, will default to ~/liminal_home)\n\n```bash\necho 'export LIMINAL_HOME=\u003c/path/to/some/folder\u003e' \u003e\u003e ~/.bash_profile \u0026\u0026 source ~/.bash_profile\n```\n\n# Authoring pipelines\n\nThis involves at minimum creating a single file called liminal.yml as in the example above.\n\nIf your pipeline requires custom python code to implement tasks, they should be organized\n[like this](https://github.com/apache/incubator-liminal/tree/master/tests/runners/airflow/liminal)\n\nIf your pipeline  introduces imports of external packages which are not already a part\nof the liminal framework (i.e. you had to pip install them yourself), you need to also provide\na requirements.txt in the root of your project.\n\n# Testing the pipeline locally\n\nWhen your pipeline code is ready, you can test it by running it locally on your machine.\n\n1. Ensure you have The Docker engine running locally, and enable a local Kubernetes cluster:\n\n  ![Kubernetes configured](https://raw.githubusercontent.com/apache/incubator-liminal/master/images/k8s_running.png)\n\n  And allocate it at least 3 CPUs (under \"Resources\" in the Docker preference UI).\n\n  If you want to execute your pipeline on a remote kubernetes cluster, make sure the cluster is configured using:\n\n  ```bash\n  kubectl config set-context \u003cyour remote kubernetes cluster\u003e\n  ```\n\n2. Build the docker images used by your pipeline.\n\nIn the example pipeline above, you can see that tasks and services have an \"image\" field - such as\n\"my_static_input_task_image\". This means that the task is executed inside a docker container, and the docker container\nis created from a docker image where various code and libraries are installed.\n\nYou can take a look at what the build process looks like, e.g.\n[here](https://github.com/apache/incubator-liminal/tree/master/liminal/build/image/python)\n\nIn order for the images to be available for your pipeline, you'll need to build them locally:\n\n```bash\ncd \u003c/path/to/your/liminal/code\u003e\nliminal build\n```\n\nYou'll see that a number of outputs indicating various docker images built.\n\n3. Create a kubernetes local volume \\\nIn case your Yaml includes working with [volumes](https://github.com/apache/incubator-liminal/blob/6253f8b2c9dc244af032979ec6d462dc3e07e170/docs/getting_started.md#mounted-volumes)\nplease first run the following command:\n\n```bash\ncd \u003c/path/to/your/liminal/code\u003e\nliminal create\n```\n\n4. Deploy the pipeline:\n\n```bash\ncd \u003c/path/to/your/liminal/code\u003e\nliminal deploy\n```\n\nNote: after upgrading liminal, it's recommended to issue the command\n\n```bash\nliminal deploy --clean\n```\n\nThis will rebuild the airlfow docker containers from scratch with a fresh version of liminal, ensuring consistency.\n\n5. Start the server\n\n```bash\nliminal start\n```\n\n6. Stop the server\n\n```bash\nliminal stop\n```\n\n7. Display the server logs\n\n```bash\nliminal logs --follow/--tail\n\nNumber of lines to show from the end of the log:\nliminal logs --tail=10\n\nFollow log output:\nliminal logs --follow\n```\n\n8. Navigate to [http://localhost:8080/admin](http://localhost:8080/admin)\n\n9. You should see your ![pipeline](https://raw.githubusercontent.com/apache/incubator-liminal/master/images/airflow.png)\nThe pipeline is scheduled to run according to the ```json schedule: 0 * 1 * *``` field in the .yml file you provided.\n\n10. To manually activate your pipeline:\n\n- Click your pipeline and then click \"trigger DAG\"\n- Click \"Graph view\"\nYou should see the steps in your pipeline getting executed in \"real time\" by clicking \"Refresh\" periodically.\n\n![Pipeline activation](https://raw.githubusercontent.com/apache/incubator-liminal/master/images/airflow_trigger.png)\n\n# Contributing\n\nMore information on contributing can be found [here](CONTRIBUTING.md)\n\n# Community\n\nThe Liminal community holds a public call every Monday\n\n- [Liminal Community Calendar](https://calendar.google.com/calendar/u/0/r?cid=jom1i20emghura6s6ookhe2skk@group.calendar.google.com)\n- [Dev-Mailing-List](https://lists.apache.org/list.html?dev@liminal.apache.org)\n\n## Running Tests (for contributors)\n\nWhen doing local development and running Liminal unit-tests, make sure to set LIMINAL_STAND_ALONE_MODE=True\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fapache%2Fincubator-liminal","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fapache%2Fincubator-liminal","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fapache%2Fincubator-liminal/lists"}