{"id":41790537,"url":"https://github.com/tripl-ai/arc","last_synced_at":"2026-01-31T00:19:55.805Z","repository":{"id":49139939,"uuid":"149208836","full_name":"tripl-ai/arc","owner":"tripl-ai","description":"Arc is an opinionated framework for defining data pipelines which are predictable, repeatable and manageable.","archived":false,"fork":false,"pushed_at":"2024-02-10T22:08:06.000Z","size":74656,"stargazers_count":162,"open_issues_count":0,"forks_count":8,"subscribers_count":8,"default_branch":"master","last_synced_at":"2024-02-11T17:29:32.059Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"https://arc.tripl.ai","language":"Scala","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/tripl-ai.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"docs/contributing/index.html","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":"docs/security/index.html","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2018-09-18T01:09:32.000Z","updated_at":"2024-02-11T17:29:32.060Z","dependencies_parsed_at":"2024-02-10T17:39:29.823Z","dependency_job_id":null,"html_url":"https://github.com/tripl-ai/arc","commit_stats":null,"previous_names":[],"tags_count":58,"template":false,"template_full_name":null,"purl":"pkg:github/tripl-ai/arc","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tripl-ai%2Farc","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tripl-ai%2Farc/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tripl-ai%2Farc/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tripl-ai%2Farc/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/tripl-ai","download_url":"https://codeload.github.com/tripl-ai/arc/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tripl-ai%2Farc/sbom","scorecard":{"id":898854,"data":{"date":"2025-08-11","repo":{"name":"github.com/tripl-ai/arc","commit":"e424f8d3163b99dd4a090801285e27ebcd13e866"},"scorecard":{"version":"v5.2.1-40-gf6ed084d","commit":"f6ed084d17c9236477efd66e5b258b9d4cc7b389"},"score":2.6,"checks":[{"name":"Maintained","score":0,"reason":"project is archived","details":["Warn: Repository is archived."],"documentation":{"short":"Determines if the project is \"actively maintained\".","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#maintained"}},{"name":"Token-Permissions","score":-1,"reason":"No tokens found","details":null,"documentation":{"short":"Determines if the project's workflows follow the principle of least privilege.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#token-permissions"}},{"name":"Dangerous-Workflow","score":-1,"reason":"no workflows found","details":null,"documentation":{"short":"Determines if the project's GitHub Action workflows avoid dangerous patterns.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#dangerous-workflow"}},{"name":"Code-Review","score":0,"reason":"Found 0/17 approved changesets -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project requires human code review before pull requests (aka merge requests) are merged.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#code-review"}},{"name":"Packaging","score":-1,"reason":"packaging workflow not detected","details":["Warn: no GitHub/GitLab publishing workflow detected."],"documentation":{"short":"Determines if the project is published as a package that others can easily download, install, easily update, and uninstall.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#packaging"}},{"name":"Binary-Artifacts","score":8,"reason":"binaries present in source code","details":["Warn: binary detected: docs-src/hugo:1","Warn: binary detected: docs-src/hugo_linux_arm64:1"],"documentation":{"short":"Determines if the project has generated executable (binary) artifacts in the source repository.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#binary-artifacts"}},{"name":"CII-Best-Practices","score":0,"reason":"no effort to earn an OpenSSF best practices badge detected","details":null,"documentation":{"short":"Determines if the project has an OpenSSF (formerly CII) Best Practices Badge.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#cii-best-practices"}},{"name":"Security-Policy","score":0,"reason":"security policy file not detected","details":["Warn: no security policy file detected","Warn: no security file to analyze","Warn: no security file to analyze","Warn: no security file to analyze"],"documentation":{"short":"Determines if the project has published a security policy.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#security-policy"}},{"name":"Vulnerabilities","score":10,"reason":"0 existing vulnerabilities detected","details":null,"documentation":{"short":"Determines if the project has open, known unfixed vulnerabilities.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#vulnerabilities"}},{"name":"Fuzzing","score":0,"reason":"project is not fuzzed","details":["Warn: no fuzzer integrations found"],"documentation":{"short":"Determines if the project uses fuzzing.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#fuzzing"}},{"name":"License","score":10,"reason":"license file detected","details":["Info: project has a license file: LICENSE:0","Info: FSF or OSI recognized license: MIT License: LICENSE:0"],"documentation":{"short":"Determines if the project has defined a license.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#license"}},{"name":"Signed-Releases","score":-1,"reason":"no releases found","details":null,"documentation":{"short":"Determines if the project cryptographically signs release artifacts.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#signed-releases"}},{"name":"Branch-Protection","score":0,"reason":"branch protection not enabled on development/release branches","details":["Warn: branch protection not enabled for branch 'master'"],"documentation":{"short":"Determines if the default and release branches are protected with GitHub's branch protection settings.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#branch-protection"}},{"name":"SAST","score":0,"reason":"SAST tool is not run on all commits -- score normalized to 0","details":["Warn: 0 commits out of 22 are checked with a SAST tool"],"documentation":{"short":"Determines if the project uses static code analysis.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#sast"}},{"name":"Pinned-Dependencies","score":0,"reason":"dependency not pinned by hash detected -- score normalized to 0","details":["Warn: containerImage not pinned by hash: src/it/resources/flask_serving/Dockerfile:2: pin your Docker image by updating python:3.9 to python:3.9@sha256:b93ed1cb6815be1d99908ab629d49680be28024c4183611ded45c425ecf7b907","Warn: containerImage not pinned by hash: src/it/resources/minio/Dockerfile:1: pin your Docker image by updating minio/minio:RELEASE.2020-04-22T00-11-12Z to minio/minio:RELEASE.2020-04-22T00-11-12Z@sha256:182ae301bbc83ac65e3e863d4059530ce5a81fa3831b4e18356b2ead56945ab4","Warn: containerImage not pinned by hash: src/it/resources/postgres/Dockerfile:1: pin your Docker image by updating postgres:12.2-alpine to postgres:12.2-alpine@sha256:9ea72265275674225b1eaa2ae897dd244028af4ee7ef6e4e89fe474938e0992e","Warn: containerImage not pinned by hash: src/it/resources/tensorflow_serving/Dockerfile:1: pin your Docker image by updating tensorflow/serving:2.3.0 to tensorflow/serving:2.3.0@sha256:a94b7e3b0e825350675e83b0c2f2fc28f34be358c34e4126a1d828de899ec44f","Warn: pipCommand not pinned by hash: src/it/resources/flask_serving/Dockerfile:9","Info:   0 out of   4 containerImage dependencies pinned","Info:   0 out of   1 pipCommand dependencies pinned"],"documentation":{"short":"Determines if the project has declared and pinned the dependencies of its build process.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#pinned-dependencies"}}]},"last_synced_at":"2025-08-24T14:45:21.878Z","repository_id":49139939,"created_at":"2025-08-24T14:45:21.878Z","updated_at":"2025-08-24T14:45:21.878Z"},"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28924375,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-30T22:32:35.345Z","status":"ssl_error","status_checked_at":"2026-01-30T22:32:31.927Z","response_time":66,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-01-25T05:00:28.986Z","updated_at":"2026-01-31T00:19:55.797Z","avatar_url":"https://github.com/tripl-ai.png","language":"Scala","funding_links":[],"categories":["大数据"],"sub_categories":["Spring Cloud框架"],"readme":"Arc is an opinionated framework for defining data pipelines which are predictable, repeatable and manageable.\n\n## Documentation\n\nFull documentation is available here: https://arc.tripl.ai\n\n## Index\n\n- [What is Arc?](#what-is-spark-etl-pipeline)\n- [Getting Started](#getting-started)\n- [Principles](#principles)\n- [Not just for data engineers](#not-just-for-data-engineers)\n- [Why abstract from code?](#why-abstract-from-code)\n- [Why SQL first?](#why-sql-first)\n- [Example pipeline](#example-pipeline)\n- [Building](#building)\n- [Contributing](#contributing)\n- [License](#license)\n\n## What is Arc?\n\nArc is an **opinionated** framework for defining **predictable**, **repeatable** and **manageable** data transformation pipelines;\n\n- **predictable** in that data is used to define transformations - not code.\n- **repeatable** in that if a job is executed multiple times it will produce the same result.\n- **manageable** in that execution considerations and logging have been baked in from the start.\n\n## Getting Started\n\n![Notebook](/docs-src/static/img/arc-starter.png)\n\nArc has an interactive [Jupyter Notebook](https://jupyter.org/) extension to help with rapid development of jobs. Start by cloning [https://github.com/tripl-ai/arc-starter](https://github.com/tripl-ai/arc-starter) and running through the [tutorial](https://arc.tripl.ai/tutorial/).\n\nThis extension is available at [https://github.com/tripl-ai/arc-jupyter](https://github.com/tripl-ai/arc-jupyter).\n\n## Principles\n\nMany of these principles have come from [12factor](https://12factor.net/) with the aim of deploying **predictable**, **repeatable** and **manageable** data transformation pipelines:\n\n- **[single responsibility](https://en.wikipedia.org/wiki/Single_responsibility_principle)** components/stages with first-class support for [extending](https://arc.tripl.ai/extend/).\n- **stateless** jobs where possible and use of [immutable](https://en.wikipedia.org/wiki/Immutable_object) datasets.\n- **precise logging** to allow management of jobs at scale.\n- **library dependencies** limited or avoided where possible.\n\n## Not just for data engineers\n\nThe intent of the pipeline is to provide a simple way of creating Extract-Transform-Load (ETL) pipelines which are able to be maintained in production, and captures the answers to simple operational questions transparently to the user.\n\n- **monitoring**: is it working each time it's run? and how much resource was consumed in creating it?\n- **devops**: is packaged as a Docker image to allow rapid deployment on ephemeral compute.\n\nThese concerns are supported at run time to ensure that as deployment grows in uses and complexity it does not become opaque and unmanageable.\n\n## Why abstract from code?\n\nFrom experience a very high proportion of data pipelines perform very similar extract, transform and load actions on datasets. Unfortunately, whilst the desired outcomes are largely similar, the implementations are vastly varied resulting in higher maintenance costs, lower test-coverage and high levels of rework.\n\nThe intention of this project is to define and implement an **opinionated** standard approach for declaring data pipelines which is open and extensible. Abstraction from underlying code allows rapid deployment, a consistent way of defining transformation tasks (such as data typing) and allows abstraction of the pipeline definition from the pipeline execution (to support changing of the underlying execution engines) - see [declarative programming](https://en.wikipedia.org/wiki/Declarative_programming).\n\nCurrently it is tightly coupled to [Apache Spark](https://spark.apache.org) due to its fault-tolerance, performance and solid API for standard data engineering tasks but the definitions are human and machine readable [HOCON](https://github.com/lightbend/config/blob/master/HOCON.md) (a JSON derivative) allowing the transformation definitions to be implemented against future execution engines.\n\n## Why SQL first?\n\nSQL first (copied from the Mobile First UX principle) is an approach where, if possible, transformations are done using Structured Query Language (SQL) as a preference. This is because SQL is a very good way of expressing standard data transformation intent in a [declarative](https://en.wikipedia.org/wiki/Declarative_programming) way. SQL is so widely known and taught that finding people who are able to understand the business context and able to write basic SQL is much easier than finding a Scala developer who also understands the business context (for example).\n\nCurrently the [HIVE](https://cwiki.apache.org/confluence/display/Hive/LanguageManual) dialect of SQL is supported as [Spark SQL](https://spark.apache.org/docs/latest/sql-programming-guide.html) uses the same SQL dialect and has a lot of the same [functions](https://spark.apache.org/docs/latest/api/sql/) that would be expected from other SQL dialects. This could change in the future.\n\n## Example pipeline\n\nThis is an example of a fairly standard pipeline:\n\n1. First load a set of CSV files from an input directory. Separator is a comma and the file has a header.\n\n2. Convert the data to the correct datatypes using metadata defined in a separate JSON.\n\n3. Execute a SQL statement that will perform custom validation to ensure the data conversion in the previous step resulted in an acceptable data conversion error rate. If not successful the job will terminate.\n\n4. Write out the aggreate resultset to a Parquet target.\n\n```json\n{\n  \"stages\": [\n    {\n      \"type\": \"DelimitedExtract\",\n      \"name\": \"extract data from green_tripdata/0\",\n      \"environments\": [\n        \"production\",\n        \"test\"\n      ],\n      \"inputURI\": ${ETL_CONF_BASE_URL}\"/data/green_tripdata/0/*\",\n      \"outputView\": \"green_tripdata0_raw\",\n      \"delimiter\": \"Comma\",\n      \"quote\": \"DoubleQuote\",\n      \"header\": true\n    },\n    {\n      \"type\": \"TypingTransform\",\n      \"name\": \"apply green_tripdata/0 data types\",\n      \"environments\": [\n        \"production\",\n        \"test\"\n      ],\n      \"inputURI\": ${ETL_CONF_BASE_URL}\"/meta/green_tripdata/0/green_tripdata.json\",\n      \"inputView\": \"green_tripdata0_raw\",\n      \"outputView\": \"green_tripdata0\",\n      \"persist\": true\n    },\n    {\n      \"type\": \"SQLValidate\",\n      \"name\": \"ensure no errors exist after data typing\",\n      \"environments\": [\n        \"production\",\n        \"test\"\n      ],\n      \"inputURI\": ${ETL_CONF_BASE_URL}\"/job/0/sqlvalidate_errors.sql\",\n      \"sqlParams\": {\n        \"table_name\": \"green_tripdata0\"\n      }\n    },\n    {\n      \"type\": \"ParquetLoad\",\n      \"name\": \"write trips back to filesystem\",\n      \"environments\": [\n        \"production\",\n        \"test\"\n      ],\n      \"inputView\": \"green_tripdata0\",\n      \"outputURI\": ${ETL_CONF_BASE_URL}\"/data/output/green_tripdata0.parquet\",\n      \"saveMode\": \"Overwrite\"\n    }\n  ]\n}\n```\n\nA full worked example job is available [here](https://github.com/tripl-ai/arc/tree/master/tutorial/gnaf/job/0). It is for the Australian Geocoded National Address File [G-NAF](https://data.gov.au/dataset/geocoded-national-address-file-g-naf) and will load the pipe-separated data, apply data types, ensure there are no errors in applying data types and write the output to Parquet. This could easily be modified to write to a JDBC target.\n\n## Building\n\n### Library\n\nTo compile the main library run:\n\n```bash\nsbt +package\n```\n\nTo build a library to use with a [Databricks Runtime](https://databricks.com/product/databricks-runtime) environment it is easiest to `assembly` Arc with all the dependencies into a single JAR to simplify the deployment.\n\nExample:\n\n```bash\nsbt assembly\n```\n\nIf you are having problems compiling it is likely due to environment setup. This command is executed in CICD and uses a predictable build environment pulled from Dockerhub:\n\n```bash\ndocker run --rm -v $(pwd):/app -w /app mozilla/sbt:8u212_1.2.8 sbt +package\n```\n\n### Tests\n\nTo run unit tests:\n\n```bash\nsbt +test\n```\n\nTo run integration tests (which have external service depenencies):\n\n```bash\n./it.sh\n```\n\n### License Report\n\nTo rebuild the license report run:\n\n```bash\nsbt dumpLicenseReport\n```\n\n### Documentation\n\nTo generate the documentation you need to download [Hugo](https://gohugo.io/) to `/docs-src` and run `./hugo` in that  directory. The `/docs` directory is the output of the docuementation generation and should not be edited by hand. The `/docs` directory is automatically published by the Github Pages process on commit.\n\nYou can also generate documentation for previous version by checking out a previous tag (`git checkout tags/1.3.0`) and running `hugo server` from the `/docs-src` directory.\n\n## Contributing\n\nIf you have suggestions of additional components or find issues that you believe need fixing then please raise an issue. An issue with a test case is even more appreciated.\n\nWhen you contribute code, you affirm that the contribution is your original work and that you license the work to the project under the project’s open source license. Whether or not you state this explicitly, by submitting any copyrighted material via pull request, email, or other means you agree to license the material under the project’s open source license and warrant that you have the legal authority to do so.\n\n## Support\n\nFor questions around use (which are not clear from the documentation) post a new 'issue' in the [questions](https://github.com/tripl-ai/questions/issues) repository. This repository acts as a forum where questions can be posted, discussed and searched.\n\nFor commercial support requests please [contact us](mailto:contact@tripl.ai) via email.\n\n## Authors/Contributors\n\n- [Mike Seddon](https://github.com/seddonm1)\n- [John Bruce](https://github.com/jbruce)\n\n\n## Attribution\n\nThanks to the following projects:\n\n- [AGL Energy](https://github.com/aglenergy) for the original development.\n- [Apache Spark](https://spark.apache.org/) for the underlying framework that has made this library possible.\n- [slf4j-json-logger](https://github.com/savoirtech/slf4j-json-logger) Copyright (c) 2016 Savoir Technologies released under the [Apache 2.0 License](https://www.apache.org/licenses/LICENSE-2.0). We have slightly altered their library to change the default logging format.\n- [nyc-taxi-data](https://github.com/toddwschneider/nyc-taxi-data) for preparing an easy to use set of real-world data for the tutorial.\n\n## License\n\nArc is released under the [MIT License](https://opensource.org/licenses/MIT).\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftripl-ai%2Farc","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftripl-ai%2Farc","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftripl-ai%2Farc/lists"}