{"id":15208244,"url":"https://github.com/vasnake/artefacts-2019_2023","last_synced_at":"2026-03-10T10:05:17.580Z","repository":{"id":246909029,"uuid":"824081851","full_name":"vasnake/artefacts-2019_2023","owner":"vasnake","description":"Collection of some interesting pieces of my projects. Spark, Scala, Python, sh","archived":false,"fork":false,"pushed_at":"2024-10-24T12:12:19.000Z","size":1161,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-03-06T17:15:44.248Z","etag":null,"topics":["catalyst","etl","ml","scala","spark","udaf","udf"],"latest_commit_sha":null,"homepage":"","language":"Scala","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/vasnake.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-07-04T10:23:04.000Z","updated_at":"2025-02-26T10:13:36.000Z","dependencies_parsed_at":"2024-08-29T10:58:03.984Z","dependency_job_id":"165030e2-62bd-4c71-beb6-ffb184c056ef","html_url":"https://github.com/vasnake/artefacts-2019_2023","commit_stats":{"total_commits":198,"total_committers":1,"mean_commits":198.0,"dds":0.0,"last_synced_commit":"024853280d6857df72320824b79927c2b4832475"},"previous_names":["vasnake/artefacts-2019_2023"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/vasnake/artefacts-2019_2023","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vasnake%2Fartefacts-2019_2023","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vasnake%2Fartefacts-2019_2023/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vasnake%2Fartefacts-2019_2023/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vasnake%2Fartefacts-2019_2023/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/vasnake","download_url":"https://codeload.github.com/vasnake/artefacts-2019_2023/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vasnake%2Fartefacts-2019_2023/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":30329698,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-03-10T05:25:20.737Z","status":"ssl_error","status_checked_at":"2026-03-10T05:25:17.430Z","response_time":106,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["catalyst","etl","ml","scala","spark","udaf","udf"],"created_at":"2024-09-28T07:01:10.111Z","updated_at":"2026-03-10T10:05:17.556Z","avatar_url":"https://github.com/vasnake.png","language":"Scala","funding_links":[],"categories":[],"sub_categories":[],"readme":"# artefacts-2019_2023\n\n\u003e Data engineering is the software engineering that enables data scientists to work effectively\n\nCollection of some interesting pieces from my projects (2019 .. 2023). Spark, Scala, Python, sh\n\nTwo main categories:\n- [Spark/Scala stuff](./etl-ml-pieces.scala/readme.md)\n- [Python scripts and apps](./scripts.python/readme.md)\n\nPlus one derivative, 'Spark/Scala stuff' after migration to Spark3:\n- [Spark3 Scala stuff](./spark3-pieces.scala/readme.md)\n\nFor full description see docs inside.\n\n## CI/CD\n\n- To build uber-jar use `sbt assembly` command.\n- To run unit tests use `sbt test` command.\n- To run integration tests ... TBD.\n\nManual directory cleanup: `find . -depth -type d \\( -name target -or -name .bloop -or -name .bsp -or -name .metals -or -name metastore_db -or -name spark-warehouse \\) -exec echo rm -rfv {} \\;`\n\nBash script, [build fat-jar for Spark](cicd/build_uber_jar.sh).\nExample: `bash -xve ./cicd/build_uber_jar.sh ./etl-ml-pieces.scala/ /tmp/workdir/PACKAGES`\n\n[Docker image](cicd/jarbuilder.Dockerfile), contains tools for uber-jar builder.\nBuild docker image: `docker buildx build -f cicd/jarbuilder.Dockerfile --tag docker/jarbuilder:0.1.0 .`\n\nBuild uber-jar using docker container:\n```s\ndocker run \\\n    -it --rm \\\n    --mount type=bind,src=./etl-ml-pieces.scala,dst=/sbtproject \\\n    --workdir /sbtproject \\\n    docker/jarbuilder:0.1.0 \\\n    sbt assembly \u0026\u0026 \\\ncp -v etl-ml-pieces.scala/target/scala-2.11/*.jar /tmp/\n```\n\nRun interactive sbt shell in docker container:\n```s\ndocker run \\\n    -it --rm \\\n    --mount type=bind,src=./etl-ml-pieces.scala,dst=/sbtproject \\\n    --workdir /sbtproject \\\n    docker/jarbuilder:0.1.0 \\\n    sbt -v --mem 4096\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvasnake%2Fartefacts-2019_2023","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fvasnake%2Fartefacts-2019_2023","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvasnake%2Fartefacts-2019_2023/lists"}