{"id":19746340,"url":"https://github.com/astrolabsoftware/fink-docker","last_synced_at":"2025-04-30T08:30:55.474Z","repository":{"id":41192205,"uuid":"467436829","full_name":"astrolabsoftware/fink-docker","owner":"astrolabsoftware","description":"Collection of Dockerfiles for Fink","archived":false,"fork":false,"pushed_at":"2024-11-06T09:51:03.000Z","size":86,"stargazers_count":0,"open_issues_count":4,"forks_count":2,"subscribers_count":3,"default_branch":"main","last_synced_at":"2024-11-06T10:55:48.697Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/astrolabsoftware.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-03-08T09:03:26.000Z","updated_at":"2024-11-06T09:51:01.000Z","dependencies_parsed_at":"2024-05-28T08:27:30.835Z","dependency_job_id":"9695fc28-edc8-420a-8a35-62250d787d4c","html_url":"https://github.com/astrolabsoftware/fink-docker","commit_stats":null,"previous_names":[],"tags_count":36,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/astrolabsoftware%2Ffink-docker","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/astrolabsoftware%2Ffink-docker/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/astrolabsoftware%2Ffink-docker/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/astrolabsoftware%2Ffink-docker/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/astrolabsoftware","download_url":"https://codeload.github.com/astrolabsoftware/fink-docker/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":224202820,"owners_count":17272807,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-12T02:14:20.181Z","updated_at":"2025-04-30T08:30:55.468Z","avatar_url":"https://github.com/astrolabsoftware.png","language":"Shell","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Dockerfile for Fink\n\nThis repository contains the Dockerfile to build all-in-one image for Fink. For k8s deployment, see https://github.com/astrolabsoftware/fink-broker instead.\n\n## Workflow\n\nWe mainly use these in the Continuous Integration of various repositories. The image is built at each release of this repository: \n\n```\n# push modifications\ngit tag \u003cnumber\u003e\ngit push origin --tags\n\n# then publish release on GH\n```\n\nThe code is currently checked on `Almalinux:9`. \n\n| | Latest |\n|-|-----|\n| OS | AlmaLinux 9 |\n| Spark | 3.4.1 |\n| Hadoop | 3.2 |\n| Java |11 |\n| HBase | 2.4.10 |\n| Kafka | 2.8.1 |\n\nThe production environment is currently in use at VirtualData, Université Paris-Saclay, to process the ZTF alert stream. For development purposes, one can also build locally the image using the wrapper:\n\n```bash\n./fink_docker -h\nBuild Dockerfile image for Fink\n\n Usage:\n ./fink_docker [-h]\n ./fink_docker --build [--os] [--tag]\n ./fink_docker --run [--tag]\n\n Specify the name of a folder with a Dockerfile with the option --os.\n Use --build to build the image from the Dockerfile in --os, with a tag (--tag).\n Use --run with a tag (--tag) to enter the container instead\n For the deployment, you need to have credentials defined.\n Use -h to display this help.\n```\n\nwhere the argument to `--os` is a folder containing necessary files (copy and modify `ztf` or `rubin` for your purposes -- see below).\n\n### Building an image\n\nTo build an image from a specific Dockerfile, use:\n\n```bash\n# e.g. build the prod image based on AlmaLinux 9\n# and name it prod\n./fink_docker --build --os ztf --tag dev\n```\n\nYou might need to modify resolvers though. In this case, just add in `/etc/resolv.conf`\n\n```\nnameserver 8.8.8.8\n```\n\nand restart docker before building your image.\n\nWithout optimisation, the images are quite big because of dependencies. Here is the breakdown for an image based on centos7 with a single build stage (size on disk):\n\n|        | size |\n|--------|------|\n| centos7| 204MB|\n| +system build dependencies|   822MB |\n| +Apache Kafka, HBase and Spark|    1.39GB  |\n| +Python dependencies|  3.62GB    |\n\nWith a multi-stage build, and some optimisation on the Python side:\n\n|        | size | Comment |\n|--------|------|---------|\n| prod   |  3.62GB    | Default\n| prod + hard multi-stage   |  2.75GB  | No Java available |\n| prod + soft multi-stage   |  3.15GB  | Java available |\n\nWith the hard multi-stage (i.e. we do not include any of the system build dependencies), we save 1GB. But the image is useless as we cannot use java-based framework. With the soft multi-stage (i.e. we keep Java, but get rid of other system build dependencies), we save 500 MB.\n\nThe current versions use the soft multi-stage strategy (about 3GB).\n\nTodo:\n- [ ] Inspect better Python dependencies.\n\n### Start a container\n\n```bash\n# Enter a container based on the prod image\n./fink_docker --run --tag dev\n```\n\nNote that when starting a container, a script is launched to automatically start Apache HBase and Apache Kafka. Several environment variables are already defined inside the container (see each Dockerfile specifically).\n\n### Deploy images\n\nSee https://docs.docker.com/docker-hub/repos/#pushing-a-docker-container-image-to-docker-hub.\n\nExample:\n\n```bash\n$ docker images\nREPOSITORY                TAG       IMAGE ID       CREATED          SIZE\njulienpeloton/fink-ci     dev       d5ee1c3b1299   20 minutes ago   3.15GB\n\n$ docker push julienpeloton/fink-ci:dev\nThe push refers to repository [docker.io/julienpeloton/fink-ci]\nd3eeb8e94cd6: Pushed\n970209ec3e0c: Pushed\n529806cc03af: Pushed\n174f56854903: Mounted from library/centos\n```\n\n### Create your own image\n\nTo create your own image with the versions you want, you would create a new folder, and copy files from an existing one (`centos7` or `centos9stream`). Then modify the values in the `ARG` in the Dockerfile. Change the base image if you wish. And then build it using:\n\n```bash\n./fink_docker --build --os \u003cfolder_name\u003e --tag \u003cwhatever\u003e\n```\n\nTodo:\n- [ ] Allow `--build-arg` to be used from the CLI\n\n## Image availability\n\nWe have deployed images in DockerHub ([julienpeloton/fink-ci](https://hub.docker.com/repository/docker/julienpeloton/fink-ci)), than can be used easily.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fastrolabsoftware%2Ffink-docker","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fastrolabsoftware%2Ffink-docker","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fastrolabsoftware%2Ffink-docker/lists"}