{"id":18818945,"url":"https://github.com/geoscienceaustralia/dea-sandbox","last_synced_at":"2025-04-13T23:32:51.397Z","repository":{"id":36964681,"uuid":"198156223","full_name":"GeoscienceAustralia/dea-sandbox","owner":"GeoscienceAustralia","description":"Digital Earth Australia Sandbox config and planning","archived":false,"fork":false,"pushed_at":"2025-03-25T05:26:48.000Z","size":346,"stargazers_count":14,"open_issues_count":23,"forks_count":7,"subscribers_count":26,"default_branch":"develop","last_synced_at":"2025-03-25T06:26:14.339Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/GeoscienceAustralia.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-07-22T05:51:46.000Z","updated_at":"2025-02-19T03:17:23.000Z","dependencies_parsed_at":"2023-11-14T05:21:45.212Z","dependency_job_id":"e914a486-a79c-4a06-9f45-50bbdc47794f","html_url":"https://github.com/GeoscienceAustralia/dea-sandbox","commit_stats":null,"previous_names":[],"tags_count":59,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/GeoscienceAustralia%2Fdea-sandbox","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/GeoscienceAustralia%2Fdea-sandbox/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/GeoscienceAustralia%2Fdea-sandbox/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/GeoscienceAustralia%2Fdea-sandbox/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/GeoscienceAustralia","download_url":"https://codeload.github.com/GeoscienceAustralia/dea-sandbox/tar.gz/refs/heads/develop","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248796962,"owners_count":21163058,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-08T00:19:33.890Z","updated_at":"2025-04-13T23:32:50.960Z","avatar_url":"https://github.com/GeoscienceAustralia.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# DEA Sandbox\n\n![Sandbox build and push latest](https://github.com/GeoscienceAustralia/dea-sandbox/workflows/Sandbox%20build%20and%20push%20latest/badge.svg)\n![Vulnerability Scan](https://github.com/GeoscienceAustralia/dea-sandbox/workflows/Vulnerability%20Scan/badge.svg)\n\nDigital Earth Australia Sandbox Docker build, configuration and planning.\n\nPlease ensure all tasks have a description that includes a clear definition of when the task is complete.\n\n[Kanban](https://github.com/GeoscienceAustralia/dea-sandbox/projects/1)\n\n[Issues](https://github.com/GeoscienceAustralia/dea-sandbox/issues)\n\n## Automated builds\n\nBuilds are run automatically out of this repository. Basic workflow is:\n\n- Create a branch and implement your changes.\n- The Docker image is built on the branch and as part of the pull request.\n- The integration test will be run on PR, merge and release events.\n- Once the PR is merged, a build will run. The resultant image will be tagged with git `head` and `latest` and pushed to ECR.\n- When you'd like to release a new stable image, create a release with a version number in the format `major.minor.patch`, e.g., `2.0.1`. This will trigger a build and the resultant image will be tagged with `2.0.1` (in this example) and `stable`, and pushed to ECR.\n\n## Packages' version maintenance and upgrade\n\nThe base environment uses Conda, and the Docker image is built in two stages:\n\n1. Create conda env and install as many as possible packages from `conda-forge`. Then `pip install` the rest, e.g., most `odc-` packages.\n2. Copies the Conda env to a new Ubuntu image.\n\nTo speed up the build, the workflow pulls images from a cache stored on ECR. However, with every build the cache layers starting from `pip install` will be discarded, so that the newest versions of `odc-` packages will be installed. Thus, to perform version upgrades on these packages, creating a release is sufficient.\n\nThe old Conda env cache is used for all builds unless `env.yml` is changed. In addition to speeding up builds, this cached environment allows us to maintain a working `odc-` codebase and defer resolving conflicts on geospatial base packages such as `GDAL` and `GEOS`, until we have a good opportunity to manually review them.\n\nThe steps for package version upgrades are as follows:\n\n- For `odc-` packages, create a new release.\n- For packages listed in `env.yml`, be specific with the version required, for example, `Shapely\u003e=2.0`.\n\n**Note**: Avoid using `==` or `\u003c=`, unless there is a hard requirement or a very good reason. Ensure you specify this reason clearly in your PR; what is it required for, your justification, and any supporting PRs (if applicable).\n\n## Local environment\n\n### Simple test environment\n\nA simple local environment that can be used to test the JupyterHub system in can be started using Docker Compose\nwith the command `docker-compose up`\n\nif the container started up successfully, it will show console log similar to the following\n\n```\ndea-sandbox-sandbox-1   | [C 2022-12-08 03:02:47.100 ServerApp]\ndea-sandbox-sandbox-1   |\ndea-sandbox-sandbox-1   |     To access the server, open this file in a browser:\ndea-sandbox-sandbox-1   |         file:///home/jovyan/.local/share/jupyter/runtime/jpserver-7-open.html\ndea-sandbox-sandbox-1   |     Or copy and paste one of these URLs:\ndea-sandbox-sandbox-1   |         http://5cf0ca7d3dd0:9988/lab?token=bedea39c6e6ef14f633a99968cf47ec891588b6e14ec0862\ndea-sandbox-sandbox-1   |      or http://127.0.0.1:9988/lab?token=bedea39c6e6ef14f633a99968cf47ec891588b6e14ec0862\n```\n\nbrowsing to http://localhost:9988 and adding the token that is displayed\non your terminal, i.e. `http://localhost:9988/lab?token=bedea39c6e6ef14f633a99968cf47ec891588b6e14ec0862`\n\n#### Tip for hosting behind reverse proxy\n\n```\nlocation / {\n        proxy_set_header   Host $http_host;\n        proxy_set_header   X-Forwarded-For $remote_addr;\n        proxy_read_timeout 300s;\n        proxy_connect_timeout 75s;\n\n        # forward to port 9988\n        proxy_pass         \"http://localhost:9988\";\n\n        # for terminal and notebook websockets\n        proxy_http_version 1.1;\n        proxy_set_header Upgrade $http_upgrade;\n        proxy_set_header Connection \"upgrade\";\n    }\n```\n\n\n### With DEA database\nTo run `docker-compose` with a DEA indexed database, you'll need to start a Kubernetes port forwading process\nwith a command like `port-forward -n service deployment/pg-proxy 5432:5432`.\n\nAnd then set up a file in the root of this folder `.env` with connection details in it. Use the\n`.env.example` as a template for this file. You then want to run the Docker Compose environment without a\npostgres database, so use the command `docker-compose -f docker-compose.yml up` to start it. This will ignore\nthe `docker-compose.override.yml` file, which provides a postgres container.\n\nAny files in the `./notebooks` folder will be mounted in the user's home folder. That is to say that `./notebooks`\nwill be mounted at `/home/jovyan`/\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgeoscienceaustralia%2Fdea-sandbox","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgeoscienceaustralia%2Fdea-sandbox","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgeoscienceaustralia%2Fdea-sandbox/lists"}