{"id":13588410,"url":"https://github.com/Unidata/ldm-docker","last_synced_at":"2025-04-08T03:33:56.524Z","repository":{"id":10055740,"uuid":"60878161","full_name":"Unidata/ldm-docker","owner":"Unidata","description":"Dockerized LDM ","archived":false,"fork":false,"pushed_at":"2023-12-19T21:43:32.000Z","size":90,"stargazers_count":12,"open_issues_count":1,"forks_count":14,"subscribers_count":8,"default_branch":"latest","last_synced_at":"2025-03-22T17:40:03.292Z","etag":null,"topics":["docker","geoscience"],"latest_commit_sha":null,"homepage":"https://hub.docker.com/r/unidata/ldm-docker/","language":"Tcl","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-3-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Unidata.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2016-06-10T21:35:37.000Z","updated_at":"2023-08-28T09:27:23.000Z","dependencies_parsed_at":"2024-01-13T07:54:20.356Z","dependency_job_id":"10b71244-ae8a-495f-b035-edf5af344fe8","html_url":"https://github.com/Unidata/ldm-docker","commit_stats":{"total_commits":74,"total_committers":6,"mean_commits":"12.333333333333334","dds":"0.17567567567567566","last_synced_commit":"ac90ca70cf00e3a0f9b9ec09f590e8615b9068e1"},"previous_names":[],"tags_count":7,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Unidata%2Fldm-docker","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Unidata%2Fldm-docker/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Unidata%2Fldm-docker/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Unidata%2Fldm-docker/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Unidata","download_url":"https://codeload.github.com/Unidata/ldm-docker/tar.gz/refs/heads/latest","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247772937,"owners_count":20993627,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["docker","geoscience"],"created_at":"2024-08-01T15:06:41.834Z","updated_at":"2025-04-08T03:33:56.195Z","avatar_url":"https://github.com/Unidata.png","language":"Tcl","funding_links":[],"categories":["Tcl"],"sub_categories":[],"readme":"- [Unidata LDM Docker](#h-62DFECEA)\n  - [Introduction](#h-18078CC5)\n  - [Versions](#h-A2FD2990)\n  - [Prerequisites](#h-AF97F2DC)\n  - [Installation](#h-CB0C0962)\n  - [Usage](#h-9349C7AF)\n    - [Upstream Data Feed from Unidata or Elsewhere](#h-62A78014)\n    - [Create LDM Directories on Docker Host](#h-E8938576)\n    - [Docker compose](#h-CF2B20CB)\n    - [Running LDM](#h-B170D5E0)\n    - [Stopping LDM](#h-81152971)\n    - [Delete LDM Container](#h-987BC10D)\n    - [Check What is Running](#h-77B9A5B4)\n    - [Running LDM (or Other Shell) Commands Inside the Container](#h-0C9E573F)\n    - [Updating the LDM](#h-796560DA)\n  - [Configuration](#h-2E05BCEF)\n    - [Docker compose](#h-18B4F365)\n    - [LDM Configuration Files](#h-487F14F2)\n    - [Configurable LDM UID and GID](#h-B2E37DB2)\n    - [Scouring from crontab](#h-5980F5C8)\n      - [Note on troubleshooting `ldmadmin scour`](#h-3A2F274D)\n    - [Additional Scouring](#h-0E2ACC51)\n  - [Self-Contained Example](#h-F23DB57C)\n  - [Citation](#h-9A911575)\n  - [Support](#h-1544BB3D)\n\n\n\n\u003ca id=\"h-62DFECEA\"\u003e\u003c/a\u003e\n\n# Unidata LDM Docker\n\nDockerized [LDM](https://www.unidata.ucar.edu/software/ldm).\n\n\n\u003ca id=\"h-18078CC5\"\u003e\u003c/a\u003e\n\n## Introduction\n\nThis repository contains files necessary to build and run a LDM Docker container. The Unidata LDM Docker images associated with this repository are [available on DockerHub](https://hub.docker.com/r/unidata/ldm-docker/).\n\n\n\u003ca id=\"h-A2FD2990\"\u003e\u003c/a\u003e\n\n## Versions\n\nSee tags listed [on dockerhub](https://hub.docker.com/r/unidata/ldm-docker/tags).\n\n\n\u003ca id=\"h-AF97F2DC\"\u003e\u003c/a\u003e\n\n## Prerequisites\n\nBefore you begin using this Docker container project, make sure your system has Docker installed. Docker Compose is optional but recommended.\n\n\n\u003ca id=\"h-CB0C0962\"\u003e\u003c/a\u003e\n\n## Installation\n\nYou can either pull the image from DockerHub with:\n\n```sh\ndocker pull unidata/ldm-docker:\u003cversion\u003e\n```\n\nOr you can build it yourself with:\n\n1.  ****Clone the repository****: `git clone https://github.com/Unidata/ldm-docker.git`\n2.  ****Navigate to the project directory****: `cd ldm-docker`\n3.  ****Build the Docker image****: `docker build -t ldm-docker:\u003cversion\u003e` .\n\n\n\u003ca id=\"h-9349C7AF\"\u003e\u003c/a\u003e\n\n## Usage\n\n\n\u003ca id=\"h-62A78014\"\u003e\u003c/a\u003e\n\n### Upstream Data Feed from Unidata or Elsewhere\n\nThe LDM operates on a push data model. You will have to find an institution who will agree to push you the data you are interested in. If you are part of the academic community please send a support email to `support-idd@unidata.ucar.edu` to discuss your LDM data requirements.\n\n\n\u003ca id=\"h-E8938576\"\u003e\u003c/a\u003e\n\n### Create LDM Directories on Docker Host\n\nYou will want to create the local directories defined in the `docker-compose.yml` for the LDM `/home/ldm/var/logs` directory and `/home/ldm/var/data` directory. For example:\n\n```\nmkdir logs data\n```\n\nIn typical LDM usage, the `data` directory is mounted on a data volume that can handle the amount of data you expect coming down the pipe. This `data` directory is usually not backed up.\n\n\n\u003ca id=\"h-CF2B20CB\"\u003e\u003c/a\u003e\n\n### Docker compose\n\nTo run the LDM Docker container, beyond a basic Docker setup, we recommend installing [docker-compose](https://docs.docker.com/compose/). `docker-compose` serves two purposes:\n\n1.  Reduce headaches involving unwieldy `docker` command lines where you are running `docker` with multiple volume mounts and port forwards. In situations like these, `docker` commands become difficult to issue and read. Instead, the lengthy `docker` command is captured in a `docker-compose.yml` that is easy to read, maintain, and can be committed to version control.\n\n2.  Coordinate the running of two or more containers. This can be useful for taking into account the same volume mountings, for example.\n\nHowever, `docker-compose` use is not mandatory. There is an example [docker-compose.yml](https://github.com/Unidata/ldm-docker/blob/master/docker-compose.yml) in this repository\n\n\n\u003ca id=\"h-B170D5E0\"\u003e\u003c/a\u003e\n\n### Running LDM\n\nOnce you have completed your `docker-compose.yml` setup, you can run the container with:\n\n```sh\ndocker-compose up -d ldm\n```\n\nNote that if you have not pulled or built the LDM Docker image, this command will implicitly pull the image.\n\nThe output of such command should be something like:\n\n```\nCreating ldm\n```\n\n\n\u003ca id=\"h-81152971\"\u003e\u003c/a\u003e\n\n### Stopping LDM\n\nTo stop this container:\n\n```sh\ndocker-compose stop\n```\n\n\n\u003ca id=\"h-987BC10D\"\u003e\u003c/a\u003e\n\n### Delete LDM Container\n\nTo clean the slate and remove the container (not the image, the container):\n\n```sh\ndocker-compose rm -f\n```\n\n\n\u003ca id=\"h-77B9A5B4\"\u003e\u003c/a\u003e\n\n### Check What is Running\n\nTo verify the LDM is alive you can run `ldmadmin config` **inside** the container. To do that, run:\n\n```sh\ndocker exec ldm gosu ldm ldmadmin config\n```\n\nwhich should give output that looks something like:\n\n```\nhostname:              docker.localhost.local\nos:                    Linux\nrelease:               4.18.0-425.19.2.el8_7.x86_64\nldmhome:               /home/ldm\nLDM version:           6.14.5\nPATH:                  /home/ldm/ldm-6.14.5/bin:/home/ldm/util:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/home/ldm/bin\nLDM conf file:         /home/ldm/etc/ldmd.conf\npqact(1) conf file:    /home/ldm/etc/pqact.conf\nscour(1) conf file:    /home/ldm/etc/scour.conf\nproduct queue:         /home/ldm/var/queues/ldm.pq\nqueue size:            6G bytes\nqueue slots:           default\nreconciliation mode:   do nothing\npqsurf(1) path:        /home/ldm/var/queues/pqsurf.pq\npqsurf(1) size:        2M\nIP address:            0.0.0.0\nport:                  388\nPID file:              /home/ldm/ldmd.pid\nLock file:             /home/ldm/.ldmadmin.lck\nmaximum clients:       256\nmaximum latency:       3600\ntime offset:           3600\nlog file:              /home/ldm/var/logs/ldmd.log\nnumlogs:               7\nlog_rotate:            1\nnetstat:               /usr/bin/netstat -A inet -t -n\ntop:                   /usr/bin/top -b -n 1\nmetrics file:          /home/ldm/var/logs/metrics.txt\nmetrics files:         /home/ldm/var/logs/metrics.txt*\nnum_metrics:           4\ncheck time:            1\ndelete info files:     0\nntpdate(1):            /usr/sbin/chronyd\nntpdate(1) timeout:    5\ntime servers:          ntp.ucsd.edu ntp1.cs.wisc.edu ntppub.tamu.edu otc1.psu.edu timeserver.unidata.ucar.edu\ntime-offset limit:     10\n```\n\n\n\u003ca id=\"h-0C9E573F\"\u003e\u003c/a\u003e\n\n### Running LDM (or Other Shell) Commands Inside the Container\n\nWhen using the LDM in any realistic scenario, you will want to execute LDM commands (e.g., `notifyme`). Make sure you are user `ldm` and not `root`. Running LDM commands as `root` user can result in anomalous, and difficult to track down behavior. There are a couple of different ways you can accomplish this goal.\n\n1.  You can enter the container with `docker exec -it \u003ccontainer name or ID\u003e bash`. For example,\n\n```sh\n$ docker exec -it ldm bash\nbash-4.4# su - ldm\n[ldm@291c06984ded ~]$ notifyme -vl- -h idd.unidata.ucar.edu\n```\n\nor\n\n```sh\n[ldm@291c06984ded ~]$ ldmadmin restart\n```\n\n1.  Or you can simply execute the command from outside the container with `docker exec \u003ccontainer name or ID\u003e gosu ldm \u003ccommand\u003e`. (The `gosu` utility enables you to \"drop down\" to another user from `root`. Remember, when you first enter the container, you are `root` user.) For example,\n\n```sh\ndocker exec ldm gosu ldm notifyme -vl- -h idd.unidata.ucar.edu\n```\n\nor\n\n```sh\ndocker exec ldm gosu ldm ldmadmin restart\n```\n\n\n\u003ca id=\"h-796560DA\"\u003e\u003c/a\u003e\n\n### Updating the LDM\n\nWhen Unidata releases a new version of the LDM, it is easy to update the container:\n\n```sh\ndocker pull unidata/ldm-docker:\u003cversion\u003e\ndocker-compose stop \u0026\u0026 docker-compose rm -f \u0026\u0026 docker-compose up -d ldm\n```\n\n\n\u003ca id=\"h-2E05BCEF\"\u003e\u003c/a\u003e\n\n## Configuration\n\n\n\u003ca id=\"h-18B4F365\"\u003e\u003c/a\u003e\n\n### Docker compose\n\nYou can customize the default `docker-compose.yml` to decide:\n\n-   which LDM image version you want to run\n-   which port will map to port `388`\n\nFor anyone who has worked with the LDM, you will be familiar with the following directories:\n\n-   `etc/`\n-   `var/data`\n-   `var/logs`\n-   `var/queues`\n\nThese directory paths will be mounted outside the container with `docker-compose.yml`.\n\n\n\u003ca id=\"h-487F14F2\"\u003e\u003c/a\u003e\n\n### LDM Configuration Files\n\nIn the `etc` directory of this repository, you will have to do the usual LDM configuration by editing:\n\n-   `ldmd.conf`\n-   `registry.xml`\n-   `scour.conf`\n-   `pqact.conf`\n\n\n\u003ca id=\"h-B2E37DB2\"\u003e\u003c/a\u003e\n\n### Configurable LDM UID and GID\n\nThe problem with mounted Docker volumes and UID/DIG mismatch headaches is best explained here: \u003chttps://denibertovic.com/posts/handling-permissions-with-docker-volumes/\u003e.\n\nThis container allows the possibility of controlling the UID/GID of the `ldm` user inside the container via `LDM_USER_ID` and `LDM_GROUP_ID` environment variables. If not set, the default UID/GID is `1000~/~1000`. For example,\n\n```sh\ndocker run --name ldm  \\\n     -e LDM_USER_ID=`id -u`  \\\n     -e LDM_GROUP_ID=`getent group $USER | cut -d':' -f3`  \\\n     -v ./etc/:/home/ldm/etc/ \\\n     -v ./data/:/home/ldm/var/data/ \\\n     -v ./data/:/home/ldm/var/queues/ \\\n     -v ./logs/:/home/ldm/var/logs/ \\\n     -v ./cron/:/var/spool/cron/ \\\n     -d -p 388:388 unidata/ldm-docker:\u003cversion\u003e\n```\n\nwhere `LDM_USER_ID` and `LDM_GROUP_ID` have been configured with the UID/GID of the user running the container. If using `docker-compose`, see `compose.env` to configure the UID/GID of user `ldm` inside the container.\n\nThis feature enables greater control of file permissions written outside the container via mounted volumes (e.g., data files written by the LDM).\n\nNote that this UID/GID configuration option will not work on operating systems where Docker is not native (e.g., macOS).\n\n\n\u003ca id=\"h-5980F5C8\"\u003e\u003c/a\u003e\n\n### Scouring from crontab\n\nThe [recommended LDM crontab entries](http://www.unidata.ucar.edu/software/ldm/ldm-current/basics/configuring.html#cron) have been installed inside the container. You can modify the LDM crontab by editing the `cron/ldm` file. This file can be mounted over `/var/spool/cron/ldm` with `docker-compose.yml`. See the `docker-compose.yml` file herein for an example.\n\n\n\u003ca id=\"h-3A2F274D\"\u003e\u003c/a\u003e\n\n#### Note on troubleshooting `ldmadmin scour`\n\nWhen running `ldmadmin scour` manually to test your scour configuration (i.e., `scour.conf`), you may run into warnings that look like the following:\n\n```\n[ldm@9bc83d08f79f etc]$ ldmadmin scour\n20221220T003634.938502Z scour[975675]               parser.c:loginHomeDir:531           WARN  loginHomeDir:getlogin() failed: No such device or address\n20221220T003634.938556Z scour[975675]               parser.c:isSameAsLoginDirectory:168 WARN  Could not determine login HOME\n```\n\nThese may be red herrings and not necessarily a sign of `ldmadmin scour` running anomalously.\n\nTo see if scour is behaving as expected, you can issue this command which gives the oldest file in the `/data` directory tree or wherever data needs to be scoured:\n\n```sh\nfind /data -type f -printf '%T+ %p\\n' | sort | head -n 1\n```\n\n\n\u003ca id=\"h-0E2ACC51\"\u003e\u003c/a\u003e\n\n### Additional Scouring\n\nThe scouring facilities built-in to the LDM mysteriously do not have the ability to scour empty directories. In this container, therefore, are included additional scouring utility scripts that will scour empty directories as well.\n\n-   `scourBYnumber`\n-   `scourBYempty`\n-   `scourBYday`\n\nTypically, these will be invoked from cron and will correspond to the same directories being scoured in `scour.conf`. For example, if you have a `scour.conf` that has the following entries:\n\n```\n/data/ldm/pub/decoded/gempak/areas/ANTARCTIC    2\n/data/ldm/pub/decoded/gempak/areas/ARCTIC   2\n/data/ldm/pub/decoded/gempak/areas/GEWCOMP  4\n```\n\nyou may wish to have corresponding entries in your crontab (e.g., `cron/ldm` file that will be mounted into the container with `docker-compose.yml`) file:\n\n```\n16 0 * * * /home/ldm/util/scourBYday /data/ldm/pub/decoded/gempak/areas/ANTARCTIC 2\n17 0 * * * /home/ldm/util/scourBYday /data/ldm/pub/decoded/gempak/areas/ARCTIC    2\n18 0 * * * /home/ldm/util/scourBYday /data/ldm/pub/decoded/gempak/areas/GEWCOMP   4\n```\n\n\n\u003ca id=\"h-F23DB57C\"\u003e\u003c/a\u003e\n\n## Self-Contained Example\n\nThis project comes with a self-contained example. To run it:\n\n1.  `docker pull unidata/ldm-docker:\u003cversion\u003e`\n2.  `cd example`\n3.  possibly edit `etc/registry.xml` to change hostname currently set at `ldm-example.jetstream-cloud.org`\n4.  possibly edit `etc/registry.xml` to change the LDM queue size currently set at `2G`\n5.  edit `compose.env` to set `UID` and `GID` of user running container\n6.  `docker-compose up -d`\n\nAssuming you have permission to request data from `iddb.unidata.ucar.edu` (see `example/etc/ldmd.conf`), after a few moments you should see data. For example:\n\n```\n./example/data/ldm/pub/native/radar/composite/grib2/N0R/20180301/Level3_Composite_N0R_20180301_2010.grib2\n./example/data/ldm/pub/native/radar/composite/grib2/N0R/20180301/Level3_Composite_N0R_20180301_2015.grib2\n./example/data/ldm/pub/native/radar/composite/grib2/N0R/20180301/Level3_Composite_N0R_20180301_2020.grib2\n```\n\n\n\u003ca id=\"h-9A911575\"\u003e\u003c/a\u003e\n\n## Citation\n\nIn order to cite this project, please simply make use of the Unidata LDM DOI: https://doi.org/10.5065/D64J0CT0 \u003chttps://doi.org/10.5065/D64J0CT0\u003e\n\n\n\u003ca id=\"h-1544BB3D\"\u003e\u003c/a\u003e\n\n## Support\n\nIf you have a question or would like support for this LDM Docker container, consider [submitting a GitHub issue](https://github.com/Unidata/ldm-docker/issues). Alternatively, you may wish to start a discussion on the LDM Community mailing list: [ldm-users@unidata.ucar.edu](mailto:ldm-users@unidata.ucar.edu).\n\nFor general LDM questions, please see the [Unidata LDM page](https://www.unidata.ucar.edu/software/ldm/).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FUnidata%2Fldm-docker","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FUnidata%2Fldm-docker","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FUnidata%2Fldm-docker/lists"}