{"id":22944197,"url":"https://github.com/52north/maridataharvest","last_synced_at":"2025-08-12T22:31:33.523Z","repository":{"id":49904422,"uuid":"344733508","full_name":"52North/MariDataHarvest","owner":"52North","description":"MariDataHarvest is a tool for scrapping and harvesting Automatic Identification System (AIS) data provided by marinecadastre then appending it with the weather and environment conditions provided by CMEMS and RDA at each geographical and UTC timestamp point.","archived":false,"fork":false,"pushed_at":"2023-06-28T08:19:58.000Z","size":3619,"stargazers_count":3,"open_issues_count":40,"forks_count":6,"subscribers_count":6,"default_branch":"main","last_synced_at":"2024-04-12T07:15:34.829Z","etag":null,"topics":["ais-data","environmental-data","marine-data"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/52North.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2021-03-05T07:44:39.000Z","updated_at":"2024-01-05T14:16:32.000Z","dependencies_parsed_at":"2023-12-08T10:47:27.312Z","dependency_job_id":null,"html_url":"https://github.com/52North/MariDataHarvest","commit_stats":{"total_commits":278,"total_committers":7,"mean_commits":"39.714285714285715","dds":"0.49640287769784175","last_synced_commit":"92465f79a29418b3727c2d3bfac9f8fbd13ecee3"},"previous_names":[],"tags_count":15,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/52North%2FMariDataHarvest","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/52North%2FMariDataHarvest/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/52North%2FMariDataHarvest/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/52North%2FMariDataHarvest/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/52North","download_url":"https://codeload.github.com/52North/MariDataHarvest/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":229711044,"owners_count":18112066,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ais-data","environmental-data","marine-data"],"created_at":"2024-12-14T14:16:51.942Z","updated_at":"2024-12-14T14:16:53.485Z","avatar_url":"https://github.com/52North.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# MariDataHarvest\n\n## TODOS\n\n- [x] List only required/directly used dependencies in `dev` branch without versions, unless they are required in THIS version.\n      Split `requirements.txt` and list only directly used ones.\n  - [x] EnvironmentalData\n  - [x] Harvester\n  - [x] EnvDataServer\n- [x] Solve context path problem (via configuration or similar)\n- [ ] Code-Cleaning: \n  - [x] Remove unused imports\n  - [] functions, variables\n- [ ] Move every config variable to config.py\n- [ ] Establish central location for variable definition: keep in weather.py; EnvDataServer/app.py accesses them via python features;  page.js accesses theme via light weight endpoint implemented in app.py\n      Convert to dictionary and provide helper functions to retrieve the required representation\n\n- [ ] Establish `dev` branch\n\n- [ ] Add Package/Component Overview to root `README.md`\n      Document, who is using what part of the utilities package.\n- [ ] Comment wind variable in webui (and API)\n      page.js:95 comment\n      EnvDataServer/app.py:159 remove Wind from list of variables if not configurable\n\n- [ ] WebUI: set time bbox to current day (limit input to hours only, no minutes, if easily possible)\n- [ ] Store last request in cookie or Browser-Storage\n- [ ] History Back does not work\n\n- [ ] Replace MOTU bei OPeNDAP\n- [ ] `Dockerfile` of harvester:\n  - [ ] reuqirements.txt -\u003e new name\n  - [ ] copy package and the dependency packages (utilities, environmentData)\n  - [ ] adjust `CMD` to match new python package structure\n    - [ ] add env pythonpath\n    - [ ] change CMD to variant with \"[\" s\n- [ ] Identify useful k8s/proxy timeout or switch to asynchronous communication pattern with managed request queue\n\n## Description\n\nMariDataHarvest is a tool for scrapping and harvesting Automatic Identification System (AIS) data provided by [marinecadastre](https://marinecadastre.gov/AIS/)\nthen appending it with the weather and environment conditions provided by [CMEMS](https://nrt.cmems-du.eu) and [RDA](https://rda.ucar.edu/index.html) at each geographical and UTC timestamp point.\nIn the following is a description of the datasets used:\n\n- \u003cdetails\u003e\n  \u003csummary\u003eDatasets Description\u003c/summary\u003e\n\n  [\u003cimg alt=\"Datasets Description\" align=\"middle\" src=\"./img/datasets_description.PNG\"/\u003e](https://docs.google.com/spreadsheets/d/1GxcBtnaAa2GQDwZibYFbWPXGi7BPpPdYLZwyetpsJOQ/edit?usp=sharing)\n\n  \u003c/details\u003e\n\n- \u003cdetails\u003e\n  \u003csummary\u003eVariables Description\u003c/summary\u003e\n\n  [\u003cimg alt=\"Variables Description\" align=\"middle\" src=\"./img/variables_description.PNG\"/\u003e](https://docs.google.com/spreadsheets/d/1GxcBtnaAa2GQDwZibYFbWPXGi7BPpPdYLZwyetpsJOQ/edit?usp=sharing)\n\n  \u003c/details\u003e\n\nThis tool is developed with in the [MariData](https://www.maridata.org) project.\n\n## Requirements\n\nMariDataHarvest requires **python 3** and **pip** to run. You can install all python requirements with the following command:\n\n```sh\npip install -r requirements.txt\n```\n\nFor a detailed list, see section License below.\n\n## Usage\n\nThe script requires accounts for the following webservices:\n\n- [CMEMS](https://resources.marine.copernicus.eu/?option=com_csw\u0026task=results?option=com_csw\u0026view=account)\n- [RDA](https://rda.ucar.edu/index.html?hash=data_user\u0026action=register)\n\nThe credentials of these services MUST be entered into a file called `.env.secret` as outlined here:\n\n   ```sh\n   UN_CMEMS=\n   PW_CMEMS=\n   UN_RDA=\n   PW_RDA=\n   ```\n\nStart harvesting with the following command:\n\n```sh\npython main.py --year=2019 --minutes=30 --dir=C:\\..\n```\n\n- `year`: the year(s) to download AIS-data. Expected input a year 'YYYY' , a range of years 'YYYY-YYYY' or multiple years 'YYYY,YYYY,YYYY'.\n\n- `minutes`: is the sub-sampling interval in minutes.\n\n- `dir`: the absolute path where to keep data. If empty, the directory is same as the project directory.\n\n- **optional** arguments:\n\n  - `step`: starts the script at a specific step:\n\n    1. Download,\n    2. Subsample,\n    3. Appending weather data.\n\n    If `step` equals `0` (default value), the script runs all steps starting from step 1.\n\n  - `clear`: clears files of `year` ONLY after step 2 is done.\n\n  - `depth_first`: runs all steps for each file, which automatically deactivates `step` argument.\n\n## Development\n\nStart the EnvDataAPI services locally for testing using the following command in the `EndDataServer` directory:\n\n```shell\nexport PYTHONPATH=\"$PYTHONPATH:.:EnvironmentalData:utilities\"\npython ./EnvDataServer/app.py\n```\n\n## Docker\n\nYou can use the [Dockerfile](./Dockerfile) to build a docker image and run the script in its own isolated environment. It is recommended to provide a volume to persist the data between each run. You can specify all arguments including the optional ones as environment variables when creating/starting the container as outlined in the following. The labels used are following the [Image And Container Label Specification](https://wiki.52north.org/Documentation/ImageAndContainerLabelSpecification) of 52°North.\n\n1. Build:\n\n   ```sh\n   docker build \\\n      --build-arg GIT_COMMIT=$(git rev-parse -q --verify HEAD) \\\n      --build-arg BUILD_DATE=$(date -u +\"%Y-%m-%dT%H:%M:%SZ\") \\\n      --file Dockerfile.harvester \\\n      -t 52north/mari-data_harvester:1.0.0 .\n   ```\n\n   Ensure, that the version of the image tag is matching the version in the Dockerfile, here: `1.0.0`.\n\n1. Create named volume:\n\n   ```sh\n   docker volume create \\\n      --label org.52north.contact=dev-opser+mari-data_harvester@example.org \\\n      --label org.52north.context=\"MariData Project: Data Harvesting Script\" \\\n      --label org.52north.end-of-life=$(date -d \"+365 days\" -u +\"%Y-%m-%dT%H:%M:%SZ\") \\\n      mari-data-harvester_data\n   ```\n\n1. Run:\n\n   ```sh\n   docker run \\\n      --label org.52north.contact=dev-opser+mari-data_harvester@example.org \\\n      --label org.52north.context=\"MariData Project: Data Harvesting Script\" \\\n      --label org.52north.end-of-life=$(date -d \"+365 days\" -u +\"%Y-%m-%dT%H:%M:%SZ\") \\\n      --label org.52north.created='$(date -u +\"%Y-%m-%dT%H:%M:%SZ\")' \\\n      --volume mari-data-harvester_data:/mari-data/data \\\n      --volume .env.secret:/mari-data/EnvironmentalData/.env.secret:ro \\\n      --env-file docker.env \\\n      --name=mari-data_harvester \\\n      --detach \\\n      52north/mari-data_harvester:1.0.0 \\\n      \u0026\u0026 docker logs --follow mari-data_harvester\n   ```\n\n   with `docker.env` containing the following information:\n\n   ```sh\n   YEAR=2015-2021\n   MINUTES=5\n   DATA_DIR=/mari-data/data\n   STEP=0\n   DEPTH_FIRST=--depth-first\n   CLEAR=--clear\n   ```\n\n## Deployment\n\nUse the following command to send the code to any server for building the image (or clone this repository using `git clone...`) and run it:\n\n```sh\nrsync --recursive --verbose --times --rsh ssh \\\n   --exclude='AIS-DATA' --exclude='*.tmp' \\\n   --exclude='*.swp' --exclude='.vscode' \\\n   --exclude='__pycache__' --delete . \\\n   mari-data-harvester.example.org:/home/user/mari-data-harvester\n```\n\n### Web Access to the Data\n\nWe are using an nginx container to provide web access to the generated data. It requires an external service to maintain the ssl certificates. The according data volume is mounted read-only and externally provided, hence docker-compose does not create it with a project prefix.\n\nJust execute the following command in root folder of the repository to start the service:\n\n```sh\ndocker-compose up -d --build \u0026\u0026 docker-compose logs --follow\n```\n\nThe data is available directly at the server root via https. All requests to http are redirected to https by default.\n\n## Contact\n\n- [Demmich, Katharina](https://github.com/kdemmich)\n- [Jürrens, Eike Hinderk](https://github.com/EHJ-52n)\n- [Pontius, Martin](https://github.com/MartinPontius)\n\n## License\n\nThis application is licensed under the GPLv2 (see [LICENSE](./LICENSE)).\n\n| Name               | Version     | License                                       |\n|--------------------|-------------|-----------------------------------------------|\n| Deprecated         | 1.2.13      | MIT License                                   |\n| Flask              | 1.1.4       | BSD License                                   |\n| Flask-Limiter      | 1.5         | MIT License                                   |\n| Jinja2             | 2.11.3      | BSD License                                   |\n| MarkupSafe         | 1.1.1       | BSD License                                   |\n| Paste              | 3.5.2       | MIT License                                   |\n| PyYAML             | 5.4.1       | MIT License                                   |\n| Werkzeug           | 1.0.1       | BSD License                                   |\n| beautifulsoup4     | 4.11.1      | MIT License                                   |\n| certifi            | 2022.12.7   | Mozilla Public License 2.0 (MPL 2.0)          |\n| cftime             | 1.6.2       | MIT License                                   |\n| charset-normalizer | 3.0.1       | MIT License                                   |\n| click              | 7.1.2       | BSD License                                   |\n| idna               | 3.4         | BSD License                                   |\n| itsdangerous       | 1.1.0       | BSD License                                   |\n| joblib             | 1.2.0       | BSD License                                   |\n| limits             | 3.2.0       | MIT License                                   |\n| motuclient         | 1.8.8       | GNU Lesser General Public License v3 (LGPLv3) |\n| netCDF4            | 1.6.2       | MIT License                                   |\n| numpy              | 1.24.1      | BSD License                                   |\n| packaging          | 23.0        | Apache Software License; BSD License          |\n| pandas             | 1.5.3       | BSD License                                   |\n| protobuf           | 4.21.12     | 3-Clause BSD License                          |\n| python-dateutil    | 2.8.2       | Apache Software License; BSD License          |\n| python-dotenv      | 0.17.0      | BSD License                                   |\n| pytz               | 2021.1      | MIT License                                   |\n| requests           | 2.28.2      | Apache Software License                       |\n| scikit-learn       | 0.24.0      | new BSD                                       |\n| scipy              | 1.10.0      | BSD License                                   |\n| siphon             | 0.9         | BSD License                                   |\n| six                | 1.16.0      | MIT License                                   |\n| soupsieve          | 2.3.2.post1 | MIT License                                   |\n| threadpoolctl      | 3.1.0       | BSD License                                   |\n| typing_extensions  | 4.4.0       | Python Software Foundation License            |\n| urllib3            | 1.26.14     | MIT License                                   |\n| waitress           | 2.1.2       | Zope Public License                           |\n| wrapt              | 1.14.1      | BSD License                                   |\n| xarray             | 0.17.0      | Apache Software License                       |\n\n\u003cdetails\u003e\n\u003csummary\u003egenerate license list\u003c/summary\u003e\n\n```sh\ndocker compose run --interactive --rm api /bin/bash \\\n   -c \"pip install --no-warn-script-location --no-cache-dir pip-licenses \u003e /dev/null \u0026\u0026 .local/bin/pip-licenses -f markdown\"\n```\n\n\u003c/details\u003e\n\n## Funding\n\n| Project/Logo | Description |\n| :-------------: | :------------- |\n| [\u003cimg alt=\"MariData\" align=\"middle\" width=\"267\" height=\"50\" src=\"https://52north.org/delivery/MariData/img/maridata_logo.png\"/\u003e](https://www.maridata.org/) | MariGeoRoute is funded by the German Federal Ministry of Economic Affairs and Energy (BMWi)[\u003cimg alt=\"BMWi\" align=\"middle\" width=\"144\" height=\"72\" src=\"https://52north.org/delivery/MariData/img/bmwi_logo_en.png\" style=\"float:right\"/\u003e](https://www.bmvi.de/) |\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2F52north%2Fmaridataharvest","html_url":"https://awesome.ecosyste.ms/projects/github.com%2F52north%2Fmaridataharvest","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2F52north%2Fmaridataharvest/lists"}