{"id":19402728,"url":"https://github.com/cscfi/metadata-submitter","last_synced_at":"2025-04-24T07:32:24.676Z","repository":{"id":36950638,"uuid":"259599504","full_name":"CSCfi/metadata-submitter","owner":"CSCfi","description":"Metadata Submission Interface for SDA","archived":false,"fork":false,"pushed_at":"2024-04-12T06:56:54.000Z","size":3392,"stargazers_count":3,"open_issues_count":30,"forks_count":2,"subscribers_count":5,"default_branch":"main","last_synced_at":"2024-04-12T14:47:29.581Z","etag":null,"topics":["api","metadata-management","python3"],"latest_commit_sha":null,"homepage":"https://metadata-submitter.rtfd.io","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/CSCfi.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2020-04-28T10:04:53.000Z","updated_at":"2024-04-15T06:18:02.796Z","dependencies_parsed_at":"2024-02-22T10:39:16.771Z","dependency_job_id":"71299bfa-4d58-4b2a-9b9d-1af762c5e5d5","html_url":"https://github.com/CSCfi/metadata-submitter","commit_stats":null,"previous_names":[],"tags_count":19,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CSCfi%2Fmetadata-submitter","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CSCfi%2Fmetadata-submitter/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CSCfi%2Fmetadata-submitter/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CSCfi%2Fmetadata-submitter/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/CSCfi","download_url":"https://codeload.github.com/CSCfi/metadata-submitter/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":223945374,"owners_count":17229617,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["api","metadata-management","python3"],"created_at":"2024-11-10T11:25:30.301Z","updated_at":"2025-04-24T07:32:24.669Z","avatar_url":"https://github.com/CSCfi.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# SD Submit API\n\n![Python Unit Tests](https://github.com/CSCfi/metadata-submitter/workflows/Python%20Unit%20Tests/badge.svg)\n![Integration Tests](https://github.com/CSCfi/metadata-submitter/workflows/Integration%20Tests/badge.svg)\n![Documentation Checks](https://github.com/CSCfi/metadata-submitter/workflows/Documentation%20Checks/badge.svg)\n![Python style check](https://github.com/CSCfi/metadata-submitter/workflows/Python%20style%20check/badge.svg)\n[![Coverage Status](https://coveralls.io/repos/github/CSCfi/metadata-submitter/badge.svg?branch=main)](https://coveralls.io/github/CSCfi/metadata-submitter?branch=main)\n[![pre-commit](https://img.shields.io/badge/pre--commit-enabled-brightgreen?logo=pre-commit\u0026logoColor=white)](https://github.com/pre-commit/pre-commit)\n[![linting: pylint](https://img.shields.io/badge/linting-pylint-yellowgreen)](https://github.com/PyCQA/pylint)\n\nMetadata submission API, which handles programmatic submissions of EGA metadata, Bigpicture metadata and SDSX (generic) metadata models. Metadata can be submitted either via XML files or via web form submissions. The submitted and processed metadata as well as other user and project data is stored in a MongoDB instance as queryable JSON documents.\n\nGraphical UI implementation for web form submissions is implemented separately here: [metadata-submitter-frontend](https://github.com/CSCfi/metadata-submitter-frontend).\n\nSD Submit API also communicates with the following external services via their respective API:\n- SD Connect ([source code](https://github.com/CSCfi/swift-browser-ui))\n- Imaging Beacon ([source code](https://github.com/CSCfi/imaging-beacon))\n- NeIC Sensitive Data Archive ([docs](https://neic-sda.readthedocs.io/en/latest/))\n- REMS ([source code](https://github.com/CSCfi/rems))\n- Metax ([docs](https://metax.fairdata.fi/docs/))\n- DataCite ([docs](https://support.datacite.org/))\n  - Additionally a separate PID microservice for DOI handling\n\n```mermaid\nflowchart LR\n    SD-Connect(SD Connect) --\u003e|Information about files| SD-Submit[SD Submit API]\n    SD-Submit --\u003e|Bigpicture metadata| Bigpicture-Discovery(Imaging Beacon)\n    SD-Submit \u003c--\u003e|Ingestion pipeline actions| NEIC-SDA(NEIC SDA)\n    REMS --\u003e|Workflows/Licenses/Organizations| SD-Submit --\u003e|Resources/Catalogue items| REMS(REMS)\n    SD-Submit --\u003e|EGA/SDSX metadata| Metax(Metax API)\n    Metax --\u003e Fairdata-Etsin(FairData Etsin)\n    SD-Submit \u003c--\u003e|DOI for Bigpicture| DataCite(DataCite)\n    SD-Submit \u003c--\u003e|DOI for EGA/SDSX| PID(PID) \u003c--\u003e DataCite\n```\n\n## 💻 Development\n\n\u003cdetails\u003e\u003csummary\u003eClick to expand\u003c/summary\u003e\n\n### Prerequisites\n\n- `Python 3.12+`\n- `Docker`\n- [`Git LFS`](https://git-lfs.com/)\n\n\u003e **Note:** Git LFS is not necessarily required to be installed but the file affected by Git LFS needs to be generated via the following command otherwise:\n```bash\n$ scripts/taxonomy/generate_name_taxonomy.sh\n```\n\n### Simple local testing environment\n\nTo get started, the quickest way to setup the API in a local environment is to do the following in a terminal:\n\n- clone the repository with `git clone`\n- go to the resulting directory: `cd metadata-submitter`\n- copy the contents of .env.example file to .env file: `cp .env.example .env`\n- launch both server and database with Docker by running: `docker compose up --build` (add `-d` flag to the command to run containers in the background).\n\nServer can then be found from `http://localhost:5430`.\n\n\u003e **If you are developing on macOS,** you will also need to reconfigure the `database` service in `docker-compose.yml` file to the following:\n\n```yml\n  database:\n    image: \"arm64v8/mongo\"\n    platform: linux/arm64/v8\n    ...\n```\n\n\u003e **If you also need to initiate the graphical UI for developing the API**, check out [metadata-submitter-frontend](https://github.com/CSCfi/metadata-submitter-frontend/) repository and follow its development instructions. You will then also need to set the `REDIRECT_URL` environment variable to the UI address (e.g. add `REDIRECT_URL=http://localhost:3000` into the `.env` file) and relaunch the development environment as specified above.\n\nAlternatively, there is a more convenient method for developing the SD Submit API via a _**Python virtual environment using a Procfile**_, which is described here below.\n\n### Developing with Python virtual environment\n\nFirst, install Python dependencies with `pip` and other development tools:\n\n```bash\n# Optional: create virtual Python environment\n$ python3 -m venv venv --prompt submitter\n$ source venv/bin/activate  # Activates virtual environment\n\n$ pip install -U pip\n$ pip install -Ue .\n$ pip install -r requirements-dev.txt\n\n# Optional: install pre-commit hooks\n$ pre-commit install\n\n# Optional: update references for metax integration\n$ scripts/metax_mappings/fetch_refs.sh\n\n# Optional: update taxonomy names for taxonomy search endpoint\n# However, this is a NECESSARY step if you have not installed Git LFS\n$ scripts/taxonomy/generate_name_taxonomy.sh\n```\n\nThen copy `.env` file and set up the environment variables.\nThe example file has hostnames for development with Docker network (via `docker compose`). You will have to change the hostnames to `localhost`.\n\n```bash\n$ cp .env.example .env  # Make any changes you need to the file\n```\n\nFinally, start the servers with code reloading enabled, so any code changes restarts the servers automatically.\n\n```bash\n$ honcho start\n```\n\nThe development server should now be accessible at `localhost:5430`.\nIf it doesn't work right away, check your settings in `.env` and restart the servers manually if you make changes to `.env` file.\n\n\u003e **Note:** This approach uses Docker to run MongoDB. You can comment it out in the `Procfile` if you don't want to use Docker.\n\n### OpenAPI Specification docs with Swagger\n\nSwagger UI for viewing the API specs is already available in the production docker image. During development, you can enable it by executing: `bash scripts/swagger/generate.sh`.\n\nRestart the server, and the swagger docs will be available at http://localhost:5430/swagger.\n\n**Swagger docs requirements:**\n- `bash`\n- `Python 3.12+`\n- `PyYaml` (installed via the development dependencies)\n- `realpath` (default Linux terminal command)\n\n### Keeping Python requirements up to date\n\nThe project Python package dependencies are automatically being kept up to date with [renovatebot](https://github.com/renovatebot/renovate). However, if there is ever a need to update the package requirements manually, you can do the following:\n\n1. Install `pip-tools`:\n    * `pip install pip-tools`\n    * if using docker compose pip-tools are installed automatically\n\n2. Add new package names to `requirements.in` or `requirements-dev.in`\n    * `requirements.in` file is reserved for all dependencies necessary to the production deployment\n    * `requirements-dev.in` file may also include all packages required for development purposes\n\n3. Update `.txt` file for the changed requirements file:\n    * `pip-compile requirements.in`\n    * `pip-compile requirements-dev.in`\n\n4. If you want to update all dependencies to their newest versions, run:\n    * `pip-compile --upgrade requirements.in`\n    * `pip-compile --upgrade requirements-dev.in`\n\n5. To install Python requirements run:\n    * `pip-sync requirements.txt`\n    * `pip-sync requirements-dev.txt`\n\n\u003c/details\u003e\n\n## 🛠️ Contributing\n\n\u003cdetails\u003e\u003csummary\u003eClick to expand\u003c/summary\u003e\n\nDevelopment team members should check internal [contributing guidelines for Gitlab](https://gitlab.ci.csc.fi/groups/sds-dev/-/wikis/Guides/Contributing).\n\nIf you are not part of CSC and our development team, your help is nevertheless very welcome. Please see [contributing guidelines for Github](CONTRIBUTING.md).\n\n\u003c/details\u003e\n\n## 🧪 Testing\n\n\u003cdetails\u003e\u003csummary\u003eClick to expand\u003c/summary\u003e\n\nMajority of the automated tests (such as unit tests, code style checks etc.) can be run with [`tox`](https://tox.wiki/en/4.24.2/) automation. Integration tests are run separately with [`pytest`](https://docs.pytest.org/en/stable/) as they require the full test environment to be running with a local database instance and all the mocked versions of related external services.\n\nBelow are minimal instructions for executing the automated tests of this project locally. Run the below commands in the project root:\n\n```bash\n# Optional: set up virtual python env\npython3 -m venv venv --prompt submitter\nsource venv/bin/activate\n\n# Install python dependencies\npip install -U pip\npip install -r requirements-dev.txt\n\n# Unit tests, linting, etc.\ntox -p auto\n\n# Integration tests\ndocker compose --env-file .env.example up --build -d\npytest tests/integration\n```\n\nAdditionally, we use pre-commit hooks in the CI/CD pipeline for automated tests in every merge/pull request. The pre-commit hooks include some extra tests such as spellchecking so installing pre-commit hooks locally (with `pre-commit install`) is also useful.\n\n\u003c/details\u003e\n\n## 🚀 Deployment\n\n\u003cdetails\u003e\u003csummary\u003eClick to expand\u003c/summary\u003e\n\nProduction version can be built and run with following docker commands:\n```bash\n$ docker build --no-cache -f dockerfiles/Dockerfile -t cscfi/metadata-submitter .\n$ docker run -p 5430:5430 cscfi/metadata-submitter\n```\n\nThe [frontend](https://github.com/CSCfi/metadata-submitter-frontend) is built and added as static files to the backend deployment with this method.\n\n\u003e Helm charts for a kubernetes cluster deployment will also be available soon™️.\n\n\u003c/details\u003e\n\n## 📜 License\n\n\u003cdetails\u003e\u003csummary\u003eClick to expand\u003c/summary\u003e\n\nMetadata submission interface is released under `MIT`, see [LICENSE](LICENSE).\n\n\u003c/details\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcscfi%2Fmetadata-submitter","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcscfi%2Fmetadata-submitter","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcscfi%2Fmetadata-submitter/lists"}