{"id":30019818,"url":"https://github.com/veupathdb/service-user-dataset-import","last_synced_at":"2025-08-23T10:36:08.128Z","repository":{"id":40420432,"uuid":"263965115","full_name":"VEuPathDB/service-user-dataset-import","owner":"VEuPathDB","description":null,"archived":true,"fork":false,"pushed_at":"2024-03-15T10:59:31.000Z","size":762,"stargazers_count":0,"open_issues_count":11,"forks_count":0,"subscribers_count":25,"default_branch":"master","last_synced_at":"2025-08-06T01:32:00.775Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/VEuPathDB.png","metadata":{"files":{"readme":"readme.adoc","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2020-05-14T16:17:06.000Z","updated_at":"2025-02-11T15:00:44.000Z","dependencies_parsed_at":"2024-03-07T20:02:46.896Z","dependency_job_id":null,"html_url":"https://github.com/VEuPathDB/service-user-dataset-import","commit_stats":null,"previous_names":[],"tags_count":26,"template":false,"template_full_name":"VEuPathDB/example-jaxrs-container-service","purl":"pkg:github/VEuPathDB/service-user-dataset-import","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/VEuPathDB%2Fservice-user-dataset-import","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/VEuPathDB%2Fservice-user-dataset-import/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/VEuPathDB%2Fservice-user-dataset-import/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/VEuPathDB%2Fservice-user-dataset-import/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/VEuPathDB","download_url":"https://codeload.github.com/VEuPathDB/service-user-dataset-import/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/VEuPathDB%2Fservice-user-dataset-import/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":271746660,"owners_count":24813575,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-23T02:00:09.327Z","response_time":69,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-08-06T01:21:37.911Z","updated_at":"2025-08-23T10:36:08.122Z","avatar_url":"https://github.com/VEuPathDB.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"= User Dataset Import Service\n:toc:\n:icons: font\n\nhttps://veupathdb.github.io/service-user-dataset-import/api.html[API Docs]\n\nHTTP service that allows users to upload datasets directly to a site,\nskipping Galaxy entirely.\n\n== Adding a New Handler\n\nNew handlers can be added to the udis stack by editing the\nhttps://github.com/VEuPathDB/service-user-dataset-import/blob/master/docker-compose.yml[docker-compose.yml]\nfile and the\nhttps://github.com/VEuPathDB/service-user-dataset-import/blob/master/config.json[config.json]\nfile with the new handler configuration.\n\nThe `docker-compose.yml` file puts the new image on the internal docker network\nso the import service can access it, and the `config.json` file tells the import\nservice how to access it.\n\n=== Prerequisites\n\nBefore a new handler can be added, it must first be registered in Jenkins and\nhave a built image published to the https://hub.docker.com/u/veupathdb[VEuPathDB Dockerhub].\n\nFor more on implementing a handler, see this https://github.com/VEuPathDB/example-dataset-import-handler[import handler template].\n\n=== `docker-compose.yml`\n\nTo add the new handler to the docker-compose file, copy/paste one of the\nexisting handler entries and edit it to suit your new handler.\n\n.Original Config\n[source, yml]\n----\n  biom-handler: \u003c1\u003e\n    image: veupathdb/user-dataset-handler-biom:${BIOME_HANDLER_TAG:-latest} \u003c2\u003e\n    networks:\n      - internal\n      - monitoring-ext\n    labels:\n      - \"com.centurylinklabs.watchtower.enable=${BIOME_HANDLER_WATCHTOWER:-false}\" \u003c3\u003e\n      - \"traefik.enable=false\"\n      - \"prometheus-scrape.enabled=true\"\n----\n\u003c1\u003e The internal network name of the handler.  This value must be unique and contain only URL safe characters.\n\u003c2\u003e The docker image name and version env-var\n\u003c3\u003e The watchtower name and version-env-var\n\nTo create a new handler in the stack, edit your copy like so:\n\n.Edited Config\n[source, yml]\n----\n  biom-handler:\n    image: veupathdb/user-dataset-handler-biom:${BIOME_HANDLER_TAG:-latest}\n    networks:\n      - internal\n      - monitoring-ext\n    labels:\n      - \"com.centurylinklabs.watchtower.enable=${BIOME_HANDLER_WATCHTOWER:-false}\"\n      - \"traefik.enable=false\"\n      - \"prometheus-scrape.enabled=true\"\n\n  my-new-handler: \u003c1\u003e\n    image: veupathdb/user-dataset-handler-my-new-handler:${MY_NEW_HANDLER_TAG:-latest} \u003c2\u003e\n    networks:\n      - internal\n      - monitoring-ext\n    labels:\n      - \"com.centurylinklabs.watchtower.enable=${MY_NEW_HANDLER_WATCHTOWER:-false}\" \u003c3\u003e\n      - \"traefik.enable=false\"\n      - \"prometheus-scrape.enabled=true\"\n----\n\u003c1\u003e New handler network name\n\u003c2\u003e Updated docker-image name and version env-var.\n\u003c3\u003e Updated watchtower env-var.\n\nAfter this change is made and committed, Ops will need to be notified of the\nchange so they can update the deployment configuration on their end.\n\n\n=== `config.json`\n\nSimilar to the docker-compose file, the easiest way to add a new service\nconfiguration is to copy an existing one and append it to the `services` array.\n\n.Original Config\n[source, json]\n----\n{\n  \"services\": [\n    {\n      \"dsType\": \"biom\", \u003c1\u003e\n      \"projects\": [\n        \"MicrobiomeDB\" \u003c2\u003e\n      ],\n      \"fileTypes\": [\n        \"biom\" \u003c3\u003e\n      ],\n      \"name\": \"biom-handler\" \u003c4\u003e\n    }\n  ]\n}\n----\n\u003c1\u003e Value that will be sent by the client to select your handler for use with\n    the upload.\n\u003c2\u003e List of projects your handler is allowed to be used from.\n\u003c3\u003e File types/extensions allowed to be handled by your handler.  In addition to\n    the file types listed here, zip and tar files are also permitted by default.\n\u003c4\u003e The network name for your handler as configured in the `docker-compose.yml`\n    file.\n\n.Edited Config\n[source, json]\n----\n{\n  \"services\": [\n    {\n      \"dsType\": \"biom\",\n      \"projects\": [\n        \"MicrobiomeDB\"\n      ],\n      \"fileTypes\": [\n        \"biom\"\n      ],\n      \"name\": \"biom-handler\"\n    },\n    {\n      \"dsType\": \"mine\",\n      \"projects\": [\n        \"PlasmoDB\",\n        \"FungiDB\"\n      ],\n      \"fileTypes\": [\n        \"myfile\"\n      ],\n      \"name\": \"my-new-handler\"\n    }\n  ]\n}\n----\n\n== Development\n\n=== Code Generation\n\nWARNING: Code generation is intentionally disabled in this project due to issues with\nthe RAML code generation creating incorrect controller methods for handling\n`multipart/form-data` inputs.\n\n\n=== In this repo\n\nFor base contents and explanations see the\nhttps://github.com/VEuPathDB/example-jaxrs-container-service[template project].\n\n[cols=\"2,1,5\"]\n|===\n| docker-compose.yml\n| Docker\n| Configuration file needed to spin up the full service stack for development\npurposes. +\nFor additional info see\nhttps://github.com/VEuPathDB/service-user-dataset-import/blob/master/stack-readme.adoc[stack-readme.adoc]\n| Dockerfile\n| Docker\n| The docker config specifically for the service-user-dataset-import container.\n| pgDockerfile\n| Docker\n| The docker config for the service's backing datastore.\n| init.sql\n| Postgres\n| The initialization script for the service's backing Postgres datastore.\n|===\n\n=== Running Locally\n\n==== Prerequisites\n\nTo bring up the eda project via docker-compose, you’ll need a few things.\n\n* a functioning docker setup and docker-compose (https://www.docker.com/products/docker-desktop)\n\n* a functioning traefik setup. clone the https://github.com/VEuPathDB/docker-traefik[VEuPathDB docker traefik repo], and run docker-compose up -d in the clone, or follow the instructions in that repo. This will run traefik locally, which handles the container routing.\n\n* a functioning sshuttle setup or see \u003c\u003c_using_the_ssh_forwarder\u003e\u003e for an alternative approach.\n\nOnce the service is brought up using `docker-compose`, the endpoints can be accessed at the host name specified in the `TRAEFIK_HOST` environment variable (default configuration is https://udis-dev.local.apidb.org:8443).\n\n==== Using the SSH forwarder\n\nThe docker-compose-forwarder.yml file defines an additional set of containers and config that will handle forwarding local db connections. This eliminates the use of sshuttle, and some of the quirks related to sshuttle.\nhow to use the forwarder containers\n\nTo bring up the stack this way:\n\n* ensure you are running an ssh-agent\n* ensure that your .env file contains the values from env-sample (or use env-sample as a template)\n* run docker-compose -f docker-compose.yml -f docker-compose-forwarder.yml up -d.\n\nhow the forwarding containers work:\n\nThere are 4 different forwarding containers:\n\n* ldapforward - responsible for forwarding ldap connections to a remote ldap server\n* dbforward1 - responsible for forwarding oracle connections to remote database server\n* dbforward2 - responsible for forwarding oracle connections to remote database server\n* irodsforward - responsible for forwarding irods connections to remote irods server\n\nThe forwarding containers map your ssh-agent socket into the container, which allows it to authenticate via ssh to our servers. It then opens an ssh connection using the vars in .env, and forwards ports to internal servers. It exposes these ports itself, so other containers in the stack can connect to it, and it mocks the hostname of the remote server as well. The ssh command is set as the containers entrypoint, which looks like this:\n\n----\n    entrypoint:\n      - \"ssh\"\n      - \"-tNn\"\n      - \"-p\"\n      - \"${DBFORWARD_SSH_PORT}\"\n      - \"-L\"\n      - \"0.0.0.0:389:${LDAPFORWARD_HOST}:389\"\n      - \"${DBFORWARD_USER}@${DBFORWARD_HOST}\"\n----\n\nThe companion to the port forwarding above, is setting the hostname as an alias to the remote server:\n\n----\n    networks:\n      internal:\n        aliases:\n          - ${LDAPFORWARD_HOST}\n      external:\n----\n\nThe mapping of your local ssh-agent socket into the container is done in the following bind volume:\n\n----\n    volumes:\n      - type: bind\n        source: /run/host-services/ssh-auth.sock\n        target: /ssh-agent\n----\n\nThis instructs docker to resolve the `${LDAPFORWARD_HOST}` name to the ldapforward container’s ip. The port forwarding then routes through ssh to the remote ldap server.\n\nThe other forwarding containers work exactly the same way, but they forward connections on the port appropriate for their service. You can copy/paste to make a dbforward3 if you like, changing the appropriate vars to the new number - but two are included because it is unlikely you’d need more.\n\n==== Local Overrides\nIn order to run the service, overriding the latest images with your locally built changes, you can make use of the `docker-compose-build-*.yml` files. If you are testing updates to an individual handler, you can include the relevant `docker-compose-build-*.yml` file in addition to the main `docker-compose.yml` file when running docker-compose:\n\n[source, shell script]\n----\ndocker-compose -f docker-compose-build-gene-list.yml -f docker-compose.yml\n----\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fveupathdb%2Fservice-user-dataset-import","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fveupathdb%2Fservice-user-dataset-import","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fveupathdb%2Fservice-user-dataset-import/lists"}