{"id":30019790,"url":"https://github.com/veupathdb/util-user-dataset-handler-server","last_synced_at":"2025-10-14T21:39:52.060Z","repository":{"id":41829068,"uuid":"260596352","full_name":"VEuPathDB/util-user-dataset-handler-server","owner":"VEuPathDB","description":null,"archived":false,"fork":false,"pushed_at":"2023-04-07T13:29:10.000Z","size":742,"stargazers_count":0,"open_issues_count":3,"forks_count":1,"subscribers_count":20,"default_branch":"master","last_synced_at":"2025-08-06T09:50:40.870Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/VEuPathDB.png","metadata":{"files":{"readme":"readme.adoc","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-05-02T02:05:19.000Z","updated_at":"2022-03-15T16:47:06.000Z","dependencies_parsed_at":"2024-06-20T01:46:07.408Z","dependency_job_id":"774cf4a1-989f-47d2-8a51-533c13dff777","html_url":"https://github.com/VEuPathDB/util-user-dataset-handler-server","commit_stats":null,"previous_names":["veupathdb/util-exporter-server"],"tags_count":29,"template":false,"template_full_name":null,"purl":"pkg:github/VEuPathDB/util-user-dataset-handler-server","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/VEuPathDB%2Futil-user-dataset-handler-server","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/VEuPathDB%2Futil-user-dataset-handler-server/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/VEuPathDB%2Futil-user-dataset-handler-server/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/VEuPathDB%2Futil-user-dataset-handler-server/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/VEuPathDB","download_url":"https://codeload.github.com/VEuPathDB/util-user-dataset-handler-server/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/VEuPathDB%2Futil-user-dataset-handler-server/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":279021376,"owners_count":26087023,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-14T02:00:06.444Z","response_time":60,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-08-06T01:21:27.980Z","updated_at":"2025-10-14T21:39:52.054Z","avatar_url":"https://github.com/VEuPathDB.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"= Exporter Script Server\n// General Doc Settings\n:toc: left\n:source-highlighter: highlightjs\n:icons: font\n// Github specifics\nifdef::env-github[]\n:toc: preamble\n:tip-caption: :bulb:\n:note-caption: :information_source:\n:important-caption: :heavy_exclamation_mark:\n:caution-caption: :fire:\n:warning-caption: :warning:\nendif::[]\nElizabeth Paige Harper \u003cepharper@upenn.edu\u003e\nv1.0.0\n\n// Custom Config\n:repo-url: https://github.com/VEuPathDB/util-user-dataset-handler-server\n:site-url: https://veupathdb.github.io/util-user-dataset-handler-server\n:repo-file-base: {repo-url}/blob/master\n\nimage:https://www.travis-ci.org/VEuPathDB/util-user-dataset-handler-server.svg?branch=master[\"Build Status\", link=\"https://www.travis-ci.org/VEuPathDB/util-user-dataset-handler-server\"]\nimage:https://goreportcard.com/badge/github.com/VEuPathDB/util-user-dataset-handler-server[\"Go Report Card\", link=\"https://goreportcard.com/report/github.com/VEuPathDB/util-user-dataset-handler-server\"]\nimage:https://img.shields.io/github/v/release/VEuPathDB/util-user-dataset-handler-server[\"Latest Release\", link=\"https://github.com/VEuPathDB/util-user-dataset-handler-server/releases/latest\"]\n\n\nExposes configured exporter / validation scripts over HTTP.\n\nifdef::env-github[]\n{site-url}[Rendered Readme] |\nendif::[]\n{site-url}/api.html[API Docs]\n\n== Usage\n\n=== In Script Container\n\n==== Standard Use\n\nhttps://github.com/VEuPathDB/dataset-handler-biom[Working example project]\n\n\n==== Custom Use\n\nTo create a custom setup not based on the demo container, this server can be\nconfigured by performing the following steps.\n\nCAUTION: Currently, this server app only supports linux \u0026 mac environments.\n\n. Download the latest release from the {repo-url}/releases/latest[releases page]\n  and unpack the archive in your project directory.\n. Run the command `./service gen-config` to generate a server configuration\n  template file.  This file will be named `config.tpl.yml`.\n. Edit the configuration file with your desired service name and command\n  configurations.\n+\nNOTE: Check the {file-config-readme}[config file readme] for more\ninformation about the configuration file.\n. Once you have edited the configuration file, rename it and place it in the\n  desired relative to the service binary.\n. Validate the configuration by running the command\n  `./service check-config --config=path/to/your/config.yml`.  This command will\n  validate the configuration and report any errors.\n+\n.Bad Config Example\n[source, bash-session]\n----\n$ ./service --config=config.yml check-config\n----\n\n== Script Expectations\n\nThis server is language agnostic when it comes to calling scripts.  Anything may\nbe used as long as it can be executed via a cli call.\n\n=== Output\n\nThe script expectations are based on the Galaxy tooling.  This means scripts are\nexpected to output a `.tgz` file suitable for being directly loaded into iRODS.\n\nThe pattern for the tar file output name must match the iRODS trigger's expected\nformat of `dataset_u\\{user-id}_t\\{timestamp-int64}_p\\{process-id}.tgz`.  For\nexample: `dataset_u12345_t1647364816_p132.tgz`.\n\nThis means the tar file must contain a `meta.json`, `dataset.json`, and a\ndirectory named `datafiles` containing the actual dataset file(s).\n\nThis server will automatically pick up that file and return it to the udis\nservice to be loaded into iRODS.\n\nThe reason for this is the HTTP server which unpacks and starts the command does\nnot know what belongs or doesn't belong in the iRODS tar.\n\n=== Error handling\n\nA script can indicate a fatal error by exiting with a non-zero status code.  An\nerror message will be generated using the text printed to `stderr` by the\nscript.\n\nErrors may be plaintext, a json object, or plaintext followed by a json object.\n\nTo indicate an error is a user error and not a script failure, a json object\nerror must be returned.\n\n.Valid `stderr` output examples\n[source, shell script]\n----\n# A plaintext error\nSome error message that will be recorded as a script/server error.\n\n# Error output followed by a final fatal error.\nSome error text followed by a {\"error\": \"json\", \"message\": \"object\"}\n\n# A structured error\n{\"error\": \"user\", \"message\": \"some user error\"}\n----\n\nWhen returning both plaintext and a json object, the plaintext will be prepended\nto the error message in the json output.\n\nFor example:\n\n.Raw `stderr` output\n----\nSome error messages\nLogged while executing the script\n{\"error\": \"fatal\", \"message\": \"failed to open input file\"}\n----\n\n.Recorded error\n[source, json]\n----\n{\n  \"error\": \"fatal\",\n  \"message\": \"Some error messages\\nLogged while executing the script\\nfailed to open input file\"\n}\n----\n\n\n==== Json Object Error Format\n\n[source,json]\n----\n{\n  \"error\": \"user\", \u003c1\u003e\n  \"message\": \"some error text\" \u003c2\u003e\n}\n----\n\u003c1\u003e The `\"error\"` field is used to define the type of error.  A value of\n    `\"user\"` indicates that the error was a user error and the user import\n    service will record the error as such.  Any other value will be recorded as\n    a script error.\n\u003c2\u003e The `\"message\"` field defines the error message to record.\n\nCAUTION: Both fields are required.\n\n== Config File\n\n=== Components\n\n.config.yml\n[source, yaml, linenums]\n----\nservice-name: my service \u003c1\u003e\ncommand:\n  executable: /some/path/to/your/executable \u003c2\u003e\n  args: \u003c3\u003e\n    - \u003c\u003cinput-files\u003e\u003e \u003c4\u003e\n----\n\u003c1\u003e The display name of the running service\n\u003c2\u003e Path to an executable script or binary to be called by the server to process\n    uploaded datasets.\n\u003c3\u003e An array of arguments that will be passed to the configured executable.\n    Each array element is effectively a single, quote wrapped cli arg.\n+\nWARNING: Space separated entries in the args array _will_ be treated as a single\n         argument.\n\u003c4\u003e An \u003c\u003cVariables,Injected Variable\u003e\u003e\n\n=== Command Context\n\nThe configured command will be executed in an isolated subshell, but will be\nprovided the same environment as the server itself, meaning it is possible to\nset environment variables for the script simply by setting them on the docker\ncontainer.\n\nThe output of the script's `stdout` will be piped through the server's logging\nmechanism and will appear in the container logs.\n\nThe output of the script's `stderr` will be both piped through the server's\nlogging mechanism (like `stdout`) but will also be captured for parsing and\nreturning to the caller.\n\n=== Variables\n\n==== `+\u003c\u003ccwd\u003e\u003e+`\n\nThe working directory for the job that is running in the current request.\n\n.Example\n----\n/workspace/12345\n----\n\n==== `+\u003c\u003cdate\u003e\u003e+`\n\nThe current date formatted as `YYYY-MM-DD`.\n\n.Example\n----\n1994-02-13\n----\n\n==== `+\u003c\u003cdate-time\u003e\u003e+`\n\nThe current datetime in RFC3339 format.\n\n.Example\n----\n2018-10-31T23:37:18.013557+0500\n----\n\n==== `+\u003c\u003cds-description\u003e\u003e+`\n\nThe user provided description for the current dataset upload.\n\n.Example\n----\nMy dataset upload containing foo and bar\n----\n\n==== `+\u003c\u003cds-name\u003e\u003e+`\n\nThe user provided name for the current dataset upload.\n\n.Example\n----\nMy Dataset 3\n----\n\n==== `+\u003c\u003cds-summary\u003e\u003e+`\n\nThe user provided summary of the current dataset upload.\n\n.Example\n----\nSome summary text for my dataset upload\n----\n\n==== `+\u003c\u003cds-origin\u003e\u003e+`\n\nThe source/origin of the user dataset should be either `galaxy` or `direct-upload`.\n\n.Example\n----\ndirect-upload\n----\n\n==== `+\u003c\u003cds-user-id\u003e\u003e+`\n\nWDK user ID of the user that uploaded the dataset.\n\n.Example\n----\n123456\n----\n\n==== `+\u003c\u003cinput-files\u003e\u003e+`\n\nA space separated list of the files that were unpacked from the uploaded zip or\ntar file sorted by name ascending.\n\n===== Examples\n\n[#upload-tgz]\n.Upload Contents\n----\ndataset.tgz\n ├─ foo.txt\n ├─ bar.xml\n ├─ fizz.json\n └─ buzz/\n     └─ fazz.yml\n----\n\n.+\u003c\u003cinput-files\u003e\u003e+\n----\nbar.xml buzz fizz.json foo.txt\n----\n\n===== `+\u003c\u003cinput-files[n]\u003e\u003e+`\n\nArray style access of the input file list allowing retrieval of a single file\nname from the input file list.\n\n====== Examples\n\nThese use \u003c\u003c#upload-tgz, this example input\u003e\u003e.\n\n.+\u003c\u003cinput-files[0]\u003e\u003e+\n----\nbar.xml\n----\n\n.+\u003c\u003cinput-files[2]\u003e\u003e+\n----\nfizz.json\n----\n\n==== `+\u003c\u003ctime\u003e\u003e+`\n\nThe current time formatted as `HH:MM:SS`.\n\n.Example\n----\n03:47:58\n----\n\n==== `+\u003c\u003ctimestamp\u003e\u003e+`\n\nThe current unix timestamp in seconds.\n\n.Example\n----\n783647299\n----\n\n==== `\u003c\u003cprojects\u003e\u003e`\n\nA comma-separated list of project identifiers.\n\n.Example\n----\nVectorBase,PlasmoDB\n----\n\n==== `+\u003c\u003chandler-params.p1\u003e\u003e+`\n\nAccess of handler-specific parameter values from the request body of the import request.\n\n===== Examples\n.Request Body\n[source, json]\n----\n{\n    \"datasetName\": \"foo\",\n    \"projects\": [\n        \"PlasmoDB\"\n    ],\n    \"datasetType\": \"gene-list\",\n    \"origin\": \"direct-upload\",\n    \"formatParams\": {\n        \"genome\": \"test-genome-1\",\n        \"otherParam\": \"other-param-value\"\n    }\n}\n----\n\n.+\u003c\u003chandler_params.genome\u003e\u003e+\n----\ntest-genome-1\n----\n\n.+\u003c\u003chandler_params.otherParam\u003e\u003e+\n----\nother-param-value\n----","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fveupathdb%2Futil-user-dataset-handler-server","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fveupathdb%2Futil-user-dataset-handler-server","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fveupathdb%2Futil-user-dataset-handler-server/lists"}