{"id":22058766,"url":"https://github.com/sd2e/pipelinejobs-agave-proxy","last_synced_at":"2025-03-23T16:46:00.415Z","repository":{"id":73350154,"uuid":"161042171","full_name":"SD2E/pipelinejobs-agave-proxy","owner":"SD2E","description":"(Mirror) Provides a generalized interface to run Agave jobs in the PipelineJobs framework","archived":false,"fork":false,"pushed_at":"2019-01-31T23:16:22.000Z","size":32,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":7,"default_branch":"master","last_synced_at":"2025-01-28T22:51:16.624Z","etag":null,"topics":["agaveapi","metadata","reactor","workflow"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/SD2E.png","metadata":{"files":{"readme":"README.rst","changelog":"CHANGELOG.rst","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-12-09T13:27:25.000Z","updated_at":"2019-01-31T23:16:23.000Z","dependencies_parsed_at":null,"dependency_job_id":"6af7c21c-d733-48a4-a8f3-00407116fa4c","html_url":"https://github.com/SD2E/pipelinejobs-agave-proxy","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SD2E%2Fpipelinejobs-agave-proxy","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SD2E%2Fpipelinejobs-agave-proxy/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SD2E%2Fpipelinejobs-agave-proxy/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SD2E%2Fpipelinejobs-agave-proxy/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/SD2E","download_url":"https://codeload.github.com/SD2E/pipelinejobs-agave-proxy/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245136355,"owners_count":20566586,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["agaveapi","metadata","reactor","workflow"],"created_at":"2024-11-30T17:24:57.939Z","updated_at":"2025-03-23T16:46:00.390Z","avatar_url":"https://github.com/SD2E.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"========================\nPipelineJobs Agave Proxy\n========================\n\nThis Reactor provides a generalized proxy for running Agave API jobs such that\ntheir inputs, parameterization, and outputs are connected to (and thus\ndiscoverable from within) the Data Catalog.\n\nRegister Agave App as a Pipeline\n--------------------------------\n\nBefore an Agave App can be run by this proxy, three things must happen:\n\n#. It must be architected to fit the PipelineJobs workflow\n#. It must be public or shared with user **sd2eadm**\n#. It must be registered as a Data Catalog ``Pipeline``\n\nApp Architecture\n^^^^^^^^^^^^^^^^\n\nThe app must generate filenames that are distinguishable between runs. This is\nenforced to prevent accidentallly over-writing of files when multiple jobs\nshare an archiving destination. Furthermore, the app definition and any\ninterior runtime logic must use fully-qualified Agave files URLs to\ndefine inputs. Finally, the app's ``id`` must be unique not only in the\n**Agave Apps Catalog** (this is automatically enforced) but also in the\n**Data Catalog Pipelines** collection.\n\nShare or Publish the App\n^^^^^^^^^^^^^^^^^^^^^^^^\n*Coming soon...*\n\nRegistering a Pipeline\n^^^^^^^^^^^^^^^^^^^^^^\n*Coming soon...*\n\nLaunching a Managed Agave Job\n-----------------------------\n\nConstruct and send a message including the following components to the\n**PipelineJobs Agave Proxy** Reactor.\n\n#. An Agave job definition\n#. A metdata linkage parameter\n#. Optional control parameters\n\n.. note:: The **agave_pipelinejob** format is documented in JSONSchemas_.\n\nAgave Job Definition\n^^^^^^^^^^^^^^^^^^^^\n\nThe Agave job definition must be included as as subdocument in the message. To\nillustrate this, start with a basic Agave job definition: Here is an\nexample for an imaginary Agave app ``tacobot9000-0.1.0u1``.\n\n.. code-block:: json\n\n    {\n      \"appId\": \"tacobot9000-0.1.0u1\",\n      \"name\": \"TACObot job\",\n      \"inputs\": {\"file1\": \"agave://data.tacc.cloud/examples/tacobot/test1.txt\"},\n      \"parameters\": {\"salsa\": true, \"avocado\": false, \"cheese\": true},\n      \"maxRunTime\": \"01:00:00\"\n    }\n\nTo launch this via the Agave, this document would be sent directly to the\n``/apps`` endpoint. To send it instead to the proxy, move it to key\n``job_definition`` in a JSON document.\n\n.. code-block:: json\n\n    {\n    \t\"job_definition\": {\n    \t\t\"appId\": \"tacobot9000-0.1.0u1\",\n    \t\t\"name\": \"TACObot job\",\n    \t\t\"inputs\": {\n    \t\t\t\"file1\": \"agave://data.tacc.cloud/examples/tacobot/test1.txt\"\n    \t\t},\n    \t\t\"parameters\": {\n    \t\t\t\"salsa\": true,\n    \t\t\t\"avocado\": false,\n    \t\t\t\"cheese\": true\n    \t\t},\n    \t\t\"maxRunTime\": \"01:00:00\"\n    \t}\n    }\n\nMetadata Linkage Parameter\n^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nAn explicit linkage to objects in the Data Catalog must be established. This is\ndone via the ``parameters`` key, which must contain a valid value for one of\nthe following:\n\n- ``experiment_id``\n- ``sample_id``\n- ``measurement_id``\n\nEither single values or an array of values may be passed, and either the\nreadable text value may be provided or the corresponding UUID.\n\nWhich Parameter to Pass\n#######################\n\nA PipelineJob is always linked to a set of measurements by way of the\nlinkage parameter. The job's archive path is also determined by the linkage\nparameter. To illustrate:.\n\nIf a job's ``measurement_id=['measurement.tacc.1234',\n'measurement.tacc.2345']``, it will linked to these two measurements\nand its archive path will end with a hash of the two ``measurement_id`` values.\n\nAssuming those measurements are children of ``sample.tacc.abcde`` and the only\nlinkage parameter sent was ``sample_id='sample.tacc.abcdef'``, the job will\nstill be linked to all the child measurements of that sample. Its archive path\nwill end with a hash of ``sample.tacc.abcde``. Howver, if both measurement_id\nand sample_id are passed, the linkages are made to the specified measurement(s)\nwhile the archive path is a function of the sample_id value(s).\n\nFor experiment_id, the specific samples are linked to the job and the\narchive path is a function of experiment_id value(s).\n\nThis design allows files generated by the job to be linked to only one level\nof the metadata hiearchy, while allowing collection of outputs at higher\nlevels of organization in the file system.\n\nHere is a worked example of the current example job request, as it stands:\n\n.. code-block:: json\n\n    {\n    \t\"parameters\": {\n    \t\t\"sample_id\": \"sample.tacc.abcde\"\n    \t},\n    \t\"job_definition\": {\n    \t\t\"appId\": \"tacobot9000-0.1.0u1\",\n    \t\t\"name\": \"TACObot job\",\n    \t\t\"inputs\": {\n    \t\t\t\"file1\": \"agave://data.tacc.cloud/examples/tacobot/test1.txt\"\n    \t\t},\n    \t\t\"parameters\": {\n    \t\t\t\"salsa\": true,\n    \t\t\t\"avocado\": false,\n    \t\t\t\"cheese\": true\n    \t\t},\n    \t\t\"maxRunTime\": \"01:00:00\"\n    \t}\n    }\n\nAdditional Control Parameters\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nJob behavior can be refined with additional control parameters.\n\ninstanced\n#########\n\nEach PipelineJob has a distinct archive path derived from its Pipeline UUID,\nthe ``data`` dictionary passed at job ``init()`` and/or ``setup()``, and a\nfunction of its linkage parameters to experiments, samples, or measurements.\nTo avoid inadvertent over-writes, the archive path is extended with an\n*instancing directory* named in the form ``adjective-animal-YYYYMMDDTHHmmssZ``.\nTo avoid use of the instancing directory, include ``instanced: false`` in the\njob request message.\n\nExample: ``\"instanced\": false``\n\nindex_patterns\n##############\n\nThe default behavior of the PipelineJobs System is to index every file found\nunder a job's archive path to be linked to that specific job. To subselect only\nspecific files, it is possible to include one or more Python regular\nexpressions in ``index_patterns``. Only files matching these patterns will be\nlinked to the job.\n\nExample: ``\"index_patterns\": []``\n\nprocessing_level\n################\n\nThe default behavior of the PipelineJobs System is to index files under a job's\narchive path as processing level \"1\". To change this, an alternative\n``processing_level`` may be passed in the job request message.\n\nExample: ``\"processing_level\": \"2\"``\n\n.. note:: Only one automatic indexing configuration can be active for a given\n          job. Additional indexing actions with other configurations may be\n          initiated by sending a message directly to **PipelineJobs Indexer**\n\nJob Life Cycle\n--------------\n\nHere is complete record from the Pipelines system showing how the information\nfrom job creation and subsequent events is stored and discoverable. A few key\nhighlights:\n\n* The top-level ``data`` field holds the original parameterization of the job\n* Three events are noted in the ``history``: create, run, finish\n* The actor and execution for the managing instance of **PipelineJobs Agave Proxy** are available under ``agent`` and ``task``, respectively\n\n.. literalinclude:: jobdocument.json\n   :language: json\n   :linenos:\n\n.. _JSONSchemas:\n\nJSON Schemas\n------------\n\n.. literalinclude:: schemas/agave2.jsonschema\n   :language: json\n   :linenos:\n   :caption: agave_pipelinejob\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsd2e%2Fpipelinejobs-agave-proxy","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsd2e%2Fpipelinejobs-agave-proxy","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsd2e%2Fpipelinejobs-agave-proxy/lists"}