{"id":20672193,"url":"https://github.com/dgruber/ubercluster","last_synced_at":"2025-04-19T19:04:25.205Z","repository":{"id":24281424,"uuid":"27676110","full_name":"dgruber/ubercluster","owner":"dgruber","description":"Simple multi-clustering tool for DRMAA2 enabled cluster schedulers and more...","archived":false,"fork":false,"pushed_at":"2018-02-11T21:26:42.000Z","size":67387,"stargazers_count":4,"open_issues_count":0,"forks_count":1,"subscribers_count":4,"default_branch":"master","last_synced_at":"2024-01-05T04:45:04.417Z","etag":null,"topics":["cloud","cloudfoundry","docker","drmaa","grid","gridengine","multiclustering"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/dgruber.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2014-12-07T16:21:25.000Z","updated_at":"2024-01-05T04:45:04.418Z","dependencies_parsed_at":"2022-08-22T12:41:14.340Z","dependency_job_id":null,"html_url":"https://github.com/dgruber/ubercluster","commit_stats":null,"previous_names":[],"tags_count":0,"template":null,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dgruber%2Fubercluster","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dgruber%2Fubercluster/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dgruber%2Fubercluster/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dgruber%2Fubercluster/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/dgruber","download_url":"https://codeload.github.com/dgruber/ubercluster/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":224970197,"owners_count":17400292,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cloud","cloudfoundry","docker","drmaa","grid","gridengine","multiclustering"],"created_at":"2024-11-16T20:32:46.828Z","updated_at":"2024-11-16T20:32:47.846Z","avatar_url":"https://github.com/dgruber.png","language":"Go","readme":"ubercluster\n===========\n[![GoDoc](http://img.shields.io/badge/godoc-reference-blue.svg)](http://godoc.org/github.com/dgruber/ubercluster)\n[![Apache V2 License](http://img.shields.io/badge/license-Apache%20V2-blue.svg)](https://raw.githubusercontent.com/dgruber/ubercluster/master/LICENSE)\n[![Go Report Card](http://goreportcard.com/badge/dgruber/ubercluster)](http://goreportcard.com/report/dgruber/ubercluster)\n\nSimple multi-clustering tool based on an open standard for job submission and cluster monitoring ([DRMAA2](http://www.drmaa.org)). Works on top of supported cluster schedulers, like Univa Grid Engine but also\nfor Cloud Foundry tasks or local Docker containers.\n\nIt consists of following components:\n- **uc** [![Build Status](https://travis-ci.org/dgruber/ubercluster.svg)](https://travis-ci.org/dgruber/ubercluster): Main command line tool to interact with the compute clusters (show status / start jobs). Pure Go - can run everywhere where you can compile Go (MacOS / Linux / Windows / ...). Communicates with proxies. \n- **d2proxy**: Proxy which runs on a submit host of a compute cluster (Grid Engine cluster). Based on Go DRMAA2 (which is based on the DRMAA2 C API).\n- **d1proxy**: Proxy for DRMAA (version 1) compatible clusters. Does not support most concepts but job submission works. Good starting point if you want to create your own proxy (which is btw. extremely easy).\n- [**cf-tasks**](https://github.com/dgruber/ubercluster/blob/master/cmd/cf-tasks): Proxy which emits Cloud Foundry tasks as jobs.\n- [**dockerproxy**](https://github.com/dgruber/ubercluster/blob/master/cmd/dockerproxy): Proxy which runs Docker containers as jobs.\n- [**processProxy**](https://github.com/dgruber/ubercluster/blob/master/cmd/processProxy): Proxy which starts local processes as jobs.\n\n![uc image](https://raw.githubusercontent.com/dgruber/ubercluster/master/img/uc.png)\n\n## Compilation of DRMAA2 Proxy\n\nMake sure you have a cluster scheduler supporting DRMAA2 C API (like Univa Grid Engine)\ninstalled. This tool is working on top of the [Go DRMAA2 API](https://github.com/dgruber/drmaa2) (which accesses the C API).\nHence during compile/runtime the tool needs access to drmaa2.h (which comes with \n[Univa Grid Engine](http://www.univa.com/resources/univa-grid-engine-trial.php) for example / $SGE_ROOT/include) and libdrmaa2.so ($SGE_ROOT/lib/lx-amd64).\n\nUpdate: Removed DRMAA2 C dependencies from **uc** tool. Hence those requirements are only needed for the DRMAA2 proxy (**d2proxy**) tool.\n\nGo to ```cmd/d2proxy```\n \n    $ source path/to/your/GE/installation\n    $ godep restore\n    $ ./build\n\n## Compilation of Cloud Foundry or Docker Proxy\n\nGo to ```cmd/cf-tasks``` or ```cmd/dockerproxy```\n\n    $ go install\n\n## Installation uc\n\nGo to ```cmd/uc```\n\n    $ go install\n\n## Example usage\n\n### Start your proxy - one per cluster:\n\n    $ source path/to/your/GE/installation\n\n    $ d2proxy \u0026\n    \nor for listening on port 8282\n    \n    $ d2proxy -port=\":8282\" \u0026\n\nOr a local Docker Proxy:\n\n    $ dockerproxy --otp \"supersecret\"\n\nStart a Docker container:\n\n    $ uc --otp=supersecret run --arg 120 --category \"ubuntu:latest\" /bin/sleep\n\n### Test the proxies by opening the address in the webbrowser.\n\nExample:\n\n    $ firefox http://localhost:8888/v1/msession/jobinfos\n\n### Update config.json \n\nThe *config.json* file (an example can be found in the **uc** directory) contains the contact details of the proxies used by **uc**. First **uc** scans the current working directory, then $HOME/.ubercluster/config.json, and finally /etc/ubercluster/config.json. The file can contain the locations of different proxies. The *default* entry is the cluster/proxy which is used when no other is specified as  __--cluster__ parameter of **uc**.\n\n### Examples\n\n#### List all jobs of your default cluster\n\n    $ uc show job\n\n#### List all running jobs of cluster \"cluster1\" (from config)\n\n    $ uc --cluster=cluster1 show job --state=r\n\n    job_number:\t\t    3000000003\n    state:\t\t\t    Running\n    submission_time:\t2014-12-06 18:02:59 +0100 CET\n    dispatch_time:\t\t2014-12-06 18:03:00 +0100 CET\n    finish_time:\t\t-\n    owner:\t\t\t    daniel\n    slots:\t\t\t    1\n    allocated_machines:\tu1010\n    exit_status:\t\t-1\n\n    job_number:\t\t    3000000004\n    state:\t\t\t    Running\n    submission_time:\t2014-12-06 18:03:01 +0100 CET\n    dispatch_time:\t\t2014-12-06 18:03:10 +0100 CET\n    finish_time:\t\t-\n    owner:\t\t\t    daniel\n    slots:\t\t\t    1\n    allocated_machines:\tu1010\n    exit_status:\t\t-1\n\n#### Let a simple process run in default cluster\n\n    $ uc run --arg=123 /bin/sleep\n\n#### Upload the job file and execute it\n\nWith recent check-ins also file staging is partially supported. By\nusing the **upload** flag of the **run** command, the file is first\nuploaded to the \"uploads\" directory (per default a subdirectory of\nwhere you are starting the proxy) by http, then marked as executable.\n\nOther file staging capabilities are accessible by the **uc fs** command.\n\n    $ uc run --upload=testjob.sh testjob.sh\n\n#### ...and now let it run in the \"cluster1\" cluster, adding a job name and selecting a queue (default is \"all.q\"):\n\n    $ uc --cluster=cluster1 run --queue=all.q --name=MyName --arg=123 /bin/sleep\n    \n#### ...more submission command parameters\n\nSince submission commands are never enough, always needs to be extended, ..., and are different between versions of cluster schedulers let's keep it simple. **uc** supports DRMAA2 job categories, which are names referencing a particular set of submission parameters. **Univa Grid Engine \u003e= 8.2** encodes job categories as job classes. In **uc** you can request such job categories / classes with the **--category** parameter.\n\n```\n$ uc run --help\nusage: uc [\u003cflags\u003e] run [\u003cflags\u003e] \u003ccommand\u003e\n\nSubmits an application to a cluster.\n\nFlags:\n  --arg=ARG            Argument of the command.\n  --name=NAME          Reference name of the command.\n  --queue=QUEUE        Queue name for the job.\n  --category=CATEGORY  Job category / job class of the job.\n  --alg=ALG            Automatic cluster selection when submitting jobs (\"rand\", \"prob\", \"load\")\n  --upload=UPLOAD      Path to job which is uploaded before execution.\n\n\nArgs:\n  \u003ccommand\u003e  Command to submit.\n```\n\n#### List all hosts of default cluster:\n\n    $ uc show machine\n    \n    HOSTNAME ARCH NSOC NCOR NTHR LOAD MEMTOT SWAPTO\n    u1010 x64 1 4 4 0.080000 504184 911731\n    ...\n\n#### Get full command description...\n\n    $ uc --help\n\n```\nusage: uc [\u003cflags\u003e] \u003ccommand\u003e [\u003cflags\u003e] [\u003cargs\u003e ...]\n\nA tool which can interact with multiple compute clusters.\n\nFlags:\n  --help               Show help.\n  --verbose            Enables enhanced logging for debugging.\n  --cluster=\"default\"  Cluster name to interact with.\n  --otp=OTP            One time password (\"yubikey\") or shared secret.\n  \nCommands:\n  help [\u003ccommand\u003e]\n    Show help for a command.\n\n  show job [\u003cflags\u003e] [\u003cid\u003e]\n    Information about a particular job.\n\n  show machine [\u003cname\u003e]\n    Information about compute hosts.\n\n  show queue [\u003cname\u003e]\n    Information about queues.\n\n  show categories [\u003cname\u003e]\n    Information about job categories\n\n  run [\u003cflags\u003e] \u003ccommand\u003e\n    Submits an application to a cluster.\n\n  terminate job [\u003cjobid\u003e]\n    Terminates (ends) a job in a cluster.\n\n  suspend job [\u003cjobid\u003e]\n    Suspends (pauses) a job in a cluster.\n\n  resume job [\u003cjobid\u003e]\n    Resumes a suspended job in a cluster.\n\n  fs ls\n    List all files in staging area.\n\n  fs up \u003cfiles\u003e\n    Upload a file to staging area.\n\n  fs down \u003cfiles\u003e\n    Download files from staging area.\n\n  config list\n    Lists all configured cluster proxies.\n\n  inception [\u003cport\u003e]\n    Run uc as compatible proxy itself. Allows to create trees of clusters.\n\n```\n\nFor detailed help on sub-commands:\n\n    $ uc show job --help\n    \n```\nusage: uc [\u003cflags\u003e] show job [\u003cflags\u003e] [\u003cid\u003e]\n\nInformation about a particular job.\n\nFlags:\n  --state=\"all\"  Show only jobs in that state (r/q/h/s/R/Rh/d/f/u/all).\n\nArgs:\n  [\u003cid\u003e]  Id of job\n\n```\n\n#### Security Considerations\n\nPlease be aware that when exporting over http also others in the same network\n(or even publicly) can access the clusters. Job modifications are only allowed\nfor jobs started in the same DRMAA2 job session (usually only those submitted\nby *uc*) so only jobs started with *uc* can be deleted, held, suspended...\n\nNo security: Just starting proxy without any parameter. All traffic is unencrypted\nand the proxy accessible by everybody who can access the network port.\n\nLow security: Starting the proxy with --otp=MySuperSecretKey.\nUnencrypted, the caller needs to know the key and add that key with \nall *uc* commands (like *uc --otp=MySuperSecretKey ..*) or in the configuration.\nThe key is part of each http request.\n\nHigh security (but no encryption): Starting the proxy with *--otp=yubikey*.\nAll client calls must have *--otp=yubikey* set. The *uc* tool is \nrequesting from the client a one-time-password which is generated\nby the yubikey USB stick (obviously this is an requirement). The \nproxy needs to be registered first as service and started with the\nsecret key and service id. Using the official servers you can register\nyour service here: https://upgrade.yubico.com/getapikey/\nAlternatively you can setup your own OTP validation server\n(like https://github.com/digintLab/yubikey-server).\n\nRudimentary mTLS support: Start **processProxy** with *--key* (points to \nserver key) and *--cert* (points to cert of server) *--clientCerts* \n(points to a directory with trusted client crts). *uc* needs to use\n*--cert* and *--key* of client certificates.\n\n#### Other\n\nA [Go Report Card](http://goreportcard.com/report/dgruber/ubercluster) is available.\n\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdgruber%2Fubercluster","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdgruber%2Fubercluster","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdgruber%2Fubercluster/lists"}