{"id":15065885,"url":"https://github.com/jjauzion/ws-worker","last_synced_at":"2026-02-10T09:41:51.874Z","repository":{"id":38400311,"uuid":"345362350","full_name":"jjauzion/ws-worker","owner":"jjauzion","description":"A worker node in golang to execute jobs in a docker container","archived":false,"fork":false,"pushed_at":"2022-06-04T19:34:43.000Z","size":26506,"stargazers_count":0,"open_issues_count":8,"forks_count":3,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-03-16T01:12:05.614Z","etag":null,"topics":["go","golang","graphql-server","grpc-server","worker-node"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jjauzion.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2021-03-07T14:06:28.000Z","updated_at":"2022-01-22T18:14:07.000Z","dependencies_parsed_at":"2022-08-18T06:12:26.248Z","dependency_job_id":null,"html_url":"https://github.com/jjauzion/ws-worker","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/jjauzion/ws-worker","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jjauzion%2Fws-worker","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jjauzion%2Fws-worker/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jjauzion%2Fws-worker/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jjauzion%2Fws-worker/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jjauzion","download_url":"https://codeload.github.com/jjauzion/ws-worker/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jjauzion%2Fws-worker/sbom","scorecard":{"id":521997,"data":{"date":"2025-08-11","repo":{"name":"github.com/jjauzion/ws-worker","commit":"ecce3a2807ff76e6fa6f66f92923e17673b4dfe1"},"scorecard":{"version":"v5.2.1-40-gf6ed084d","commit":"f6ed084d17c9236477efd66e5b258b9d4cc7b389"},"score":3.2,"checks":[{"name":"Maintained","score":0,"reason":"0 commit(s) and 0 issue activity found in the last 90 days -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project is \"actively maintained\".","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#maintained"}},{"name":"Pinned-Dependencies","score":-1,"reason":"no dependencies found","details":null,"documentation":{"short":"Determines if the project has declared and pinned the dependencies of its build process.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#pinned-dependencies"}},{"name":"Binary-Artifacts","score":10,"reason":"no binaries found in the repo","details":null,"documentation":{"short":"Determines if the project has generated executable (binary) artifacts in the source repository.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#binary-artifacts"}},{"name":"Code-Review","score":8,"reason":"Found 4/5 approved changesets -- score normalized to 8","details":null,"documentation":{"short":"Determines if the project requires human code review before pull requests (aka merge requests) are merged.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#code-review"}},{"name":"Dangerous-Workflow","score":-1,"reason":"no workflows found","details":null,"documentation":{"short":"Determines if the project's GitHub Action workflows avoid dangerous patterns.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#dangerous-workflow"}},{"name":"Packaging","score":-1,"reason":"packaging workflow not detected","details":["Warn: no GitHub/GitLab publishing workflow detected."],"documentation":{"short":"Determines if the project is published as a package that others can easily download, install, easily update, and uninstall.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#packaging"}},{"name":"Token-Permissions","score":-1,"reason":"No tokens found","details":null,"documentation":{"short":"Determines if the project's workflows follow the principle of least privilege.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#token-permissions"}},{"name":"CII-Best-Practices","score":0,"reason":"no effort to earn an OpenSSF best practices badge detected","details":null,"documentation":{"short":"Determines if the project has an OpenSSF (formerly CII) Best Practices Badge.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#cii-best-practices"}},{"name":"Security-Policy","score":0,"reason":"security policy file not detected","details":["Warn: no security policy file detected","Warn: no security file to analyze","Warn: no security file to analyze","Warn: no security file to analyze"],"documentation":{"short":"Determines if the project has published a security policy.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#security-policy"}},{"name":"Fuzzing","score":0,"reason":"project is not fuzzed","details":["Warn: no fuzzer integrations found"],"documentation":{"short":"Determines if the project uses fuzzing.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#fuzzing"}},{"name":"License","score":10,"reason":"license file detected","details":["Info: project has a license file: LICENSE:0","Info: FSF or OSI recognized license: Apache License 2.0: LICENSE:0"],"documentation":{"short":"Determines if the project has defined a license.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#license"}},{"name":"Signed-Releases","score":-1,"reason":"no releases found","details":null,"documentation":{"short":"Determines if the project cryptographically signs release artifacts.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#signed-releases"}},{"name":"Branch-Protection","score":-1,"reason":"internal error: error during branchesHandler.setup: internal error: githubv4.Query: Resource not accessible by integration","details":null,"documentation":{"short":"Determines if the default and release branches are protected with GitHub's branch protection settings.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#branch-protection"}},{"name":"SAST","score":0,"reason":"SAST tool is not run on all commits -- score normalized to 0","details":["Warn: 0 commits out of 30 are checked with a SAST tool"],"documentation":{"short":"Determines if the project uses static code analysis.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#sast"}},{"name":"Vulnerabilities","score":0,"reason":"30 existing vulnerabilities detected","details":["Warn: Project is vulnerable to: GO-2022-0921 / GHSA-c72p-9xmj-rx3w","Warn: Project is vulnerable to: GO-2022-0938 / GHSA-c2h3-6mxw-7mvq","Warn: Project is vulnerable to: GO-2022-0360 / GHSA-5j5w-g665-5m35","Warn: Project is vulnerable to: GO-2022-0344 / GHSA-crp2-qrr5-8pq7","Warn: Project is vulnerable to: GO-2024-2846 / GHSA-c9cp-9c75-9v8c","Warn: Project is vulnerable to: GO-2022-0482 / GHSA-5ffw-gxpp-mxpf","Warn: Project is vulnerable to: GO-2022-1147 / GHSA-2qjp-425j-52j9","Warn: Project is vulnerable to: GO-2023-1573 / GHSA-259w-8hf6-59c2","Warn: Project is vulnerable to: GO-2023-1574 / GHSA-hmfx-3pcx-653p","Warn: Project is vulnerable to: GO-2023-2412 / GHSA-7ww5-4wqc-m92c","Warn: Project is vulnerable to: GO-2025-3528 / GHSA-265r-hfxg-fhmg","Warn: Project is vulnerable to: GO-2022-0379 / GHSA-qq97-vm5h-rrhg","Warn: Project is vulnerable to: GHSA-hqxw-f8mx-cpmw","Warn: Project is vulnerable to: GO-2024-2500 / GHSA-3fwx-pjgw-3558","Warn: Project is vulnerable to: GO-2024-2913 / GHSA-v994-f8vw-g7j4","Warn: Project is vulnerable to: GO-2024-2914 / GHSA-xmmx-7jpf-fx42","Warn: Project is vulnerable to: GO-2022-0390 / GHSA-2mm7-x5h6-5pvq","Warn: Project is vulnerable to: GO-2022-0985 / GHSA-rc4r-wh2q-q6c4","Warn: Project is vulnerable to: GO-2022-1107 / GHSA-vp35-85q5-9f25","Warn: Project is vulnerable to: GO-2023-1699 / GHSA-232p-vwff-86mp","Warn: Project is vulnerable to: GO-2023-1700 / GHSA-33pg-m6jh-5237","Warn: Project is vulnerable to: GO-2023-1701 / GHSA-6wrf-mxfj-pf5p","Warn: Project is vulnerable to: GHSA-jq35-85cj-fj4p","Warn: Project is vulnerable to: GHSA-mq39-4gv4-mvpx","Warn: Project is vulnerable to: GO-2024-3005 / GHSA-v23v-6jw2-98fq","Warn: Project is vulnerable to: GO-2024-2512 / GHSA-xw73-rw38-6vjc","Warn: Project is vulnerable to: GO-2025-3829 / GHSA-4vq8-7jfc-9cvp","Warn: Project is vulnerable to: GHSA-77vh-xpmg-72qh","Warn: Project is vulnerable to: GO-2023-2153 / GHSA-m425-mq94-257g / GHSA-qppj-fm5r-hxr3","Warn: Project is vulnerable to: GO-2024-2611 / GHSA-8r3f-844c-mc37"],"documentation":{"short":"Determines if the project has open, known unfixed vulnerabilities.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#vulnerabilities"}}]},"last_synced_at":"2025-08-20T03:16:21.531Z","repository_id":38400311,"created_at":"2025-08-20T03:16:21.531Z","updated_at":"2025-08-20T03:16:21.531Z"},"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":273227970,"owners_count":25067691,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-09-02T02:00:09.530Z","response_time":77,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["go","golang","graphql-server","grpc-server","worker-node"],"created_at":"2024-09-25T00:56:49.373Z","updated_at":"2026-02-10T09:41:46.831Z","avatar_url":"https://github.com/jjauzion.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Workstation\nThe workstation project is machine learning job management system.  \n\nIt consists of a task queue where user can create new jobs and one or more worker nodes that will pull\n jobs from this queue, run the algorithm and return the result when it is finished.\n\nThe jobs are submitted via a Docker image that shall be available on a public container registry.\n\nThe project is made of three repositories:\n- the worker node: https://github.com/jjauzion/ws-worker\n- the backend server: https://github.com/jjauzion/ws-backend\n- the frontend: *not yet implemented*\n\n# How to run\n\nThis tutorial will guide you through running the whole project.\n\n## Pre requisite\n- macOS or linux\n- Docker installed\n- Docker-compose installed\n- go installed\n- makefile support\n\n## Start the backend\nHere we will run the entire project on your local machine from scratch, including the database.\n The database will be boostrapped with default users.\n- Clone the backend repository: `git clone https://github.com/jjauzion/ws-backend`\n- Open the repo: `cd ws-backend`\n- Create the `.env` file in the project root folder.\n  For a dev environment running in localhost, use this:\n```dotenv\nWS_ES_HOST=http://localhost\nWS_ES_PORT=9200\nWS_KIBANA_PORT=5601\nWS_API_HOST=localhost\nWS_API_PORT=8080\nWS_GRPC_HOST=localhost\nWS_GRPC_PORT=8090\nIS_DEV_ENV=true\nTOKEN_DURATION_HOURS=24\nWS_ES_USERNAME=\"\"\nWS_ES_PWD=\"\"\n```\n- Start the elastic and kibana cluster: `make elastic`\n- Check kibana container logs: `docker logs ws-backend_kibana_1 -f`   \n- Wait until you see:\n```dockerfile\n{\"type\":\"log\",\"@timestamp\":\"2021-03-28T15:11:50+00:00\",\"tags\":[\"listening\",\"info\"],\"pid\":7,\"message\":\"Server running at http://0:5601\"}\n{\"type\":\"log\",\"@timestamp\":\"2021-03-28T15:11:51+00:00\",\"tags\":[\"info\",\"http\",\"server\",\"Kibana\"],\"pid\":7,\"message\":\"http server running at http://0:5601\"}\n{\"type\":\"log\",\"@timestamp\":\"2021-03-28T15:11:54+00:00\",\"tags\":[\"warning\",\"plugins\",\"reporting\"],\"pid\":7,\"message\":\"Enabling the Chromium sandbox provides an additional layer of protection.\"}\n```\n- Start the GraphQL server `make gql FLAG=\"--bootstrap\"`  \n  The bootstrap option initialise the DB by creating the required index and indexing default users\n- Open a new terminal in the same repo \n- Start the gRPC server: `make grpc`\n  \nAt this point you have started the database, the graphQL server that interact with the frontend\n and the gRPC server that interact with the worker nodes.  \n\n## Kibana\nBefore starting the worker node we will learn how to interact with the backend. First, lets check the\ndatabase:\n- Open Kibana: http://localhost:5601  \n- Click on the burger menu in the top left corner and go to the `Dev Tools`\n- Copy / Paste the following in the console and run it: `GET _cat/indices?v`  \n  This list all the index existing in the DB. You should see an index called `ws_task` and one\n  called `ws_user`. Index starting with a dot `.` are system index.\n- Now run the following to list all the existing users:\n```\nGET ws_user/_search\n{\n  \"query\": {\n    \"match_all\": {}\n  }\n}\n```\n- To get all the task, replace in the previous query `ws_user` by `ws_task`\n- You can use this console for debug purpose if you need to check the content of your database.\n  You could also create or delete task and user manually from here but it is better to use the\n  GraphQL API.\n  \n\n## Login\nBefore being able to create user and task you will need to login. As we started the GraphQL server\nwith bootstrap option, two default users have been created in the DB.  \nWe will login with the admin user using the GraphQL API.\n- open the GraphQL playground: http://localhost:8080/playground\n- you can find the doc and schema of our API thanks to the \"DOCS\" and \"SCHEMA\" tabs on the right side\n  of the screen. This will help you later to build your own request\n- copy / paste the following request to log in as the admin user:\n```graphql\nquery login {\n  login (id: \"admin-user@email.com\", pwd: \"\") {\n    ... on Token {\n      token\n      userId\n      username\n    }\n    ... on Error {\n      code\n      message\n    }\n  }\n}\n```\nYou should get a response similar to this:\n```json\n{\n  \"data\": {\n    \"login\": {\n      \"token\": \"eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdXRob3JpemVkIjp0cnVlLCJleHAiOjE2MTczODMzNDksInVzZXJfaWQiOiJkZjljNDYzZC00ZmIwLTRmYzAtYTU5OC00YmQ3NzEzMzg2ZDAifQ.Xj_rUGIB7l90kiXD_U12ni2kf9U-afARaCZKbEao-oU\",\n      \"userId\": \"df9c463d-4fb0-4fc0-a598-4bd7713386d0\",\n      \"username\": \"admin-user@email.com\"\n    }\n  }\n}\n```\nAs you can see the server successfully authenticated your request and have generated a JWT token \nthat you can use for further request to prove that you are authenticated.  \n\nCopy the `token` value and `userId` somewhere as you will need those later.\n  \n## Create a user\nWe will now see how to create user and task with the GraphQL API.\n- To create a new user, paste the following in the console:\n```graphql\nmutation tuto_create_user {\n  create_user(input:{email:\"just-for-test@email.com\"}) {\n    id\n    email\n  }\n}\n```\n- if you send the request like this you will get a `403` error because you are not authenticated\n- you need to pass the token we generated in the chapter before: on the bottom of the console, \n  click on `HTTP HEADERS` and paste the following (replace the token value with yours):\n```json\n{\n  \"auth\": \"eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdXRob3JpemVkIjp0cnVlLCJleHAiOjE2MTczODMwNTUsInVzZXJfaWQiOiI0MzVmNTA3OC02NjFlLTRkOGMtODJjZS0zNDJhZTQ1ZTQ4MzcifQ.RTzseF7mSjR8aop-9CCiNt1-IkqFGem9nNWymaJKBRo\"\n}\n```\n- you can now send the request. You should have a response like this:\n```json\n{\n  \"data\": {\n    \"create_user\": {\n      \"id\": \"86c776ec-9abe-43a0-93f1-4dac0997ba90\",\n      \"email\": \"just-for-tes3t@email.com\"\n    }\n  }\n}\n```\n## Create a task\n- now let's create a task. Run the following command (replace the user id with yours):\n```graphql\nmutation createTask {\n  create_task(input:{docker_image:\"jjauzion/ws-mock-container\", dataset:\"s3//\"}) {\n    id\n    user_id\n  \tcreated_at\n  \tstarted_at\n  \tended_at\n  \tstatus\n    job { dataset, docker_image }\n  }\n}\n```\n- if you got a `403` error, check you didn't forget the `auth` Header in your request (see previous chapter)\n\nCongratulations !! You have created a user and a new jobs :) You can go to the kibana console and run \nthe search to see your creations.\n\n## Run a worker node\nNow that we have created a new task, it would be nice to have a worker to actually run that task right?  \nBut before starting a worker node, we need to start the gRPC server:\n- Go in the `ws-backend` repository and run: `make grpc`  \n\nNow let's run the worker:  \n- Clone the worker repository: `git clone https://github.com/jjauzion/ws-worker.git`\n- go in the `ws-worker` repo: `cd ws-worker`  \n- Create a `.env` file at the project root. \n  For a dev environment running in localhost with the following values:\n```dotenv\nWS_GRPC_HOST=localhost\nWS_GRPC_PORT=8090\n```  \n- Start the worker: `make run`  \n\nThis will start the worker and it will automatically pull the task you have created in the \n  previous chapter and run it.\n\nYou can go to kibana and check your task, you will see the status going from \"NOT_STARTED\" \nto \"RUNNING\" and \"ENDED\"\n\n## Create a Machine Learning task\nLet's create a real job: running a ML jobs and tracking your jobs parameters while it is running.\n\nFor this we will use wandb (https://wandb.ai/site) so you must create a user and copy your private key.\n\nThen paste the following in the playground console and put your wandb key in the env variable.\nYour key will be encrypted on the server and will never be stored in clear (WIP, not done yet)\n\n```graphql\nmutation createTask {\n  create_task(input:{env:\"WANDB_API_KEY=putYourKeyHere\", docker_image:\"jjauzion/wandb-test\", dataset:\"s3//\"}) {\n    id\n    user_id\n        created_at\n        started_at\n        ended_at\n        status\n    job { dataset, docker_image, env }\n  }\n}\n```\nWait until the task status is updated to \"RUNNING\" (can take up to 30sec), then log to wandb.\nYou should see your work ongoing.\n\nWhen you are done, go in the ws-backend repo and run `make down` to stop and the elastic containers\n\nThat's it folks :)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjjauzion%2Fws-worker","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjjauzion%2Fws-worker","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjjauzion%2Fws-worker/lists"}