{"id":17964791,"url":"https://github.com/joergen7/cf_worker","last_synced_at":"2025-04-10T03:36:07.502Z","repository":{"id":59150118,"uuid":"73930034","full_name":"joergen7/cf_worker","owner":"joergen7","description":"Cuneiform worker implementation","archived":false,"fork":false,"pushed_at":"2023-03-20T15:43:05.000Z","size":61,"stargazers_count":3,"open_issues_count":0,"forks_count":1,"subscribers_count":3,"default_branch":"master","last_synced_at":"2024-10-29T12:59:22.381Z","etag":null,"topics":["cuneiform","erlang","otp","worker"],"latest_commit_sha":null,"homepage":"https://hex.pm/packages/cf_worker","language":"Erlang","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/joergen7.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2016-11-16T14:54:04.000Z","updated_at":"2022-05-27T07:45:28.000Z","dependencies_parsed_at":"2024-10-29T12:28:48.318Z","dependency_job_id":null,"html_url":"https://github.com/joergen7/cf_worker","commit_stats":{"total_commits":97,"total_committers":2,"mean_commits":48.5,"dds":0.07216494845360821,"last_synced_commit":"d28a3d177cb2c505e310571cd014da9236b36186"},"previous_names":[],"tags_count":8,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/joergen7%2Fcf_worker","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/joergen7%2Fcf_worker/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/joergen7%2Fcf_worker/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/joergen7%2Fcf_worker/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/joergen7","download_url":"https://codeload.github.com/joergen7/cf_worker/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247426531,"owners_count":20937141,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cuneiform","erlang","otp","worker"],"created_at":"2024-10-29T12:08:40.687Z","updated_at":"2025-04-10T03:36:07.491Z","avatar_url":"https://github.com/joergen7.png","language":"Erlang","funding_links":[],"categories":[],"sub_categories":[],"readme":"# cf_worker\n###### Cuneiform worker implementation\n\n[![hex.pm](https://img.shields.io/hexpm/v/cf_worker.svg?style=flat-square)](https://hex.pm/packages/cf_worker)\n\nThe Cuneiform worker is a worker implementation for the common runtime environment (CRE). The worker uses the Erlang foreign function interface (Effi) to execute tasks and interacts with the (distributed) filesystem via a Posix interface.\n\n## Integration and Build\n\n### Adding the Worker to a Project\n\nAlthough the Cuneiform worker application can be imported also directly from GitHub, we recommend adding a dependency via [hex.pm](https://hex.pm). Here, we show how this can be done using the build tools [rebar3](https://www.rebar3.org) or mix.\n\n#### rebar3\n\nTo integrate the Cuneiform worker application into a rebar3-managed project change the `deps` entry in your application's `rebar.config` file to include the tuple `{cf_worker, \"0.1.8\"}`.\n\n```erlang\n{deps, [{cf_worker, \"0.1.8\"}]}.\n```\n\n#### mix\n\nTo integrate effi into a mix-managed project include the following\n\n```elixir\n{:cf_worker, \"~\u003e 0.1.8\"}\n\n```\n### Compiling\n\nHaving rebar3 available on your system, compile the project as an Erlang project by entering\n\n    rebar3 compile\n\nIf you want to drive the project from the command line please compile the project by entering\n\n    rebar3 escriptize\n\n\n## Command Line Synopsis\n\nCompiling the Cuneiform client using `escriptize` creates an Erlang script file `cf_worker` which allows starting the Cuneiform client via the command line.\n\nTo display a help text enter\n\n    ./cf_worker --help\n\nThis will show the command line synopsis, which looks like the following:\n\n    Usage: cf_worker [-v] [-h] [-s \u003csuppl_file\u003e] [-c \u003ccre_node\u003e] [-n \u003cn_wrk\u003e]\n                     [-w \u003cwrk_dir\u003e] [-r \u003crepo_dir\u003e] [-d \u003cdata_dir\u003e]\n\n      -v, --version     Show cf_worker version.\n      -h, --help        Show command line options.\n      -s, --suppl_file  Supplementary configuration file.\n      -c, --cre_node    Erlang node running the CRE application (must be \n                        specified).\n      -n, --n_wrk       Number of worker processes to start. 0 means \n                        auto-detect available processors.\n      -w, --wrk_dir     Working directory in which workers store temporary \n                        files.\n      -r, --repo_dir    Repository directory for intermediate and output data.\n      -d, --data_dir    Data directory where input data is located.\n\n\nTo start the worker application from the command line and connect with a running CRE instance enter\n\n    ./cf_worker -c cre@my_node\n\nHere, we assume that the CRE runs on an Erlang node identified as `cre@my_node`.\n\n## Erlang API\n\nIf a CRE instance is already running on the same Erlang node you can start the Cuneiform worker application by calling\n\n```erlang\ncf_worker:start().\n```\n\nWhich is exactly the same as calling\n\n```erlang\napplication:start( cf_worker ).\n```\n\n### Starting Under the Default Supervisor\n\nTo start the Cuneiform worker default supervisor under a custom supervision tree enter\n\n```erlang\nCreNode = node().\nNWrk    = 4.\nWrkDir  = \"./_cuneiform/wrk\".\nRepoDir = \"./_cuneiform/repo\".\nDataDir = \"./\".\n\ncf_client_sup:start_link( CreNode, NWrk, WrkDir, RepoDir, DataDir ).\n```\n\nThis starts a worker supervisor with four workers using `WrkDir` to store temporal data, `RepoDir` for intermediate and output data, and `DataDir` to look up input data. Also, we expect a CRE to be running on the same node.\n\n### Starting Directly\n\nThe Cuneiform client process can be started directly. There are several ways to do this. The first is to start the process with a function that allows it to locate the CRE:\n\n```erlang\nCreNode = node().\nF = fun() -\u003e cre:pid( CreNode ) end.\n\nWrkDir  = \"./_cuneiform/wrk\".\nRepoDir = \"./_cuneiform/repo\".\nDataDir = \"./\".\n\n{ok, WorkerPid} = cf_worker_process:start_link( F, WrkDir, RepoDir, DataDir ).\n```\n\nGiving a function instead of a plain CRE process identifier has the advantage, that if the CRE crashes, taking the Cuneiform worker with it, the restarted worker instance uses the output of the function, which offers the possibility of locating the CRE under its new process identifier.\n\nIf this is too tedious, one can start it giving the CRE process identifier directly:\n\n```erlang\nCrePid = cre:pid( node() ).\n\nWrkDir  = \"./_cuneiform/wrk\".\nRepoDir = \"./_cuneiform/repo\".\nDataDir = \"./\".\n\n{ok, WorkerPid} = cf_worker_process:start_link( CrePid ).\n```\n\nBoth previous direct starting methods do not register the Cuneiform client with any registry service.\n\n## System Requirements\n\n- [Erlang](https://www.erlang.org) OTP 19.0 or higher\n- [Rebar3](https://www.rebar3.org) 3.0.0 or higher\n\n## Resources\n\n- [joergen7/cre](https://github.com/joergen7/cre). A common runtime environment (CRE) for distributed workflow languages.\n- [joergen7/cuneiform](https://github.com/joergen7/cuneiform). A functional language for large-scale data analysis whose distributed execution environment is implemented on top of the CRE.\n- [joergen7/effi](https://github.com/joergen7/effi). Erlang foreign function interface.\n\n\n## Authors\n\n- Jörgen Brandt ([@joergen7](https://github.com/joergen7/)) [joergen.brandt@onlinehome.de](mailto:joergen.brandt@onlinehome.de)\n\n## License\n\n[Apache 2.0](https://www.apache.org/licenses/LICENSE-2.0.html)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjoergen7%2Fcf_worker","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjoergen7%2Fcf_worker","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjoergen7%2Fcf_worker/lists"}