{"id":16577474,"url":"https://github.com/alchemine/distributed_computing","last_synced_at":"2025-08-03T10:07:07.679Z","repository":{"id":55055909,"uuid":"453674994","full_name":"alchemine/distributed_computing","owner":"alchemine","description":"분산 처리를 위한 예제 코드","archived":false,"fork":false,"pushed_at":"2022-08-09T06:50:48.000Z","size":1930,"stargazers_count":0,"open_issues_count":0,"forks_count":1,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-01-16T07:15:40.620Z","etag":null,"topics":["distributed-computing","parallel-computing","python"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/alchemine.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2022-01-30T12:33:12.000Z","updated_at":"2023-09-02T18:10:57.000Z","dependencies_parsed_at":"2022-08-14T10:20:35.146Z","dependency_job_id":null,"html_url":"https://github.com/alchemine/distributed_computing","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alchemine%2Fdistributed_computing","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alchemine%2Fdistributed_computing/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alchemine%2Fdistributed_computing/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alchemine%2Fdistributed_computing/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/alchemine","download_url":"https://codeload.github.com/alchemine/distributed_computing/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":242097417,"owners_count":20071252,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["distributed-computing","parallel-computing","python"],"created_at":"2024-10-11T22:11:00.119Z","updated_at":"2025-03-05T19:58:24.012Z","avatar_url":"https://github.com/alchemine.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# 분산 처리를 위한 예제 코드\n\n# 1. Usage\n```bash\n## 1. Server 설정(3.1 참고)\nserver_config.ini 생성\n\n\n## 2. Cluster 생성\n$ python cluster_start.py\n\n\n## 3. 작업 수행\n$ python example.py\n$ python example_connection_error_handled.py  # client.gather() error is handled\n\n\n## 4. Cluster 해제\n$ python cluster_stop.py \n```\n\n# 2. Example python script code\n- [example.py](example.py)\n- [example_connection_error_handled.py](example_connection_error_handled.py)\n- [example_connection_error_handled.ipynb](example_connection_error_handled.ipynb)\n\n\n# 3. File structure\n```\ndistributed_computing\n├── README.md\n├── cluster_run.py\n├── cluster_start.py\n├── cluster_stop.py\n├── common.py\n├── example.py\n├── example_connection_error_handled.py\n├── example_connection_error_handled.ipynb\n└── server_config.ini (private)\n```\n\n### 3.1 `server_config.ini`\nCluster를 구성하는 server 정보가 저장된 파일\n- `ip`, `port`, `username` 등을 저장\n- 비공개 file 혹은 DB로 관리하여 server의 정보가 online 상에 upload 되지 않도록 주의 (`.gitignore`에 추가)\n\ne.g. \n```\n[server1]\nhost=123.456.78.910\nssh_port=10022\nusername=root\nscheduler_port=18786\ngpus=4\n\n\n[server2]\nhost=123.456.78.911\nssh_port=10022\nusername=root\nscheduler_port=18786\ngpus=4\n\n...\n```\n\n### 3.2 `cluster_run.py --cmd \u003ccommand\u003e [--ini \u003cserver config path\u003e]`\n모든 server들에 병렬적으로 명령을 실행시키는 python script\n- Arguments\n    1. `--cmd` : 수행할 명령어 (복수의 명령을 실행시킬 경우, `;`를 구분자로 사용)\n    2. `--ini` : server 정보가 저장된 ini 파일 경로 (default: `server_config.ini`)\n\n### 3.3 `cluster_start.py [--scheduler \u003cid of scheduler server\u003e] [--worker \u003cids of worker server\u003e] [--ini \u003cserver config path\u003e]`\nCluster를 연결하는 python script\n- Arguments\n    1. `--scheduler` : scheduler server id (default: `server1`)\n    2. `--worker` : worker server ids (복수의 server를 사용할 경우, ` `를 구분자로 사용) (default: `server1 server2 server3 server4`)\n    3. `--ini` : server 정보가 저장된 ini 파일 경로 (default: `server_config.ini`)\n\n### 3.4 `cluster_stop.py [--scheduler \u003cid of scheduler server\u003e] [--ini \u003cserver config path\u003e]`\n생성된 cluster를 구성하는 server들 간의 연결을 끊는 python script\n- Arguments\n    1. `--scheduler` : scheduler server id (default: `server1`)\n    2. `--ini` : server 정보가 저장된 ini 파일 경로 (default: `server_config.ini`)\n\n### 3.5 `common.py`\nUtility function 등이 있는 공용 module\n\n### 3.6 `example.py`\n분산 처리 예제 python script\n\n### 3.7 `example_connection_error_handled.py`, `example_connection_error_handled.ipynb`\n분산 처리 예제 python script (connection error가 발생하는 경우 사용)\n- worker port 접근이 제한되어 있는 경우, `client.gather()`이 사용불가하기 때문에 파일 전송 등을 통해 worker에서 실행된 결과를 받아와야 한다.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Falchemine%2Fdistributed_computing","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Falchemine%2Fdistributed_computing","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Falchemine%2Fdistributed_computing/lists"}