{"id":13536614,"url":"https://github.com/zhenrong-wang/hpc-now","last_synced_at":"2025-04-02T03:30:55.862Z","repository":{"id":200240518,"uuid":"705089895","full_name":"zhenrong-wang/hpc-now","owner":"zhenrong-wang","description":"A Cross-Platform, Multi-Cloud High-Performance Computing Platform","archived":false,"fork":false,"pushed_at":"2024-05-11T02:15:18.000Z","size":24595,"stargazers_count":229,"open_issues_count":1,"forks_count":108,"subscribers_count":74,"default_branch":"master","last_synced_at":"2024-05-11T03:24:15.317Z","etag":null,"topics":["aliyun","aws","azure","baiduyun","c","cloud","cluster","devops","google-cloud","hpc","huaweicloud","linux","opentofu","scripts","slurm","tencent-cloud","terraform"],"latest_commit_sha":null,"homepage":"https://www.hpc-now.com","language":"C","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/zhenrong-wang.png","metadata":{"files":{"readme":"README-CN.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"COPYING","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-10-15T01:37:19.000Z","updated_at":"2024-05-11T03:24:17.301Z","dependencies_parsed_at":"2023-10-16T07:15:00.569Z","dependency_job_id":"5a67513e-fbe7-445b-8789-45dca06f3dcd","html_url":"https://github.com/zhenrong-wang/hpc-now","commit_stats":{"total_commits":1812,"total_committers":2,"mean_commits":906.0,"dds":"0.13520971302428253","last_synced_commit":"da9e25ccc7d1317f2e2ebba4d746ab2a294a829e"},"previous_names":["zhenrong-wang/hpc-now"],"tags_count":20,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zhenrong-wang%2Fhpc-now","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zhenrong-wang%2Fhpc-now/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zhenrong-wang%2Fhpc-now/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zhenrong-wang%2Fhpc-now/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/zhenrong-wang","download_url":"https://codeload.github.com/zhenrong-wang/hpc-now/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246750938,"owners_count":20827804,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aliyun","aws","azure","baiduyun","c","cloud","cluster","devops","google-cloud","hpc","huaweicloud","linux","opentofu","scripts","slurm","tencent-cloud","terraform"],"created_at":"2024-08-01T09:00:45.516Z","updated_at":"2025-04-02T03:30:55.069Z","avatar_url":"https://github.com/zhenrong-wang.png","language":"C","funding_links":[],"categories":["Management Tool"],"sub_categories":[],"readme":"高性能计算，即刻开始\n\n*全栈的云 HPC 平台*\n\n- [1. 项目背景](#1-项目背景)\n- [2. 核心组件](#2-核心组件)\n- [3. 如何部署、使用、卸载、构建与开发](#3-如何部署使用卸载构建与开发)\n- [4. Bug与技术沟通](#4-bug与技术沟通)\n\n### 1. 项目背景\n\n云上的高性能计算（ Cloud High-Performance Computing, Cloud HPC ），与本地集群相比，有多方面的区别。其中最为显著的区别，就是云资源的弹性、动态与灵活性。理论上讲，您可以轻松使用多个云计算机房的计算、存储、网络资源，您的超算集群规模可以扩展到数百、数千核心，也可能根据实际情况减小到 0 个计算核心。这种动态特点就意味着，您在一定程度上需要关心底层资源 —— 例如集群的规模、集群的数量等。\n\n然而，资源管理对于广大的 Cloud HPC 用户来说，是比较陌生的；尤其是云资源的管理，更是无从下手。云提供海量的底层资源，但是 Cloud HPC 用户首先需要让这些资源组成具备超级计算能力的集群。这也就意味着需要一个  **构建和运行** 的过程。这个过程，对于大多数用户来说，难度非常高，原因在于它涉及到多方面的 IT 应用知识和云计算技能，包括但不限于：\n\n- 什么是网络、虚拟私有网络、子网、网段、ACL、公有和私有 IP ……\n- 什么是云主机、主机镜像、安全组、公钥私钥 ……\n- 什么是云硬盘、文件存储、对象存储、挂载不同类型的存储 ……\n- 什么是任务调度器、调度器如何配置和运行、超算用户如何使用和提交任务 ……\n- ……\n\n以上种种，都成为了制约 Cloud HPC 走向更广阔应用的一道门槛。目前，这些工作要么是由云厂商以自研的方式、面向 HPC 用户推出自有的 Cloud HPC 服务，要么是由第三方服务商进行集成开发之后以类 SaaS 平台的方式提供给客户。无论是哪一种方案，都旨在将客户留在各自的平台上；尽管在商业方面是完全合理而且完全值得尊重的，但是无形之中会限制最终用户对于超算的掌控权以及选择 Cloud HPC 服务的自由度。\n\n为了打造全面开放且超级简单的 Cloud HPC 平台，让用户能够以极低的门槛开启 Cloud HPC，加速科研创新，我们打造了开源的云超算平台 HPC-NOW。这里，NOW 有两层含义：\n\n- 现在即可开始，无需等待\n- No Operation Workload，无需繁重的运行维护工作，即可在云上轻松管理您的超算集群和超算服务\n\n目前，HPC-NOW 服务支持的云平台列表如下：\n\n- 阿里云，Alibaba，Cloud，内部代号 CLOUD_A\n- 腾讯云，Tencent Cloud，内部代号 CLOUD_B\n- 亚马逊，Amazon Web Services，内部代号 CLOUD_C\n- 华为云，Huawei Cloud，内部代号 CLOUD_D\n- 百度云，Baidu BCE，内部代号 CLOUD_E\n- 微软云，Azure，内部代号 CLOUD_F\n- 谷歌云，Google Cloud Platform，内部代号 CLOUD_G\n- 火山引擎，Volcano Engine，内部代号 CLOUD_H\n\n希望与您共同携手打造开源、开放的 Cloud HPC 生态！\n\n本项目基于 MIT 协议开源，协议条款详见 COPYING 。\n\n### 2. 核心组件\n\n首先，在此鸣谢卓越、开源、面向多云的 IaC（ 基础设施即代码 ）平台 [Terraform](http://www.terraform.io)。正是 Terraform 强大的功能和良好的生态为 HPC-NOW 提供了坚实的基础，我们可以不必重复造轮子。本项目的核心工作，在于如何驱动 Terraform 来构建和管理一个或多个 Cloud HPC 集群。\n\n本项目的核心组件如下：\n\n-  **installer**  ：HPC-NOW 服务的安装器。主要负责服务的安装、卸载、更新三项工作。该安装器被设计为必须由管理员权限执行。\n-  **hpcopr**     ：意即 HPC Operator，是 HPC-NOW 的核心程序，也是用户需要执行的主程序。为了确保安全性和隔离性，该程序被设计为必须由专属 OS 用户 \"hpc-now\" 执行，其他用户，即使是管理员用户或者根用户也无法执行。由 hpcopr 管理基础设施代码，并调用 Terraform 对云资源进行全生命周期管理。\n-  **now-crypto** ：核心程序，主要作用是文本文件加密和解密，以确保敏感信息不以明文形式存放。请注意，该程序通过偏移字符的方式对文本信息进行随机修改，您的密文文件仍需要妥善保管。\n-  **hpcmgr**    ：强大的集群内管理工具，包括集群的连接、SLURM 服务的启动、以及 HPC 软件包的自动化编译安装等。\n-  **infra-as-code** ：模板文件是 IaC（基础设施及代码）的核心要素，我们已经针对阿里云、腾讯云、AWS、华为云、百度云、Azure（国际区）、Google Cloud Platform 七家公有云厂商制作了专用的资源模板。\n-  **scripts**    ：启动脚本包含了集群各个节点启动过程的编排，包括各类必要组件的自动化安装。\n\n### 3. 如何部署、使用、卸载、构建与开发\n\n请查阅 docs/UserManual-CN.pdf ，**请注意：该文档可能不是最新的**。\n\n### 4. Bug与技术沟通\n\n欢迎向该仓库提交 issues，或者邮箱联系 zhenrongwang@live.com | wangzhenrong@hpc-now.com | info@hpc-now.com","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzhenrong-wang%2Fhpc-now","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fzhenrong-wang%2Fhpc-now","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzhenrong-wang%2Fhpc-now/lists"}