{"id":20713358,"url":"https://github.com/linaro/hpc_lustreci","last_synced_at":"2025-07-24T05:35:55.999Z","repository":{"id":90096723,"uuid":"221906047","full_name":"Linaro/hpc_lustreci","owner":"Linaro","description":"HPC-SIG Lustre CI","archived":false,"fork":false,"pushed_at":"2020-01-31T15:14:33.000Z","size":69,"stargazers_count":3,"open_issues_count":0,"forks_count":4,"subscribers_count":5,"default_branch":"master","last_synced_at":"2025-04-09T03:11:23.178Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Linaro.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-11-15T11:00:37.000Z","updated_at":"2024-01-30T18:47:38.000Z","dependencies_parsed_at":"2024-04-21T04:50:00.805Z","dependency_job_id":null,"html_url":"https://github.com/Linaro/hpc_lustreci","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/Linaro/hpc_lustreci","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Linaro%2Fhpc_lustreci","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Linaro%2Fhpc_lustreci/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Linaro%2Fhpc_lustreci/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Linaro%2Fhpc_lustreci/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Linaro","download_url":"https://codeload.github.com/Linaro/hpc_lustreci/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Linaro%2Fhpc_lustreci/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":266796865,"owners_count":23985487,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-07-24T02:00:09.469Z","response_time":99,"last_error":null,"robots_txt_status":null,"robots_txt_updated_at":null,"robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-17T02:24:49.828Z","updated_at":"2025-07-24T05:35:55.991Z","avatar_url":"https://github.com/Linaro.png","language":"Shell","funding_links":[],"categories":[],"sub_categories":[],"readme":"# HPC-SIG Lustre CI recipes\n\n## General Overview\n\nThis repository contains recipes to build and test Lustre on AArch64 platforms.\nIt also contains a script and kickstart file(s) to provision an environment (a VM) with the necessary requirements to **build**, **install** and **test** Lustre from its git repository.\nCurrently, the supported platform for **building**, **installing** and **testing** is **CentOS8**.\nThere are also *old* scripts to build on **RHEL8** and **build the client only** on **CentOS 7 altarch**.\n\nBut it is **highly** recommended to use CentOS8.\n\nPlease also note that since Lustre Server's **LDISKFS** requires modifications to the *ext4 kernel driver sources*, it is tied to the kernel version.\nAs of the writing of this README, **on CentOS8 AArch64, the latest kernel supported is 4.18.0-80.11.2 (that is the latest 8.0 release)**.\n\n## Requirements\n\nThe script to install the VMs : *centos8vm/create_centos8_qemuvirt.sh* installs its own software dependencies to install the VM (that would be libvirt, qemu-kvm, virt-install and related dependencies).\nOne thing it **omits is the installation of Mellanox OFED stack**, and the configuration of the Virtual Functions. The configuration of the Virtual Functions can be done using *helper_scripts/ib-enable-sriov.sh* (Thanks to @rafaeldtinoco).\n\nPlease note that as of the writing of this README, **SRIOV/Virtual Functions proper operation requires the M(ellanox)OFED stack to be installed on CentOS7, else the VM UEFI will get stuck on initializing the VF** (*tested on ThunderX2*).\n\nTo summarize, the only hard requirement on the machine side is to have **MOFED installed (and be sure the VFs do work)**.\nAnother requirement, on the infrastructure side, is to have a fileserver/webserver able to serve the resulting RPMs from Lustre building (for the install and subsequent testing), as well as the kernel sources (to build Lustre against).\nThe kickstart files also assume that you are using [MrProvisioner](https://github.com/mr-provisioner/mr-provisioner) as a provisioner (as well as URLs corresponding to the HPC-SIGs)\nThe kickstart file containing all requirements for Lustre building, installing and testing is **centos8vm/Centos8.Upstream.LustreVM.ks**\n\n*Note: this could be substituted by a PXE/TFTP server and a web server able to jinja template the correct variables into the kickstart and serve it*\n\n*Note2: it would require some changes to the URLs, name of the initrd/kernel images and no doubt variables in the script as well as the kickstarts*\n\n\n## Operation of the recipes\n\n### I - Installing the VMs\n\nTo install the VMs, make sure the above requirements are satisfied.\nOnce done, as a user that as full access to the hypervisor :\n```bash\n$ VMNUM=\"The index of the VM to be created : an integer is expected, default is 1\"\n$ IBVF=\"The index of the Virtual Function to be used by the VM : an integer is expected, default is 16-$VMNUM\"\n$ centos8vm/create_centos8_vm.sh\n```\n\nThis operation will create the VM alongside two disks : a 45GB and a 15GB one.\nThe 45GB one is sda which CentOS8 will boot on.\nThe 15GB one, sdb, is the disk reserved for Lustre.\n\nPlease note that the LustreVM kickstart also downgrades the kernel to the latest supported version and blacklists it in yum.conf.\nThis requires a repository containing the full set of RPMs.\n\n### II - Building Lustre\n\nRequires user with sudo access (preferably passwordless):\n```bash\n$ ./build_lustre_centos8_latest.sh\n```\n\nThe Lustre (Server **and** client) build done here builds both **LDISKFS** and **ZFS** support.\nThus it builds ZFS, which is not available in the upstream repositiories for AArch64.\nCurrently, the version built by the scripts is the latest : **0.8.2** (*so no SPL*)\nZFS requires access to the Kernel sources.\n(*Author's note: the build probably can be streamlined since it seems to repeat some steps*)\n\nIf ZFS is already installed, it will not build it (*this can be easily toggled with a '!' placed accordingly*).\n\nIt also builds Lustre's accompanying e2fsprogs as well as packages it (*following upstream documentation*).\nNote that as of the writing of this README, the version built is above CentOS8's default one, so it upgrades easily.\nAlso note that, as is the case with ZFS, building can be avoided with a '!' placed accordingly.\n(*Author's note: e2fsprogs' configuration takes quite a while*)\n\nOnce the above are built, Lustre is built. First the client, then the server. It is built from upstream Whamcloud's git master (but if a sourcetree is already there the script will not overwrite it).\nThe server requires access to the **FULL** kernel sources (in CentOS7, debuginfo contained those, but CentOS8 doesn't seem to).\nAs a rule of thumb, always have the SRPM of the kernel lying somewhere (even though they might be hard to track down...)\n\nEvery RPM produced is put into a directory ($DIR_REPO) and createrepo is run against it.\nA .repo file is added to yum and installation of ZFS and e2fsprogs is done that way.\n**NOTE: Lustre is not installed at the end of the build**\n\n### III - Installing Lustre\n\nRequires user with sudo access (preferably passwordless):\n```bash\n$ ./install_lustre_centos8.sh\n```\n\nThis step is the only one that requires some human intervention :\nOne needs to get the IP of the nodes provisioned using the kickstart to serve as testing (note that it should be displayed on the login prompt, for ease of use).\nThen one needs to fill the centos8vm/cluster_template with the IPs, hostnames, and partition name of the Lustre dedicated volume.\n\nThe RPMs produced by the above process have to put on a webserver to be served to the installation environment.\n\nThe installation script is intended to be run on each node of the testing cluster before testing.\nThus, not only does it install Lustre but it also installs requirements, amends the hosts file, puts the testing configuration in the correct place.\n\n**NOTE: One of the requirements is pdsh, which is not present in the upstream CentOS8 repos as far as I'm aware. Thankfully, building the rpms for it is straightforward**\n\nCurrently the test configuration assumes 1 MDS/MGS, 1 OSS, 1 Client (in addition to the MDS/MGS node acting as a client).\nIt also assumes LDISKFS as a backend.\n\n### IV - Testing Lustre\n\nRequires user with sudo access (preferably passwordless):\n```bash\n$ vim templates/cluster_template\n$ ./testing_lustre_centos8.sh\n```\n\nThis script is intended to be run on **ONLY** one node of the testing cluster **AT ANY ONE TIME**.\nIt will first bring up the Lustre cluster with llmount.sh (in the case of any errors at this stage, make sure to either run llmountcleanup.sh, or reboot all nodes).\nThen it will run the sanity test suite.\n*Note: it is run directly, not through auster*\n\n## Conclusion\n\nThese recipes serve as a basis for running CI on Lustre.\nThere is still work to be done to fully automate the pipeline, most notably :\n\n- [ ] Automate IP (rest can be infered by Jenkins) recuperation for templating\n- [ ] Automate transfer of produced RPMs to the webserver\n- [ ] Automate recuperation of test suite results, eventual kdumps and error reports to some place\n- [ ] Automate booting back up of the VMs after they have been provisioned\n- [ ] Ansiblelize what makes sense to be transposed to Ansible\n- [ ] Add ZFS backend support for testing configuration \n- [ ] Add InfiniBand link support for testing configuration (via VM's VFs) \n- [ ] Add multiple OSS, MDS/MGS support in testing configuration\n- [ ] Add support for testing at scale\n\nAlso, openSUSE support still needs to be added.\n\n## Note to developers\n\nI have tried to make the scripts as easily readable as possible, and did strive for adaptability (although it could be a lot better).\nThe goal here is to have a lean, mean and straightforward approach to build this stack. \nSo I have been quite lazy and haven't done a full write up on all the variables in the scripts (since they should be quite evident, and are *only* spread around 4 files).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flinaro%2Fhpc_lustreci","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flinaro%2Fhpc_lustreci","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flinaro%2Fhpc_lustreci/lists"}