{"id":28581449,"url":"https://github.com/aqlaboratory/llnl-docs","last_synced_at":"2026-01-29T09:10:33.424Z","repository":{"id":280491286,"uuid":"926653358","full_name":"aqlaboratory/llnl-docs","owner":"aqlaboratory","description":null,"archived":false,"fork":false,"pushed_at":"2025-03-05T18:17:29.000Z","size":55,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2026-01-13T18:26:25.993Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/aqlaboratory.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-02-03T16:28:09.000Z","updated_at":"2025-03-05T18:17:33.000Z","dependencies_parsed_at":null,"dependency_job_id":"133d1d94-4fdc-4c7d-8274-c5ece43cc523","html_url":"https://github.com/aqlaboratory/llnl-docs","commit_stats":null,"previous_names":["aqlaboratory/llnl-docs"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/aqlaboratory/llnl-docs","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aqlaboratory%2Fllnl-docs","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aqlaboratory%2Fllnl-docs/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aqlaboratory%2Fllnl-docs/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aqlaboratory%2Fllnl-docs/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/aqlaboratory","download_url":"https://codeload.github.com/aqlaboratory/llnl-docs/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aqlaboratory%2Fllnl-docs/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28769887,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-26T07:45:00.504Z","status":"ssl_error","status_checked_at":"2026-01-26T07:45:00.070Z","response_time":59,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-06-11T04:17:34.163Z","updated_at":"2026-01-29T09:10:33.415Z","avatar_url":"https://github.com/aqlaboratory.png","language":null,"funding_links":[],"categories":[],"sub_categories":[],"readme":"# llnl-docs (GLM-geared)\n\n## 0. Ask Mohammed to be onboarded to LLNL. \n\nYou will receive an email from Barry Chen (most likely) and will need to provide your address for them to ship you an RSA token. \n\n## 0.5. Check your mail for a physical RSA token. \n\nMeet with Barry following mail receipt of the RSA token. \n\n## 1. ssh onto an LLNL compute cluster, accessing a compute node like [Dane](https://hpc.llnl.gov/hardware/compute-platforms/dane):\n```\ndhcp-10-118-155-112:~ mayavenkatraman$ ssh venkatraman2@dane.llnl.gov\n```\nYou will be prompted for your 8-digit pin appended to the 6-digit number on the RSA token you received in the mail.  \n\n## 2. ssh onto Tuolomne, which has the same architecture as exascale supercomputer El Capitán.\n```\n[venkatraman2@dane6:~]$ ssh tuolumne.llnl.gov\n```\n\n## 2.5 Set up your environment using mamba.\n\n### Install mamba using miniforge.\n\nMiniforge is an edition of Conda that only uses free and openly-licensed packages from the conda-forge project. It is already available on LLNL. Conda is an open-source package and environment manager that helps users install, update, and remove software. It allows you to isntall software WITHOUT needing root access (`sudo`), which is critical on systems like LLNL. \n\nMamba is like Conda, but more efficient. \n```conda install -n base -c conda-forge mamba```\n\nCheck that mamba is installed correctly by running\n```mamba --version```\n\n### Optional: Improve your shell\n\nRun `mamba install oh-my-bash` or some variant. \n\n### Install tmux\n\nRun `mamba install tmux` or some variant.\n\n### Optimize vim \n\n#### Vim deletion issue on Mac\n\nIf you are on Mac, you might observe that whenever you try to delete characters in vim, the terminal adds `^?`. By default, the macOS Terminal and some remote SSH sessions send ^? for Delete, which Vim does not interpret as a delete command. Run `stty erase ^?` to circumvent this. \n\nTo make this change permanent, run `echo 'stty erase ^?' \u003e\u003e ~/.bashrc`.\n\n#### Enable scrolling with mouse\n\nRun `echo \"set mouse=a\" \u003e\u003e ~/.vimrc`.\n\n### Cache your git credentials\n\nAdd your git userame and personal access token to `~/.git-credentials` like so:\n`echo \"https://your-username:your-personal-access-token@gitlab.com\" \u003e\u003e ~/.git-credentials`\n\nThis way, you won't have to log in every time you pull or push. \n\n## 3. Find GLM model of interest\n\nTo find a model, check out `/p/vast1/OpenFoldCollab/genome_lm/training_output/`. It is probably there. \n\n## 4. Use sbatch to run a script.\n\nLike on Manitou, you can use sbatch to run a script.\n\nAn example training script `train_model.sh` can be found in `/p/vast1/OpenFoldCollab/genome_lm/experiments/emb_exp/test_fsdp/train_model.sh`.\n\nRun it like so:\n```\n[venkatraman2@tuolumne1003:scripts]$ sbatch train_model.sh \nfaKoR2uSfgB\n```\n## 5. Check on the success of your job using your username\n`[venkatraman2@tuolumne1004:scripts]$ squeue -u {username}`\n\nNOTE: If your job has an \"S\" under \"ST\" (Status), this means \"Suspended.\" It is possible that all jobs on Tuolomne are being suspended, due to maintenance. \n\n### Check whether all jobs are suspended\n\nBy running squeue without specifying a username:\n`[venkatraman2@tuolumne1004:scripts]$ squeue`\n\n### To search for your job by its id\n`flux jobs | grep {job_id}`\n\n## 6. Kill jobs you don't want\n\nRun `scancel {job_id}`.\n\n# Debugging\n\nIf your job fails silently (many of mine did at first), try the following.\n\n### 1. Add echo statements to your original script. \n\n### 2. Try running in interactive mode using `srun` rather than `sbatch`. \n\n#### Example: training glm using srun\n```\nsrun python /p/vast1/OpenFoldCollab/genome_lm/glm/glm/train/training.py \\\n  --config-yaml=\"/g/g14/venkatraman2/scripts/mvenkat_glm_12l_20k.yaml\" \\\n  --limit-val-batches 50     --inference-mode-off     --skip-last-val \\\n  --compile-off\n```\n\n### 3. Consider changing the `pl_strategy` defined in your model config.\n\n`ModelSeqParallelStrategy` seems much less reliable than `DDP`, so if your `pl_strategy` `class` is set to `ModelSeqParallelStrategy`, try changing it.\n\n### 4. If debugging NCCL errors, consider the following.\n\n#### a. Set `export NCCL_ASYNC_ERROR_HANDLING=1`\n\nNormally, when NCCL detects an error (e.g., a timeout, GPU failure, or network issue), it does not immediately report the error. Instead, it waits for all ranks to reach the same failure point, which can cause deadlocks where some GPUs hang indefinitely.\n\nWith `NCCL_ASYNC_ERROR_HANDLING=1:`\n\n- NCCL immediately detects errors and reports them asynchronously.\n- If a GPU fails or hangs, the job is aborted automatically instead of waiting for other ranks.\n- This prevents silent hangs in distributed training jobs.\n\n## Selective Learning\n\nDirectory: `/p/vast1/OpenFoldCollab/genome_lm/experiments/SL-GLM_exp`.\n\nExperiment 1: `/SL-GLM_exp/02.10.2025_experiment_1`.\n\nSubmission/training scripts: `/02.10.2025_experiment_1/submit_SL/`.\n\nCheck `/02.10.2025_experiment_1/configs_SL/esm3s_12l_varlen20k_spanmask01_student_teacher_token.yaml` for the token selection config.\n* Note the `student_teacher:` key\n\nCurrently SL can only handle standard-rope so set data params like `return_contig_indices: false`  correctly in both student and teacher configs. The key in ['student_teacher']['selection_scheme']  can only have two values — token or batch . You can see the batch selection config in `configs_SL/esm3s_12l_varlen20k_spanmask01_student_teacher_batch.yaml`.\n\n## FSDP Training\n\nTo test FSDP, you need to change `pl_strategy` in the model config. You must also specify args that multiply to the world size.\n```\npl_strategy:\n  class:  ModelSeqParallelStrategy\n  args:\n    data_parallel_size: 64 \n    sequence_parallel_size: 1\n    tensor_parallel_size: 1\n```\n\nThis is a valid setting for args if `trainer` is configured like so:\n```\ntrainer:\n    log_every_n_steps: 400  # Log every n steps\n    max_steps: 210005 # Maximum steps\n    precision: bf16-mixed # Precision\n    gradient_clip_val: # Gradient clip value\n    devices: 4 # Devices\n    num_nodes: 16 # Number of nodes\n```\n\n\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faqlaboratory%2Fllnl-docs","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Faqlaboratory%2Fllnl-docs","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faqlaboratory%2Fllnl-docs/lists"}