{"id":20779441,"url":"https://github.com/freedomintelligence/try_phoenix2","last_synced_at":"2026-06-20T16:32:31.329Z","repository":{"id":204476021,"uuid":"711932686","full_name":"FreedomIntelligence/try_Phoenix2","owner":"FreedomIntelligence","description":"Phoenix2 code in dev","archived":false,"fork":false,"pushed_at":"2023-10-30T14:25:58.000Z","size":69,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-10-23T18:38:12.258Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/FreedomIntelligence.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2023-10-30T13:18:18.000Z","updated_at":"2023-11-01T01:55:09.000Z","dependencies_parsed_at":null,"dependency_job_id":"8a8407dc-7e8b-45a6-be9d-ab5ea965a923","html_url":"https://github.com/FreedomIntelligence/try_Phoenix2","commit_stats":null,"previous_names":["wangxidong06/try_phoenix2"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/FreedomIntelligence/try_Phoenix2","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/FreedomIntelligence%2Ftry_Phoenix2","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/FreedomIntelligence%2Ftry_Phoenix2/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/FreedomIntelligence%2Ftry_Phoenix2/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/FreedomIntelligence%2Ftry_Phoenix2/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/FreedomIntelligence","download_url":"https://codeload.github.com/FreedomIntelligence/try_Phoenix2/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/FreedomIntelligence%2Ftry_Phoenix2/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34578089,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-20T02:00:06.407Z","response_time":98,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-17T13:27:56.526Z","updated_at":"2026-06-20T16:32:31.311Z","avatar_url":"https://github.com/FreedomIntelligence.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# try_Phoenix2\nPhoenix2 code in dev\n\n## Dependency\n- module load cuda11.8/toolkit/11.8.0\n\n- pip3 install -U xformers --index-url https://download.pytorch.org/whl/cu118\n- pip install packaging\n- pip uninstall -y ninja \u0026\u0026 pip install ninja\n- pip install flash-attn --no-build-isolation\n\n- pip install -r requirements.txt tokenizers sentencepiece\n\n## Structure\n\n- TinyLlama: 原来的预训练代码\n- TinyLlama_deepspeed_check1： 利用Deepspeed策略\n    - pip install deepspeed\n    - 文档:\n        - 代码: https://lightning.ai/docs/pytorch/stable/advanced/model_parallel/deepspeed.html\n        - 参数: https://lightning.ai/docs/pytorch/stable/api/lightning.pytorch.strategies.DeepSpeedStrategy.html#lightning.pytorch.strategies.DeepSpeedStrategy\n    - 改动: \n        - 更换策略: pretrain.py line 10, 95\n        - 为了用checkpointing技术: pretrain.py line 141\n        - scripts/convert_zero_checkpoint.py Zero3 checkpoint转换\n- TinyLlama_deepspeed_check2： 利用Deepspeed策略\n    - 改动: \n        - 为了用checkpointing技术: lit-gpt/model/gpt.foward line 69-119\n- TinyLlama_collosal： \n    - 改动: \n        - 更换策略: pretrain.py line 10, 95\n\n## Usage\n\n```\nsbatch multinode_pretrain.sh\n```\n\n### 数据\n```\npython scripts/prepare_starcoder.py --source_path /path/to/starcoderdata/ --tokenizer_path data/llama --destination_path data/slim_star_combined --split train --percentage 1.0\npython scripts/prepare_slimpajama.py --source_path /path/to/SlimPajama --tokenizer_path data/llama  --destination_path data/slim_star_combined --split validation --percentage 1.0\npython scripts/prepare_slimpajama.py --source_path /path/to/SlimPajama --tokenizer_path data/llama  --destination_path data/slim_star_combined --split train --percentage 1.0\n```\n\n### 多节点训练\n\n```\nsbatch multinode_pretrain.sh\n```\n\n\n## Calculate index for Strategy choosing\n```\npython pre_train_math.py\n```\n```\n-----------Model_Size and GPU_Mem-----------\n+--------------+------------------------+----------------------+\n| Model size/B | ratio(NHIDDEN/NLAYERS) | Usable_mem_per_GPU/G |\n+--------------+------------------------+----------------------+\n|     1.18     |           93           |          79          |\n+--------------+------------------------+----------------------+\n-----------With Mixed Precision(bp16)-----------\n-----Memory_reference_indicator(Batch_size=1)-----\n+-------------------------+----------+------------------+-------------------+\n| Module                  |   Size/B |   Eval_memory/GB |   Train_momery/GB |\n+=========================+==========+==================+===================+\n| emb                     |     0.07 |             0.14 |              1.12 |\n+-------------------------+----------+------------------+-------------------+\n| one_layer               |     0.05 |             0.1  |              0.81 |\n+-------------------------+----------+------------------+-------------------+\n| input                   |     0    |             0.01 |              0.01 |\n+-------------------------+----------+------------------+-------------------+\n| activation(batchsize=1) |     1.77 |             3.54 |              3.54 |\n+-------------------------+----------+------------------+-------------------+\n| ALL                     |     2.95 |             5.91 |             22.39 |\n+-------------------------+----------+------------------+-------------------+\n-----Strategy_reference_indicator(Batch_size=1)-----\n+------------+--------------------------+---------------------------+\n| Strategy   |   Eval_memory_per_gpu/GB |   Train_momery_per_gpu/GB |\n+============+==========================+===========================+\n| Zero1      |                     2.35 |                      8.44 |\n+------------+--------------------------+---------------------------+\n| Zero2      |                     2.35 |                      6.11 |\n+------------+--------------------------+---------------------------+\n| Zero3      |                     0.03 |                      3.79 |\n+------------+--------------------------+---------------------------+\n---------------------Strategy_Recommand---------------------\nRecommand_Strategy:\n+--------+------+------+------+---------------------------+-----------------+\n| Zero   |   DP |   TP |   PP |   Train_momery_per_gpu/GB |   Trianing_days |\n+========+======+======+======+===========================+=================+\n| Zero1  |   80 |    1 |    1 |                      8.44 |            0.01 |\n+--------+------+------+------+---------------------------+-----------------+\nPlease find the best batch_size by adjusting BATCH_SIZE\n```\n\n\n## To do list\nUse Deepspeed/ Collasal AI/ Megatron-deepspeed to reconstruct the code respectively. ","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffreedomintelligence%2Ftry_phoenix2","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ffreedomintelligence%2Ftry_phoenix2","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffreedomintelligence%2Ftry_phoenix2/lists"}