{"id":18533660,"url":"https://github.com/lonepatient/albert_pytorch","last_synced_at":"2025-04-05T00:07:29.820Z","repository":{"id":106806966,"uuid":"211853338","full_name":"lonePatient/albert_pytorch","owner":"lonePatient","description":"A Lite Bert For Self-Supervised Learning Language Representations","archived":false,"fork":false,"pushed_at":"2020-05-13T01:36:34.000Z","size":290,"stargazers_count":716,"open_issues_count":31,"forks_count":150,"subscribers_count":12,"default_branch":"master","last_synced_at":"2025-03-28T23:05:53.077Z","etag":null,"topics":["albert","bert","language-model","mask","ngram","nlp","pytorch"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/lonePatient.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2019-09-30T12:21:14.000Z","updated_at":"2025-03-25T11:30:57.000Z","dependencies_parsed_at":"2023-07-30T08:16:40.484Z","dependency_job_id":null,"html_url":"https://github.com/lonePatient/albert_pytorch","commit_stats":null,"previous_names":[],"tags_count":2,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lonePatient%2Falbert_pytorch","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lonePatient%2Falbert_pytorch/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lonePatient%2Falbert_pytorch/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lonePatient%2Falbert_pytorch/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/lonePatient","download_url":"https://codeload.github.com/lonePatient/albert_pytorch/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247266563,"owners_count":20910836,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["albert","bert","language-model","mask","ngram","nlp","pytorch"],"created_at":"2024-11-06T19:12:33.975Z","updated_at":"2025-04-05T00:07:29.798Z","avatar_url":"https://github.com/lonePatient.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"[**English Version**](./README.md) | [**中文版说明**](./README_zh.md)\n\n## albert_pytorch\n\nThis repository contains a PyTorch implementation of the albert model from the paper \n\n[A Lite Bert For Self-Supervised Learning Language Representations](https://arxiv.org/pdf/1909.11942.pdf)\n\nby Zhenzhong Lan. Mingda Chen....\n\n## Dependencies\n\n- pytorch=1.10\n- cuda=9.0\n- cudnn=7.5\n- scikit-learn\n- sentencepiece\n\n## Download Pre-trained Models of English\n\nOfficial download links: [google albert](https://github.com/google-research/ALBERT)\n\nAdapt to this version，download pytorch model (google drive):\n\n**v1**\n\n- [albert_base_v1.zip](https://drive.google.com/open?id=1dVsVd6j8rCTpqF4UwnqWuUpmkhxRkEie)\n- [albert_large_v1.zip](https://drive.google.com/open?id=18dDXuIHXYWibCLlKX5_rZkFxa3VSc5j1)\n- [albert_xlarge_v1.zip](https://drive.google.com/open?id=1jidZkLLFeDuQJsXVtenTvV_LU-AYprJn)\n- [albert_xxlarge_v1.zip](https://drive.google.com/open?id=1PV8giuCEAR2Lxaffp0cuCjXh1tVg7Vj_)\n\n**v2**\n\n- [albert_base_v2.zip](https://drive.google.com/open?id=1byZQmWDgyhrLpj8oXtxBG6AA52c8IHE-)\n- [albert_large_v2.zip](https://drive.google.com/open?id=1KpevOXWzR4OTviFNENm_pbKfYAcokl2V)\n- [albert_xlarge_v2.zip](https://drive.google.com/open?id=1W6PxOWnQMxavfiFJsxGic06UVXbq70kq)\n- [albert_xxlarge_v2.zip](https://drive.google.com/open?id=1o0EhxPqjd7yRLIwlbH_UAuSAV1dtIXBM)\n\n## Fine-tuning\n\n１. Place `config.json` and `30k-clean.model` into the `prev_trained_model/albert_base_v2` directory.\nexample:\n```text\n├── prev_trained_model\n|  └── albert_base_v2\n|  |  └── pytorch_model.bin\n|  |  └── config.json\n|  |  └── 30k-clean.model\n```\n2．convert albert tf checkpoint to pytorch\n```python\npython convert_albert_tf_checkpoint_to_pytorch.py \\\n    --tf_checkpoint_path=./prev_trained_model/albert_base_tf_v2 \\\n    --bert_config_file=./prev_trained_model/albert_base_v2/config.json \\\n    --pytorch_dump_path=./prev_trained_model/albert_base_v2/pytorch_model.bin\n```\nThe [General Language Understanding Evaluation (GLUE) benchmark](https://gluebenchmark.com/) is a collection of nine sentence- or sentence-pair language understanding tasks for evaluating and analyzing natural language understanding systems.\n\nBefore running anyone of these GLUE tasks you should download the [GLUE data](https://gluebenchmark.com/tasks) by running [this script](https://gist.github.com/W4ngatang/60c2bdb54d156a41194446737ce03e2e) and unpack it to some directory $DATA_DIR.\n\n3．run `sh scripts/run_classifier_sst2.sh`to fine tuning albert model\n\n## Result\n\nPerformance of ALBERT on GLUE benchmark results using a single-model setup on **dev**:\n\n|  | Cola| Sst-2| Mnli| Sts-b|\n| :------- | :---------: | :---------: |:---------: | :---------: |\n| metric | matthews_corrcoef |accuracy |accuracy | pearson |\n\n| model | Cola| Sst-2| Mnli| Sts-b|\n| :------- | :---------: | :---------: |:---------: | :---------: |\n| albert_base_v2 | 0.5756 | 0.926 | 0.8418 | 0.9091 |\n| albert_large_v2 | 0.5851 |0.9507 |  |0.9151 |\n| albert_xlarge_v2 | 0.6023 | |  |0.9221 |\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flonepatient%2Falbert_pytorch","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flonepatient%2Falbert_pytorch","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flonepatient%2Falbert_pytorch/lists"}