{"id":13845941,"url":"https://github.com/mmmwhy/pure_attention","last_synced_at":"2025-07-12T03:33:12.246Z","repository":{"id":42991161,"uuid":"90704096","full_name":"mmmwhy/pure_attention","owner":"mmmwhy","description":"使用 attention 实现 nlp 和 cv 相关模型。","archived":false,"fork":false,"pushed_at":"2022-03-24T12:31:29.000Z","size":2910,"stargazers_count":804,"open_issues_count":0,"forks_count":643,"subscribers_count":59,"default_branch":"master","last_synced_at":"2024-07-30T14:19:11.576Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mmmwhy.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2017-05-09T04:59:18.000Z","updated_at":"2024-07-04T06:01:04.000Z","dependencies_parsed_at":"2022-08-29T23:21:01.223Z","dependency_job_id":null,"html_url":"https://github.com/mmmwhy/pure_attention","commit_stats":null,"previous_names":["mmmxcc/ss-panel-and-ss-py-mu"],"tags_count":7,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mmmwhy%2Fpure_attention","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mmmwhy%2Fpure_attention/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mmmwhy%2Fpure_attention/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mmmwhy%2Fpure_attention/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mmmwhy","download_url":"https://codeload.github.com/mmmwhy/pure_attention/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":225791465,"owners_count":17524794,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-04T17:04:02.988Z","updated_at":"2024-11-21T19:30:55.496Z","avatar_url":"https://github.com/mmmwhy.png","language":"Python","funding_links":[],"categories":["Python"],"sub_categories":[],"readme":"# 介绍\nattention 在 cv 和 nlp 领域都有很多的应用，比如在 cv 中，可以使用 detr 进行目标检测任务，使用 vit / mae 进行图片预训练任务。\n\n在 nlp 领域中的作用更不用提， bert 以及后续的更多工作将 attention 彻底的发扬光大。\n\ncv 和 nlp 中的很多方法和技巧也在相互影响，比如大规模的预训练、mask 的设计(mae 、vilbert)、自监督学习的设计(从 imageNet 做有监督的预训练到纯粹的自监督预训练)。\n\n这些方面都非常的有趣，我希望可以设计一个 backbone 结构，让其可以在 cv 任务和 nlp 任务上均取到 sota 的效果。\n\n从而为之后的任务提供一个 baseline。\n\n# 目标\n提供一套完整的的基础算法服务\n\n1、python 训练任务，包含 NLP 和 CV 任务。\n\n2、java 环境下使用 onnx 的在线推理部署。\n\n# todo\n第一阶段：实现 NLP 和 CV 的典型任务，并评估下游效果。\n- [x]  Pytorch 实现 Transformer 的 encode 阶段，并实现 bert ;\n\n  \u003e 参考 [transformers](https://github.com/huggingface/transformers) 的设计，但只保留与关键 encode 相关的代码，简化代码量。\n  保持与原始 huggingface encode 的结果一致, 使用方法和一致性校验可以参考 [backbone_bert](pure_attention/backbone_bert/README.md) 。\n\n  - [x] 提供 [transformers](https://github.com/huggingface/transformers) 中 [bert-base-chinese](https://huggingface.co/bert-base-chinese) 、[chinese-roberta-wwm-ext](https://huggingface.co/hfl/chinese-roberta-wwm-ext) 、[chinese-roberta-wwm-ext-large](https://huggingface.co/hfl/chinese-roberta-wwm-ext-large) 、[ernie 1.0](https://huggingface.co/nghuyong/ernie-1.0) 的国内下载镜像,  下载方式具体可参考 [transformers国内下载镜像](pure_attention/backbone_bert/README.md#transformers国内下载镜像) 。\n\n- [x]  Pytorch 实现 Transformer 的 decode 阶段，并实现 seq2seq 任务。\n  \u003e todo\n- [ ]  NLP 下游任务 序列标注、分类 的实现，并在公开数据集上进行评估，这里主要是想证明实现的 backbone 效果是符合预期的；\n  \u003e todo\n- [ ]  实现 Vit，并在下游任务上验证实现 Vit 的效果是否符合预期；\n  \u003e todo\n\n 第二阶段：增加 NLP 和 CV 的其余常见任务，扩增项目的能力范围。\n- [ ] UNILM；\n- [ ] MAE；\n- [ ] GPT系列；\n- [ ] seq2seq，搞一个翻译任务；\n- [ ] 实现模型的 onnx export； \n- [ ] 实现 java 下的 onnx 推理过程；\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmmmwhy%2Fpure_attention","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmmmwhy%2Fpure_attention","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmmmwhy%2Fpure_attention/lists"}