{"id":13595165,"url":"https://github.com/ZhuiyiTechnology/simbert","last_synced_at":"2025-04-09T10:33:02.072Z","repository":{"id":37418586,"uuid":"264445386","full_name":"ZhuiyiTechnology/simbert","owner":"ZhuiyiTechnology","description":"a bert for retrieval and generation","archived":false,"fork":false,"pushed_at":"2021-02-26T02:47:55.000Z","size":21,"stargazers_count":843,"open_issues_count":28,"forks_count":150,"subscribers_count":8,"default_branch":"master","last_synced_at":"2024-11-06T17:46:13.435Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ZhuiyiTechnology.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-05-16T13:39:34.000Z","updated_at":"2024-11-05T08:53:07.000Z","dependencies_parsed_at":"2022-07-07T23:11:16.470Z","dependency_job_id":null,"html_url":"https://github.com/ZhuiyiTechnology/simbert","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ZhuiyiTechnology%2Fsimbert","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ZhuiyiTechnology%2Fsimbert/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ZhuiyiTechnology%2Fsimbert/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ZhuiyiTechnology%2Fsimbert/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ZhuiyiTechnology","download_url":"https://codeload.github.com/ZhuiyiTechnology/simbert/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248020593,"owners_count":21034459,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-01T16:01:45.154Z","updated_at":"2025-04-09T10:32:57.064Z","avatar_url":"https://github.com/ZhuiyiTechnology.png","language":"Python","readme":"# SimBERT\n基于UniLM思想、融检索与生成于一体的BERT模型。\n\n权重下载：https://github.com/ZhuiyiTechnology/pretrained-models\n\n## 模型简介\n\n\u003cimg src=\"https://kexue.fm/usr/uploads/2020/05/2840550561.png\" width=600 alt=\"SimBERT训练方式示意图.png\" /\u003e\n\n假设SENT_a和SENT_b是一组相似句，那么在同一个batch中，把\u003cstong\u003e[CLS] SENT_a [SEP] SENT_b [SEP]\u003c/stong\u003e和\u003cstong\u003e[CLS] SENT_b [SEP] SENT_a [SEP]\u003c/stong\u003e都加入训练，做一个相似句的生成任务，这是Seq2Seq部分。\n\n另一方面，把整个batch内的[CLS]向量都拿出来，得到一个bxd的句向量矩阵V（b是batch_size，d是hidden_size），然后对d维度做l2归一化，得到新的V，然后两两做内积，得到bxv的相似度矩阵VV^T，接着乘以一个scale（我们取了30），并mask掉对角线部分，最后每一行进行softmax，作为一个分类任务训练，每个样本的目标标签是它的相似句（至于自身已经被mask掉）。说白了，就是把batch内所有的非相似样本都当作负样本，借助softmax来增加相似样本的相似度，降低其余样本的相似度。\n\n详细介绍请看：[https://kexue.fm/archives/7427](https://kexue.fm/archives/7427)\n\n## 训练环境\ntensorflow 1.14 + keras 2.3.1 + bert4keras 0.7.7\n\n## 如何引用\n\nBibtex：\n\n```tex\n@techreport{simbert,\n  title={SimBERT: Integrating Retrieval and Generation into BERT},\n  author={Jianlin Su},\n  year={2020},\n  url=\"https://github.com/ZhuiyiTechnology/simbert\",\n}\n```\n\n ## 联系我们\n\n邮箱：ai@wezhuiyi.com\n\n## 相关链接\n\n追一科技：https://zhuiyi.ai\n","funding_links":[],"categories":["Pretrained Language Model","Python"],"sub_categories":["Repository"],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FZhuiyiTechnology%2Fsimbert","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FZhuiyiTechnology%2Fsimbert","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FZhuiyiTechnology%2Fsimbert/lists"}