{"id":20642502,"url":"https://github.com/aditeyabaral/calbert","last_synced_at":"2025-04-16T01:42:14.943Z","repository":{"id":41060387,"uuid":"420726011","full_name":"aditeyabaral/calbert","owner":"aditeyabaral","description":"CalBERT - Code-mixed Adaptive Language representations using BERT, published at AAAI-MAKE 2022","archived":false,"fork":false,"pushed_at":"2023-12-18T16:06:48.000Z","size":124114,"stargazers_count":12,"open_issues_count":7,"forks_count":3,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-04-11T20:44:02.454Z","etag":null,"topics":["bert","code-mixed","deep-learning","machine-learning","natural-language-processing","nlp","transformer"],"latest_commit_sha":null,"homepage":"https://ceur-ws.org/Vol-3121/short3.pdf","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/aditeyabaral.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2021-10-24T15:45:41.000Z","updated_at":"2024-04-11T13:33:49.000Z","dependencies_parsed_at":"2023-12-18T17:46:24.662Z","dependency_job_id":"d2980f4d-13c5-463b-bbf6-189110ecdafd","html_url":"https://github.com/aditeyabaral/calbert","commit_stats":{"total_commits":81,"total_committers":5,"mean_commits":16.2,"dds":0.2962962962962963,"last_synced_commit":"78ac4c7fc578e89de874bcb6e1812235ed38b6c1"},"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aditeyabaral%2Fcalbert","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aditeyabaral%2Fcalbert/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aditeyabaral%2Fcalbert/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aditeyabaral%2Fcalbert/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/aditeyabaral","download_url":"https://codeload.github.com/aditeyabaral/calbert/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":249182803,"owners_count":21226123,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bert","code-mixed","deep-learning","machine-learning","natural-language-processing","nlp","transformer"],"created_at":"2024-11-16T16:09:18.473Z","updated_at":"2025-04-16T01:42:14.906Z","avatar_url":"https://github.com/aditeyabaral.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# CalBERT - Code-mixed Adaptive Language representations using BERT\n\n\t\n[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/calbert-code-mixed-adaptive-language-1/sentiment-analysis-on-iitp-product-reviews)](https://paperswithcode.com/sota/sentiment-analysis-on-iitp-product-reviews?p=calbert-code-mixed-adaptive-language-1)\n\t\n[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/calbert-code-mixed-adaptive-language-1/sentiment-analysis-on-sail-2017)](https://paperswithcode.com/sota/sentiment-analysis-on-sail-2017?p=calbert-code-mixed-adaptive-language-1)\n\nThis repository contains the source code\nfor [CalBERT - Code-mixed Adaptive Language representations using BERT](http://ceur-ws.org/Vol-3121/short3.pdf),\npublished at AAAI-MAKE 2022, Stanford University.\n\nCalBERT can be used to adapt existing Transformer language representations into another similar language by minimising\nthe semantic space between equivalent sentences in those languages, thus allowing the Transformer to learn\nrepresentations for words across two languages. It relies on a novel pre-training architecture named Siamese Pre-training to learn task-agnostic and language-agnostic\nrepresentations. For more information, please refer to the paper.\n\nThis framework allows you to perform CalBERT's Siamese Pre-training to learn representations for your own data and can be used to obtain dense vector representations for words, sentences or paragraphs. The base models used to \ntrain CalBERT consist of BERT-based Transformer models such as BERT, RoBERTa, XLM, XLNet, DistilBERT, and so on. \nCalBERT achieves state-of-the-art results on the SAIL and IIT-P Product Reviews datasets. CalBERT is also one of the\nonly models able to learn code-mixed language representations without the need for traditional pre-training methods and \nis currently one of the few models available for Indian code-mixing such as Hinglish.\n\n# Installation\n\nWe recommend `Python 3.9` or higher for CalBERT.\n\n## Install PyTorch\n\nFollow [PyTorch - Get Started](https://pytorch.org/get-started/locally/) for further details on how to install PyTorch \nwith or without CUDA.\n\n## Install CalBERT\n\n### Install with pip\n   ```bash\n   pip install calbert\n   ```\n\n### Install from source\nYou can also clone the current version from the [repository](https://github.com/aditeyabaral/calbert) and then directly \ninstall the package.\n   ```bash\n   pip install -e .\n   ```\n\n# Getting Started\n\nYou can read the [docs](https://calbert.readthedocs.io/en/latest/) to learn more about how to train CalBERT for your own\nuse case.\n\nThe following example shows you how to use CalBERT to obtain sentence embeddings.\n\n# Training\n\nThis framework allows you to also train your own CalBERT models on your own code-mixed data so you can learn\nembeddings for your custom code-mixed languages. There are various options to choose from in order to get the best\nembeddings for your language.\n\nFirst, initialise a model with the base Transformer\n```python\nfrom calbert import CalBERT\nmodel = CalBERT('bert-base-uncased')\n```\n\nCreate a CalBERTDataset using your sentences\n```python\nfrom calbert import CalBERTDataset\nbase_language_sentences = [\n   \"I am going to Delhi today via flight\",\n   \"This movie is awesome!\"\n]\ntarget_language_sentences = [\n   \"Main aaj flight lekar Delhi ja raha hoon.\",\n   \"Mujhe yeh movie bahut awesome lagi!\"\n]\ndataset = CalBERTDataset(base_language_sentences, target_language_sentences)\n```\n\nThen create a trainer and train the model\n```python\nfrom calbert import SiamesePreTrainer\ntrainer = SiamesePreTrainer(model, dataset)\ntrainer.train()\n```\n\n# Performance\n\nOur models achieve state-of-the-art results on the SAIL and IIT-P Product Reviews datasets.\n\nMore information will be added soon.\n\n# Application and Uses\n\nThis framework can be used for:\n\n- Computing code-mixed as well as plain sentence embeddings\n- Obtaining semantic similarities between any two sentences\n- Other textual tasks such as clustering, text summarization, semantic search and many more.\n\n# Citing and Authors\n\nIf you find this repository useful, please cite our publication [CalBERT - Code-mixed Apaptive Language representations using BERT](http://ceur-ws.org/Vol-3121/short3.pdf).\n\n```bibtex\n@inproceedings{calbert-baral-et-al-2022,\n  author    = {Aditeya Baral and\n               Aronya Baksy and\n               Ansh Sarkar and\n               Deeksha D and\n               Ashwini M. Joshi},\n  editor    = {Andreas Martin and\n               Knut Hinkelmann and\n               Hans{-}Georg Fill and\n               Aurona Gerber and\n               Doug Lenat and\n               Reinhard Stolle and\n               Frank van Harmelen},\n  title     = {CalBERT - Code-Mixed Adaptive Language Representations Using {BERT}},\n  booktitle = {Proceedings of the {AAAI} 2022 Spring Symposium on Machine Learning\n               and Knowledge Engineering for Hybrid Intelligence {(AAAI-MAKE} 2022),\n               Stanford University, Palo Alto, California, USA, March 21-23, 2022},\n  series    = {{CEUR} Workshop Proceedings},\n  volume    = {3121},\n  publisher = {CEUR-WS.org},\n  year      = {2022},\n  url       = {http://ceur-ws.org/Vol-3121/short3.pdf},\n  timestamp = {Fri, 22 Apr 2022 14:55:37 +0200}\n}\n```\n\n# Contact\n\nPlease feel free to contact us by emailing us to report any issues or suggestions, or if you have any further\nquestions.\n\nContact: - [Aditeya Baral](https://aditeyabaral.github.io/), [aditeya.baral@gmail.com](mailto:aditeya.baral@gmail.com)\n\nYou can also contact the other maintainers listed below.\n\n- [Aronya Baksy](mailto:abaksy@gmail.com)\n- [Ansh Sarkar](mailto:anshsarkar1@gmail.com)\n- [Deeksha D](mailto:deekshad132@gmail.com)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faditeyabaral%2Fcalbert","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Faditeyabaral%2Fcalbert","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faditeyabaral%2Fcalbert/lists"}