awesome-nlp

:book: A curated list of resources dedicated to Natural Language Processing (NLP)
https://github.com/keon/awesome-nlp

Last synced: 9 days ago
JSON representation

Datasets
- Speech and Text
  - HuggingFace Datasets Hub - the central index for modern NLP datasets, with versioned, streamable loaders.
  - The Pile - 825 GiB diverse text corpus.
  - RedPajama / RedPajama-V2 - 2024) - reproductions of LLaMA pretraining data; V2 is 30T tokens with quality signals.
  - Dolma - 2024) - 3T-token open pretraining corpus with documented filtering pipeline.
  - FineWeb / FineWeb-Edu - 15T-token cleaned web corpus; FineWeb-Edu filters for educational quality.
  - CulturaX - 6.3T tokens across 167 languages.
  - Common Corpus - 2T-token open-license multilingual corpus.
  - Tülu 3 SFT Mixture - open instruction-tuning data behind Tülu 3.
  - tiny_qa_benchmark_pp - tiny NLP multi-lingual QA datasets and library to generate your own synthetic copies.
Language Models for NLP
- Bias, Fairness, Safety in NLP
  - StereoSet - measuring stereotypical bias in pretrained LMs.
  - CrowS-Pairs - social bias measurement in masked LMs.
  - WinoBias - gender bias in coreference resolution.
  - HolisticBias - bias measurement across many demographic axes.
  - RealToxicityPrompts - toxicity in LM generation.
  - Sycophancy in Language Models - models tailoring answers to user beliefs.
  - Alignment Faking in Large Language Models - models strategically complying during training.
  - WildGuard - open safety moderation model and benchmark.
  - Emergent Misalignment - finetuning on a narrow task (insecure code) unexpectedly produces broad alignment failures across unrelated domains.
  - SafeDialBench - multilingual (Chinese/English) safety benchmark of 4000+ multi-turn dialogues across 22 scenarios and 7 jailbreak strategies.
  - TeleAI-Safety - modular jailbreak evaluation framework integrating 19 attacks, 29 defenses, and 19 evaluation methods across 14 models and 12 risk categories.
  - IndicSafe - multilingual safety benchmark across 12 Indic languages; reveals 12.8% cross-language agreement, with over-refusal in low-resource scripts.
  - VLAF: Value-Conflict Alignment Faking - alignment faking occurs in models as small as 7B in 37% of cases when policy conflicts with internalized values; steering-vector mitigation reduces it 94%.
- Efficient and Small Language Models
  - DistilBERT - distilled encoders for production NLP.
  - Phi-3 / Phi-4 - small models trained on curated data, competitive with much larger ones on NLP benchmarks.
  - SmolLM2 - fully open small-LM family with reproducible training data.
  - SmolLM3 - 3B fully open decoder pretrained on 11.2T tokens with NoPE and YaRN for 128K context; competitive with 4B-class models.
  - Gemma 3 Technical Report - 1B-27B open models with high local-to-global attention ratio to keep KV-cache tractable at 128K context.
  - Qwen3 Technical Report - dense and MoE models 0.6B-235B with unified thinking/non-thinking modes; the 30B-A3B MoE matches larger dense models while activating only 3B parameters.
  - Apple Intelligence Foundation Language Models - on-device 3B model using KV-cache sharing and 2-bit QAT for 37.5% cache memory reduction without accuracy loss.
  - Sentence-Transformers - sentence and paragraph embeddings via Siamese BERT.
  - GTE - compact text embedding models near the top of MTEB.
  - GPTQ - post-training quantization for transformers.
  - AWQ - activation-aware weight quantization.
  - KVTuner - sensitivity-aware layer-wise mixed-precision KV-cache quantization; up to 21% throughput improvement over uniform KV8.
  - GGUF / llama.cpp - portable quantized inference.
  - vLLM - PagedAttention-based high-throughput LM serving.
  - SGLang - structured generation and efficient serving.
  - Text Generation Inference (TGI) - HF production serving for LMs.
  - LoRA - low-rank adapters and quantized fine-tuning; the standard for adapting LMs to NLP tasks on modest hardware.
  - DoRA - weight-decomposed low-rank adaptation.
  - PEFT - HuggingFace library bundling LoRA, prefix tuning, IA3, and others.
- Evaluation and Benchmarks
  - GLUE - English NLU benchmarks.
  - XTREME - cross-lingual NLU.
  - XNLI - cross-lingual natural language inference, 15 languages.
  - FLORES-200 - MT evaluation across 200 languages.
  - MTEB - Massive Text Embedding Benchmark; standard for sentence/document encoders.
  - BEIR - heterogeneous IR benchmark for retrieval models.
  - HELM - holistic evaluation across NLP tasks, accuracy and beyond.
  - BIG-bench - 200+ tasks probing language model capabilities.
  - MMLU - multitask knowledge evaluation across 57 subjects.
  - MMLU-Pro - harder, more discriminative successor to MMLU.
  - GPQA - graduate-level Q&A, "Google-proof" reasoning evaluation.
  - IFEval - verifiable instruction-following evaluation.
  - Chatbot Arena (LMSYS) - human-preference ELO leaderboard for chat models.
  - LiveBench - contamination-resistant benchmark with monthly refresh.
  - LM Evaluation Harness - unified framework for LM benchmark evaluation.
  - MMLU-ProX - multilingual extension of MMLU-Pro to 29 typologically diverse languages; reveals up to 24.3% performance gap between high- and low-resource languages.
  - MultiChallenge - multi-turn conversational benchmark exposing simultaneous instruction-following and in-context-reasoning failures; all tested frontier models score below 50%.
  - FRAMES - unified RAG evaluation: 824 multi-hop questions requiring factuality, retrieval accuracy, and cross-document reasoning together.
  - Needle in a Haystack - retrieval probe for long-context windows.
  - RULER - synthetic long-context tasks beyond simple retrieval.
  - LongBench - bilingual long-context benchmark across NLP tasks.
  - LongBench v2 - 503 expert-crafted multiple-choice questions spanning 8K-2M-word contexts with deep multi-hop reasoning; humans score 53.7% under time pressure.
  - U-NIAH - extends needle-in-haystack with multi-needle and nested configurations; shows RAG mitigates lost-in-the-middle for smaller LLMs but degrades reasoning models.
  - REFUTE - scientific reasoning benchmark for evidence-grounded critique, overclaim detection, missing-evidence refusal, and calibration.
- Factuality, Hallucination, Calibration
  - Survey of Hallucination in Natural Language Generation - taxonomy and mitigation strategies.
  - TruthfulQA - benchmark for truthfulness in question answering.
  - FActScore - fine-grained factual precision in long-form generation.
  - LongFact / SAFE - long-form factuality benchmark and search-augmented evaluator.
  - SelfCheckGPT - sampling-based hallucination detection.
  - RAGAS - reference-free evaluation for RAG and QA pipelines.
  - Lookback Lens - attention-pattern-based hallucination detection in long-context generation.
  - Calibration of LLMs on Multiple Choice - calibration analysis under format effects.
  - HalluLens - hallucination benchmark with extrinsic/intrinsic taxonomy and dynamic test-set regeneration to resist data leakage.
  - Atomic Calibration - claim-level calibration analysis for long-form generation; models are substantially worse-calibrated on extended outputs than on single claims.
  - FRANQ - faithfulness-aware uncertainty quantification for RAG fact-checking; formally separates faithfulness from factuality.
  - MUCH - multilingual claim-hallucination benchmark across English, French, Spanish, German with token-level logits released for principled UQ evaluation.
  - HalluHard - hard multi-turn hallucination benchmark for citation-required responses; ~30% hallucination rates persist even with web search.
  - CURE: Think Through Uncertainty - trains models to reason about claim-level uncertainty before generating; large gains on biography factuality and FactBench AUROC.
- Instruction Tuning and Preference Optimization
  - FLAN - finetuned language models as zero-shot learners.
  - InstructGPT - training LMs to follow instructions with human feedback.
  - Self-Instruct - bootstrapping instruction data from LMs.
  - Super-NaturalInstructions - 1600+ NLP tasks with instructions.
  - Constitutional AI - training LMs with AI-generated feedback against a written constitution.
  - Direct Preference Optimization - simpler alternative to RLHF; widely adopted.
  - Tülu 3 - fully open post-training recipe with state-of-the-art results among open models.
  - LIMA - "less is more for alignment"; small high-quality SFT data goes a long way.
  - TRL - reference library for SFT, DPO, GRPO, and RLHF.
  - Magpie - 2025) - synthesizes high-quality instruction-response pairs by prompting aligned LMs with nothing; SFT on the filtered subset matches official Llama-3-Instruct.
- Long Context and Alternative Architectures
  - Mamba - 2](https://arxiv.org/abs/2405.21060) - selective state-space models, linear-time long-context alternative to attention.
  - RWKV - RNN-transformer hybrid scaling to large parameter counts.
  - Jamba - hybrid Mamba-Transformer-MoE architecture.
  - RoPE - rotary position embeddings and context-length extension.
  - Position Interpolation - extending context windows with minimal fine-tuning.
  - Lost in the Middle - long-context degradation patterns in NLP tasks.
  - RAG vs Long-Context LLMs - tradeoffs for QA over long inputs.
  - Titans: Learning to Memorize at Test Time - neural long-term memory module that learns to memorize historical context at test time; scales beyond 2M tokens, outperforms transformers and modern linear-recurrent models on language modeling and reasoning.
  - MiniMax-01 - 456B-parameter hybrid combining lightning (linear) attention with sparse softmax attention; matches GPT-4o-level NLP performance at up to 4M-token inference contexts.
  - Native Sparse Attention (NSA) - trainable sparse attention combining coarse-grained compression with fine-grained selection; large speedups at 64K with no NLP-benchmark degradation.
  - LongRoPE2 - identifies undertraining of high-frequency RoPE dimensions and applies evolutionary-search rescaling; extends LLaMA3-8B to 128K with 80x fewer training tokens than Meta's recipe.
  - Characterizing SSM and Hybrid LM Long-Context Performance - first comprehensive memory and speed analysis of transformer, SSM, and hybrid models up to 220K tokens; SSMs are up to 4x faster, hybrids balance recall and efficiency.
- Multilingual and Cross-Lingual Models
  - XLM-R - cross-lingual masked LM trained on CommonCrawl, 100 languages.
  - mT5 - multilingual T5 covering 101 languages.
  - BLOOM - 176B-parameter open multilingual LM, 46 natural languages.
  - Aya 23 / Aya Expanse - massively multilingual instruction-tuned models covering 23-101 languages.
  - Glot500 - encoder for 500+ languages, focus on low-resource.
  - SEA-LION / SeaLLM - 2025) - LMs targeting Southeast Asian languages.
  - Babel - open multilingual LLMs (9B and 83B) covering the top 25 languages by speaker population (~90% of global speakers); surpasses comparably-sized open multilingual models on XCOPA, XNLI, MGSM, FLORES-200.
  - Lugha-Llama - Llama-3.1-8B adapted for low-resource African languages via the curated WURA corpus; SOTA open-source results on IrokoBench and AfriQA.
  - AfriqueLLM - suite of open LLMs (4B-14B) continued-pretrained on 26B tokens across 20 African languages with a comprehensive empirical study of data mixing.
  - TranslateGemma - open translation-specialized models built on Gemma 3, covering 55 language pairs via SFT and RL with quality-reward models.
  - MiLMMT-46 - open multilingual MT scaled across 46 languages, matching commercial systems like Google Translate and Gemini 3 Pro.
- Pretraining and Adaptation
  - BERT - bidirectional transformer pretraining; foundation for most encoder-based NLP work since 2018.
  - RoBERTa - robustly optimized BERT pretraining; common encoder baseline.
  - ELECTRA - replaced-token-detection pretraining, sample-efficient.
  - ModernBERT - modernized encoder with rotary embeddings, FlashAttention, 8K context; current go-to encoder for classification, NER, retrieval.
  - NeoBERT - 250M-parameter encoder integrating modern architecture improvements (RoPE, 4K context, optimized depth-to-width); state of the art on MTEB, surpasses ModernBERT and RoBERTa-large under identical fine-tuning.
  - T5 - T5](https://arxiv.org/abs/2210.11416) - text-to-text framing for NLP tasks; strong instruction-tuned encoder-decoder baselines.
  - Llama 3 / 3.1 / 3.3 - 2025) - widely adopted open-weight family; default base for fine-tuning across NLP tasks.
  - Qwen 2.5 / Qwen 3 - 2025) - strong multilingual coverage, especially Chinese; often top open model on multilingual benchmarks.
  - DeepSeek-V3 - efficient MoE pretraining; competitive open base model.
  - OLMo 2 - fully open: weights, training data, code; reproducibility benchmark.
  - Gemma 2 / Gemma 3 - 2025) - open small/mid-size models with strong NLP-task performance.
  - Mistral / Mixtral - efficient dense and sparse-MoE open models.
  - What Language Model Architecture and Pretraining Objective Work Best for Zero-Shot Generalization? - encoder vs decoder vs encoder-decoder for NLP transfer.
- Probing and Interpretability
  - A Primer in BERTology - what BERT learns about language.
  - Probing Classifiers (Belinkov) - methodology, limitations, alternatives.
  - Locating and Editing Factual Associations in GPT (ROME) - causal tracing of factual recall.
  - The Pyramid of NLP Probes - structural probing for linguistic knowledge.
  - Toy Models of Superposition - foundation for the sparse-feature view of transformer representations.
  - Towards Monosemanticity / Scaling Monosemanticity - sparse autoencoders extracting interpretable features from production-scale LMs.
  - Sparse Autoencoders Find Highly Interpretable Features - SAE methodology for LM interpretability.
  - Neuronpedia - open platform browsing SAE features across models.
  - Influence Functions Scale to LLMs - identifying training examples driving model behavior.
  - Circuit Tracing: Revealing Computational Graphs in Language Models - introduces cross-layer transcoders and attribution graphs to construct an interpretable replacement model; enables prompt-level circuit tracing of feature-to-feature causal interactions.
  - On the Biology of a Large Language Model - applies attribution graphs to Claude 3.5 Haiku across multi-hop reasoning, rhyme planning, and jailbreak case studies.
  - Transcoders Beat Sparse Autoencoders for Interpretability - shows transcoders (reconstructing layer outputs from inputs) yield more interpretable features than SAEs; introduces skip transcoders.
  - Survey on Sparse Autoencoders for LLM Interpretability - reference survey of SAE architectures, training strategies, feature explanation, and evaluation.
  - Finding Highly Interpretable Prompt-Specific Circuits - identifies circuits at the per-prompt level (rather than per-task); reveals mechanism clustering by prompt family.
- Reasoning and Test-Time Compute
  - Chain-of-Thought Prompting - foundational result; intermediate reasoning steps improve performance.
  - Self-Consistency - majority vote over sampled CoT chains.
  - Tree of Thoughts - search over reasoning trees.
  - Self-Refine - self-correction at inference time.
  - Large Language Models are Zero-Shot Reasoners - chain-of-thought for NLP reasoning tasks.
  - Let's Verify Step by Step - process-supervised reward models for reasoning.
  - DeepSeek-R1 - open reasoning model trained with pure RL; replicated o1-style behavior in the open.
  - OpenAI o1 / o3 - 2025) - test-time-compute reasoning systems.
  - Scaling LLM Test-Time Compute Optimally - systematic study of inference-time compute tradeoffs.
  - s1: Simple Test-Time Scaling - small open reasoning recipe via budget-forcing.
  - Kimi k1.5 - long-context RL with policy optimization (no MCTS, no PRM) reaching o1-level performance; introduces long-CoT distillation into short-CoT models.
  - rStar-Math - small policy model paired with a process preference model trained via MCTS rollouts; enables small LMs to bootstrap reasoning without distilling from larger models.
  - DAPO - open GRPO-based RL training system with four key improvements (decoupled clipping, dynamic sampling, token-level loss, entropy bonus); reproduces and surpasses DeepSeek-R1-Zero-level reasoning.
  - VAPO - value-model-based RL with length-adaptive GAE and token-level clipping; surpasses value-free GRPO methods on AIME 2024 with stable training.
  - ThinkPRM - generative process reward models that produce chain-of-thought verification per step, matching discriminative PRMs with 1% of the supervision labels.
  - OpenThoughts - 1000+ controlled experiments on data recipes for open reasoning models; SOTA on AIME 2025 matching closed distillation baselines.
Libraries
- Annotation Tools
  - doccano - doccano is free, open-source, and provides annotation features for text classification, sequence labeling and sequence to sequence
  - Datasaur
  - Konfuzio - team-first hosted and on-prem text, image and PDF annotation tool powered by active learning, freemium based, costs $
  - UBIAI - Easy-to-use text annotation tool for teams with most comprehensive auto-annotation features. Supports NER, relations and document classification as well as OCR annotation for invoice labeling, costs $
  - Shoonya - Shoonya is free and open source data annotation platform with wide varities of organization and workspace level management system. Shoonya is data agnostic, can be used by teams to annotate data with various level of verification stages at scale.
  - Annotation Lab - Free End-to-End No-Code platform for text annotation and DL model training/tuning. Out-of-the-box support for Named Entity Recognition, Classification, Relation extraction and Assertion Status Spark NLP models. Unlimited support for users, teams, projects, documents. Not FOSS.
  - FLAT - FLAT is a web-based linguistic annotation environment based around the [FoLiA format](http://proycon.github.io/folia), a rich XML-based format for linguistic annotation. Free and open source.
  - Label Studio - open-core multi-modal labeling platform; widely used for NLP labeling.
  - Potato - Free, open-source annotation tool covering 21+ task types (classification, span, coreference, entity linking, agent trace evaluation) with built-in MACE quality control, attention checks, AI-assisted labeling, and 300+ example tasks.
- Books
  - NLP.js - An NLP library for building bots
  - node-question-answering - Fast and production-ready question answering w/ DistilBERT in Node.js
  - sentimental-onix
  - TextAttack - Adversarial attacks, adversarial training, and data augmentation in NLP
  - GluonNLP - A deep learning toolkit for NLP, built on MXNet/Gluon.
  - foliapy - Python library for working with [FoLiA](https://proycon.github.io/folia/), an XML format for linguistic annotation.
  - PySS3 - Python package implementing the SS3 white-box text classifier; ships with interactive visualization tools that explain predictions.
  - Natural Language Toolkit (NLTK) - A library containing a wide variety of NLP functionality, supporting over 50 corpora.
  - FARM - Fast & easy transfer learning for NLP. Harvesting language models for the industry. Focus on Question Answering.
  - Haystack - End-to-end Python framework for building natural language search interfaces to data. Leverages Transformers and the State-of-the-Art of NLP. Supports DPR, Elasticsearch, HuggingFace’s Modelhub, and much more!
  - Rita DSL - a DSL, loosely based on [RUTA on Apache UIMA](https://uima.apache.org/ruta.html). Allows to define language patterns (rule-based NLP) which are then translated into [spaCy](https://spacy.io/), or if you prefer less features and lightweight - regex patterns.
  - Transformers - Natural Language Processing for TensorFlow 2.0 and PyTorch.
  - Tokenizers - Tokenizers optimized for Research and Production.
  - corex_topic - Hierarchical Topic Modeling with Minimal Domain Knowledge
  - Sockeye - Neural Machine Translation (NMT) toolkit that powers Amazon Translate.
  - DL Translate - A deep learning-based translation library for 50 languages, built on `transformers` and Facebook's mBART Large.
  - Jury - Evaluation of NLP model outputs offering various automated metrics.
  - python-ucto - Unicode-aware regular-expression based tokenizer for various languages. Python binding to C++ library, supports [FoLiA format](https://proycon.github.io/folia).
  - Pearmut - Human annotation tool for multilingual NLP tasks, such as machine translation.
  - Stanza - Stanford NLP's Python toolkit for tokenization, POS, lemma, dependency parsing, and NER across 70+ languages.
  - Sentence-Transformers - sentence/document embeddings, semantic search, and re-ranking; current standard for retrieval-style NLP.
  - Argilla - open-source data annotation and feedback collection platform for LLM and NLP datasets.
  - HuggingFace Datasets - standardized loaders and processing for thousands of NLP datasets.
  - HuggingFace Evaluate - reference implementations for NLP metrics.
  - sacrebleu - reproducible BLEU/chrF/TER scoring for machine translation.
  - COMET - learned MT metrics, current de-facto standard.
  - LangTest - 60+ test types for NLP model robustness, bias, and fairness.
  - InsNet - A neural network library for building instance-dependent NLP models with padding-free dynamic batching.
  - Spark NLP - Spark NLP is a natural language processing library built on top of Apache Spark ML that provides simple, performant & accurate NLP annotations for machine learning pipelines that scale easily in a distributed environment.
  - corporaexplorer - An R package for dynamic exploration of text collections
  - tidytext - Text mining using tidy tools
  - spacyr - R wrapper to spaCy NLP
  - CRAN Task View: Natural Language Processing
  - prose - Text processing library supporting tokenization, part-of-speech tagging, and named-entity extraction.
  - gojieba - Go implementation of the jieba Chinese word segmentation algorithm.
  - kagome - Japanese morphological analyzer written in pure Go.

Programming Languages

Python 116 Jupyter Notebook 18 C++ 14 JavaScript 13 Java 8 HTML 6 R 6 Scala 5 Shell 5 Go 4

ecosyste.ms

Data

Tools

Indexes

Applications

Experiments

Awesome

awesome-nlp

Datasets

Speech and Text

Language Models for NLP

Bias, Fairness, Safety in NLP

Efficient and Small Language Models

Evaluation and Benchmarks

Factuality, Hallucination, Calibration

Instruction Tuning and Preference Optimization

Long Context and Alternative Architectures

Multilingual and Cross-Lingual Models

Pretraining and Adaptation

Probing and Interpretability

Reasoning and Test-Time Compute

Libraries

Annotation Tools

Books