Projects in Awesome Lists tagged with summarization

https://github.com/deepset-ai/haystack

:mag: AI orchestration framework to build customizable, production-ready LLM applications. Connect components (models, vector DBs, file converters) to pipelines or agents that can interact with your data. With advanced retrieval methods, it's best suited for building RAG, question answering, semantic search or conversational agent chatbots.

ai bert chatgpt generative-ai gpt-3 information-retrieval language-model large-language-models llm machine-learning nlp python pytorch question-answering rag retrieval-augmented-generation semantic-search squad summarization transformers

Last synced: 17 Dec 2024

https://github.com/miso-belica/sumy

Module for automatic summarization of text documents and HTML pages.

html-extraction html-extractor html-page lsa nlp pagerank-algorithm python reduction summarization summarizer summary sumy text-extraction textteaser

Last synced: 16 Dec 2024

https://github.com/sylphai-inc/adalflow

AdalFlow: The library to build & auto-optimize LLM applications.

agent ai bm25 chatbot faiss framework generative-ai information-retrieval llm machine-learning nlp optimizer python question-answering rag reranker retriever summarization trainer

Last synced: 17 Dec 2024

https://github.com/allenai/rl4lms

A modular RL library to fine-tune language models to human preferences

dialogue-generation language-modeling machine-translation natural-language-processing nlp reinforcement-learning summarization table-to-text text-generation

Last synced: 20 Dec 2024

https://github.com/allenai/RL4LMs

A modular RL library to fine-tune language models to human preferences

dialogue-generation language-modeling machine-translation natural-language-processing nlp reinforcement-learning summarization table-to-text text-generation

Last synced: 31 Oct 2024

https://github.com/SylphAI-Inc/LightRAG

AdalFlow: The library to build & auto-optimize LLM applications.

agent ai bm25 chatbot faiss framework generative-ai information-retrieval llm machine-learning nlp optimizer python question-answering rag reranker retriever summarization trainer

Last synced: 01 Dec 2024

https://github.com/derwenai/pytextrank

Python implementation of TextRank algorithms ("textgraphs") for phrase extraction

graph-algorithms machine-learning natural-language natural-language-processing nlp python spacy spacy-extension summarization textgraphs textrank

Last synced: 17 Dec 2024

https://github.com/DerwenAI/pytextrank

Python implementation of TextRank algorithms ("textgraphs") for phrase extraction

graph-algorithms machine-learning natural-language natural-language-processing nlp python spacy spacy-extension summarization textgraphs textrank

Last synced: 29 Oct 2024

https://github.com/SylphAI-Inc/AdalFlow

AdalFlow: The library to build & auto-optimize LLM applications.

agent ai bm25 chatbot faiss framework generative-ai information-retrieval llm machine-learning nlp optimizer python question-answering rag reranker retriever summarization trainer

Last synced: 13 Nov 2024

https://github.com/mesolitica/NLP-Models-Tensorflow

Gathers machine learning and Tensorflow deep learning models for NLP problems, 1.13 < Tensorflow < 2.0

attention chatbot deep-learning dnc-seq2seq embedded language-detection lstm lstm-seq2seq-tf luong-api machine-learning neural-machine-translation nlp optical-character-recognition pos-tagging speech-to-text summarization

Last synced: 06 Nov 2024

https://github.com/mesolitica/nlp-models-tensorflow

Gathers machine learning and Tensorflow deep learning models for NLP problems, 1.13 < Tensorflow < 2.0

attention chatbot deep-learning dnc-seq2seq embedded language-detection lstm lstm-seq2seq-tf luong-api machine-learning neural-machine-translation nlp optical-character-recognition pos-tagging speech-to-text summarization

Last synced: 26 Sep 2024

https://github.com/chiphuyen/sotawhat

Returns latest research results by crawling arxiv papers and summarizing abstracts. Helps you stay afloat with so many new papers everyday.

arxiv python research-tool script summarization

Last synced: 21 Dec 2024

https://github.com/xcfcode/Summarization-Papers

Summarization Papers

chatgpt natural-language-processing nlp pretrained-language-model summarization text-generation

Last synced: 13 Nov 2024

https://github.com/IntelLabs/fastRAG

Efficient Retrieval Augmentation and Generation Framework

benchmark colbert diffusion generative-ai information-retrieval knowledge-graph llm multi-modal nlp question-answering semantic-search sentence-transformers summarization transformers

Last synced: 28 Oct 2024

https://github.com/intellabs/fastrag

Efficient Retrieval Augmentation and Generation Framework

benchmark colbert diffusion generative-ai information-retrieval knowledge-graph llm multi-modal nlp question-answering semantic-search sentence-transformers summarization transformers

Last synced: 21 Dec 2024

https://github.com/georgian-io/llm-finetuning-toolkit

Toolkit for fine-tuning, ablating and unit-testing open-source LLMs.

ablation-study classification falcon fine-tuning finetuning flan-t5 large-language-models llama2 llm-test lora mistral-7b nlp nlp-machine-learning qlora redpajama summarization unit-testing zephyr

Last synced: 19 Dec 2024

https://github.com/georgian-io/LLM-Finetuning-Toolkit

Toolkit for fine-tuning, ablating and unit-testing open-source LLMs.

ablation-study classification falcon fine-tuning finetuning flan-t5 large-language-models llama2 llm-test lora mistral-7b nlp nlp-machine-learning qlora redpajama summarization unit-testing zephyr

Last synced: 07 Nov 2024

https://github.com/namuan/dr-doc-search

Converse with book - Built with GPT-3

gpt3 langchain nlp-machine-learning python summarization

Last synced: 20 Dec 2024

https://github.com/juliastats/statsbase.jl

Basic statistics for Julia

julia statistical-models statistics summarization

Last synced: 19 Dec 2024

https://github.com/JuliaStats/StatsBase.jl

Basic statistics for Julia

julia statistical-models statistics summarization

Last synced: 17 Nov 2024

https://github.com/udibr/headlines

Automatically generate headlines to short articles

generation keras nlp rnn summarization

Last synced: 27 Nov 2024

https://github.com/SKT-AI/KoBART

Korean BART

korean-nlp language-model summarization

Last synced: 09 Nov 2024

https://github.com/cognitivetech/ollama-ebook-summary

LLM for Long Text Summary (Comprehensive Bulleted Notes)

generative-ai gpt llm localai localgpt ollama ollama-app privategpt privategpt4linux summarization

Last synced: 17 Dec 2024

https://github.com/HHousen/TransformerSum

Models to perform neural summarization (extractive and abstractive) using machine learning transformers and a tool to convert abstractive summarization datasets to the extractive task.

albert automatic-summarization bert distilbert extractive-summarization machine-learning pytorch-lightning roberta summarization summarization-dataset text-summarization transformer-models

Last synced: 11 Nov 2024

https://github.com/santhoshkolloju/Abstractive-Summarization-With-Transfer-Learning

Abstractive summarisation using Bert as encoder and Transformer Decoder

abstractive-summarization abstractive-text-summarization bert bert-model nlg nlp summarization transfer-learning transformer

Last synced: 02 Nov 2024

https://github.com/ymfa/seq2seq-summarizer

Pointer-generator reinforced seq2seq summarization in PyTorch

abstractive-summarization attention attention-mechanism coverage coverage-mechanism pointer-network pytorch reinforcement-learning seq2seq summarization summarizer

Last synced: 13 Nov 2024

https://github.com/nlpodyssey/cybertron

Cybertron: the home planet of the Transformers in Go

bart bert bert-as-service deep-learning huggingface machine-learning machine-translation named-entity-recognition natural-language-processing nlp question-answering summarization text-categorization text-classification text-similarity transformers translation zero-shot-classification

Last synced: 21 Dec 2024

https://github.com/awslabs/generative-ai-cdk-constructs

AWS Generative AI CDK Constructs are sample implementations of AWS CDK for common generative AI patterns.

agent amazon-bedrock appsync aws aws-cdk content-generation generative-ai huggingface knowledge-base langchain llm opensearch question-answering-system retrieval-augmented-generation sagemaker step-functions summarization

Last synced: 17 Dec 2024

https://github.com/shreyaskarnik/distillama

Chrome Extension to Summarize or Chat with Web Pages/Local Documents Using locally running LLMs. Keep all of your data and conversations private. 🔐

chrome-extension langchain llama2 llms local-llm mistral-7b ollama privacy-preserving-computing private readability retrieval-augmented-generation summarization zephyr

Last synced: 16 Dec 2024

https://github.com/shreyaskarnik/DistiLlama

Chrome Extension to Summarize or Chat with Web Pages/Local Documents Using locally running LLMs. Keep all of your data and conversations private. 🔐

chrome-extension langchain llama2 llms local-llm mistral-7b ollama privacy-preserving-computing private readability retrieval-augmented-generation summarization zephyr

Last synced: 06 Nov 2024

https://github.com/alex-fabbri/multi-news

Large-scale multi-document summarization dataset and code

multi-document-summarization multi-news summarization

Last synced: 17 Dec 2024

https://github.com/Alex-Fabbri/Multi-News

Large-scale multi-document summarization dataset and code

multi-document-summarization multi-news summarization

Last synced: 03 Nov 2024

https://github.com/opea-project/GenAIExamples

Generative AI Examples is a collection of GenAI examples such as ChatQnA, Copilot, which illustrate the pipeline capabilities of the Open Platform for Enterprise AI (OPEA) project.

chatqna codegen copilot gaudi2 genai llms rag summarization tgi xeon

Last synced: 05 Nov 2024

https://github.com/abelriboulot/onnxt5

Summarization, translation, sentiment-analysis, text-generation and more at blazing speed using a T5 version implemented in ONNX.

inference nlp nlp-machine-learning onnx onnxruntime sentiment-analysis summarization text-classification text-generation transformer transformers translation

Last synced: 07 Nov 2024

https://github.com/bheinzerling/pyrouge

A Python wrapper for the ROUGE summarization evaluation package

evaluation-metrics nlp rouge summarization

Last synced: 18 Dec 2024

https://github.com/DavidBelicza/PHP-Science-TextRank

:zap: :elephant: TextRank (resource-efficient and low-cost automatic text summarisation) for PHP

ai airtificialintelligence algorithm php science search summarization textrank

Last synced: 25 Nov 2024

https://github.com/davidbelicza/php-science-textrank

:zap: :elephant: TextRank (resource-efficient and low-cost automatic text summarisation) for PHP

ai airtificialintelligence algorithm php science search summarization textrank

Last synced: 15 Dec 2024

https://github.com/j-min/adversarial_video_summary

Unofficial PyTorch Implementation of SUM-GAN from "Unsupervised Video Summarization with Adversarial LSTM Networks" (CVPR 2017)

gan pytorch summarization vae video

Last synced: 19 Nov 2024

https://github.com/j-min/Adversarial_Video_Summary

Unofficial PyTorch Implementation of SUM-GAN from "Unsupervised Video Summarization with Adversarial LSTM Networks" (CVPR 2017)

gan pytorch summarization vae video

Last synced: 07 Nov 2024

https://github.com/crabcamp/lexrank

LexRank algorithm for text summarization

algorithm lex lexrank markov-chains rank summarization summarizer summary text

Last synced: 03 Nov 2024

https://github.com/Hellisotherpeople/CX_DB8

a contextual, biasable, word-or-sentence-or-paragraph extractive summarizer powered by the latest in text embeddings (Bert, Universal Sentence Encoder, Flair)

contextual-summarization cuda debate-evidence embeddings extractive-summarization flair python semantic-search semantic-summarization summarization summarizer token-level-summarization universal-sentence-encoder

Last synced: 22 Nov 2024

https://github.com/davidbelicza/textrank

:wink: :cyclone: :strawberry: TextRank implementation in Golang with extendable features (summarization, phrase extraction) and multithreading (goroutine).

go golang pagerank phrase-extraction sentence-classification summarization textrank token

Last synced: 17 Nov 2024

https://github.com/DavidBelicza/TextRank

:wink: :cyclone: :strawberry: TextRank implementation in Golang with extendable features (summarization, phrase extraction) and multithreading (goroutine).

go golang pagerank phrase-extraction sentence-classification summarization textrank token

Last synced: 26 Oct 2024

https://github.com/IlyaGusev/summarus

Models for automatic abstractive summarization

deep-learning machine-learning nlp pytorch summarization

Last synced: 04 Nov 2024

https://github.com/magic282/NeuSum

Code for the ACL 2018 paper "Neural Document Summarization by Jointly Learning to Score and Select Sentences"

natural-language-processing summarization

Last synced: 16 Nov 2024

https://github.com/dmmiller612/lecture-summarizer

Lecture summarization with BERT

bert bert-model extractive-summarization flask pytorch summarization

Last synced: 27 Oct 2024

https://github.com/pszemraj/textsum

CLI & Python API to easily summarize text-based files with transformers

batch-processing inference inference-api pipeline summarization summary text text-to-text-transformer transformer transformers

Last synced: 18 Dec 2024

https://github.com/cosmoquester/2021-dialogue-summary-competition

[2021 훈민정음 한국어 음성•자연어 인공지능 경진대회] 대화요약 부문 알라꿍달라꿍 팀의 대화요약 학습 및 추론 코드를 공유하기 위한 레포입니다.

dialogue huggingface-transformers nlp pytorch-lightning summarization

Last synced: 09 Nov 2024

https://github.com/charlesdedampierre/BunkaTopics

🗺️ Data Cleaning and Textual Data Visualization 🗺️

cartography data-cleaning explainability fine-tuning llms machine-learning natural-language-processing nlp summarization topic-modeling

Last synced: 03 Sep 2024

https://github.com/RevanthRameshkumar/CRD3

The repo containing the Critical Role Dungeons and Dragons Dataset.

acl2020 dataset dialogue-systems machine-learning nlp storytelling summarization

Last synced: 03 Nov 2024

https://github.com/sean1832/SumGPT

Bulk summarization of documents using ChatGPT API

artificial-intelligence bulk-operation gpt-3 gpt-4 knowledgemanagement note-taking openai openai-chatgpt parallel-computing python summarization ultrafast

Last synced: 31 Oct 2024

https://github.com/soptikha2/video-summarizer

Summarizes videos into much shorter videos. Ideal for long lecture videos.

ffmpeg rust summarization video video-processing

Last synced: 18 Dec 2024

https://github.com/santosh-gupta/scientificsummarizationdatasets

Datasets I have created for scientific summarization, and a trained BertSum model

bert bertsum dataset extractive-summarization pointer-generator summarization tensor2tensor transformer

Last synced: 12 Nov 2024

https://github.com/clovaai/focusseq2seq

[EMNLP 2019] Mixture Content Selection for Diverse Sequence Generation (Question Generation / Abstractive Summarization)

emnlp2019 generation nlp pytorch question-generation summarization

Last synced: 12 Nov 2024

https://github.com/muhtasham/summarization-eval

📝 Reference-Free automatic summarization evaluation with potential hallucination detection

evaluation-metrics summarization

Last synced: 05 Dec 2024

https://github.com/tanyuqian/ctc-gen-eval

EMNLP 2021 - CTC: A Unified Framework for Evaluating Natural Language Generation

dialogue-generation evaluation-metrics style-transfer summarization text-generation

Last synced: 18 Dec 2024

https://github.com/ravi-teja-konda/surveillance_video_summarizer

VLM driven tool that processes surveillance videos, extracts frames, and generates insightful annotations using a fine-tuned Florence-2 Vision-Language Model. Includes a Gradio-based interface for querying and analyzing video footage.

ai chatgpt florence-2 gpt-4 gradio gradio-python-llm huggingface summarization surviellance video vision-and-language vlm

Last synced: 17 Dec 2024

https://github.com/jddunn/frame

Notetaking Electron app that can answer your questions and makes summaries for you

notetaking summarization

Last synced: 13 Nov 2024

https://github.com/tingofurro/summac

Codebase, data and models for the SummaC paper in TACL

factual-consistency inconsistency-detection summarization text-generation

Last synced: 20 Nov 2024

https://github.com/nezhar/speech-condenser

A tool for summarizing dialogues from videos or audio

asr speach-recognition speaker-diarization speaker-identification summarization

Last synced: 01 Nov 2024

https://github.com/hyunwoongko/summarizers

Package for controllable summarization

nlp summarization

Last synced: 27 Oct 2024

https://github.com/salesforce/summary-of-a-haystack

Codebase accompanying the Summary of a Haystack paper.

evaluation llm need-in-a-haystack summarization summary-of-a-haystack

Last synced: 18 Dec 2024

https://github.com/yuhaozhang/summarize-radiology-findings

Code and pretrained model for paper "Learning to Summarize Radiology Findings"

medical-informatics medical-natural-language-processing medicine natural-language-processing nlp radiology summarization

Last synced: 08 Nov 2024

https://github.com/akashp1712/nlp-akash

Natural Language Processing notes and implementations.

natural-language-processing nlp nlp-akash nltk summarization text-summarization

Last synced: 17 Nov 2024

https://github.com/RUCAIBox/MVP

This repository is the official implementation of our paper MVP: Multi-task Supervised Pre-training for Natural Language Generation.

data-to-text dialog multi-task-learning natural-language-generation natural-language-processing nlg nlp plm pre-trained-model question-answering question-generation seq2seq sequence-to-sequence story-generation summarization text-generation

Last synced: 16 Nov 2024

https://github.com/transitive-bullshit/text-summarization

Automagically generates summaries from html or text.

extractive-summarization extractive-text-summarization summarization summarize summary text

Last synced: 01 Nov 2024

https://github.com/mirfan899/Urdu

Collection of Urdu datasets for POS, NER, Sentiment, Summarization and NLP tasks.

machine-learning ner nlp sentiment-analysis spacy-models summarization urdu-language urdu-model

Last synced: 17 Nov 2024

https://github.com/cobanov/easy-web-summarizer

Summarize webpages from specified URLs using the LangChain framework and the ChatOllama model

langchain llm ollama summarization

Last synced: 20 Oct 2024

https://github.com/stangirard/speechdigest

Audio to summary with openAI Whisper & GPT 3.5/4 using streamlit

audio-processing gpt gpt-3 gpt-4 llm openai recording summarization whisper

Last synced: 14 Nov 2024

https://github.com/alisafaya/mukayese

State-of-the-art NLP tools for Turkish

language-model machine-translation ner nlp segmentation sentiment-analysis spelling-correction summarization text-classification

Last synced: 10 Oct 2024

https://github.com/kardbord/hfapigo

Unofficial (Golang) Go bindings for the Hugging Face Inference API

api audio-classification conversational-ai go golang hacktoberfest huggingface inference-api natural-language-processing nlp object-detection question-answering speech-recognition speech-recognition-api summarization text-classification text-generation token-classification translation zero-shot-classification

Last synced: 18 Dec 2024

https://github.com/salesforce/factualnlg

Code for the arXiv paper: "LLMs as Factual Reasoners: Insights from Existing Benchmarks and Beyond"

factual-consistency factuality large-language-models llm nlp summarization

Last synced: 08 Nov 2024

https://github.com/salesforce/factualNLG

Code for the arXiv paper: "LLMs as Factual Reasoners: Insights from Existing Benchmarks and Beyond"

factual-consistency factuality large-language-models llm nlp summarization

Last synced: 27 Oct 2024

https://github.com/pcyin/pyrouge

A python library to compute rouge score for summarization

rouge summarization

Last synced: 01 Dec 2024

https://github.com/richawo/chain-of-density

Implementing the Chain Of Density text summarisation technique from recent NLP research by researchers at Salesforce, MIT, Columbia, etc. Takes a long text input and iteratively generates increasingly concise, entity-dense summaries using LLMs (default: OpenAI's GPT-4). Feedback and contributions welcome!

ai llm openai poetry python summarization summarizer

Last synced: 08 Nov 2024

https://github.com/iwasakiyuuki/bert-abstractive-text-summarization

Japanese Sentence Summarization with BERT

bert deep-learning japanese-language pytorch summarization

Last synced: 20 Dec 2024

https://github.com/cannylab/summary_loop

Codebase for the Summary Loop paper at ACL2020

abstractive-summarization acl2020 fluency-model news reinforcement-learning summarization unsupervised-learning

Last synced: 16 Nov 2024

https://github.com/saasify-sh/synopsis

Automagical summarization for webpages and articles. 🔥

saas saasify summarization summarizer summary text text-summarization

Last synced: 13 Nov 2024

https://github.com/webis-de/summary-explorer

Summary Explorer is a tool to visually explore the state-of-the-art in text summarization.

evaluation summarization

Last synced: 06 Nov 2024

https://github.com/Ravi-Teja-konda/Surveillance_Video_Summarizer

VLM driven tool that processes surveillance videos, extracts frames, and generates insightful annotations using a fine-tuned Florence-2 Vision-Language Model. Includes a Gradio-based interface for querying and analyzing video footage.

ai chatgpt florence-2 gpt-4 gradio gradio-python-llm huggingface summarization surviellance video vision-and-language vlm

Last synced: 22 Sep 2024

https://github.com/kenlimmj/rouge

A Javascript implementation of the Recall-Oriented Understudy for Gisting Evaluation (ROUGE) evaluation metric for summaries.

bootstrapping-statistics evaluation-metric jackknifing nlp rouge summarization

Last synced: 10 Nov 2024

https://github.com/rudrakshi99/farmer-call-center

AI-based voice-assisted Contact Center for assisting Farmers for their problems.

ai alan-ai call-center crop crop-recommendation disease-prediction farmers fertilizer-recommendation flask hackathon machine-learning nlp pytorch react sms summarization twillio unesco-india-africa-hackathon-2022 voice-assistant

Last synced: 22 Nov 2024

https://github.com/salesforce/query-focused-sum

Official code repository for "Exploring Neural Models for Query-Focused Summarization".

deep-learning machine-learning neural-network nlp question-answering summarization

Last synced: 08 Nov 2024

https://github.com/webis-de/summary-workbench

Framework for unified summarisation and evaluation of English documents using state-of-the-art models and measures.

evaluation-metrics summarization text-generation visualization

Last synced: 06 Nov 2024

https://github.com/hyunwoongko/bert2bert-summarization

Abstractive summarization using Bert2Bert framework.

bert nlp summarization

Last synced: 28 Oct 2024

https://github.com/easonlai/google_news_content_scrape_and_analyze_with_gpt

This demo repository illustrates how to use Python to scrape news articles from Google based on a given keyword. The scraped articles are then processed by Azure OpenAI Service (AOAI)'s GPT-3 model, which generates concise summaries of the main points. The summaries are then formatted and sent via email using MailJet API.

azure azure-openai azure-openai-api azureopenai googlenews googlenewsscraper gpt-3 mailjet news-scraper news-scraping news-summarization newspaper3k nltk openai python python3 summarization

Last synced: 10 Nov 2024

https://github.com/locaal-ai/note-taker

A lo-fi AI-first note taker running locally on-device

ai llm local-llm note-taking notes-app notes-tool summarization transcription

Last synced: 10 Nov 2024

https://github.com/megagonlabs/coop

☘️ Code for Convex Aggregation for Opinion Summarization (Iso et al; Findings of EMNLP 2021)

natural-language-generation opinion-summarization summarization text-generation unsupervised-learning vae variational-autoencoder

Last synced: 10 Nov 2024

https://github.com/mcdallas/summarize

Summarize audio/video files

chatgpt cli command-line-tool openai rust rust-lang summarization summarize summary whisper-ai

Last synced: 12 Oct 2024

https://github.com/microsoft/verseagility

Ramp up your custom natural language processing (NLP) task, allowing you to bring your own data, use your preferred frameworks and bring models into production.

classification dstoolkit machine-reading-comprehension ner nlp question-answering summarization transformer

Last synced: 04 Dec 2024

https://github.com/tcsenpai/youlama

YouTube Summarizer is a Streamlit-based web application that allows users to generate summaries of YouTube videos using AI-powered language models.

ai llama local ollama privacy summarization youtube

Last synced: 03 Dec 2024

https://github.com/webis-de/webis-tldr-17-corpus

Code for constructing TLDR corpus from Reddit dataset

corpus summarization

Last synced: 06 Nov 2024

https://github.com/shrebox/Personified-Chatbot

A personified chatbot responding to a query based on the answering pattern of Dr. APJ Abdul Kalam using Information Retrieval, Natural Language Processing, and Deep Learning techniques.

apj-abdul-kalam chatbot deep-learning information-retrieval lstm natural-language-processing nlp ranking-algorithm seq2seq-chatbot seq2seq-model summarization word2vec

Last synced: 11 Nov 2024

https://github.com/amsqr/NaiveSumm

NaiveSumm is a naive summarization approach based on Luhn1958 work "The Automatic Creation of Literature Abstracts" It uses the frequencies of words in the document in order to calculate and extract the sentences that include the most frequent words.

natural-language-processing nlp python summarization

Last synced: 31 Oct 2024

https://github.com/mainakrepositor/paribhasha

heroku-deployment nlp-machine-learning pos-tagging sentiment-analysis streamlit-webapp summarization word-cloud

Last synced: 12 Nov 2024

https://github.com/twardoch/split-markdown4gpt

A Python tool for splitting large Markdown files into smaller sections based on a specified token limit. This is particularly useful for processing large Markdown files with GPT models, as it allows the models to handle the data in manageable chunks.

data-preprocessing gpt gpt-3 gpt-35-turbo gpt-35-turbo-16k gpt-4 markdown markdown-processing mistletoe natural-language-processing nlp openai openai-gpt python split-text summarization text-analysis text-processing text-summarization text-tokenization