{"id":19162757,"url":"https://github.com/centre-for-humanities-computing/llm-tweet-classification","last_synced_at":"2026-03-06T07:03:09.363Z","repository":{"id":188198616,"uuid":"678260375","full_name":"centre-for-humanities-computing/llm-tweet-classification","owner":"centre-for-humanities-computing","description":"Classifying tweets with large language models with zero- and few-shot learning.","archived":false,"fork":false,"pushed_at":"2025-02-24T09:45:09.000Z","size":2775,"stargazers_count":8,"open_issues_count":3,"forks_count":1,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-09-09T23:59:11.381Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/centre-for-humanities-computing.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2023-08-14T06:18:52.000Z","updated_at":"2025-05-12T09:22:58.000Z","dependencies_parsed_at":"2023-10-17T05:51:10.301Z","dependency_job_id":"6f920ffc-4718-4ff8-b36d-34b2d4642b33","html_url":"https://github.com/centre-for-humanities-computing/llm-tweet-classification","commit_stats":null,"previous_names":["centre-for-humanities-computing/llm-tweet-classification"],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/centre-for-humanities-computing/llm-tweet-classification","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/centre-for-humanities-computing%2Fllm-tweet-classification","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/centre-for-humanities-computing%2Fllm-tweet-classification/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/centre-for-humanities-computing%2Fllm-tweet-classification/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/centre-for-humanities-computing%2Fllm-tweet-classification/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/centre-for-humanities-computing","download_url":"https://codeload.github.com/centre-for-humanities-computing/llm-tweet-classification/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/centre-for-humanities-computing%2Fllm-tweet-classification/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":30164901,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-03-06T04:43:31.446Z","status":"ssl_error","status_checked_at":"2026-03-06T04:40:30.133Z","response_time":250,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-09T09:13:04.556Z","updated_at":"2026-03-06T07:03:09.338Z","avatar_url":"https://github.com/centre-for-humanities-computing.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"[![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.14916576.svg)](https://doi.org/10.5281/zenodo.14916576)\n\n# llm-tweet-classification\nClassifying tweets with large language models with zero- and few-shot learning with custom and generic prompts, as well as supervised learning algorithms for comparison.\n\n### Our results on annotating tweets with labels `exemplar` and `political`:\n\n| F1-scores \u0026 Accuracies | Precision-Recall |\n|:---------:|:----------------:|\n| ![F1 scores](figures/f1_acc.png) | ![prec-rec scores](figures/prec_rec.png) |\n\n## Getting Started\n\nInstall all requirements for the LLM classification script.\n```bash\npip install -r requirements.txt\n```\n\n**NB:** This will only install a minimal set of requirements to create figures for reproducability sake with the code [below](#plotting_results). A more complete requirements file for running the full pipeline can be found in [configs](configs/requirements.txt).\n\n## Inference\nThe repo contains a CLI script `llm_classification.py`.\nYou can use it for running arbitrary classification tasks in `.tsv` or `.csv` files with Large Language models from either\nHuggingFace or OpenAI.\n\nIf you intend to use OpenAI models, you will have to specify your API key and ORG as environment variables.\n\n```bash\nexport OPENAI_API_KEY=\"...\"\nexport OPENAI_ORG=\"...\"\n```\n\nThe script has one command-line argument, namely a config file of the following format:\n\n```\n[paths]\nin_file=\"labelled_data.csv\"\nout_dir=\"predictions/\"\n\n[system]\nseed=0\ndevice=\"cpu\"\n\n[model]\nname=\"google/flan-t5-base\"\ntask=\"few-shot\"\n\n[inference]\nx_column=\"raw_text\"\ny_column=\"exemplar\"\nn_examples=5\n```\n\nIf you intend to use a custom prompt for a given model, you can save it in a txt file and add its path to the\n`paths` section of the config.\n\n ```\n[paths]\nin_file=\"labelled_data.csv\"\nout_dir=\"predictions/\"\nprompt_file=\"custom_prompt.txt\"\n ```\n\nIf you want to use hand-selected examples for few-shot learning, pass along a subset of the original data int the paths section of\nthe config.\nExamples have to be in the same format as the data.\n\n ```\n[paths]\nin_file=\"labelled_data.csv\"\nout_dir=\"predictions/\"\nexamples=\"examples.csv\"\n ```\n\n\nYou can run the CLI like this:\n\n```bash\npython3 llm_classification.py \"config.cfg\"\n```\n\n### Config Documentation\n- Paths:\n    - in_file: `str` - Path to input file, either `.csv` or `.tsv`\n    - out_dir: `str` - Output directory. The script creates one if not already there.\n- System:\n    - seed: `int` - Random seed for selecting few-shot examples. Is ignored when `task==\"zero-shot\"`\n    - device: `str` - Device to run inference on. Change to `cuda:0` if you want to run on GPU.\n- Model:\n    - name: `str` - Name of the model from OpenAI or HuggingFace.\n    - task: `{\"few-shot\", \"zero-shot\"}` - Indicates whether zero-shot or few-shot inference should be run.\n- Inference:\n    - x_column: `str` - Name of independent variable in the table.\n    - y_column: `str` - Name of dependent variable in the table.\n    - n_examples: `int` - Number of examples to give to few-shot models. Is ignored when `task==\"zero-shot\"`\n\n## OpenAI script\n\nFor ease of use we have developed a script that generates predictions for all OpenAI models in one run. We did this, because OpenAI inference can run on low performance instances, as such it isn't a problem if it takes a long time to run.\nAdditionally since all instances access the same API, and there are rate limits, we could not start multiple instances and run them in parallel.\n\nPaths in this script are hardcoded and you might need to adjust it for personal use.\n\n```bash\npython3 run_gpt_inference.py\n```\n\n## Supervised Classification\n\nFor supervised models we made a separate script. This includes running and evaluating Glove-200d with logistic regression and finetuning DistilBert for classification.\n\nThis script requires different requirements, therefore you should install these from the appropriate file:\n\n```bash\npip install -r supervised_requirements.txt\n```\n\nPaths in this script are hardcoded and you might need to adjust it for personal use.\n\n```bash\npython3 supervised_classification.py\n```\n\n\n## Output\n\nThis will output a table with predictions added to the `out_dir` folder in the config.\n\nThe file name format is as follows:\n\n```python\nf\"predictions/{task}_pred_{column}_{model}.csv\"\n```\n\nEach table will have a `pred_\u003cy_column\u003e` and also a `train_test_set` column that is labelled `train` for all examples included in the prompt for few-shot\nlearning and `test` everywhere else.\n\n## Evaluating results\nTo evaluate the performance of the model(s), you can run the CLI `evaluation.py` script. It has two command line arguments: --in_dir and --out_dir. These, respectively, refer to the folder in which the predictions from the llm_classification.py script has been saved (i.e., your predictions folder), and the folder where the classification report(s) should be saved. \n--in_dir defaults to 'predictions/' and --out_dir defaults to 'output/' (which is a folder that is created if it does not exist already)\n\nIt can be run as follows:\n\n```python\npython3 evaluation.py --in_dir \"your/data/path\" --out_dir \"your/out/path\"\n```\n\nIt expects the output file(s) from `llm_classification.py` in the specified file name format and placement. \nIt will output two files to the specified out folder: \n- a txt file with the classification report for the test data for each of the files in the --in_dir folder. \n- a csv file with the same information as the txt file, but which can be used for plotting the results. \n\n\u003ca name=\"plotting_results\"\u003e\u003c/a\u003e\n## Plotting results\nThe `plotting.py` script takes the csv-file produced by the evaluation script and makes three plots:\n- acc_figure.png: The accuracy for each of the 8 models on each outcome (political, exemplar) in each task (zero-shot, few-shot) with each prompt type (generic, custom). It's split into four quadrants, with the left side being the exemplar column, the right being political, the upper line being custom prompts and the lower column being generic prompts. \n- f1_figure.png: The f1-score for positive labels for each model in each task – again split into political and exemplar + generic and custom prompt. \n- prec_rec_figure.png: Precision plotted against recall for each of the models, split into three rows and four columns. Rows indicate task (zero-shot, few-shot, supervised classification), columns indiciate label column (political, exemplar) and prompt type (generic, custom)\n\n\n```python\npython3 plotting.py\n```\n\n\nThese are all saved in a figures/ folder.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcentre-for-humanities-computing%2Fllm-tweet-classification","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcentre-for-humanities-computing%2Fllm-tweet-classification","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcentre-for-humanities-computing%2Fllm-tweet-classification/lists"}