{"id":28790699,"url":"https://github.com/lucasfrag/fact-checking-dataset-translator","last_synced_at":"2025-08-01T15:14:39.955Z","repository":{"id":297654309,"uuid":"993440838","full_name":"lucasfrag/fact-checking-dataset-translator","owner":"lucasfrag","description":"This repository provides scripts and workflows for translating fact-checking datasets and automating claim classification using large language models (LLMs).","archived":false,"fork":false,"pushed_at":"2025-07-01T08:59:15.000Z","size":13616,"stargazers_count":2,"open_issues_count":0,"forks_count":1,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-07-01T09:44:13.976Z","etag":null,"topics":["dataset","fact-checking","few-shot","few-shot-learning","llm","nlp","ollama","pt-br","python","translation","zero-shot","zero-shot-learning"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/lucasfrag.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-05-30T19:48:53.000Z","updated_at":"2025-07-01T08:59:19.000Z","dependencies_parsed_at":"2025-06-17T23:18:06.041Z","dependency_job_id":null,"html_url":"https://github.com/lucasfrag/fact-checking-dataset-translator","commit_stats":null,"previous_names":["lucasfrag/dataset-translator-ollama","lucasfrag/fact-checking-dataset-translator"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/lucasfrag/fact-checking-dataset-translator","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lucasfrag%2Ffact-checking-dataset-translator","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lucasfrag%2Ffact-checking-dataset-translator/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lucasfrag%2Ffact-checking-dataset-translator/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lucasfrag%2Ffact-checking-dataset-translator/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/lucasfrag","download_url":"https://codeload.github.com/lucasfrag/fact-checking-dataset-translator/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lucasfrag%2Ffact-checking-dataset-translator/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":268248938,"owners_count":24219559,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-01T02:00:08.611Z","response_time":67,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["dataset","fact-checking","few-shot","few-shot-learning","llm","nlp","ollama","pt-br","python","translation","zero-shot","zero-shot-learning"],"created_at":"2025-06-17T23:07:58.381Z","updated_at":"2025-08-01T15:14:39.950Z","avatar_url":"https://github.com/lucasfrag.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Fact-Checking Dataset Translator and Fine-tuning Project\r\n\r\nThis project focuses on fact-checking using various language models, particularly for English and Brazilian Portuguese (PT-BRr) datasets. It includes scripts for zero-shot and few-shot inference, dataset translation, model fine-tuning, and result analysis.\r\n\r\n## Project Structure\r\n\r\nThe project is organized into three main directories:\r\n\r\n  * **`averitec/`**: Contains scripts and results related to the Averitec fact-checking dataset.\r\n  * **`liar/`**: Contains scripts and results related to the LIAR fact-checking dataset.\r\n  * **`fine-tuning/`**: Houses scripts and configurations for fine-tuning language models on fact-checking datasets.\r\n\r\n### Detailed Directory Breakdown:\r\n\r\n  * **`averitec/` and `liar/`**\r\n\r\n      * **`dataset/`**: Stores the raw fact-checking datasets (`dev.json`, `train.json`, `test.jsonl`, `valid.jsonl`).\r\n      * **`results/`**: Contains subdirectories for different language models (e.g., `gemma3-4b/`, `llama3-8b/`, `deepseek-r1-7b/`) and JSON files with inference results (e.g., `few_shot_all.json`, `zero_shot_only_statement.json`).\r\n      * **`scripts/`**: Includes Python scripts for running inference and generating reports:\r\n          * `few_shot_all.py`: Performs few-shot inference including claim, subjects, speaker, job title, state, party affiliation, and context.\r\n          * `few_shot_only_statement.py`: Performs few-shot inference using only the claim statement.\r\n          * `zero_shot_all.py`: Performs zero-shot inference including all available context fields.\r\n          * `zero_shot_only_statement.py`: Performs zero-shot inference using only the claim statement.\r\n          * `report.py`: Generates classification reports, confusion matrices, and ordinal error metrics from inference results.\r\n          * `translate.py`: Translates datasets using a specified language model.\r\n      * **`results_overview.ipynb`**: Jupyter notebooks for visualizing and analyzing results of zero-shot and few-shot experiments.\r\n\r\n  * **`fine-tuning/`**\r\n\r\n      * **`configs/`**: Stores JSON configuration files for various fine-tuning experiments, specifying model names, dataset paths, hyperparameters (learning rate, batch size, epochs), and other training settings (e.g., `BERT_LARGE_cased_ENG_LIAR_answer_LR_2e-5_1kepochs.json`, `finetune_mBERT_cased_PTBR_averitec_answers_LR_2e-5_2kepochs.json`).\r\n      * **`datasets/`**: Contains Python scripts for custom dataset classes:\r\n          * `averitec.py`: Dataset class for the Averitec dataset.\r\n          * `liar.py`: Dataset class for the LIAR dataset.\r\n      * **`errors/`**: Stores CSV files detailing errors from specific fine-tuning runs (e.g., `BERT_LARGE_cased_ENG_LIAR_answer_gold_errors.csv`).\r\n      * **`LIAR_LR_hyperparam_search_csvs/`**: Contains CSV files with results from hyperparameter search for learning rates on the LIAR dataset (e.g., `BERT_LARGE_cased_ENG_LIAR_answer_LR_2e-5_1kepochs_optuna_trials.csv`).\r\n      * **`hyperparam_search.py`**: Script for performing hyperparameter optimization using Optuna.\r\n      * **`main.py`**: The main script for running fine-tuning experiments based on a specified configuration file.\r\n      * **`model.py`**: Defines the model architecture and loading functions for pre-trained language models.\r\n      * **`results.ipynb`**: Jupyter notebook for analyzing fine-tuning results.\r\n      * **`training.py`**: Contains functions for training and evaluating models.\r\n      * **`utils.py`**: Provides utility functions such as loading configurations, setting random seeds, saving metadata, and finalizing test results.\r\n      * **`.gitignore`**: Specifies intentionally untracked files to ignore.\r\n\r\n  * **`requirements.txt`**: Lists Python dependencies required to run the project.\r\n\r\n## Datasets\r\n\r\nThis project utilizes the following fact-checking datasets:\r\n\r\n  * **[LIAR Dataset](https://www.cs.ucsb.edu/~william/data/liar_dataset.zip)**: A publicly available dataset for fake news detection, consisting of 12.8K manually fact-checked short statements from PolitiFact.com. Each statement is accompanied by meta-information and a label indicating its veracity.\r\n\r\n\r\n  * **[Averitec Dataset](https://fever.ai/dataset/averitec.html)**: A dataset used for evidence-based fact-checking, typically involving claims that require external evidence for verification.\r\n\r\n\r\n## Getting Started\r\n\r\nThese instructions will get you a copy of the project up and running on your local machine for development and testing purposes.\r\n\r\n### Prerequisites\r\n\r\nYou need Python 3.8+ installed. It is recommended to use a virtual environment.\r\n\r\n### Installation\r\n\r\n1.  **Clone the repository (if not already done):**\r\n\r\n    ```bash\r\n    git clone https://github.com/lucasfrag/fact-checking-dataset-translator\r\n    cd fact-checking-dataset-translator\r\n    ```\r\n\r\n2.  **Create and activate a virtual environment:**\r\n\r\n    ```bash\r\n    python -m venv venv\r\n    # On Windows:\r\n    .\\venv\\Scripts\\activate\r\n    # On macOS/Linux:\r\n    source venv/bin/activate\r\n    ```\r\n\r\n3.  **Install the required dependencies:**\r\n\r\n    ```bash\r\n    pip install -r requirements.txt\r\n    ```\r\n\r\n### Ollama Setup\r\n\r\nThis project uses `ollama` for running language models locally. You need to install `ollama` and pull the necessary models.\r\n\r\n1.  **Download and install Ollama:**\r\n    Follow the instructions on the [Ollama website](https://ollama.com/download) to install it for your operating system.\r\n\r\n2.  **Pull the required models:**\r\n    The scripts frequently use `gemma3:4b`. You can pull them using the following commands:\r\n\r\n    ```bash\r\n    ollama pull gemma3:4b\r\n    ```\r\n\r\n    Ensure your Ollama server is running before executing the project scripts.\r\n\r\n## Usage\r\n\r\n### Running Inference (Zero-Shot and Few-Shot)\r\n\r\nNavigate to either the `averitec/` or `liar/` directory and then into the `scripts/` folder.\r\n\r\n#### Zero-Shot Inference:\r\n\r\nTo run zero-shot inference, execute the respective Python script. For example, for Portuguese LIAR dataset:\r\n\r\n```bash\r\ncd liar/pt-br/scripts/\r\npython zero_shot_all.py\r\n# or for only statement:\r\npython zero_shot_only_statement.py\r\n```\r\n\r\nFor English Averitec dataset:\r\n\r\n```bash\r\ncd averitec/english/scripts/\r\npython zero_shot_claim_Youtubes.py\r\n# or for only claim and question:\r\npython zero_shot_claim_question.py\r\n```\r\n\r\n#### Few-Shot Inference:\r\n\r\nTo run few-shot inference, execute the respective Python script. For example, for Portuguese LIAR dataset:\r\n\r\n```bash\r\ncd liar/pt-br/scripts/\r\npython few_shot_all.py\r\n# or for only statement:\r\npython few_shot_only_statement.py\r\n```\r\n\r\nFor English Averitec dataset:\r\n\r\n```bash\r\ncd averitec/english/scripts/\r\npython few_shot_claim_Youtubes.py\r\n# or for only claim and question:\r\npython few_shot_claim_question.py\r\n```\r\n\r\n### Hyperparameter Search\r\n\r\nTo perform hyperparameter optimization, execute the `hyperparam_search.py` script with a configuration file.\r\n\r\n```bash\r\npython hyperparam_search.py \u003cpath_to_config_json\u003e\r\n# Example:\r\npython hyperparam_search.py configs/example.json\r\n```\r\n\r\nThis script uses Optuna to find optimal learning rates, and results are saved in CSV files within the `LIAR_LR_hyperparam_search_csvs/` directory.\r\n\r\n### Dataset Translation\r\n\r\nTo translate datasets, navigate to the `averitec/pt-br/scripts/` or `liar/pt-br/scripts/` directory and run the `translate.py` script.\r\n\r\n```bash\r\ncd averitec/pt-br/scripts/\r\npython translate.py\r\n```\r\n\r\nThis script will translate specified datasets using the configured Ollama model.\r\n\r\n## Analysis and Results\r\n\r\nJupyter notebooks located in `averitec/english/results_overview.ipynb`, `averitec/pt-br/results_overview.ipynb` and `liar/english/results_overview.ipynb`, `liar/pt-br/results_overview.ipynb` and `fine-tuning/results.ipynb` provide detailed analyses and visualizations of the experimental results. You can open these notebooks to explore:\r\n\r\n  * Overall evaluation metrics (exact matches, accuracy).\r\n  * Confusion matrices.\r\n  * Classification reports (precision, recall, f1-score).\r\n  * Ordinal error metrics (mean, median, mode error, Quadratic Weighted Kappa).\r\n\r\nTo view these notebooks, make sure you have Jupyter installed (`pip install jupyter`) and run:\r\n\r\n```bash\r\njupyter notebook\r\n```\r\n\r\nThen navigate to the respective `.ipynb` file.\r\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flucasfrag%2Ffact-checking-dataset-translator","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flucasfrag%2Ffact-checking-dataset-translator","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flucasfrag%2Ffact-checking-dataset-translator/lists"}