{"id":16162908,"url":"https://github.com/fabriziosalmi/text-boundaries","last_synced_at":"2025-04-07T04:23:27.606Z","repository":{"id":236246855,"uuid":"792194689","full_name":"fabriziosalmi/text-boundaries","owner":"fabriziosalmi","description":"A Python-based tool for preprocessing, cleaning, and analyzing text datasets, designed to filter, deduplicate, sort data, and generate statistical insights. ","archived":false,"fork":false,"pushed_at":"2024-09-16T20:18:44.000Z","size":7274,"stargazers_count":0,"open_issues_count":0,"forks_count":1,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-02-13T08:52:21.528Z","etag":null,"topics":["data-automation","data-deduplication","data-preprocessing","data-sorting","data-statistics-generation","data-validation","dataset-boundaries","dataset-cleaning","machine-learning","natural-language-processing","text-data-analysis"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/fabriziosalmi.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-04-26T07:04:51.000Z","updated_at":"2024-09-16T20:18:48.000Z","dependencies_parsed_at":null,"dependency_job_id":"a9c8b74e-c114-47cc-9aa6-4bf925c98ad9","html_url":"https://github.com/fabriziosalmi/text-boundaries","commit_stats":null,"previous_names":["fabriziosalmi/dataset-ita"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fabriziosalmi%2Ftext-boundaries","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fabriziosalmi%2Ftext-boundaries/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fabriziosalmi%2Ftext-boundaries/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fabriziosalmi%2Ftext-boundaries/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/fabriziosalmi","download_url":"https://codeload.github.com/fabriziosalmi/text-boundaries/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247590266,"owners_count":20963114,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["data-automation","data-deduplication","data-preprocessing","data-sorting","data-statistics-generation","data-validation","dataset-boundaries","dataset-cleaning","machine-learning","natural-language-processing","text-data-analysis"],"created_at":"2024-10-10T02:33:50.995Z","updated_at":"2025-04-07T04:23:27.561Z","avatar_url":"https://github.com/fabriziosalmi.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# 📝 Text boundaries \n\n## 📖 Overview\n\n**Text boundaries ** is a powerful Python script designed to help you clean, preprocess, and analyze text data! 🚀 This script reads a text file, filters lines containing only alphabetical characters, removes duplicates, sorts them, and generates detailed statistics about the data. It’s perfect for finding the boundaries of datasets for machine learning models, natural language processing, or validating dataset quality. 🧑‍💻\n\n## ✨ Features\n\n- 🔍 **Filter Lines**: Retains only lines with alphabetical characters to ensure data consistency.\n- 🗑️ **Remove Duplicates**: Ensures every line is unique for effective data preprocessing.\n- 🔠 **Sort Case-Sensitively**: Keeps the data order sensitive to case, which is useful for NLP tasks.\n- 📊 **Generate Statistics**: Provides insights such as:\n  - Number of lines starting or ending with each letter.\n  - Longest word starting with each letter.\n  - First and last word starting with each letter.\n- 📂 **Outputs**:\n  - Cleaned dataset is saved back to `dataset.txt`.\n  - Detailed statistics are written to `boundaries.log`.\n\n## 🛠️ Prerequisites\n\n- **Python 3.9+**: Make sure Python is installed and available in your system's PATH.\n\n## 🚀 Quick Start\n\n1. **Clone the Repository**:\n   ```bash\n   git clone https://github.com/fabriziosalmi/text-boundaries.git\n   cd text-boundaries\n   ```\n\n2. **Add Your Data**: Place your text data in a file named `dataset.txt` in the repository directory.\n\n3. **Run the Script**:\n   ```bash\n   python process_dataset.py\n   ```\n\n4. **Check the Outputs**:\n   - Updated data in `dataset.txt`.\n   - Statistics in `boundaries.log`.\n\n## 🔍 Use Cases and Examples\n\n### 1. **Finding Dataset Boundaries for NLP Preprocessing**\n\nWhen working with text datasets, such as those used to train or fine-tune language models, it's essential to clean the data and identify the boundaries of valid input. This script helps in preprocessing text data by removing invalid entries and generating useful statistics that help define the boundaries of a valid dataset.\n\n#### **Example: NLP Dataset Validation**\n\nGiven an input file `dataset.txt` containing:\n\n```\napple\nBanana\n!@#$%^\ncarrot\napple\nApricot\n123456\ndragonfruit\napple\nBanana\n```\n\nRunning the script will produce:\n\n- **Filtered and Cleaned Data** in `dataset.txt`:\n\n    ```\n    Apricot\n    Banana\n    apple\n    carrot\n    dragonfruit\n    ```\n\n- **Detailed Statistics** in `boundaries.log`:\n\n    ```\n    Total rows: 5\n    A: Starts 2, Ends 0, Longest Apricot, First Apricot, Last Apricot\n    B: Starts 1, Ends 0, Longest Banana, First Banana, Last Banana\n    C: Starts 1, Ends 0, Longest carrot, First carrot, Last carrot\n    D: Starts 1, Ends 0, Longest dragonfruit, First dragonfruit, Last dragonfruit\n    ```\n\nThis allows you to quickly understand the composition and range of your dataset, helping ensure that your data meets the requirements for training NLP models.\n\n### 2. **Identifying Boundaries in Large Language Model (LLM) Training Data**\n\nWhen preparing datasets for LLMs, understanding the range of data (such as the first and last words alphabetically, or the longest word per letter) can be critical in optimizing data loading, tokenization strategies, or even debugging unexpected behavior in trained models.\n\n#### **Example: Preparing Data for LLM Training**\n\nImagine you have a dataset of vocabulary terms that will be used to fine-tune an LLM. The dataset contains words from various domains, but you need to identify the longest word and the range of starting letters to manage tokenization effectively.\n\nGiven the input file `dataset.txt`:\n\n```\nneuroscience\nabracadabra\nxylophone\nQuantum\nalgorithm\nArtificial\nZoo\nnightmare\nalpha\n```\n\nAfter running the script, `boundaries.log` provides insights like:\n\n```\nTotal rows: 9\nA: Starts 3, Ends 1, Longest Artificial, First Artificial, Last algorithm\nN: Starts 2, Ends 0, Longest neuroscience, First neuroscience, Last nightmare\nQ: Starts 1, Ends 0, Longest Quantum, First Quantum, Last Quantum\nX: Starts 1, Ends 0, Longest xylophone, First xylophone, Last xylophone\nZ: Starts 1, Ends 0, Longest Zoo, First Zoo, Last Zoo\n```\n\nFrom this output, you can see that:\n- The dataset has a balanced mix of starting letters.\n- The longest word is **\"neuroscience\"**.\n- Words start with different letters, from **A** to **Z**.\n\n### 3. **Finding Boundaries in Multi-language Datasets**\n\nFor multi-language datasets, especially when developing multilingual LLMs, you might want to check that your dataset contains a balanced range of entries across different languages. This script helps in identifying such patterns.\n\n#### **Example: Multi-language Dataset Analysis**\n\nInput file `dataset.txt`:\n\n```\nこんにちは\nHello\nBonjour\nHola\n안녕하세요\nHallo\n你好\nHola\nBonjour\n```\n\nRunning the script will yield:\n\n- **Filtered Data** (since non-Latin characters will be filtered out):\n\n    ```\n    Bonjour\n    Hello\n    Hola\n    ```\n\n- **Statistics**:\n\n    ```\n    Total rows: 3\n    B: Starts 1, Ends 0, Longest Bonjour, First Bonjour, Last Bonjour\n    H: Starts 2, Ends 0, Longest Hello, First Hello, Last Hola\n    ```\n\nYou can use this information to adjust your dataset for balanced language representation or decide on additional preprocessing steps.\n\n## 🖥️ Automate with GitHub Actions\n\nAutomate your dataset processing with GitHub Actions! 🤖 Whenever `dataset.txt` is updated, the script runs and updates the results automatically.\n\n### 🔄 Workflow Configuration\n\nTo set up automation, create a `.github/workflows/process_dataset.yml` file with:\n\n```yaml\nname: Process Dataset\n\non:\n  workflow_dispatch:\n  push:\n    paths:\n      - 'dataset.txt'  # Trigger the workflow when dataset.txt is updated\n\njobs:\n  update-dataset:\n    runs-on: ubuntu-latest\n    steps:\n    - name: Checkout code\n      uses: actions/checkout@v2\n\n    - name: Set up Python\n      uses: actions/setup-python@v2\n      with:\n        python-version: '3.9'\n\n    - name: Install dependencies\n      run: |\n        python -m pip install --upgrade pip\n        echo 'No dependencies to install'\n\n    - name: Process dataset\n      run: python process_dataset.py\n\n    - name: Commit changes\n      run: |\n        git config --global user.name 'Fabrizio Salmi'\n        git config --global user.email 'fabrizio.salmi@gmail.com'\n        git add dataset.txt boundaries.log\n        git commit -m \"Update dataset and boundaries\"\n        git push\n```\n\n## 🤝 Contributing\n\nContributions are welcome! 🎉 If you have ideas to improve the script or find a bug, feel free to open an issue or submit a pull request.\n\n## 📜 License\n\nThis project is licensed under the MIT License. See the [LICENSE](LICENSE) file for details.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffabriziosalmi%2Ftext-boundaries","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ffabriziosalmi%2Ftext-boundaries","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffabriziosalmi%2Ftext-boundaries/lists"}