{"id":18993525,"url":"https://github.com/infinitode/duplipy","last_synced_at":"2026-04-15T10:30:18.639Z","repository":{"id":179469203,"uuid":"663551393","full_name":"Infinitode/duplipy","owner":"Infinitode","description":"DupliPy is a quick and easy-to-use package that can handle text formatting and data augmentation tasks for NLP in Python. It now offers support for image augmentation tasks as well.","archived":false,"fork":false,"pushed_at":"2023-12-27T15:52:57.000Z","size":39,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2024-04-25T14:21:38.545Z","etag":null,"topics":["ai","augmentation","data-analysis","data-preprocessing","data-science","images","language-models","nlp","preprocessing","text-data","text-datasets","text-formatting"],"latest_commit_sha":null,"homepage":"https://infinitode.netlify.app","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Infinitode.png","metadata":{"files":{"readme":"readme.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2023-07-07T14:58:51.000Z","updated_at":"2024-01-08T09:17:47.000Z","dependencies_parsed_at":"2024-01-19T10:13:43.103Z","dependency_job_id":"4df0f6c9-dcf2-465a-88e3-68ff1541a5ce","html_url":"https://github.com/Infinitode/duplipy","commit_stats":{"total_commits":15,"total_committers":2,"mean_commits":7.5,"dds":0.06666666666666665,"last_synced_commit":"d8a70bb8f35c32034b9fd67589234b61c8167af7"},"previous_names":["infinitode/duplipy"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Infinitode%2Fduplipy","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Infinitode%2Fduplipy/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Infinitode%2Fduplipy/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Infinitode%2Fduplipy/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Infinitode","download_url":"https://codeload.github.com/Infinitode/duplipy/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":240008112,"owners_count":19733260,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai","augmentation","data-analysis","data-preprocessing","data-science","images","language-models","nlp","preprocessing","text-data","text-datasets","text-formatting"],"created_at":"2024-11-08T17:21:46.434Z","updated_at":"2026-04-15T10:30:18.587Z","avatar_url":"https://github.com/Infinitode.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# DupliPy 0.2.4\r\n![Python Version](https://img.shields.io/badge/python-3.12-blue.svg)\r\n![Code Size](https://img.shields.io/github/languages/code-size/infinitode/duplipy)\r\n![Downloads](https://pepy.tech/badge/duplipy)\r\n![License Compliance](https://img.shields.io/badge/license-compliance-brightgreen.svg)\r\n![PyPI Version](https://img.shields.io/pypi/v/duplipy)\r\n\r\nAn open source Python library for text formatting, augmentation, and similarity calculation tasks in NLP, the package now also includes additional methods for image augmentation.\r\n\r\n## Changes in DupliPy 0.2.4\r\n\r\n- Created new functions in `duplipy.replication` for image augmentation: `random_flip`, `random_color_jitter`, and `noise_overlay`.\r\n- Created a new function (`post_format_text`) for post-formatting after DupliPy processing or augmentation tasks that cleans up extra whitespace and normalizes punctuation spacing.\r\n\r\n## Changes to DupliPy 0.2.3\r\n\r\nDuplipy now utilizes another one of our Python packages, called ValX, which provides quick methods we can use to clean and format our text data before training in preprocessing steps.\r\n\r\n## Installation\r\n\r\nYou can install DupliPy using pip:\r\n\r\n```bash\r\npip install duplipy\r\n```\r\n\r\n## Supported Python Versions\r\n\r\nDupliPy supports the following Python versions:\r\n\r\n- Python 3.6\r\n- Python 3.7\r\n- Python 3.8\r\n- Python 3.9\r\n- Python 3.10\r\n- Python 3.11\r\n- Python 3.12 or later\r\n\r\nPlease ensure that you have one of these Python versions installed before using DupliPy. DupliPy may not work as expected on lower versions of Python than the supported.\r\n\r\n## Features\r\n\r\n- Text Formatting: Remove special characters, standardize text formatting.\r\n- Text Replication: Generate replicated instances of text for data augmentation.\r\n- Sentiment Analysis: Find impressions within sentences.\r\n- Similarity Calculation: Calculate text similarity using various metrics.\r\n- BLEU Score Calculation: Calculate how well your text-based NLP model performs.\r\n- Image Augmentation Tasks.\r\n- Profanity removal, hate speech removal, offensive speech removal, and sensitive information removal.\r\n\r\n*For full reference documentation view [DupliPy's official documentation](https://infinitode-docs.gitbook.io/documentation/package-documentation/duplipy-package-documentation).*\r\n\r\n## Usage\r\n\r\n### Text Formatting\r\n\r\n```python\r\nfrom duplipy.formatting import remove_special_characters, standardize_text\r\n\r\ntext = \"Hello! This is some text.\"\r\n\r\n# Remove special characters\r\nformatted_text = remove_special_characters(text)\r\nprint(formatted_text)  # Output: Hello This is some text\r\n\r\n# Standardize text formatting\r\nstandardized_text = standardize_text(text)\r\nprint(standardized_text)  # Output: hello! this is some text\r\n```\r\n\r\n### Text Replication\r\n\r\n```python\r\nfrom duplipy.replication import replace_word_with_synonym, augment_text_with_synonyms\r\n\r\ntext = \"Hello! This is some text.\"\r\n\r\n# Replace words with synonyms\r\naugmented_text = augment_text_with_synonyms(text, augmentation_factor=3, probability=0.5)\r\nprint(augmented_text)\r\n\r\n# Output:\r\n# ['Hello! This is some text.', 'Hi! This is some text.', 'Hello! This is certain text.']\r\n\r\n# Load text from a file and augment it\r\nfile_path = \"path/to/file.txt\"\r\naugmented_file_text = augment_file_with_synonyms(file_path, augmentation_factor=3, probability=0.5)\r\nprint(augmented_file_text)\r\n```\r\n\r\n### Sentiment Analysis\r\n\r\n```python\r\nfrom duplipy.text_analysis import analyze_sentiment\r\n\r\ntext = \"I love this product! It's amazing!\"\r\n\r\n# Analyze sentiment\r\nsentiment = analyze_sentiment(text)\r\nprint(sentiment)  # Output: Positive\r\n```\r\n\r\n### Similarity Calculation\r\n\r\n```python\r\nfrom duplipy.similarity import edit_distance_score\r\n\r\ntext1 = \"Hello! How are you?\"\r\ntext2 = \"Hi! How are you doing?\"\r\n\r\n# Calculate edit distance\r\nedit_distance = edit_distance_score(text1, text2)\r\nprint(edit_distance)  # Output: 4\r\n```\r\n\r\n### BLEU Score Calculation\r\n\r\n```python\r\nfrom duplipy.similarity import bleu_score\r\n\r\ntext1 = \"Hello! How are you?\"\r\ntext2 = \"Hi! How are you doing?\"\r\n\r\n# Calculate cosine similarity\r\nbleu_value = bleu_score(text1, text2)\r\nprint(bleu_value)  # Output: 0.434\r\n```\r\n\r\n### Image Augmentation\r\n\r\n```python\r\nfrom PIL import Image\r\nfrom duplipy.replication import flip_horizontal, flip_vertical, rotate, random_rotation, resize, crop, random_crop\r\n\r\n# Load an image for testing\r\nimage_path = \"path/to/image.jpg\"\r\nimage = Image.open(image_path)\r\n\r\n# Flip the image horizontally\r\nflipped_horizontal_image = flip_horizontal(image)\r\n\r\n# Flip the image vertically\r\nflipped_vertical_image = flip_vertical(image)\r\n\r\n# Rotate the image by a specific angle (e.g., 45 degrees)\r\nrotated_image = rotate(image, 45)\r\n\r\n# Apply a random rotation to the image within a specified range of angles (e.g., -30 to 30 degrees)\r\nrandomly_rotated_image = random_rotation(image, max_angle=30)\r\n\r\n# Resize the image to a specific target size (e.g., 224x224 pixels)\r\nresized_image = resize(image, target_size=(224, 224))\r\n\r\n# Crop a random region from the image (e.g., 150x150 pixels)\r\nrandomly_cropped_image = random_crop(image, crop_size=(150, 150))\r\n\r\n# Save the augmented images (optional, if you want to view the results)\r\nflipped_horizontal_image.save(\"path/to/flipped_horizontal.jpg\")\r\nflipped_vertical_image.save(\"path/to/flipped_vertical.jpg\")\r\nrotated_image.save(\"path/to/rotated.jpg\")\r\nrandomly_rotated_image.save(\"path/to/randomly_rotated.jpg\")\r\nresized_image.save(\"path/to/resized.jpg\")\r\nrandomly_cropped_image.save(\"path/to/randomly_cropped.jpg\")\r\n```\r\n\r\n### Hate speech and Offensive speech removal using AI\r\n\r\n```python\r\nfrom duplipy.formatting import remove_hate_speech_from_text\r\n\r\ntext = \"I hate all of you bad word! Can't you just bad word leave me alone! Hi, I'm Katy.\"\r\n\r\nprint(remove_hate_speech_from_text(text))\r\n\r\n### Output\r\n# \"Hi, I'm Katy.\"\r\n```\r\n\r\n## Contributing\r\n\r\nContributions are welcome! If you encounter any issues, have suggestions, or want to contribute to DupliPy, please open an issue or submit a pull request on [GitHub](https://github.com/infinitode/duplipy).\r\n\r\n## License\r\n\r\nDupliPy is released under the terms of the **MIT License (Modified)**. Please see the [LICENSE](https://github.com/infinitode/duplipy/blob/main/LICENSE) file for the full text.\r\n\r\n**Modified License Clause**\r\n\r\n\r\n\r\nThe modified license clause grants users the permission to make derivative works based on the DupliPy software. However, it requires any substantial changes to the software to be clearly distinguished from the original work and distributed under a different name.\r\n\r\nBy enforcing this distinction, it aims to prevent direct publishing of the source code without changes while allowing users to create derivative works that incorporate the code but are not exactly the same.\r\n\r\nPlease read the full license terms in the [LICENSE](https://github.com/infinitode/duplipy/blob/main/LICENSE) file for complete details.\r\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Finfinitode%2Fduplipy","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Finfinitode%2Fduplipy","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Finfinitode%2Fduplipy/lists"}