{"id":14885492,"url":"https://github.com/itsmejayd/filter-duplicate-lines","last_synced_at":"2025-03-01T20:13:39.898Z","repository":{"id":245555440,"uuid":"818588156","full_name":"itsmejayd/filter-duplicate-lines","owner":"itsmejayd","description":"A Python tool with a Flask web interface designed to remove duplicate entries from line break delimited lists while maintaining the relative position of blank lines in the input text. Originally created to clean up OneTab exports (maintaining tab groups), it can be used for any text list with similar requirements.","archived":false,"fork":false,"pushed_at":"2024-07-22T21:09:14.000Z","size":142,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-01-12T11:14:13.442Z","etag":null,"topics":["flask-application","onetab","parser","python"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/itsmejayd.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-06-22T09:00:52.000Z","updated_at":"2024-11-21T16:53:03.000Z","dependencies_parsed_at":"2024-09-21T16:02:31.532Z","dependency_job_id":"b4ad73df-0fda-4b9d-a268-eb43a9b522e8","html_url":"https://github.com/itsmejayd/filter-duplicate-lines","commit_stats":null,"previous_names":["itsmejayd/filter-duplicate-lines"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/itsmejayd%2Ffilter-duplicate-lines","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/itsmejayd%2Ffilter-duplicate-lines/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/itsmejayd%2Ffilter-duplicate-lines/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/itsmejayd%2Ffilter-duplicate-lines/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/itsmejayd","download_url":"https://codeload.github.com/itsmejayd/filter-duplicate-lines/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":241418357,"owners_count":19959736,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["flask-application","onetab","parser","python"],"created_at":"2024-09-21T16:01:14.071Z","updated_at":"2025-03-01T20:13:39.881Z","avatar_url":"https://github.com/itsmejayd.png","language":"Python","funding_links":[],"categories":["Python"],"sub_categories":[],"readme":"\u003ca name=\"readme-top\"\u003e\u003c/a\u003e\n\u003c!-- PROJECT SHIELDS --\u003e\n\u003c!--\n*** I'm using markdown \"reference style\" links for readability.\n*** Reference links are enclosed in brackets [ ] instead of parentheses ( ).\n*** See the bottom of this document for the declaration of the reference variables\n*** for contributors-url, forks-url, etc. This is an optional, concise syntax you may use.\n*** https://www.markdownguide.org/basic-syntax/#reference-style-links\n--\u003e\n[![Forks][forks-shield]][forks-url]\n[![Stargazers][stars-shield]][stars-url]\n[![Issues][issues-shield]][issues-url]\n[![MIT License][license-shield]][license-url]\n[![LinkedIn][linkedin-shield]][linkedin-url]\n\n\n\n\u003c!-- PROJECT LOGO --\u003e\n\u003cbr /\u003e\n\u003cdiv align=\"center\"\u003e\n  \u003ca href=\"https://github.com/itsmejayd/filter-duplicate-lines\"\u003e\n    \u003cimg src=\"images/logo.png\" alt=\"Logo\" width=\"80\" height=\"80\"\u003e\n  \u003c/a\u003e\n\n\u003ch3 align=\"center\"\u003eDuplicate Line Remover\u003c/h3\u003e\n\n  \u003cp align=\"center\"\u003e\n    A Python tool with a Flask web interface designed to remove duplicate entries from line break delimited lists while maintaining the relative position of blank lines in the input text. Originally created to clean up OneTab exports (maintaining tab groups), it can be used for any text list with similar requirements.\n    \u003cbr /\u003e\n    \u003ca href=\"https://github.com/itsmejayd/filter-duplicate-lines\"\u003e\u003cstrong\u003eExplore the docs »\u003c/strong\u003e\u003c/a\u003e\n    \u003cbr /\u003e\n    \u003cbr /\u003e\n    \u003c!--\n    \u003ca href=\"https://github.com/itsmejayd/filter-duplicate-lines\"\u003eView Demo\u003c/a\u003e\n    --\u003e\n    ·\n    \u003ca href=\"https://github.com/itsmejayd/filter-duplicate-lines/issues/new?labels=bug\u0026template=bug-report---.md\"\u003eReport Bug\u003c/a\u003e\n    ·\n    \u003ca href=\"https://github.com/itsmejayd/filter-duplicate-lines/issues/new?labels=enhancement\u0026template=feature-request---.md\"\u003eRequest Feature\u003c/a\u003e\n  \u003c/p\u003e\n\u003c/div\u003e\n\n\n\n\u003c!-- TABLE OF CONTENTS --\u003e\n\u003cdetails\u003e\n  \u003csummary\u003eTable of Contents\u003c/summary\u003e\n  \u003col\u003e\n    \u003cli\u003e\n      \u003ca href=\"#about-the-project\"\u003eAbout The Project\u003c/a\u003e\n      \u003cul\u003e\n        \u003cli\u003e\u003ca href=\"#built-with\"\u003eBuilt With\u003c/a\u003e\u003c/li\u003e\n      \u003c/ul\u003e\n    \u003c/li\u003e\n    \u003cli\u003e\n      \u003ca href=\"#getting-started\"\u003eGetting Started\u003c/a\u003e\n      \u003cul\u003e\n        \u003cli\u003e\u003ca href=\"#prerequisites\"\u003ePrerequisites\u003c/a\u003e\u003c/li\u003e\n        \u003cli\u003e\u003ca href=\"#installation\"\u003eInstallation\u003c/a\u003e\u003c/li\u003e\n      \u003c/ul\u003e\n    \u003c/li\u003e\n    \u003cli\u003e\u003ca href=\"#usage\"\u003eUsage\u003c/a\u003e\u003c/li\u003e\n    \u003cli\u003e\u003ca href=\"#contributing\"\u003eContributing\u003c/a\u003e\u003c/li\u003e\n    \u003cli\u003e\u003ca href=\"#license\"\u003eLicense\u003c/a\u003e\u003c/li\u003e\n    \u003cli\u003e\u003ca href=\"#contact\"\u003eContact\u003c/a\u003e\u003c/li\u003e\n    \u003cli\u003e\u003ca href=\"#acknowledgments\"\u003eAcknowledgments\u003c/a\u003e\u003c/li\u003e\n  \u003c/ol\u003e\n\u003c/details\u003e\n\n\n\n\u003c!-- ABOUT THE PROJECT --\u003e\n## About The Project\n\n[![Product Name Screen Shot][product-screenshot]](https://github.com/itsmejayd/filter-duplicate-lines)\n\n\n\n### Built With\n\n\n* [![Python][Python.org]][Python-url]\n* [![Flask][flask.palletsprojects.com]][Flask-url]\n\n\n\n\u003c!-- GETTING STARTED --\u003e\n## Getting Started\n\nThis guide provides instructions on setting up and using the \"Duplicate Line Remover\" tool locally.\n\n### Prerequisites\n\nTo run this project, you'll need:\n- Python (version 3.11.0 recommended)\n- Flask (for the web application)\n\nNote: I've added `.python-version` to `.gitignore` to ensure compatibility across different Python versions. For reference, the version used during development was 3.11.0.\n\nYou may want to set up a virtual environment (venv) to manage dependencies, for exmaple:\n```sh\npython -m venv env\n```\n```sh\nsource env/bin/activate\n```\n\nInstall Flask:\n```sh\npip install flask\n```\n\n### Installation\n\n   Clone the repo:\n   ```sh\n   git clone https://github.com/itsmejayd/filter-duplicate-lines.git\n   ```\n   ```sh\n   cd filter-duplicate-lines\n   ```\n\n\n\u003c!-- USAGE EXAMPLES --\u003e\n## Usage\n\n1. Removing Duplicate Lines:\n  - Use `remove_duplicates(input_file, output_file)` function from `filter_duplicate_lines.py` to process your input file and generate a cleaned output file without duplicate lines.\n\n  Example usage:\n  ```python\n  from filter_duplicate_lines import remove_duplicates\n\n  input_file = \"your_input_file_name.txt\"\n  output_file = \"cleaned_output.txt\"\n\n  remove_duplicates(input_file, output_file)\n  ```\n\n2. Using the Web Application:\n\n  - Start the Flask application by running python app.py in your terminal.\n  - Access the application in your web browser at http://localhost:5000.\n  - Upload a text file or paste text into the text box, then click download button to remove duplicates and download the cleaned file.\n\n\n\n\n\u003c!-- CONTRIBUTING --\u003e\n## Contributing\n\nIf you have a suggestion that would make this better, please fork the repo and create a pull request. You can also simply open an issue with the tag \"enhancement\".\nDon't forget to give the project a star! Thanks again!\n\n1. Fork the Project\n2. Create your Feature Branch (`git checkout -b feature/AmazingFeature`)\n3. Commit your Changes (`git commit -m 'Add some AmazingFeature'`)\n4. Push to the Branch (`git push origin feature/AmazingFeature`)\n5. Open a Pull Request\n\nSee the [open issues](https://github.com/itsmejayd/filter-duplicate-lines/issues) for a full list of proposed features (and known issues).\n\n\n\n\u003c!-- LICENSE --\u003e\n## License\n\nDistributed under the MIT License. See `LICENSE` for more information.\n\n\n\n\u003c!-- CONTACT --\u003e\n## Contact\n\nJoseph Dwyer - josephdwyer20@gmail.com - [Twitter](https://twitter.com/jdlately)\n\n\n\n\u003c!-- ACKNOWLEDGMENTS --\u003e\n## Acknowledgments\n\n* [OneTab](https://www.one-tab.com/) Browser Extension: A wonderful tool I love, inspired me to make my tool specifically to increase OneTab's functionality and to share it for others to use.\n\n\n\u003cp align=\"right\"\u003e(\u003ca href=\"#readme-top\"\u003eback to top\u003c/a\u003e)\u003c/p\u003e\n\n\u003c!-- https://www.markdownguide.org/basic-syntax/#reference-style-links --\u003e\n[forks-shield]: https://img.shields.io/github/forks/itsmejayd/filter-duplicate-lines.svg?style=for-the-badge\n[forks-url]: https://github.com/itsmejayd/filter-duplicate-lines/network/members\n[stars-shield]: https://img.shields.io/github/stars/itsmejayd/filter-duplicate-lines.svg?style=for-the-badge\n[stars-url]: https://github.com/itsmejayd/filter-duplicate-lines/stargazers\n[issues-shield]: https://img.shields.io/github/issues/itsmejayd/filter-duplicate-lines.svg?style=for-the-badge\n[issues-url]: https://github.com/itsmejayd/filter-duplicate-lines/issues\n[license-shield]: https://img.shields.io/github/license/itsmejayd/filter-duplicate-lines.svg?style=for-the-badge\n[license-url]: https://github.com/itsmejayd/filter-duplicate-lines/blob/master/LICENSE.txt\n[linkedin-shield]: https://img.shields.io/badge/-LinkedIn-black.svg?style=for-the-badge\u0026logo=linkedin\u0026colorB=555\n[linkedin-url]: https://linkedin.com/in/josephdwyer20\n[product-screenshot]: images/flask_web_app.png\n[Python.org]: https://img.shields.io/badge/Python-FFD43B?style=for-the-badge\u0026logo=python\u0026logoColor=blue\n[Python-url]: https://www.python.org/\n[Flask.palletsprojects.com]: https://img.shields.io/badge/Flask-000000?style=for-the-badge\u0026logo=flask\u0026logoColor=white\n[Flask-url]: https://flask.palletsprojects.com/en/3.0.x/\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fitsmejayd%2Ffilter-duplicate-lines","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fitsmejayd%2Ffilter-duplicate-lines","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fitsmejayd%2Ffilter-duplicate-lines/lists"}