Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/itsmejayd/filter-duplicate-lines

A Python tool with a Flask web interface designed to remove duplicate entries from line break delimited lists while maintaining the relative position of blank lines in the input text. Originally created to clean up OneTab exports (maintaining tab groups), it can be used for any text list with similar requirements.
https://github.com/itsmejayd/filter-duplicate-lines

flask-application onetab parser python

Last synced: 17 days ago
JSON representation

Host: GitHub
URL: https://github.com/itsmejayd/filter-duplicate-lines
Owner: itsmejayd
License: mit
Created: 2024-06-22T09:00:52.000Z (7 months ago)
Default Branch: master
Last Pushed: 2024-07-22T21:09:14.000Z (6 months ago)
Last Synced: 2024-11-13T04:13:32.678Z (3 months ago)
Topics: flask-application, onetab, parser, python
Language: Python
Homepage:
Size: 139 KB
Stars: 1
Watchers: 1
Forks: 0
Open Issues: 0
Metadata Files:
- Readme: README.md
- License: LICENSE

Awesome Lists containing this project

README

        

[![Forks][forks-shield]][forks-url]

[![Stargazers][stars-shield]][stars-url]

[![Issues][issues-shield]][issues-url]

[![MIT License][license-shield]][license-url]

[![LinkedIn][linkedin-shield]][linkedin-url]






  

    

  

Duplicate Line Remover


  


    A Python tool with a Flask web interface designed to remove duplicate entries from line break delimited lists while maintaining the relative position of blank lines in the input text. Originally created to clean up OneTab exports (maintaining tab groups), it can be used for any text list with similar requirements.

    


    Explore the docs »

    


    


    

    ·

    Report Bug

    ·

    Request Feature

  



  Table of Contents

  


    

      About The Project

      

        Built With

      

    

    

      Getting Started

      

        Prerequisites

        Installation

      

    

    Usage

    Contributing

    License

    Contact

    Acknowledgments

  


## About The Project

[![Product Name Screen Shot][product-screenshot]](https://github.com/itsmejayd/filter-duplicate-lines)

### Built With

* [![Python][Python.org]][Python-url]

* [![Flask][flask.palletsprojects.com]][Flask-url]

## Getting Started

This guide provides instructions on setting up and using the "Duplicate Line Remover" tool locally.

### Prerequisites

To run this project, you'll need:

- Python (version 3.11.0 recommended)

- Flask (for the web application)

Note: I've added `.python-version` to `.gitignore` to ensure compatibility across different Python versions. For reference, the version used during development was 3.11.0.

You may want to set up a virtual environment (venv) to manage dependencies, for exmaple:

```sh

python -m venv env

```

```sh

source env/bin/activate

```

Install Flask:

```sh

pip install flask

```

### Installation

   Clone the repo:

   ```sh

   git clone https://github.com/itsmejayd/filter-duplicate-lines.git

   ```

   ```sh

   cd filter-duplicate-lines

   ```

## Usage

1. Removing Duplicate Lines:

  - Use `remove_duplicates(input_file, output_file)` function from `filter_duplicate_lines.py` to process your input file and generate a cleaned output file without duplicate lines.

  Example usage:

  ```python

  from filter_duplicate_lines import remove_duplicates

  input_file = "your_input_file_name.txt"

  output_file = "cleaned_output.txt"

  remove_duplicates(input_file, output_file)

  ```

2. Using the Web Application:

  - Start the Flask application by running python app.py in your terminal.

  - Access the application in your web browser at http://localhost:5000.

  - Upload a text file or paste text into the text box, then click download button to remove duplicates and download the cleaned file.

## Contributing

If you have a suggestion that would make this better, please fork the repo and create a pull request. You can also simply open an issue with the tag "enhancement".

Don't forget to give the project a star! Thanks again!

1. Fork the Project

2. Create your Feature Branch (`git checkout -b feature/AmazingFeature`)

3. Commit your Changes (`git commit -m 'Add some AmazingFeature'`)

4. Push to the Branch (`git push origin feature/AmazingFeature`)

5. Open a Pull Request

See the [open issues](https://github.com/itsmejayd/filter-duplicate-lines/issues) for a full list of proposed features (and known issues).

## License

Distributed under the MIT License. See `LICENSE` for more information.

## Contact

Joseph Dwyer - [email protected] - [Twitter](https://twitter.com/jdlately)

## Acknowledgments

* [OneTab](https://www.one-tab.com/) Browser Extension: A wonderful tool I love, inspired me to make my tool specifically to increase OneTab's functionality and to share it for others to use.

(back to top)


[forks-shield]: https://img.shields.io/github/forks/itsmejayd/filter-duplicate-lines.svg?style=for-the-badge

[forks-url]: https://github.com/itsmejayd/filter-duplicate-lines/network/members

[stars-shield]: https://img.shields.io/github/stars/itsmejayd/filter-duplicate-lines.svg?style=for-the-badge

[stars-url]: https://github.com/itsmejayd/filter-duplicate-lines/stargazers

[issues-shield]: https://img.shields.io/github/issues/itsmejayd/filter-duplicate-lines.svg?style=for-the-badge

[issues-url]: https://github.com/itsmejayd/filter-duplicate-lines/issues

[license-shield]: https://img.shields.io/github/license/itsmejayd/filter-duplicate-lines.svg?style=for-the-badge

[license-url]: https://github.com/itsmejayd/filter-duplicate-lines/blob/master/LICENSE.txt

[linkedin-shield]: https://img.shields.io/badge/-LinkedIn-black.svg?style=for-the-badge&logo=linkedin&colorB=555

[linkedin-url]: https://linkedin.com/in/josephdwyer20

[product-screenshot]: images/flask_web_app.png

[Python.org]: https://img.shields.io/badge/Python-FFD43B?style=for-the-badge&logo=python&logoColor=blue

[Python-url]: https://www.python.org/

[Flask.palletsprojects.com]: https://img.shields.io/badge/Flask-000000?style=for-the-badge&logo=flask&logoColor=white

[Flask-url]: https://flask.palletsprojects.com/en/3.0.x/