https://github.com/dibyendubiswas1998/document-tagging
Creare CI-CD pipeline that helps to automate the process of assigning relevant tags or categories to large volumes of unstructured textual data.
https://github.com/dibyendubiswas1998/document-tagging
aws-ec2 docker dvc git mlflow python3 pytorch s3-bucket
Last synced: 3 months ago
JSON representation
Creare CI-CD pipeline that helps to automate the process of assigning relevant tags or categories to large volumes of unstructured textual data.
- Host: GitHub
- URL: https://github.com/dibyendubiswas1998/document-tagging
- Owner: dibyendubiswas1998
- License: gpl-3.0
- Created: 2023-10-20T03:29:18.000Z (over 2 years ago)
- Default Branch: main
- Last Pushed: 2023-12-26T14:12:02.000Z (over 2 years ago)
- Last Synced: 2025-01-29T07:14:10.928Z (over 1 year ago)
- Topics: aws-ec2, docker, dvc, git, mlflow, python3, pytorch, s3-bucket
- Language: Jupyter Notebook
- Homepage: http://65.0.74.77:8080/
- Size: 38.2 MB
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# Document Tagging
## Problem Statement:
Document-Tagging system that leverages advanced natural language processing (NLP) techniques and transformer-based pre-trained models for efficient document categorization. This system aims to automatically generate the relevant tags or categories based on large volumes of unstructured textual data. Users can upload documents, which undergo preprocessing and are then classified into multiple relevant categories using NLP models. [Download Problem Statement PDF](./documents/problem%20statement.pdf)
Here I build CI-CD pipeline for generating relevant tags based on given textual data.
[DagsHub](https://dagshub.com/dibyendubiswas1998/Document-Tagging.mlflow)
[WebApp](http://65.0.74.77:8080/)
## Project Documentation:
[High Level Design Dcocument (HLD)](./documents/High%20Level%20Design%20(HLD).pdf)
[Architecture Document](./documents/Architecture.pdf)
[Wireframe Document](./documents/Wireframe.pdf)
[Low Level Design Document (LLD)](./documents/Low%20Level%20Design%20(LLD).pdf)
[Detailed Project Report (DPR)](./documents/Detailed%20Project%20Report%20(DPR).docx.pdf)
## Project Workflow:
* **Step-01:** Load the raw or custom data from **AWS S3 Bucket**, provided by user. And save the data into particular directory.
* **Step-02:** Preprocessed the raw data, like handle the missing values, duplicate values, text-preprocessing, vectorization, separate the X and Y, create tensor dataset and split them into train, test and validation sets.
* **Step-03:** Create the model (default: bert-base-uncased), and train the model. After that save the pre-trained model & tokenizer in a particular directory.
* **Step-04:** Evaluate the model baed on test datasets and save the inflormation on [DagsHub](https://dagshub.com/dibyendubiswas1998/Document-Tagging.mlflow) using mlflow.
* **Step-05:** Create a Web Application for generating the tags and host the entier application on AWS.
## Tech Stack:
## How to Run the Application:
```bash
# For Windows OS:
docker pull dibyendubiswas1998/document_tagging
docker run -p 8080:8080 dibyendubiswas1998/document_tagging
# For Ubuntu OS:
sudo docker pull dibyendubiswas1998/document_tagging
sudo docker run -p 8080:8080 dibyendubiswas1998/document_tagging
```
## Web Interface:
