{"id":24333093,"url":"https://github.com/kianoushamirpour/end_to_end_text_classification","last_synced_at":"2026-04-08T16:01:58.898Z","repository":{"id":272603057,"uuid":"887900905","full_name":"KianoushAmirpour/End_to_End_Text_Classification","owner":"KianoushAmirpour","description":"Developing feature engineering pipelines, building packages, automating tests, and creating FastAPI endpoints.","archived":false,"fork":false,"pushed_at":"2025-01-15T12:55:11.000Z","size":51,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-03-11T13:48:04.162Z","etag":null,"topics":["apache-airflow","ci","docker-compose","factory-design-pattern","fastapi","feast","grafana","hyperopt","mlflow","prometheus","pytorch","scikit-learn","tox","transformers","xgboost-classifier"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/KianoushAmirpour.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-11-13T13:32:57.000Z","updated_at":"2025-01-15T12:58:19.000Z","dependencies_parsed_at":null,"dependency_job_id":"fe7d0226-96f8-4633-b290-360d757fdea0","html_url":"https://github.com/KianoushAmirpour/End_to_End_Text_Classification","commit_stats":null,"previous_names":["kianoushamirpour/end_2_end_text_classification"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/KianoushAmirpour/End_to_End_Text_Classification","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/KianoushAmirpour%2FEnd_to_End_Text_Classification","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/KianoushAmirpour%2FEnd_to_End_Text_Classification/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/KianoushAmirpour%2FEnd_to_End_Text_Classification/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/KianoushAmirpour%2FEnd_to_End_Text_Classification/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/KianoushAmirpour","download_url":"https://codeload.github.com/KianoushAmirpour/End_to_End_Text_Classification/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/KianoushAmirpour%2FEnd_to_End_Text_Classification/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31562697,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-08T14:31:17.711Z","status":"ssl_error","status_checked_at":"2026-04-08T14:31:17.202Z","response_time":54,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["apache-airflow","ci","docker-compose","factory-design-pattern","fastapi","feast","grafana","hyperopt","mlflow","prometheus","pytorch","scikit-learn","tox","transformers","xgboost-classifier"],"created_at":"2025-01-18T03:13:07.220Z","updated_at":"2026-04-08T16:01:58.893Z","avatar_url":"https://github.com/KianoushAmirpour.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"## Table of contents\r\n1. [Description](#description)\r\n2. [Dataset](#dataset)\r\n3. [Feature store](#feature-store)\r\n4. [Feature Engineering Pipeline](#feature-engineering-pipeline)\r\n5. [Training Module](#training-module)\r\n6. [Deployment](#deployment)\r\n7. [CI](#ci)\r\n\r\n## Description\r\nThis project addresses an end-to-end text classification problem based on a Kaggle dataset. It features a pipeline built using Airflow and Feast (feature store) for preprocessing and feature generation. Training and hyperparameter tuning leverage tools like scikit-learn, PyTorch, Transformers, Hyperopt, and Llama.cpp. The best-performing model is packaged for reuse and deployed via FastAPI for predictions. Code quality and functionality are ensured using tox and GitHub Actions.  \r\n\r\n## Dataset\r\nThe dataset is from  [Quora Insincere Questions Classification](https://www.kaggle.com/competitions/quora-insincere-questions-classification/overview).\r\n\r\n## [Feature Store](https://github.com/KianoushAmirpour/End_2_End_Text_Classification/tree/main/feature_store)\r\n**Feast** is used as the feature store, with SQLite serving as the online store.  \r\nFeatures are defined as shown in the example below:\r\n```\r\nquestion_entity = Entity(name='question_id',\r\n                         join_keys=['id'],\r\n                         value_type=ValueType.STRING,\r\n                         description='A key to fetch features for each question.')\r\n\r\nmeta_features_file_source = FileSource(name='meta_features_file_source',\r\n                                       path='data/meta_features.parquet',\r\n                                       timestamp_field='event_timestamp')\r\n\r\nmeta_features = FeatureView(\r\n    name='meta_features_extracted_from_text',\r\n    entities=[question_entity],\r\n    ttl=timedelta(days=20),\r\n    schema=[\r\n        Field(name='num_words', dtype=Int32),\r\n        Field(name='num_unique_words', dtype=Int32),\r\n        Field(name='num_stop_words', dtype=Int32),\r\n        Field(name='num_title_case', dtype=Int32),\r\n        Field(name='ave_length_words', dtype=Float32),\r\n        Field(name='num_characters', dtype=Int32),\r\n        Field(name='num_punctuations', dtype=Int32)\r\n    ],\r\n    source=meta_features_file_source,\r\n    online=True\r\n)\r\n``` \r\n\r\n## [Feature Engineering Pipeline](https://github.com/KianoushAmirpour/End_2_End_Text_Classification/tree/main/feature_pipeline)\r\nThe image below illustrates the feature engineering pipeline:\r\n\r\n![feature_engineering_pipelines](https://github.com/user-attachments/assets/7c78a4c5-73f9-4b4e-a7b4-12b52a49e142)\r\n\r\nThe feature engineering pipeline is implemented with **Apache Airflow** to orchestrate four DAGs for preprocessing, cleaning, and feature generation,\r\nalong with a dedicated DAG for registering features in **Feast (feature store)**.  \r\n\r\nKey components and features:  \r\n* Data Processing: Utilizes **pandas, NLTK, and sentence-transformers** for text preprocessing and feature extraction.\r\n* Data Quality Checks (Each DAG performs validations, including checks for):\r\n   * Row count and column constraints\r\n   * Minimum and null values\r\n   * Presence of required columns\r\n* Error Handling and Alerts: **Automatic email notifications** are triggered upon pipeline failure.\r\n* Monitoring and Logging:\r\n   * **Grafana, Prometheus, and StatsD** are integrated to monitor pipeline health and performance metrics.\r\n   * A custom logging captures detailed process logs and stored locally.\r\n\r\n## [Training Module](https://github.com/KianoushAmirpour/End_2_End_Text_Classification/tree/main/training)\r\nTo streamline training and experimentation, a custom module was developed with inspiration from the **Factory Method design pattern**.  \r\nWhile not all aspects strictly adhere to the pattern, the module is designed to make it easier to scale and add new components,  \r\nallowing experiments to be configured through a single configuration file that specifies parameters and feature store settings.\r\n\r\nCurrently Supported Methods and Features:\r\n* Preprocessing: StandardScaler\r\n* Models:\r\n  * Logistic Regression\r\n  * Random Forest\r\n  * XGBoost\r\n  * DistilBERT (via Transformers and PyTorch)\r\n* Few-Shot Classification: Experimented with Meta-Llama-3-8B-Instruct locally using llama.cpp.\r\n* Hyperparameter Optimization:\r\n  * Random Search\r\n  * Hyperopt\r\n\r\nAfter experimentation, the best model is trained on the complete dataset for production use.\r\n\r\n#### Experiment Tracking:  \r\n**MLflow** is integrated throughout the experimentation process for seamless tracking of experiments, parameters, and results.\r\n\r\n#### Usage: \r\nFor experimentation, you simply need to modify the configuration file in [run_experiment.py](https://github.com/KianoushAmirpour/End_2_End_Text_Classification/blob/main/training/run_experiment.py).  \r\nFor example, the following setup will:  \r\nTrain an XGBoost model, tune its parameters using the Hyperopt library, and apply StandardScaler for preprocessing. Retrieve the specified features from Feast for model training.  \r\nNote: The values used in this example are arbitrary and are intended solely for showcasing the process.\r\n\r\n```\r\nif __name__ == \"__main__\":\r\n\r\nconfigs = {\r\n        'experiment_name': 'xgboost_experiment',\r\n        'train_with_tuning': True,\r\n        'model_name': 'xgboost',\r\n        'preprocessor_method': 'standard_scaler',\r\n        'tuning_method': 'hyperopt',\r\n        'model_params': {\r\n                        'max_depth': hp.choice(\"max_depth\", np.arange(1,20,1,dtype=int)),\r\n                        'eta': hp.uniform(\"eta\", 0, 1),\r\n                        'gamma': hp.uniform(\"gamma\", 0, 10e1),\r\n                        'colsample_bytree': hp.uniform(\"colsample_bytree\", 0.5,1),\r\n                        'colsample_bynode': hp.uniform(\"colsample_bynode\", 0.5,1), \r\n                        'colsample_bylevel': hp.uniform(\"colsample_bylevel\", 0.5,1),\r\n                        'n_estimators': hp.choice(\"n_estimators\", np.arange(100,1000,10,dtype='int')),\r\n                        'seed' : 44\r\n                         }}\r\n    \r\n    \r\nfeatures = [\"meta_features_extracted_from_text:num_words\",\r\n                \"meta_features_extracted_from_text:num_unique_words\",\r\n                \"meta_features_extracted_from_text:num_stop_words\",\r\n                \"meta_features_extracted_from_text:num_title_case\",\r\n                \"meta_features_extracted_from_text:ave_length_words\",\r\n                \"meta_features_extracted_from_text:num_characters\",\r\n                ]\r\nrun(configs, features)\r\n\r\n```\r\n\r\n## [Deployment](https://github.com/KianoushAmirpour/End_2_End_Text_Classification/tree/main/deployment)\r\nOnce the best model is trained, it is converted into a reusable package for seamless deployment in production.  \r\nThe same pipeline used for training is leveraged for making predictions, ensuring consistency and reliability.\r\n\r\nTesting and Code Quality\r\n* Testing: The package is tested using **pytest** to ensure functionality.\r\n* Code Quality: Tools like **flake8, isort, and mypy** are used for linting, sorting imports, and type checking, respectively.\r\n* Automation: **tox** is utilized to automate testing and enforce code quality standards across environments.\r\n\r\nConfiguration files used to build the package can be found [here](https://github.com/KianoushAmirpour/End_2_End_Text_Classification/tree/main/deployment/inference).\r\n\r\nTo install the package locally, navigate to the directory containing setup.py and run the following command: `pip install -e .`\r\n\r\nFor more information regarding building and publishing the package, visit [here](https://packaging.python.org/en/latest/tutorials/packaging-projects/).\r\nThe following commands are described there:  \r\n\r\n```\r\npy -m pip install --upgrade pip\r\npy -m pip install --upgrade build\r\npy -m build\r\n```\r\n\r\nAfter building the model package, prediction endpoints were developed using FastAPI to serve the model in production. These endpoints can be found [here](https://github.com/KianoushAmirpour/End_2_End_Text_Classification/tree/main/deployment/serving_api).\r\n\r\n## [CI](https://github.com/KianoushAmirpour/End_2_End_Text_Classification/tree/main/.github/workflows) \r\nThe CI pipeline is triggered on both push and pull request events to test multiple components, such as inference and serving API deployments.  \r\nCurrently it just runs tox for code quality for inference and serving API.\r\n\r\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkianoushamirpour%2Fend_to_end_text_classification","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkianoushamirpour%2Fend_to_end_text_classification","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkianoushamirpour%2Fend_to_end_text_classification/lists"}