{"id":23485747,"url":"https://github.com/tahmid-saj/ml-repository","last_synced_at":"2025-04-15T01:27:57.274Z","repository":{"id":242802869,"uuid":"810492384","full_name":"tahmid-saj/ml-repository","owner":"tahmid-saj","description":"ML repository delivering both scheduling via cron jobs and API endpoints for running ETL, training, prediction, evaluation, etc, of various end-to-end models. Developed with FastAPI, TensorFlow, PyTorch, AWS, MongoDB.","archived":false,"fork":false,"pushed_at":"2024-10-18T01:34:24.000Z","size":735,"stargazers_count":7,"open_issues_count":0,"forks_count":4,"subscribers_count":4,"default_branch":"main","last_synced_at":"2025-03-28T13:21:18.024Z","etag":null,"topics":["aws","cron-jobs","fastapi","ml","mongodb","pytorch","tensorflow"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/tahmid-saj.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-06-04T19:57:55.000Z","updated_at":"2024-10-23T15:53:39.000Z","dependencies_parsed_at":"2024-06-05T04:28:08.226Z","dependency_job_id":"5dcec119-0680-4402-b915-5e4749ddb6ec","html_url":"https://github.com/tahmid-saj/ml-repository","commit_stats":null,"previous_names":["tahmid-saj/ml-job-scheduler","tahmid-saj/ml-repository"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tahmid-saj%2Fml-repository","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tahmid-saj%2Fml-repository/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tahmid-saj%2Fml-repository/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tahmid-saj%2Fml-repository/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/tahmid-saj","download_url":"https://codeload.github.com/tahmid-saj/ml-repository/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248987977,"owners_count":21194332,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aws","cron-jobs","fastapi","ml","mongodb","pytorch","tensorflow"],"created_at":"2024-12-24T22:13:57.694Z","updated_at":"2025-04-15T01:27:57.247Z","avatar_url":"https://github.com/tahmid-saj.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# ML Repository\n\nML job scheduler delivering both scheduling via cron jobs and API endpoints for running ETL, training, prediction, evaluation, etc, of various end-to-end models. Developed with FastAPI, TensorFlow, PyTorch, AWS, MongoDB.\n\n\u003cbr/\u003e\n\u003cbr/\u003e\n\n## Directory structure\n\nThe directory structure is as follows:\n\n```\nml-job-scheduler/\n├── api/\n│   ├── controllers/\n│   │   ├── btc_forecast_controller.py\n│   │   └── __pycache__/\n│   │       └── btc_forecast_controller.cpython-312.pyc\n│   ├── models/\n│   │   ├── btc_forecast/\n│   │   │   ├── btc_forecast_models.py\n│   │   │   └── __pycache__/\n│   │   │       └── btc_forecast_models.cpython-312.pyc\n│   │   ├── sp500_forecast/\n│   │   ├── text_analyzer/\n│   │   └── text_summarizer/\n│   └── services/\n│       ├── mongodb.py\n│       └── __pycache__/\n│           └── mongodb.cpython-312.pyc\n├── btc_forecast.py\n├── conf/\n│   ├── aws/\n│   ├── cron_jobs/\n│   │   └── cron_jobs_conf.yml\n│   └── mongodb/\n│       ├── mongodb_conf.py\n│       └── __pycache__/\n│           └── mongodb_conf.cpython-312.pyc\n├── data/\n│   └── food_classifier/\n│       └── models/\n│           ├── keras_metadata.pb\n│           └── saved_model.pb\n├── Dockerfile\n├── food_classifier.py\n├── index.py\n├── monitoring/\n├── notebooks/\n│   └── btc_forecast.py\n├── README.md\n├── requirements.txt\n├── scripts/\n│   ├── data_ops/\n│   │   └── text_summarizer_data_ops.py\n│   ├── etl/\n│   │   ├── btc_forecast/\n│   │   │   ├── btc_forecast_multivariate_etl.py\n│   │   │   └── btc_forecast_univariate_etl.py\n│   │   ├── food_classifier/\n│   │   │   └── food_classifier_etl.py\n│   │   ├── sp500_forecast/\n│   │   │   ├── sp500_forecast_multivariate_etl.py\n│   │   │   └── sp500_forecast_univariate_etl.py\n│   │   ├── text_analyzer/\n│   │   │   └── text_analyzer_etl.py\n│   │   └── text_summarizer/\n│   │       ├── text_summarizer_etl.py\n│   │       ├── text_summarizer_positional_embedding.py\n│   │       └── text_summarizer_tribid_embedding.py\n│   ├── evaluation/\n│   ├── full_pipeline/\n│   │   ├── btc_forecast_multivariate_2_weeks.py\n│   │   ├── btc_forecast_multivariate_current_day.py\n│   │   ├── sp500_forecast_multivariate_2_weeks.py\n│   │   └── __pycache__/\n│   │       ├── btc_forecast_multivariate_2_weeks.cpython-312.pyc\n│   │       └── btc_forecast_multivariate_current_day.cpython-312.pyc\n│   ├── postprocessing/\n│   ├── prediction/\n│   │   └── text_analyzer_ensemble_prediction.py\n│   ├── training/\n│   │   └── text_analyzer_ensemble_training.py\n│   └── training_prediction/\n│       ├── btc_forecast_multivariate_training_prediction.py\n│       ├── food_classifier_efficientb0_training_prediction.py\n│       ├── sp500_forecast_multivariate_training_prediction.py\n│       ├── text_summarizer_prediction_evaluation.py\n│       └── text_summarizer_test_prediction.py\n├── sp500_forecast.py\n├── src/\n│   └── mls/\n│       ├── assets/\n│       │   ├── btc_forecast_assets.py\n│       │   ├── food_classifier_assets.py\n│       │   ├── sp500_forecast_assets.py\n│       │   └── __pycache__/\n│       │       └── btc_forecast_assets.cpython-312.pyc\n│       ├── data_ops/\n│       │   ├── btc_forecast_load_prices.py\n│       │   ├── sp500_load_prices.py\n│       │   └── __pycache__/\n│       │       └── btc_forecast_load_prices.cpython-312.pyc\n│       ├── etl/\n│       │   ├── btc_forecast_etl.py\n│       │   ├── sp500_forecast_etl.py\n│       │   ├── text_summarizer_etl.py\n│       │   └── __pycache__/\n│       │       └── btc_forecast_etl.cpython-312.pyc\n│       ├── evaluation/\n│       │   ├── btc_forecast_evaluation.py\n│       │   ├── food_classifier_evaluation.py\n│       │   ├── sp500_forecast_evaluation.py\n│       │   ├── text_analyzer_evaluation.py\n│       │   ├── text_summarizer_evaluation.py\n│       │   └── __pycache__/\n│       │       └── btc_forecast_evaluation.cpython-312.pyc\n│       ├── model/\n│       │   ├── btc_forecast_ensemble_model.py\n│       │   ├── food_classifier_efficientb0_model.py\n│       │   ├── sp500_forecast_ensemble_model.py\n│       │   ├── text_analyzer_ensemble_model.py\n│       │   ├── text_summarizer_tribid_embedding_model.py\n│       │   └── __pycache__/\n│       │       └── btc_forecast_ensemble_model.cpython-312.pyc\n│       ├── postprocessing/\n│       ├── prediction/\n│       │   ├── btc_forecast_prediction.py\n│       │   ├── sp500_forecast_prediction.py\n│       │   └── __pycache__/\n│       │       └── btc_forecast_prediction.cpython-312.pyc\n│       └── training/\n├── tests/\n├── text_analyzer.py\n├── text_summarizer.py\n├── utils/\n│   ├── api-requests/\n│   ├── constants/\n│   │   ├── btc_forecast_constants.py\n│   │   └── __pycache__/\n│   │       └── btc_forecast_constants.cpython-312.pyc\n│   ├── errors/\n│   └── helpers/\n├── vercel.json\n└── vercel_dev.json\n```\n\n\u003cbr/\u003e\n\u003cbr/\u003e\n\n## Overview\n\n### Design\n\nThe usage of the service in other applications can be found below. Similar services can be found \u003ca href=\"https://whimsical.com/web-microservices-6uqvwWZtcBFsNJB2hepGy1\"\u003ehere\u003c/a\u003e and below:\n\n#### Similar services\n\n\u003cfigure\u003e\n  \u003cimg width=\"1000\" alt=\"image\" src=\"https://github.com/user-attachments/assets/80cf4ed7-58f7-4576-a3cd-d63ef584e0fd\"\u003e\n\u003c/figure\u003e\nFigure 1: High level view and usage in other applications\n\n\u003cbr/\u003e\n\u003cbr/\u003e\n\n\u003cimg width=\"834\" alt=\"image\" src=\"https://github.com/user-attachments/assets/b54088e7-870c-46dd-9cf6-2e5ec27d9d5c\"\u003e\n\n\u003cbr/\u003e\n\u003cbr/\u003e\n\n### The ML job scheduler consists of the following models:\n\n1. __BTC forecast__: Bitcoin forecasting\n2. __S\u0026P 500__: S\u0026P 500 forecasting\n3. __Food classifier__: Food detection / classification\n4. __Text analyzer__: NLP text analysis\n5. __Text summarizer__: NLP text summarization\n\n### Running the jobs / models:\n\nThe jobs / models can be both manually triggered or scheduled via API calls or cron jobs respectively:\n\n1. __BTC forecast__: \u003cem\u003ebtc_forecast.py\u003c/em\u003e\n2. __S\u0026P 500__: \u003cem\u003esp500_forecast.py\u003c/em\u003e\n3. __Food classifier__: \u003cem\u003efood_classifier.py\u003c/em\u003e\n4. __Text analyzer__: \u003cem\u003etext_analyzer.py\u003c/em\u003e\n5. __Text summarizer__: \u003cem\u003ebtc_summarizer.py\u003c/em\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftahmid-saj%2Fml-repository","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftahmid-saj%2Fml-repository","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftahmid-saj%2Fml-repository/lists"}