{"id":16830158,"url":"https://github.com/mwoss/mlflow-stock-market-example","last_synced_at":"2025-07-21T03:02:56.219Z","repository":{"id":43651986,"uuid":"189107826","full_name":"mwoss/mlflow-stock-market-example","owner":"mwoss","description":"Stock market prediction - machine learning pipeline using MLFlow.","archived":false,"fork":false,"pushed_at":"2023-05-01T13:43:54.000Z","size":540,"stargazers_count":2,"open_issues_count":2,"forks_count":1,"subscribers_count":0,"default_branch":"master","last_synced_at":"2025-03-17T22:47:32.814Z","etag":null,"topics":["anaconda","data-analysis","databricks","example","lstm","mlflow","python","stock-market","stock-price-prediction","tutorial"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mwoss.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-05-28T21:47:24.000Z","updated_at":"2024-02-24T13:17:44.000Z","dependencies_parsed_at":"2024-11-24T06:01:59.943Z","dependency_job_id":null,"html_url":"https://github.com/mwoss/mlflow-stock-market-example","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/mwoss/mlflow-stock-market-example","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mwoss%2Fmlflow-stock-market-example","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mwoss%2Fmlflow-stock-market-example/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mwoss%2Fmlflow-stock-market-example/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mwoss%2Fmlflow-stock-market-example/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mwoss","download_url":"https://codeload.github.com/mwoss/mlflow-stock-market-example/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mwoss%2Fmlflow-stock-market-example/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":266231762,"owners_count":23896473,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["anaconda","data-analysis","databricks","example","lstm","mlflow","python","stock-market","stock-price-prediction","tutorial"],"created_at":"2024-10-13T11:37:40.803Z","updated_at":"2025-07-21T03:02:56.196Z","avatar_url":"https://github.com/mwoss.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"MLFlow stock market prediction PoC\n----\n\nStock market prediction - machine learning pipeline with MlFlow.  \nRepository contains POC of machine learning pipeline using MlFlow library, PoC was based on multistep mlflow example from origin repository.  \n\nMLflow is an open source platform to manage the ML lifecycle, including experimentation, reproducibility and deployment.  \nMlflow consists of three components:\n* Mlflow tracking - experiment tracking module\n* Mlflow projects - reproducible runs\n* Mlflow model - model packaging  \n\nML lifecycle = **(-\u003e raw data -\u003e data preparation -\u003e model training -\u003e deployment -\u003e raw_data -\u003e ...)**\nMLflow aims to take any codebase written in its format and make it reproducible and reusable by multiple data scientists.  \n\n\nGetting started \n----\nIn order to use mlflow you have to setup python environment with requirements stored in `pre_requirements.txt`.  \n```\n\u003csetup your new environment using your favourite tooo\u003e\npip install -r pre_requirements.txt\n```\nRunning mlflow pipeline is pretty straight forward. Run `mlflow run` command from project directory and that's all.\nMlflow will execute pipline definied in `MLproject` file.\n\n```bash\nmlflow run .\n```\n```bash\nmlflow run git@github.com:mwoss/mlflow-stock-market-example.git [Yes, you run flows via github uri :3]\n```\nBy default mlflow gather all local pipeline execution into one experiment (group), which can be useful\nfor comparing runs intended to tackle a particular task. In order to create new group use mlflow CLI: \n```bash\nexport MLFLOW_EXPERIMENT_NAME=new-experiment\n\nmlflow experiments create --experiment-name new-experiemnt [this arguemnt is optional if you export above env var]\n```\n\nYou can also compare the results or check your run metrics/artifacts using `mlflow ui`  \nParametrized runs can be executed using `-P` attribute, for example:\n\n```bash\nmlflow run . -P lstm_units=60\n```\nYou can also run whole application using main script, just simply execute main.py\n```bash\npython main.py\n```\nExample overview\n----\nStarting from data_analysis directory. It contains jupyter notebook with a few different stock market prediction (linear regression, LSTM, moving avarage, knn).\nI've ended up using LSTM. The last chart shows how well LSTM copes with the prediction on given Microsoft dataset.  \n\nAfter choosing a proper prediction approach I've started defining pipeline.\nPipeline got split into 3 individual steps:\n* download_raw_data - download dataset using Quanld API\n* transform_data - prepare data for training purpose\n* train_model - train LSTM network and upload model\n* deploy_model - deploy trained model on S3, this step require aws credentials (optional step, can be removed from pipeline)\n\nEach model log metrics/artifact that can be used by next flow step. All information about steps is saved in mlruns directory (artifacts, metadata etc).  \nIf you want to log metrics/artifacts etc. to remote servers, you can do it easily by setting MLFLOW_TRACKING_URI env variable or\nusing `mlflow.set_tracking_uri()` (more info here: [Where run are recorded](https://mlflow.org/docs/latest/tracking.html?fbclid=IwAR0E3Ozpn52sNheoW7OmS3GkYf0iOBVgoxOB8cKI-iQKbo2hK-tBGEjUSpA#where-runs-are-recorded))\n\n**Mlproject file schema**  \nMlproject file it's probably the most important file in Mlflow app. The file that defines whole pipeline.  \nBasic structure:\n```text\nname: [PIPELINE_NAME]\nconda_env: [PATH TO yaml file with anaconda env]\ndocker_env:\n    image: [IMAGE_NAME] (conda_env or docker_env, not both at once) \n\nentry_points:\n    step1:\n        parameters:\n            PARAMETER_NAME: {type: TYPE, default: DEFAULT}\n        command: \"COMMAND TO EXECUTE\"\n    step2:\n        parameters:\n            PARAMETER_NAME1: TYPE\n            PARAMETER_NAME2: {type: TYPE, default: DEFAULT}\n        command: \"COMMAND TO EXECUTE\"\n\n```\n\n`conda.yml` contains requirements for project virtual environment. Mlflow creates venv on its own with given requirements file and run whole multistep flow within it.\n \nBeyond example\n----\nIn `additional_examples` directory I cover cool functionalities beyond this simple multistep pipeline.\n* `experiments_management.py` - short demonstration of MlflowClient's capabilities\n* `rest_api.py` - simple REST API example, useful for services written in different languages\n\nNotes\n----\nPoC is based on mlflow/examples/multistep_workflow exmaple from mlflow repository.\n\nFiles placed in flow_steps directory use relative imports due to Mlflow execution logic.\n\nInteresting and useful resources\n----\n* [Mlflow documentation](https://mlflow.org/docs/latest/index.html)\n* [Mlflow overview - Spark AI Summit 2019](https://www.youtube.com/watch?v=QJW_kkRWAUs)\n* [Complete Machine Learning Lifecycle with MLflow - workshop](https://www.youtube.com/watch?v=VVnCyPOlrbk)\n* [How to Utilize MLflow and Kubernetes](https://www.youtube.com/watch?v=cDtzu4WBzWA)\n* [Bunch of Mlflow examples from Spark Summit 2019](https://github.com/amesar/mlflow-spark-summit-2019)\n* [Mlflow quickstart - Python](https://docs.azuredatabricks.net/_static/notebooks/mlflow/mlflow-quick-start-python.html)\n* [Mlflow quickstart - Scala](https://docs.azuredatabricks.net/_static/notebooks/mlflow/mlflow-quick-start-scala.html)","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmwoss%2Fmlflow-stock-market-example","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmwoss%2Fmlflow-stock-market-example","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmwoss%2Fmlflow-stock-market-example/lists"}