{"id":19582945,"url":"https://github.com/tomgorb/project-template-for-production","last_synced_at":"2026-05-15T06:35:49.065Z","repository":{"id":39853657,"uuid":"281380257","full_name":"tomgorb/project-template-for-production","owner":"tomgorb","description":"project template to (help) put a Machine/Deep learning algorithm into production","archived":false,"fork":false,"pushed_at":"2024-07-31T15:30:46.000Z","size":78,"stargazers_count":1,"open_issues_count":2,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-01-09T05:25:28.237Z","etag":null,"topics":["airflow","bigquery","gcp"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/tomgorb.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-07-21T11:35:00.000Z","updated_at":"2024-07-31T15:30:50.000Z","dependencies_parsed_at":"2024-11-11T07:38:53.473Z","dependency_job_id":"bc1ac476-40ad-435c-9971-5e1e90996dbf","html_url":"https://github.com/tomgorb/project-template-for-production","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tomgorb%2Fproject-template-for-production","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tomgorb%2Fproject-template-for-production/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tomgorb%2Fproject-template-for-production/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tomgorb%2Fproject-template-for-production/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/tomgorb","download_url":"https://codeload.github.com/tomgorb/project-template-for-production/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":240849785,"owners_count":19867755,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["airflow","bigquery","gcp"],"created_at":"2024-11-11T07:38:44.317Z","updated_at":"2026-05-15T06:35:49.024Z","avatar_url":"https://github.com/tomgorb.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"template\n-----\n\nThis template can be used to productionize a *Machine/Deep learning* algorithm (with a few adjustments).\n\n```main.py``` contains **tasks** to be run: some tasks can be run locally (*e.g.* for developement purpose) or on the cloud (this template uses GCP AI platform).\n\nThere is a sample DAG ```dag.py``` that can be used on an Airflow - production - cluster (\\*Airflow is an orchestrator).\n\n![DAG](./DAG.png)\n\nAfter cloning this repo, you should create a python3 venv (named *venv* at the repo's root) and install all 3 requirements file:\n\n  - *requirements* contains modules needed by our main.py and custom model **except** tensorflow ;\n\n  - *requirements-ml* contains our custom library (not available on PyPI) that will be needed by GCP AI platform (will be uploaded to Google Storage when needed) ;\n\n  - *requirement-extra* contains the same tensorflow version that will be used on AI platform (depending on the runtime environment and installed by default).\n\nThere is a **MyModel** class containing *preprocess*, *train* and *predict* methods skeletons.\n\n```template.yaml.template``` is the configuration file template containing (mainly) credentials.\n\n```queries.yaml``` contains templated queries to be used to access or get data from any database (BigQuery in this case).\n\n**mymodel** is our custom library .\n\n### PACKAGING DAG AIRFLOW AND CODE\n\n```build``` folder contains code to make debian packages for DAG and CODE alongside files for jenkins pipelines.\n\n**DAG**: make-dag-package.sh\n\n  * version number\n  \u003e hard coded\n\n  * path\n  \u003e/opt/airflow/dags/\n\n**CODE**: make-code-package.sh\n\n  * version number\n  \u003e hard coded\n\n  * path\n  \u003e /opt/\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftomgorb%2Fproject-template-for-production","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftomgorb%2Fproject-template-for-production","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftomgorb%2Fproject-template-for-production/lists"}