{"id":24300587,"url":"https://github.com/aiscalate/aiscalator","last_synced_at":"2025-09-26T02:31:46.876Z","repository":{"id":33138759,"uuid":"152393348","full_name":"Aiscalate/aiscalator","owner":"Aiscalate","description":"Tools to streamline Jupyter Notebook Prototypes into robust Data Products","archived":false,"fork":false,"pushed_at":"2022-12-26T21:31:04.000Z","size":1600,"stargazers_count":5,"open_issues_count":4,"forks_count":2,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-01-16T11:26:21.919Z","etag":null,"topics":["airflow","airflow-docker","data-engineering","data-science","jupyter","jupyter-notebook","jupyterlab"],"latest_commit_sha":null,"homepage":"http://www.aiscalate.com","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Aiscalate.png","metadata":{"files":{"readme":"README.rst","changelog":"HISTORY.rst","contributing":"CONTRIBUTING.rst","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.rst","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2018-10-10T08:57:57.000Z","updated_at":"2020-07-24T19:28:37.000Z","dependencies_parsed_at":"2023-01-14T23:34:36.667Z","dependency_job_id":null,"html_url":"https://github.com/Aiscalate/aiscalator","commit_stats":null,"previous_names":[],"tags_count":18,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Aiscalate%2Faiscalator","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Aiscalate%2Faiscalator/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Aiscalate%2Faiscalator/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Aiscalate%2Faiscalator/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Aiscalate","download_url":"https://codeload.github.com/Aiscalate/aiscalator/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":234286150,"owners_count":18808433,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["airflow","airflow-docker","data-engineering","data-science","jupyter","jupyter-notebook","jupyterlab"],"created_at":"2025-01-16T23:14:48.924Z","updated_at":"2025-09-26T02:31:41.210Z","avatar_url":"https://github.com/Aiscalate.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"==========\nAIscalator\n==========\n\n\n.. start-badges\n\n.. list-table::\n    :stub-columns: 1\n\n    * - Docs\n      - |docs|\n    * - Tests\n      - | |travis| |requires|\n        | |coveralls| |codecov|\n        | |scrutinizer| |codacy| |codeclimate|\n    * - Package\n      - | |version| |commits-since|\n\n.. |docs| image:: https://readthedocs.org/projects/aiscalator/badge/?version=latest\n    :target: https://aiscalator.readthedocs.io/en/latest/?badge=latest\n    :alt: Documentation Status\n\n.. |travis| image:: https://api.travis-ci.org/Aiscalate/aiscalator.svg?branch=master\n    :alt: Travis-CI Build Status\n    :target: https://travis-ci.org/Aiscalate/aiscalator\n\n.. |requires| image:: https://requires.io/github/Aiscalate/aiscalator/requirements.svg?branch=master\n    :target: https://requires.io/github/Aiscalate/aiscalator/requirements/?branch=master\n    :alt: Requirements Status\n\n.. |coveralls| image:: https://coveralls.io/repos/Aiscalate/aiscalator/badge.svg?branch=master\u0026service=github\n    :alt: Coverage Status\n    :target: https://coveralls.io/github/Aiscalate/aiscalator\n\n.. |codecov| image:: https://codecov.io/gh/Aiscalate/aiscalator/branch/master/graphs/badge.svg?branch=master\n    :alt: Coverage Status\n    :target: https://codecov.io/github/Aiscalate/aiscalator\n\n.. |codacy| image:: https://api.codacy.com/project/badge/Grade/16c9117a680041f382ba48bd5cf73ca2\n    :target: https://www.codacy.com/app/ChristopheDuong/aiscalator\n\n.. |codeclimate| image:: https://codeclimate.com/github/Aiscalate/aiscalator/badges/gpa.svg\n   :target: https://codeclimate.com/github/Aiscalate/aiscalator\n   :alt: CodeClimate Quality Status\n\n.. |version| image:: https://img.shields.io/pypi/v/aiscalator.svg\n    :alt: PyPI Package latest release\n    :target: https://pypi.org/project/aiscalator/\n\n.. |commits-since| image:: https://img.shields.io/github/commits-since/Aiscalate/aiscalator/v0.1.18.svg\n    :alt: Commits since latest release\n    :target: https://github.com/Aiscalate/aiscalator/compare/v0.1.18...master\n\n.. |scrutinizer| image:: https://img.shields.io/scrutinizer/quality/g/Aiscalate/aiscalator/master.svg\n    :alt: Scrutinizer Status\n    :target: https://scrutinizer-ci.com/g/Aiscalate/aiscalator/\n\n.. end-badges\n\n\n* Free software: Apache Software License 2.0\n* Website: http://www.aiscalate.com\n* Documentation: https://aiscalator.readthedocs.io/en/latest/\n* Bugs: https://github.com/aiscalate/aiscalator/issues\n\nKey Features\n------------\n\nAiscalator is a toolbox to enable your team streamlining\nprocesses from innovation to productization with:\n\n* Jupyter workbench\n    * Explore Data, Prototype Solutions\n* Docker wrapper tools\n    * Share Code, Deploy Reproducible Environments\n* Airflow machinery\n    * Schedule Tasks, Refine Products\n* Data Science and Data Engineering best practices\n\n.. image:: _static/aiscalator_process.png\n        :target: _static/aiscalator_process.png\n        :align: center\n        :alt: From Prototype to Production Workflow\n\n===========\nQuick Start\n===========\n\nInstallation\n------------\n\nTest if prerequisite softwares are installed:\n\n.. code-block:: shell\n\n    docker --version\n    docker-compose --version\n    pip --version\n\nInstall AIscalator tool::\n\n    git clone https://github.com/Aiscalate/aiscalator.git\n    cd aiscalator/\n    make install\n\nGreat, we are now ready to use the AIscalator!\n\nThe following setup commands are completely optional because they are dealing with\nprebuilding Docker images. If you choose not to do it at this point, they\nwill get built later on whenever they are required.\n\nHowever, since producing a Docker image requires a certain amount of time\nto download, install packages, and sometimes even compiling them, these\ninstallation steps can be initiated right away all at once. Thus, you\nshould be free to go enjoy a nice coffee break!\n\nYou might want to customize your environment with the AIscalator, this\nwill ask you few questions::\n\n    aiscalator setup\n\nBuild docker images to run Jupyter environments::\n\n    aiscalator jupyter setup\n\nBuild docker image to run Airflow::\n\n    # aiscalator airflow setup \u003cpath-to-workspace-folder\u003e\n    # for example,\n    aiscalator airflow setup $PWD\n\n\nStart working\n-------------\n\nAIscalator commands dealing with jupyter are defining tasks in Airflow jargon;\nIn our case, they are all wrapped inside a Docker container. We also refer to\nthem as Steps.\n\nWhereas AIscalator commands about airflow are made to author, schedule and monitor\nDAGs (Directed Acyclic Graphs). They define how a workflow is composed of multiple\nsteps, their dependencies and execution times or triggers.\n\nJupyter\n-------\n\nCreate a new Jupyter notebook to work on, define corresponding AIscalator step::\n\n    # aiscalator jupyter new \u003cpath-to-store-new-files\u003e\n    # For example,\n    aiscalator jupyter new project\n    # (CTRL + c to kill when done)\n\nOr you can edit an existing AIscalator step::\n\n    # aiscalator jupyter edit \u003caiscalator step\u003e\n    # For example, if you cloned the git repository:\n    aiscalator jupyter edit resources/example/example.conf\n    # (CTRL + c to kill when done)\n\nRun the step without GUI::\n\n    # aiscalator jupyter run \u003caiscalator task\u003e\n    # For example, if you cloned the git repository:\n    aiscalator jupyter run resources/example/example.conf\n\nAirflow\n-------\n\nStart Airflow services::\n\n    aiscalator airflow start\n\nCreate a new AIscalator DAG, define the airflow job::\n\n    # aiscalator airflow new \u003cpath-to-store-new-files\u003e\n    # For example,\n    aiscalator airflow new project\n    # (CTRL + c to kill when done)\n\nOr you can edit an existing AIscalator DAG::\n\n    # aiscalator airflow edit \u003caiscalator DAG\u003e\n    # For example, if you cloned the git repository:\n    aiscalator airflow edit resources/example/example.conf\n    # (CTRL + c to kill when done)\n\nSchedule AIscalator DAG into local airflow dags folder::\n\n    # aiscalator airflow push \u003caiscalator DAG\u003e\n    # For example, if you cloned the git repository:\n    aiscalator airflow push resources/example/example.conf\n\nStop Airflow services::\n\n    aiscalator airflow stop\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faiscalate%2Faiscalator","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Faiscalate%2Faiscalator","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faiscalate%2Faiscalator/lists"}