{"id":13936790,"url":"https://github.com/jay-johnson/sci-pype","last_synced_at":"2025-07-14T11:12:18.073Z","repository":{"id":10145899,"uuid":"64637961","full_name":"jay-johnson/sci-pype","owner":"jay-johnson","description":"A Machine Learning API with native redis caching and export + import using S3. Analyze entire datasets using an API for building, training, testing, analyzing, extracting, importing, and archiving. This repository can run from a docker container or from the repository.","archived":false,"fork":false,"pushed_at":"2022-12-07T23:39:54.000Z","size":13957,"stargazers_count":100,"open_issues_count":27,"forks_count":21,"subscribers_count":13,"default_branch":"master","last_synced_at":"2025-05-13T07:52:48.677Z","etag":null,"topics":["data-science","devops-for-data-science","docker","docker-compose","ipython","ipython-notebook","jupyter","jupyter-notebook","jupyter-themes","machine-learning","machine-learning-api","predictive","python","red10","redis","s3","seaborn","stock-price-prediction","xgb","xgboost"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jay-johnson.png","metadata":{"files":{"readme":"README.rst","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2016-08-01T05:18:25.000Z","updated_at":"2024-11-21T21:59:06.000Z","dependencies_parsed_at":"2023-01-12T18:15:49.807Z","dependency_job_id":null,"html_url":"https://github.com/jay-johnson/sci-pype","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/jay-johnson/sci-pype","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jay-johnson%2Fsci-pype","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jay-johnson%2Fsci-pype/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jay-johnson%2Fsci-pype/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jay-johnson%2Fsci-pype/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jay-johnson","download_url":"https://codeload.github.com/jay-johnson/sci-pype/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jay-johnson%2Fsci-pype/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":265281723,"owners_count":23739884,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["data-science","devops-for-data-science","docker","docker-compose","ipython","ipython-notebook","jupyter","jupyter-notebook","jupyter-themes","machine-learning","machine-learning-api","predictive","python","red10","redis","s3","seaborn","stock-price-prediction","xgb","xgboost"],"created_at":"2024-08-07T23:03:00.123Z","updated_at":"2025-07-14T11:12:18.050Z","avatar_url":"https://github.com/jay-johnson.png","language":"Python","readme":"=======================================================================\nSci-Pype - A Machine Learning Framework for Sharing Models and Analysis\n=======================================================================\n\nThis is now deployed under the https://redten.io cloud service for analyzing datasets.\n\nSci-Pype is a framework for analyzing datasets using Python 2.7 and extended from the `Jupyter Scipy-Notebook`_ with a supported command line version (no docker or Jupyter required). It was built to make data analysis easier by providing an API to build, train, test, predict, validate, analyze, extract, archive, and import Models and Analysis datasets with: S3 and redis (Kafka coming soon). After building and training the requested Models with a dataset, they are cached in redis along with their respective Analysis. After they are cached, they can be extracted and shared using S3. From S3, the Models can be imported back into redis for making new predictions using the same API. \n\n.. figure:: ./examples/images/v2/Scipype-Seaborn-Visuals.png\n    :alt: Sci-Pype - A Machine Learning Framework for Sharing Models and Analysis\n    :align: center\n    \n    Analyzing the IRIS dataset with Sci-Pype\n\nCommon use cases for this framework are sharing Analysis notebooks and then automating new predictions with email delivery using AWS SES. With this native caching + deployment layer, you can build, train and use the supported Machine Learning Algorithms and Models across multiple environments (including multi-tenant ones). Once trained, you can extract the Models as a compressed, serialized ``Model file`` (like a build artifact) that is uploaded to S3. Importing a ``Model file`` decompresses the file and stores the Pickle-serialized Models + Analysis objects in redis. In production, it might be useful to house larger Models in something like a `load-balanced redis cluster`_ for sharing and making new predictions across a team or by automation.\n\nPlease note this is a `large docker container`_ so it may take some time to download and it extracts to ~8.1 GB on disk.\n\n.. _Jupyter Scipy-Notebook: https://github.com/jupyter/docker-stacks/tree/master/scipy-notebook\n.. _load-balanced redis cluster: https://github.com/jay-johnson/docker-redis-haproxy-cluster\n.. _large docker container: https://hub.docker.com/r/jayjohnson/jupyter/tags/\n\nNotebook Examples\n=================\n\nPlease refer to the `examples directory`_ for the latest notebooks. Most of the notebooks and command line tools require running with a redis server listening on port 6000 (``\u003crepo base dir\u003e/dev-start.sh`` will start one). \n\n.. _examples directory: https://github.com/jay-johnson/sci-pype/tree/master/examples\n\n#.  `ML-IRIS-Analysis-Workflow-Classification.ipynb`_\n\n    Build a unique Machine Learning Classifier (parameterized XGB by default) for each column in the IRIS dataset. After training and testing the Models, perform a general analysis on each column and save + display images generated during each step. After running, the Models + Analysis are Pickled into a set of objects stored in a set of unique redis cache keys. These leaf nodes are organized into a set of redis keys contained in the ``manifest`` node for retrieval as needed in the future (like a tree of Machine Learning Algorithm Models with their associated pre-computed Analysis in memory).\n\n#.  `ML-IRIS-Analysis-Workflow-Regression.ipynb`_\n    \n    Build a unique Machine Learning Regressor (parameterized XGB by default) for each column in the IRIS dataset. After training and testing the Models, perform a general analysis on each column and save + display images generated during each step. After running, the Models + Analysis are Pickled into a set of objects stored in a set of unique redis cache keys. These leaf nodes are organized into a set of redis keys contained in the ``manifest`` node for retrieval as needed in the future (like a tree of Machine Learning Algorithm Models with their associated pre-computed Analysis in memory).\n\n#.  `ML-IRIS-Extract-Models-From-Cache.ipynb`_\n\n    Extract all Models and Analysis records from redis and compile a large Pickle-serialized dictionary. Create a ``manifest`` for decoupling Model + Analysis nodes and compress the dictionary object (using zlib) and write it to disk as a ``Model file`` (``*.cache.pickle.zlib``). After creating the file on disk, upload it to the configured S3 Bucket and Key.\n    \n    Once uploaded to the S3 Bucket you should be able to view, download and share the ``Model files``:\n\n    .. figure:: ./examples/images/scipype_s3_bucket_with_xgb_classifier_and_regressor_models_as_pickled_object_files.png\n\n        S3 Bucket containing the IRIS ``Model Files``\n\n#.  `ML-IRIS-Import-and-Cache-Models-From-S3.ipynb`_\n\n    Download the S3 IRIS ``Model file`` from the configured S3 Bucket + Key and decompress the previously-built Analysis and Models using Pickle to store them all in the redis cache according to the ``manifest``. This includes examples from the IRIS sample dataset and requires you to have a valid S3 Bucket storing the Models and are comfortable paying for the download costs to retrieve the ``Model file`` from S3 (https://aws.amazon.com/s3/pricing/).\n\n#.  `ML-IRIS-Predict-From-Cache-for-New-Predictions-and-Analysis-Classifier.ipynb`_\n\n    This notebook shows how to make new predictions with cached IRIS Classifier Models + Analysis housed in redis. \n    \n#.  `ML-IRIS-Predict-From-Cache-for-New-Predictions-and-Analysis-Regressor.ipynb`_\n\n    This notebook shows how to make new predictions with cached IRIS Regressor Models + Analysis housed in redis. \n    \n\nCommand Line Examples\n=====================\n\nMost of the notebooks and command line tools require running with a redis server listening on port 6000 (``\u003crepo base dir\u003e/dev-start.sh`` will start one). The command line versions that do not require docker or Jupyter can be found:\n\n::\n    \n    \u003crepo base dir\u003e\n    ├── bins\n    │   ├── demo-running-locally.py - Simple validate env is working test\n    │   ├── ml\n    │   │   ├── builders - Build and Train Models then Analyze Predictions without display any plotted images (automation examples)\n    │   │   │   ├── build-classifier-iris.py\n    │   │   │   ├── build-regressor-iris.py\n    │   │   │   ├── rl-build-regressor-iris.py\n    │   │   │   └── secure-rl-build-regressor-iris.py\n    │   │   ├── demo-ml-classifier-iris.py - Command line version of: ML-IRIS-Analysis-Workflow-Classification.ipynb\n    │   │   ├── demo-ml-regressor-iris.py - Command line version of: ML-IRIS-Analysis-Workflow-Regression.ipynb\n    │   │   ├── demo-rl-regressor-iris.py - Command line version of: ML-IRIS-Redis-Labs-Cache-XGB-Regressors.ipynb\n    │   │   ├── demo-secure-ml-regressor-iris.py - Demo with a Password-Required Redis Server running locally\n    │   │   ├── demo-secure-rl-regressor-iris.py - Demo with a Password-Required Redis Labs Cloud endpoint\n    │   │   ├── downloaders\n    │   │   │   ├── download_boston_house_prices.py\n    │   │   │   └── download_iris.py - Command line tool for downloading + preparing the IRIS dataset\n    │   │   ├── extractors\n    │   │   │   ├── extract_and_upload_iris_classifier.py - Command line version of: ML-IRIS-Extract-Models-From-Cache.ipynb (Classifier)\n    │   │   │   ├── extract_and_upload_iris_regressor.py - Command line version of: ML-IRIS-Extract-Models-From-Cache.ipynb (Regressor)\n    │   │   │   ├── rl_extract_and_upload_iris_regressor.py - Command line version of:  ML-IRIS-Redis-Labs-Extract-From-Cache.ipynb\n    │   │   │   └── secure_rl_extract_and_upload_iris_regressor.py - Command line version with a password for: ML-IRIS-Redis-Labs-Extract-From-Cache.ipynb \n    │   │   ├── importers\n    │   │   │   ├── import_iris_classifier.py - ML-IRIS-Import-and-Cache-Models-From-S3.ipynb (Classifier)\n    │   │   │   ├── import_iris_regressor.py - ML-IRIS-Import-and-Cache-Models-From-S3.ipynb (Regressor)\n    │   │   │   ├── rl_import_iris_regressor.py - Command line version of: ML-IRIS-Redis-Labs-Import-From-S3.ipynb\n    │   │   │   └── secure_rl_import_iris_regressor.py - Command line version with a password for: ML-IRIS-Redis-Labs-Import-From-S3.ipynb\n    │   │   └── predictors\n    │   │       ├── predict-from-cache-iris-classifier.py - ML-IRIS-Predict-From-Cache-for-New-Predictions-and-Analysis-Classifier.ipynb (Classifier)\n    │   │       ├── predict-from-cache-iris-regressor.py - ML-IRIS-Predict-From-Cache-for-New-Predictions-and-Analysis-Regressor.ipynb (Regressor)\n    │   │       ├── rl-predict-from-cache-iris-regressor.py - Command line version of: ML-IRIS-Redis-Labs-Predict-From-Cached-XGB.ipynb\n    │   │       └── secure-rl-predict-from-cache-iris-regressor.py - Command line version with a password for: ML-IRIS-Redis-Labs-Predict-From-Cached-XGB.ipynb\n\n\nNow you can share, test, and deploy Models and their respective Analysis from a file in S3 for other Sci-Pype users running on different environments.\n\n.. _ML-IRIS-Analysis-Workflow-Classification.ipynb: ./examples/ML-IRIS-Analysis-Workflow-Classification.ipynb\n.. _ML-IRIS-Analysis-Workflow-Regression.ipynb: ./examples/ML-IRIS-Analysis-Workflow-Regression.ipynb\n.. _ML-IRIS-Extract-Models-From-Cache.ipynb: ./examples/ML-IRIS-Extract-Models-From-Cache.ipynb\n.. _ML-IRIS-Import-and-Cache-Models-From-S3.ipynb: ./examples/ML-IRIS-Import-and-Cache-Models-From-S3.ipynb\n.. _ML-IRIS-Predict-From-Cache-for-New-Predictions-and-Analysis-Classifier.ipynb: ./examples/ML-IRIS-Predict-From-Cache-for-New-Predictions-and-Analysis-Classifier.ipynb\n.. _ML-IRIS-Predict-From-Cache-for-New-Predictions-and-Analysis-Regressor.ipynb: ./examples/ML-IRIS-Predict-From-Cache-for-New-Predictions-and-Analysis-Regressor.ipynb\n\n\nOverview\n========\n\nThe docker container runs a Jupyter web application. The web application runs `Jupyter Notebooks`_ as kernels. For now the examples and core included in this repository will only work with Python 2.\n\n.. _Jupyter Notebooks: http://jupyter-notebook.readthedocs.io/en/latest/\n\nThis container can run in four modes:\n\n#.  Default development\n\n    This mode will mount your changes from the repository into the container at runtime for local testing.\n\n    To start the local development version run: dev-start.sh_\n\n    ::\n\n        ./dev-start.sh\n\n    You can login to the container with: ``./ssh.sh``\n    \n    .. _dev-start.sh: https://github.com/jay-johnson/sci-pype/blob/s3_and_machine_learning_api/dev-start.sh\n\n#.  Docker Run Single Container\n\n    To start the local development version run: start.sh_\n\n    ::\n\n        ./start.sh\n\n    You can login to the container with: ``./ssh.sh``\n    \n    .. _start.sh: https://github.com/jay-johnson/sci-pype/blob/master/start.sh\n\n#.  Full Stack\n\n    To start the full stack mode run: compose-start-full.sh_\n    \n    ::\n\n        ./compose-start-full.sh\n\n    The full-stack-compose.yml_ will deploy three docker containers using docker compose:\n    \n    - **MySQL Database container with phpMyAdmin** for Stock Data (from the `schemaprototyping repo`_) \n    - **Jupyter**\n    - **Redis server** (`jayjohnson/redis-single-node`_) on port 6000\n    \n    .. _compose-start-full.sh: https://github.com/jay-johnson/sci-pype/blob/master/compose-start-full.sh\n    .. _full-stack-compose.yml: https://github.com/jay-johnson/sci-pype/blob/master/full-stack-compose.yml\n    .. _schemaprototyping repo: https://github.com/jay-johnson/docker-schema-prototyping-with-mysql\n    .. _jayjohnson/redis-single-node: https://hub.docker.com/r/jayjohnson/redis-single-node\n\n#.  Standalone Testing\n    \n    To start the full stack mode run: compose-start-jupyter.sh_\n\n    ::\n\n        ./compose-start-jupyter.sh\n\n    The jupyter-docker-compose.yml_ is used to deploy a single Jupyter container.\n\n    .. _compose-start-jupyter.sh: https://github.com/jay-johnson/sci-pype/blob/master/compose-start-jupyter.sh\n    .. _jupyter-docker-compose.yml: https://github.com/jay-johnson/sci-pype/blob/master/jupyter-docker-compose.yml\n\nRunning Locally without Docker\n==============================\n\nHere is how to run locally without using docker (and Lambda deployments in the future).\n\n#.  Clone the repo without the dash character in the name\n\n    ::\n\n        $ git clone git@github.com:jay-johnson/sci-pype.git scipype\n\n#.  Go to the base dir of the repository\n\n    ::\n\n        dev$ cd scipype\n\n#.  Set up a local virtual environment using the installer\n\n    This will take some time and may fail due to missing packages on your host. Please refer to the `Coming Soon and Known Issues`_ section for help getting passed these issues.\n\n    ::\n    \n        scipype$ ./setup-new-dev.sh\n\n    After this finishes you should see the lines:\n\n    ::\n\n        ---------------------------------------------------------\n        Activate the new Scipype virtualenv with:\n        \n        source ./dev-properties.sh\"\n           or:\n        source ./properties.sh\n\n#.  Activate the ``scipype`` virtual environment for development:\n\n    ::\n\n        $ source ./dev-properties.sh\n\n#.  Confirm your virtual environment is ready for use\n\n    ::\n\n        (scipype) scipype$ pip list --format=columns | grep -E -i \"tensorflow|pandas|redis|kafka|xgboost|scipy|scikit\"\n        confluent-kafka                    0.9.2                                                 \n        kafka-python                       1.3.1                                                 \n        pandas                             0.19.2                                                \n        pandas-datareader                  0.2.2                                                 \n        pandas-ml                          0.4.0                                                 \n        redis                              2.10.5                                                \n        scikit-image                       0.12.3                                                \n        scikit-learn                       0.18.1                                                \n        scikit-neuralnetwork               0.7                                                   \n        scipy                              0.18.1                                                \n        tensorflow                         0.12.0                                                \n        xgboost                            0.6a2                                                 \n        (scipype) scipype$ \n\n#.  Setup the /opt/work symlink \n\n    When running outside docker, I find it easiest to just symlink the repo's base dir to ``/opt/work`` to emulate the container's internal directory deployment structure. In a future release, a local-properties.sh file will set all the environment variables relative to the repository, but for now this works.\n    \n    ::\n\n        scipype$ ln -s $(pwd) /opt/work\n\n#.  Confirm the symlink is setup\n\n    ::\n\n        scipype$ ll /opt/work\n        lrwxrwxrwx 1 driver driver 32 Mar  6 22:38 /opt/work -\u003e /home/driver/dev/scipype/\n        scipype$ \n\n#.  If you want to always use this virtual environment add this to your ``~/.bashrc``\n\n    ::\n\n        echo 'source /opt/venv/scipype/bin/activate' \u003e\u003e ~/.bashrc\n\n#.  Confirm the Demo downloader works using the Virtual Environment\n\n    Please note: this assumes running from a new terminal to validate the virtual environment activation\n\n    Activate it\n\n    ::\n\n        scipype$ source ./dev-properties.sh\n\n    Run the Demo\n\n    ::\n\n        (scipype) scipype$ ./bins/demo-running-locally.py \n        Downloading(SPY) Dates[Jan, 02 2016 - Jan, 02 2017]\n        Storing CSV File(/opt/scipype/data/src/spy.csv)\n        Done Downloading CSV for Ticker(SPY)\n        Success File exists: /opt/scipype/data/src/spy.csv\n\n    Deactivate it\n\n    ::\n\n        (scipype) scipype$ deactivate \n        scipype$ \n\n\n#.  If you want to automatically load the full Scipype environment ``properties.sh`` for any new shell terminal add this to your user's ``~/.bashrc``\n\n    ::\n\n        echo 'source /opt/work/properties.sh' \u003e\u003e ~/.bashrc\n\n.. _Coming Soon and Known Issues: https://github.com/jay-johnson/sci-pype/blob/master/README.rst#coming-soon-and-known-issues\n\nAuthenticated Redis Examples\n============================\n\nYou can lock redis down with a password by setting it in the redis.conf before starting the redis server (https://redis.io/topics/security#authentication-feature). Here is how to use the machine learning API with a password-locked Redis Labs endpoint or a local one.\n\nEnvironment Variables\n---------------------\n\nIf you are running sci-pype in a docker container it will load the following env vars to ensure the redis application system's clients are setup with the password and database:\n\n::\n\n    # Redis Password where Empty = No Password like:\n    # ENV_REDIS_PASSWORD=\n    ENV_REDIS_PASSWORD=2603648a854c4f3ba7c93e8449319380\n    ENV_REDIS_DB_ID=0\n\nYou can run without a password by either not defining the ``ENV_REDIS_PASSWORD`` environment variable or `making it set to an empty string`_.\n\n.. _making it set to an empty string: https://github.com/jay-johnson/sci-pype/blob/d8c750a25d7c46f34d86e8ff833580842b607366/src/connectors/redis/base_redis_application.py#L18-L21\n\nUsing a Password-locked Redis Labs Cloud endpoint\n-------------------------------------------------\n\n#.  Run the Secure Redis Labs Cloud Demo\n\n    ::\n\n        bins/ml$ ./demo-secure-rl-regressor-iris.py\n\n#.  Connect to the Redis Labs Cloud endpoint\n\n    After running it you can verify the models were stored on the secured endpoint:\n\n    ::\n\n        bins/ml$ redis-cli -h pub-redis-12515.us-west-2-1.1.ec2.garantiadata.com -p 12515\n\n#.  Verify the server is enforcing the password\n\n    ::\n\n        pub-redis-12515.us-west-2-1.1.ec2.garantiadata.com:12515\u003e KEYS *\n        (error) NOAUTH Authentication required\n\n#.  Authenticate with the password\n\n    ::\n\n        pub-redis-12515.us-west-2-1.1.ec2.garantiadata.com:12515\u003e auth 2603648a854c4f3ba7c93e8449319380\n        OK\n\n#.  View the redis keys\n\n    ::\n\n        pub-redis-12515.us-west-2-1.1.ec2.garantiadata.com:12515\u003e KEYS *\n        1) \"_MD_IRIS_REGRESSOR_PetalWidth\"\n        2) \"_MD_IRIS_REGRESSOR_PredictionsDF\"\n        3) \"_MD_IRIS_REGRESSOR_SepalWidth\"\n        4) \"_MODELS_IRIS_REGRESSOR_LATEST\"\n        5) \"_MD_IRIS_REGRESSOR_ResultTargetValue\"\n        6) \"_MD_IRIS_REGRESSOR_Accuracy\"\n        7) \"_MD_IRIS_REGRESSOR_PetalLength\"\n        8) \"_MD_IRIS_REGRESSOR_SepalLength\"\n        pub-redis-12515.us-west-2-1.1.ec2.garantiadata.com:12515\u003e exit\n        bins/ml$\n\nLocal\n-----\n\n#.  You can run a password-locked, standalone redis server with docker compose using this script:\n        \n    https://github.com/jay-johnson/sci-pype/blob/master/bins/redis/auth-start.sh\n\n#.  Once the redis server is started you can run the local secure demo with the script:\n\n    ::\n\n        bins/ml$ ./demo-secure-ml-regressor-iris.py\n\n#.  After the demo finishes you can authenticate with the local redis server and view the cached models:\n\n    ::\n\n        bins/ml$ redis-cli -p 6400\n        127.0.0.1:6400\u003e KEYS *\n        (error) NOAUTH Authentication required.\n        127.0.0.1:6400\u003e AUTH 2603648a854c4f3ba7c93e8449319380\n        OK\n        127.0.0.1:6400\u003e KEYS *\n        1) \"_MD_IRIS_REGRESSOR_PetalWidth\"\n        2) \"_MD_IRIS_REGRESSOR_PetalLength\"\n        3) \"_MD_IRIS_REGRESSOR_PredictionsDF\"\n        4) \"_MD_IRIS_REGRESSOR_SepalWidth\"\n        5) \"_MODELS_IRIS_REGRESSOR_LATEST\"\n        6) \"_MD_IRIS_REGRESSOR_Accuracy\"\n        7) \"_MD_IRIS_REGRESSOR_ResultTargetValue\"\n        8) \"_MD_IRIS_REGRESSOR_SepalLength\"\n        127.0.0.1:6400\u003e exit\n        bins/ml$ \n    \n#.  If you want to stop the redis server run:\n\n    https://github.com/jay-johnson/sci-pype/blob/master/bins/redis/stop.sh\n\nPrevious Examples\n=================\n\n\nVersion 1 Examples\n-------------------\n\n#.  `example-core-demo.ipynb`_ \n\n    How to use the python core from a Jupyter notebook. It also shows how to debug the JSON application configs which are used to connect to external database(s) and redis server(s).\n    \n    .. figure:: https://jaypjohnson.com/_images/image_2016-08-01_core-integration.png\n\n#.  `example-spy-downloader.ipynb`_\n\n    **Jupyter + Downloading the SPY Pricing Data**\n\n    Download the SPY ETF Pricing Data from Google Finance and store it in the shared ``ENV_PYTHON_SRC_DIR`` directory that is mounted from the host and into the Jupyter container. It uses a script that `downloads the SPY daily pricing data as a csv file`_.\n\n    .. figure:: https://jaypjohnson.com/_images/image_2016-08-01_download-spy-pricing-data.png\n\n#.  `example-plot-stock-data.ipynb`_\n\n    **Download SPY and use Pandas + Matlab to Plot Pricing by the Close**\n\n    This shows how to download the SPY daily prices from Google Finance as a csv then load it using Pandas for plotting on the Close prices with Matlab.\n\n    .. figure:: https://jaypjohnson.com/_images/image_2016-08-01_plot-spy-by-close-prices.png\n\n#.  `example-redis-cache-demo.ipynb`_\n\n    **Building a Jupyter + Redis Data Pipeline**\n\n    This extends the previous SPY pricing demo and publishes + retreives the pricing data by using a targeted ``CACHE`` redis server (that runs inside the Jupyter container). It stores the Pandas dataframe as JSON in the ``LATEST_SPY_DAILY_STICKS`` redis key.\n\n    .. figure:: https://jaypjohnson.com/_images/image_2016-08-01_redis-data-pipeline-with-spy-prices.png\n\n#.  `example-db-extract-and-cache.ipynb`_\n\n    **Building a Jupyter + MySQL + Redis Data Pipeline**\n\n    This requires running the **Full Stack** which uses the https://github.com/jay-johnson/sci-pype/blob/master/full-stack-compose.yml to deploy three docker containers on the same host:\n    \n    - MySQL (https://hub.docker.com/r/jayjohnson/schemaprototyping/)\n    - Jupyter (https://hub.docker.com/r/jayjohnson/jupyter/)\n    - Redis (https://hub.docker.com/r/jayjohnson/redis-single-node/)\n\n    **How it works**\n\n    .. figure:: https://jaypjohnson.com/_images/image_2016-08-01_using-jupyter-for-stock-analysis.png\n\n    1. Extract the IBM stock data from the MySQL dataset and store it as a csv inside the **/opt/work/data/src/ibm.csv** file\n    2. Load the IBM pricing data with Pandas\n    3. Plot the pricing data with Matlab\n    4. Publish the Pandas Dataframe as JSON to Redis\n    5. Retrieve the Pandas Dataframe from Redis\n    6. Test the cached pricing data exists outside the Jupyter container with:\n\n        ::\n\n            $ ./redis.sh \n            SSH-ing into Docker image(redis-server)\n            [root@redis-server container]# redis-cli -h localhost -p 6000\n            localhost:6000\u003e LRANGE LATEST_IBM_DAILY_STICKS 0 0\n            1) \"(dp0\\nS'Data'\\np1\\nS'{\\\"Date\\\":{\\\"49\\\":971136000000,\\\"48\\\":971049600000,\\\"47\\\":970790400000,\\\"46\\\":970704000000,\\\"45\\\":970617600000,\\\"44\\\":970531200000,\\\"43\\\":970444800000,\\\"42\\\":970185600000,\\\"41\\\":970099200000,\\\"40\\\":970012800000,\\\"39\\\":969926400000,\\\"38\\\":969\n\n             ... removed for docs ... \n\n            localhost:6000\u003e exit\n            [root@redis-server container]# exit\n            exit\n            $\n\n#.  `example-slack-debugging.ipynb`_\n\n    **Jupyter + Slack Driven Development**\n\n    This example shows how environment variables allow the python core to publish a message into Slack to notify the associated user with a message containing the line number and source code that threw the exception.\n\n    .. figure:: https://jaypjohnson.com/_images/image_2016-08-01_slack-debugging.png\n\n    .. _example-core-demo.ipynb: https://github.com/jay-johnson/sci-pype/blob/master/examples/example-core-demo.ipynb\n    .. _example-slack-debugging.ipynb: https://github.com/jay-johnson/sci-pype/blob/master/examples/example-slack-debugging.ipynb\n    .. _example-spy-downloader.ipynb: https://github.com/jay-johnson/sci-pype/blob/master/examples/example-spy-downloader.ipynb\n    .. _example-plot-stock-data.ipynb: https://github.com/jay-johnson/sci-pype/blob/master/examples/example-plot-stock-data.ipynb\n    .. _example-redis-cache-demo.ipynb: https://github.com/jay-johnson/sci-pype/blob/master/examples/example-redis-cache-demo.ipynb\n    .. _example-db-extract-and-cache.ipynb: https://github.com/jay-johnson/sci-pype/blob/master/examples/example-db-extract-and-cache.ipynb\n    .. _downloads the SPY daily pricing data as a csv file: https://github.com/jay-johnson/sci-pype/blob/master/bins/download-spy-csv.py\n\n\nComponents\n==========\n\n#.  Python 2 Core\n\n    The PyCore_ uses a `JSON config file`_ for connecting to `redis servers`_ and `configurable databases`_ (MySQL and Postgres) using SQLAlchemy. It has only been tested with the Python 2.7 kernel.\n\n    .. _PyCore: https://github.com/jay-johnson/sci-pype/blob/master/src/pycore.py\n    .. _JSON config file: https://github.com/jay-johnson/sci-pype/blob/master/configs/jupyter.json\n    .. _redis servers: https://github.com/jay-johnson/sci-pype/blob/master/configs/redis.json\n    .. _configurable databases: https://github.com/jay-johnson/sci-pype/blob/master/configs/db.json\n\n#.  Local Redis Server\n\n    When starting the container with ``ENV_DEPLOYMENT_TYPE`` set to anything not ``JustDB``, the container will start a local redis server inside the container on port ``6000`` for iterating on your pipeline analysis, Model deployment and caching strategies.\n\n#.  Loading Database and Redis Applications\n\n    By default the `jupyter.json`_ config supports multiple environments for integrating notebooks with external resources. Here is table on what they define:\n\n    +-----------+---------------------------------------------------+-----------------------+-----------------------+\n    | Name      | Purpose                                           | Redis Applications    | Database Applications | \n    +===========+===================================================+=======================+=======================+\n    | Local     | Use the internal redis server with the stock db   | `local-redis.json`_   | `db.json`_            |\n    +-----------+---------------------------------------------------+-----------------------+-----------------------+\n    | NoApps    | Run the core without redis servers or databases   | `empty-redis.json`_   | `empty-db.json`_      |\n    +-----------+---------------------------------------------------+-----------------------+-----------------------+\n    | JustRedis | Run with just the redis servers and no databases  | `local-redis.json`_   | `empty-db.json`_      |\n    +-----------+---------------------------------------------------+-----------------------+-----------------------+\n    | JustDB    | Run without redis servers and load the databases  | `empty-redis.json`_   | `db.json`_            |\n    +-----------+---------------------------------------------------+-----------------------+-----------------------+\n    | Test      | Connect to external redis servers and databases   | `redis.json`_         | `db.json`_            |\n    +-----------+---------------------------------------------------+-----------------------+-----------------------+\n    | Live      | Connect to external redis servers and databases   | `redis.json`_         | `db.json`_            |\n    +-----------+---------------------------------------------------+-----------------------+-----------------------+\n\n    Inside a notebook you can target a different environment before loading the core with:\n\n    - Changing to the **JustRedis** Environment:\n\n        ::\n            \n            import os\n            os.environ[\"ENV_DEPLOYMENT_TYPE\"] = \"JustRedis\"\n            core = PyCore()\n\n    - Changing to the **NoApps** Environment:\n\n        ::\n            \n            import os\n            os.environ[\"ENV_DEPLOYMENT_TYPE\"] = \"NoApps\"\n            core = PyCore()\n\n    .. _jupyter.json: https://github.com/jay-johnson/sci-pype/blob/master/configs/jupyter.json\n    .. _local-redis.json: https://github.com/jay-johnson/sci-pype/blob/master/configs/local-redis.json\n    .. _redis.json: https://github.com/jay-johnson/sci-pype/blob/master/configs/redis.json\n    .. _db.json: https://github.com/jay-johnson/sci-pype/blob/master/configs/db.json\n    .. _empty-redis.json: https://github.com/jay-johnson/sci-pype/blob/master/configs/empty-redis.json\n    .. _empty-db.json: https://github.com/jay-johnson/sci-pype/blob/master/configs/empty-db.json\n\n#.  Customize the Jupyter Container Lifecycle\n\n    The following environment variables can be used for defining pre-start, start, and post-start Jupyter actions as needed.\n\n    +---------------------------+-----------------------------------------------+--------------------------------------------+\n    | Environment Variable      | Default Value                                 | Purpose                                    |\n    +===========================+===============================================+============================================+\n    | **ENV_PRESTART_SCRIPT**   | `/opt/containerfiles/pre-start-notebook.sh`_  | Run custom actions before starting Jupyter |\n    +---------------------------+-----------------------------------------------+--------------------------------------------+\n    | **ENV_START_SCRIPT**      | `/opt/containerfiles/start-notebook.sh`_      | Start Jupyter                              |\n    +---------------------------+-----------------------------------------------+--------------------------------------------+\n    | **ENV_POSTSTART_SCRIPT**  | `/opt/containerfiles/post-start-notebook.sh`_ | Run custom actions after starting Jupyter  |\n    +---------------------------+-----------------------------------------------+--------------------------------------------+\n\n    .. _/opt/containerfiles/pre-start-notebook.sh: https://github.com/jay-johnson/sci-pype/blob/master/containerfiles/pre-start-notebook.sh\n    .. _/opt/containerfiles/start-notebook.sh: https://github.com/jay-johnson/sci-pype/blob/master/containerfiles/start-notebook.sh\n    .. _/opt/containerfiles/post-start-notebook.sh: https://github.com/jay-johnson/sci-pype/blob/master/containerfiles/post-start-notebook.sh\n\n#.  Slack Debugging\n\n    The core supports publishing exceptions into Slack based off the environment variables passed in using docker or docker compose.\n\n#.  Tracking Installed Dependencies for Notebook Sharing\n\n    This docker container uses these files for tracking Python 2 and Python 3 pips:\n\n    - /opt/work/pips/python2-requirements.txt\n\n    - /opt/work/pips/python3-requirements.txt\n\n#.  Shared Volumes\n\n    These are the mounted volumes and directories that can be changed as needed. Also the core uses them as environment variables.\n\n    +-------------------------------+---------------------------+-----------------------------------------------------------+\n    | Host Mount                    | Container Mount           | Purpose                                                   |\n    +===============================+===========================+===========================================================+\n    | **/opt/project**              | /opt/project              | Sharing a project from the host machine                   |\n    +-------------------------------+---------------------------+-----------------------------------------------------------+\n    | **/opt/work/data**            | /opt/work/data            | Sharing a common data dir between host and containers     |\n    +-------------------------------+---------------------------+-----------------------------------------------------------+\n    | **/opt/work/data/src**        | /opt/work/data/src        | Passing data source files into the container              |\n    +-------------------------------+---------------------------+-----------------------------------------------------------+\n    | **/opt/work/data/dst**        | /opt/work/data/dst        | Passing processed data files outside the container        |\n    +-------------------------------+---------------------------+-----------------------------------------------------------+\n    | **/opt/work/data/bin**        | /opt/work/data/bin        | Exchanging data binaries from the host into the container |\n    +-------------------------------+---------------------------+-----------------------------------------------------------+\n    | **/opt/work/data/synthesize** | /opt/work/data/synthesize | Sharing files used for synthesizing data                  |\n    +-------------------------------+---------------------------+-----------------------------------------------------------+\n    | **/opt/work/data/tidy**       | /opt/work/data/tidy       | Sharing files used to tidy and marshall data              |\n    +-------------------------------+---------------------------+-----------------------------------------------------------+\n    | **/opt/work/data/analyze**    | /opt/work/data/analyze    | Sharing files used for data analysis and processing       |\n    +-------------------------------+---------------------------+-----------------------------------------------------------+\n    | **/opt/work/data/output**     | /opt/work/data/output     | Sharing processed files and analyzed output               |\n    +-------------------------------+---------------------------+-----------------------------------------------------------+\n\nGetting Started\n===============\n\nLocal Jupyter\n-------------\n\n#.  Start the Container in Local development mode\n\n    ::\n\n        $ ./start.sh \n        Starting new Docker image(docker.io/jayjohnson/jupyter)\n        4275447ef6a3aa06fb06097837deeb202bd80b15969a9c1269a5ee042d8df13d\n        $ \n\n#.  Browse to the local Jupyter website\n\n    http://localhost:82/\n\nFull Stack\n----------\n\nThe full-stack-compose.yml_ patches the Jupyter and redis containers to ensure the MySQL database is listening on port 3306 before starting. It does this by defining a custom `entrypoint`_ wrapper for each in the `wait-for-its tools`_ directory.\n\n.. _full-stack-compose.yml: https://github.com/jay-johnson/sci-pype/blob/master/full-stack-compose.yml\n.. _entrypoint: https://github.com/jay-johnson/sci-pype/blob/master/full-stack-compose.yml#L25\n.. _wait-for-its tools: https://github.com/jay-johnson/sci-pype/tree/master/wait-for-its\n\n#.  Start the Composition\n\n    This can take around 20 seconds for MySQL to set up the seed pricing records, and it requires assigning the shared data directory permissions for read/write access from inside the Jupyter container.\n\n    ::\n\n        $ ./compose-start-full.sh \n        Before starting changing permissions with:\n           chown -R driver:users /opt/work/data/*\n        [sudo] password for driver: \n        Starting Composition: full-stack-compose.yml\n        Starting stocksdb\n        Starting jupyter\n        Starting redis-server\n        Done\n        $ \n\n#.  Check the Composition\n\n    ::\n\n        $ docker ps\n        CONTAINER ID        IMAGE                                COMMAND                  CREATED             STATUS              PORTS                                        NAMES\n        1fd9bd22987f        jayjohnson/redis-single-node:1.0.0   \"/wait-for-its/redis-\"   12 minutes ago      Up 25 seconds       0.0.0.0:6000-\u003e6000/tcp                       redis-server\n        2bcb6b8d2994        jayjohnson/jupyter:1.0.0             \"/wait-for-its/jupyte\"   12 minutes ago      Up 25 seconds       0.0.0.0:8888-\u003e8888/tcp                       jupyter\n        b7bce846b9af        jayjohnson/schemaprototyping:1.0.0   \"/root/start_containe\"   25 minutes ago      Up 25 seconds       0.0.0.0:81-\u003e80/tcp, 0.0.0.0:3307-\u003e3306/tcp   stocksdb\n        $\n\n    - Optional - Login to the database container\n\n    ::\n\n        $ ./db.ssh \n        SSH-ing into Docker image(stocksdb)\n        [root@stocksdb db-loaders]# ps auwwx | grep mysql | grep -v grep\n        root        28  0.0  0.0  11648  2752 ?        S    17:00   0:00 /bin/sh /usr/bin/mysqld_safe\n        mysql      656  1.3 12.0 1279736 474276 ?      Sl   17:00   0:01 /usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib64/mysql/plugin --user=mysql --log-error=/var/log/mysql/error.log --pid-file=/var/lib/mysql/mysqld.pid --socket=/var/lib/mysql/mysqld.sock --port=3306\n        [root@stocksdb db-loaders]# exit\n\n    View the Stocks Database with phpMyAdmin: http://localhost:81/phpmyadmin/sql.php?db=stocks\u0026table=stocks\n\n    .. note:: By default the login to this sample db is: ``dbadmin`` / ``dbadmin123`` which can be configured in the db.env_\n\n    .. _db.env: https://github.com/jay-johnson/sci-pype/blob/master/test/db.env#L6-L7\n\n    - Optional - Login to the Redis container\n\n    ::\n      \n        $ ./redis.sh \n        SSH-ing into Docker image(redis-server)\n        [root@redis-server container]# ps auwwx | grep redis\n        root         1  0.0  0.0  11644  2616 ?        Ss   17:00   0:00 bash /wait-for-its/redis-wait-for-it.sh\n        root        28  0.0  0.2 114800 11208 ?        Ss   17:00   0:00 /usr/bin/python /usr/bin/supervisord -c /etc/supervisor.d/rediscluster.ini\n        root        30  0.3  0.0  37268  3720 ?        Sl   17:00   0:00 redis-server *:6000\n        root        47  0.0  0.0   9044   892 ?        S+   17:02   0:00 grep --color=auto redis\n        [root@redis-server container]# exit\n\n    - Optional - Login to the Jupyter container\n\n    ::\n\n        $ ./ssh.sh \n        SSH-ing into Docker image(jupyter)\n        jovyan:/opt/work$ ps auwwx | grep jupyter\n        jovyan       1  0.0  0.0  13244  2908 ?        Ss   17:00   0:00 bash /wait-for-its/jupyter-wait-for-it.sh\n        jovyan      38  0.3  1.2 180564 48068 ?        S    17:00   0:00 /opt/conda/bin/python /opt/conda/bin/jupyter-notebook\n        jovyan:/opt/work$ exit\n\n#.  Run the Database Extraction Jupyter Demo\n\n    Open the notebook with this url: http://localhost:82/notebooks/examples/example-db-extract-and-cache.ipynb\n\n#.  Click the **Run** Button\n    \n    This example will connect to the ``stocksdb`` MySQL container and pull 50 records from IBM's pricing data. It will then render plot lines for Open, Close, High, and Low using Pandas and Matlab. Next it will cache the IBM records in the ``redis-server`` container and then verify those records were cached correctly by retrieving it again.\n\n#.  From outside the Jupyter container confirm the redis key holds the processed IBM data\n        \n    ::\n\n        $ ./redis.sh \n        SSH-ing into Docker image(redis-server)\n        [root@redis-server container]# redis-cli -h localhost -p 6000\n        localhost:6000\u003e LRANGE LATEST_IBM_DAILY_STICKS 0 0\n        1) \"(dp0\\nS'Data'\\np1\\nS'{\\\"Date\\\":{\\\"49\\\":971136000000,\\\"48\\\":971049600000,\\\"47\\\":970790400000,\\\"46\\\":970704000000,\\\"45\\\":970617600000,\\\"44\\\":970531200000,\\\"43\\\":970444800000,\\\"42\\\":970185600000,\\\"41\\\":970099200000,\\\"40\\\":970012800000,\\\"39\\\":969926400000,\\\"38\\\":969\n\n         ... removed for docs ... \n\n        localhost:6000\u003e exit\n        [root@redis-server container]# exit\n        exit\n        $\n\n#.  Stop the Composition\n\n    ::\n        \n        $ ./compose-stop-full.sh \n        Stopping Composition: full-stack-compose.yml\n        Stopping redis-server ... done\n        Stopping jupyter ... done\n        Stopping stocksdb ... done\n        Done\n        $\n\nStandalone\n----------\n\n#.  Start Standalone\n\n    Start the standalone Jupyter container using the `jupyter-docker-compose.yml`_ file. This compose file requires access to ``/opt/work/data`` host directory like the Full Stack version for sharing files between the container and the host.\n\n    ::\n\n        $ ./compose-start-jupyter.sh \n        Before starting changing permissions with:\n           chown -R driver:users /opt/work/data/*\n        [sudo] password for driver: \n        Starting Composition: jupyter-docker-compose.yml\n        Starting jupyter\n        Done\n        $\n\n#.  Stop Standalone\n\n    Stop the standalone Jupyter composition with:\n\n    ::\n\n        $ ./compose-stop-jupyter.sh \n        Stopping Composition: jupyter-docker-compose.yml\n        Stopping jupyter ... done\n        Done\n        $\n\nDeleting the Containers \n-----------------------\n\nRemove the containers with the command:\n\n    ::\n    \n        $ docker rm jupyter redis-server stocksdb\n        jupyter\n        redis-server\n        stocksdb\n        $ \n\nDelete them from the host with: \n\n    ::\n\n        $ docker rmi jayjohnson/schemaprototyping\n        $ docker rmi jayjohnson/jupyter\n        $ docker rmi jayjohnson/redis-single-node\n\n\nSharing between the Host and the Jupyter Container\n--------------------------------------------------\n\nBy default, the host will have this directory structure available for passing files in and out of the container:\n\n::\n\n    $ tree /opt/work\n    /opt/work\n    └── data\n        ├── analyze\n        ├── bin\n        ├── dst\n        ├── output\n        ├── src\n        │   └── spy.csv\n        ├── synthesize\n        └── tidy\n\n    8 directories, 1 file\n\nFrom inside the container here is where the directories are mapped:\n\n::\n\n    $ ./ssh.sh \n    SSH-ing into Docker image(jupyter)\n    driver:/opt/work$ tree data/\n    data/\n    ├── analyze\n    ├── bin\n    ├── dst\n    ├── output\n    ├── src\n    │   └── spy.csv\n    ├── synthesize\n    └── tidy\n\n    7 directories, 1 file\n\nComing Soon and Known Issues\n============================\n\n#.  Missing xattr.h\n    \n    If you see this error:\n\n    ::\n\n        xattr.c:29:24: fatal error: attr/xattr.h: No such file or directory\n\n    Install RPM:\n\n    ::\n        \n        sudo yum install -y libattr-devel\n\n    Install Deb:\n\n    ::\n\n        sudo apt-get install -y libattr1-dev\n\n    Retry the install\n\n#.  Local Install Confluent:\n\n    If you're trying to setup the local development environment and missing the kafka headers:\n\n    ::\n\n        In file included from confluent_kafka/src/confluent_kafka.c:17:0:\n        confluent_kafka/src/confluent_kafka.h:21:32: fatal error: librdkafka/rdkafka.h: No such file or directory\n        #include \u003clibrdkafka/rdkafka.h\u003e\n\n    Please install Kafka by adding their repository and then installing: \n    \n    ::\n    \n        $ sudo yum install confluent-platform-oss-2.11\n        $ sudo yum install librdkafka1 librdkafka-devel\n\n    Official RPM Guide: http://docs.confluent.io/3.1.1/installation.html#rpm-packages-via-yum\n\n    Official DEB Guide: http://docs.confluent.io/3.1.1/installation.html#deb-packages-via-apt\n\n    For ``Fedora 24/RHEL 7/CentOS 7`` users here's a tool to help:\n    \n    ::\n        \n        scipype/python2$ sudo ./install_confluent_platform.sh\n\n#.  Install PyQt4 for ``ImportError: No module named PyQt4`` errors:\n\n\t::\n\n\t\t(python2) jovyan:/opt/work/bins$ conda install -y pyqt=4.11\n\t\tFetching package metadata .........\n\t\tSolving package specifications: ..........\n\n\t\tPackage plan for installation in environment /opt/conda/envs/python2:\n\n\t\tThe following packages will be downloaded:\n\n\t\t\tpackage                    |            build\n\t\t\t---------------------------|-----------------\n\t\t\tqt-4.8.7                   |                3        31.3 MB  conda-forge\n\t\t\tpyqt-4.11.4                |           py27_2         3.5 MB  conda-forge\n\t\t\t------------------------------------------------------------\n\t\t\t\t\t\t\t\t\t\t\t\tTotal:        34.8 MB\n\n\t\tThe following NEW packages will be INSTALLED:\n\n\t\t\tpyqt: 4.11.4-py27_2 conda-forge\n\t\t\tqt:   4.8.7-3       conda-forge (copy)\n\n\t\tPruning fetched packages from the cache ...\n\t\tFetching packages ...\n\t\tqt-4.8.7-3.tar 100% |##########################################################################################################################################| Time: 0:00:06   5.23 MB/s\n\t\tpyqt-4.11.4-py 100% |##########################################################################################################################################| Time: 0:00:02   1.28 MB/s\n\t\tExtracting packages ...\n\t\t[      COMPLETE      ]|#############################################################################################################################################################| 100%\n\t\tLinking packages ...\n\t\t[      COMPLETE      ]|#############################################################################################################################################################| 100%\n\n\n\tNow try running a script from the shell:\n\n\t::\n\n\t\t(python2) jovyan:/opt/work/bins$ ./download-spy-csv.py \n\t\tDownloading(SPY) Dates[Jan, 02 2016 - Jan, 02 2017]\n\t\tStoring CSV File(/opt/work/data/src/spy.csv)\n\t\tDone Downloading CSV for Ticker(SPY)\n\t\tSuccess File exists: /opt/work/data/src/spy.csv\n\t\t(python2) jovyan:/opt/work/bins$ \n\n#.  How to build a customized Python Core mounted from outside the Jupyter container\n\n#.  Fixing the docker compose networking so the stocksdb container does not need to know the compose-generated docker network.\n\n    Right now it is defining the `sci-pype_datapype`_ as the expected docker network. This may not work on older versions of docker.\n    \n    .. _sci-pype_datapype: https://github.com/jay-johnson/sci-pype/blob/master/test/db.env#L1\n\n#.  Building Jupyter containers that are smaller and only run one kernel to reduce the overall size of the image\n\n#.  Testing on an older docker version\n\n    This was tested with ``1.12.1``\n\n    ::\n\n        $ docker -v\n        Docker version 1.12.1, build 23cf638\n        $ \n\n\n#.  Setting up the Jupyter wait-for-it to ensure the stocks database is loaded before starting...not just the port is up\n\n    For now just shutdown the notebook kernel if you see an error related to the stocks database not being there when running the full stack.\n\nComing Soon\n===========\n\n-   Examples for using the Timestamp Forecasting API\n\n-   Post Processing Event API support\n\n-   Confluent Kafka integration notebooks (the python client is already installed in the virtual env and docker container https://github.com/confluentinc/confluent-kafka-python)\n\n-   PySpark integration notebooks\n\n-   Tensorflow integration notebooks\n\n-   Adding more support and optional third-party mounting for customized Machine Learning Algorithms (like https://github.com/pandas-ml/pandas-ml)\n\n-   Odo (http://odo.readthedocs.io/en/latest/) examples \n\n-   Lambda deployment integration (http://docs.aws.amazon.com/lambda/latest/dg/lambda-python-how-to-create-deployment-package.html) and likely FPM (https://github.com/jordansissel/fpm) for package building.\n\nLicense\n=======\n\nThis project is not related to SciPy.org or the scipy library. It was originally built for exchanging and loading datasets using Redis for creating near-realtime data pipelines for streaming analysis (like a scientific pypeline).\n\nThis repo is Apache 2.0 License: https://github.com/jay-johnson/sci-pype/blob/master/LICENSE\n\nJupyter - BSD: https://github.com/jupyter/jupyter/blob/master/LICENSE\n\nPlease refer to the Conda Licenses for individual Python libraries: https://docs.continuum.io/anaconda/pkg-docs\n\nRedis - https://redis.io/topics/license\n\nzlib - https://opensource.org/licenses/zlib-license.php\n\n\n","funding_links":[],"categories":["Python"],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjay-johnson%2Fsci-pype","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjay-johnson%2Fsci-pype","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjay-johnson%2Fsci-pype/lists"}