{"id":15159318,"url":"https://github.com/polyaxon/polyaxon-lib","last_synced_at":"2025-09-30T09:30:34.132Z","repository":{"id":57454029,"uuid":"94631683","full_name":"polyaxon/polyaxon-lib","owner":"polyaxon","description":"Deep Learning and Reinforcement learning library for TensorFlow for building end to end models and experiments.","archived":true,"fork":false,"pushed_at":"2018-04-11T15:35:34.000Z","size":849,"stargazers_count":8,"open_issues_count":1,"forks_count":6,"subscribers_count":4,"default_branch":"master","last_synced_at":"2024-12-16T16:07:09.343Z","etag":null,"topics":["data-science","deep-learning","machine-learning","reinforcement-learning","tensorflow","tensorflow-experiments"],"latest_commit_sha":null,"homepage":"http://polyaxon.com/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/polyaxon.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2017-06-17T15:18:04.000Z","updated_at":"2023-01-28T19:45:05.000Z","dependencies_parsed_at":"2022-08-29T06:51:24.722Z","dependency_job_id":null,"html_url":"https://github.com/polyaxon/polyaxon-lib","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/polyaxon%2Fpolyaxon-lib","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/polyaxon%2Fpolyaxon-lib/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/polyaxon%2Fpolyaxon-lib/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/polyaxon%2Fpolyaxon-lib/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/polyaxon","download_url":"https://codeload.github.com/polyaxon/polyaxon-lib/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":234575251,"owners_count":18854926,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["data-science","deep-learning","machine-learning","reinforcement-learning","tensorflow","tensorflow-experiments"],"created_at":"2024-09-26T21:04:29.123Z","updated_at":"2025-09-30T09:30:28.783Z","avatar_url":"https://github.com/polyaxon.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"[![License: MIT](https://img.shields.io/badge/License-MIT-green.svg)](LICENSE)\n[![Build Status](https://travis-ci.org/polyaxon/polyaxon-lib.svg?branch=master)](https://travis-ci.org/polyaxon/polyaxon-lib)\n[![PyPI version](https://badge.fury.io/py/polyaxon-lib.svg)](https://badge.fury.io/py/polyaxon-lib)\n[![Slack](https://img.shields.io/badge/chat-on%20slack-aadada.svg?logo=slack\u0026longCache=true)](https://join.slack.com/t/polyaxon/shared_invite/enQtMzQ0ODc2MDg1ODc0LWY2ZTdkMTNmZjBlZmRmNjQxYmYwMTBiMDZiMWJhODI2ZTk0MDU4Mjg5YzA5M2NhYzc5ZjhiMjczMDllYmQ2MDg)\n\n# Polyaxon-Lib\n\nDeep Learning and Reinforcement learning library for TensorFlow for building end to end models and experiments.\n\n# Design Goals\n\nPolyaxon-Lib was built with the following goals:\n\n * Modularity: The creation of a computation graph based on modular and understandable modules,\n    with the possibility to reuse and share the module in subsequent usage.\n\n * Usability: Training a model should be easy enough, and should enable quick experimentations.\n\n * Configurable: Models and experiments could be created using a YAML/Json file, but also in python files.\n\n * Extensibility: The modularity and the extensive documentation of the code makes it easy to build and extend the set of provided modules.\n\n * Performance: Polyaxon is based on internal `tensorflow` code base and leverage the builtin distributed learning.\n\n * Data Preprocessing: Polyaxon provides many pipelines and data processor to support different data inputs.\n\n\n# Quick start\n\n## A simple linear regression\n\n```python\nfrom polyaxon_schemas.losses import MeanSquaredErrorConfig\nfrom polyaxon_schemas.optimizers import SGDConfig\n\nimport polyaxon_lib as plx\n\nX = np.linspace(-1, 1, 100)\ny = 2 * X + np.random.randn(*X.shape) * 0.33\n\n# Test a data set\nX_val = np.linspace(1, 1.5, 10)\ny_val = 2 * X_val + np.random.randn(*X_val.shape) * 0.33\n\n\ndef graph_fn(mode, inputs):\n    return plx.layers.Dense(units=1,)(inputs['X'])\n\n\ndef model_fn(features, labels, mode):\n    model = plx.models.Regressor(\n        mode, \n        graph_fn=graph_fn, \n        loss=MeanSquaredErrorConfig(),\n        optimizer=SGDConfig(learning_rate=0.009),\n        summaries='all', \n        name='regressor')\n    return model(features, labels)\n\n\nestimator = plx.estimators.Estimator(model_fn=model_fn, model_dir=\"/tmp/polyaxon_logs/linear\")\n\nestimator.train(input_fn=numpy_input_fn(\n    {'X': X}, y, shuffle=False, num_epochs=10000, batch_size=len(X)))\n```\n\n\n## A reinforcement learning problem\n\n```python\nfrom polyaxon_schemas.losses import HuberLossConfig\nfrom polyaxon_schemas.optimizers import SGDConfig\nfrom polyaxon_schemas.rl.explorations import DecayExplorationConfig\n\nimport polyaxon_lib as plx\n\nenv = plx.envs.GymEnvironment('CartPole-v0')\n\ndef graph_fn(mode, features):\n    return plx.layers.Dense(units=512)(features['state'])\n\ndef model_fn(features, labels, mode):\n    model = plx.models.DDQNModel(\n        mode,\n        graph_fn=graph_fn,\n        loss=HuberLossConfig(),\n        num_states=env.num_states,\n        num_actions=env.num_actions,\n        optimizer=SGDConfig(learning_rate=0.01),\n        exploration_config=DecayExplorationConfig(),\n        target_update_frequency=10,\n        summaries='all')\n    return model(features, labels)\n\nmemory = plx.rl.memories.Memory()\nagent = plx.estimators.Agent(\n    model_fn=model_fn, memory=memory, model_dir=\"/tmp/polyaxon_logs/ddqn_cartpole\")\n\nagent.train(env)\n```\n\n\n## A classification problem\n\n```python\nimport tensorflow as tf\nimport polyaxon_lib as plx\n\nfrom polyaxon_schemas.optimizers import AdamConfig\nfrom polyaxon_schemas.losses import SigmoidCrossEntropyConfig\nfrom polyaxon_schemas.metrics import AccuracyConfig\n\n\ndef graph_fn(mode, features):\n    x = plx.layers.Conv2D(filters=32, kernel_size=5)(features['image'])\n    x = plx.layers.MaxPooling2D(pool_size=2)(x)\n    x = plx.layers.Conv2D(filters=64, kernel_size=5)(x)\n    x = plx.layers.MaxPooling2D(pool_size=2)(x)\n    x = plx.layers.Flatten()(x)\n    x = plx.layers.Dense(units=10)(x)\n    return x\n\n\ndef model_fn(features, labels, params, mode, config):\n    model = plx.models.Classifier(\n        mode=mode,\n        graph_fn=graph_fn,\n        loss=SigmoidCrossEntropyConfig(),\n        optimizer=AdamConfig(\n            learning_rate=0.007, decay_type='exponential_decay', decay_rate=0.1),\n        metrics=[AccuracyConfig()],\n        summaries='all',\n        one_hot_encode=True,\n        n_classes=10)\n    return model(features=features, labels=labels, params=params, config=config)\n\n\ndef experiment_fn(output_dir):\n    \"\"\"Creates an experiment using Lenet network.\n\n    Links:\n        * http://yann.lecun.com/exdb/publis/pdf/lecun-01a.pdf\n    \"\"\"\n    dataset_dir = '../data/mnist'\n    plx.datasets.mnist.prepare(dataset_dir)\n    train_input_fn, eval_input_fn = plx.datasets.mnist.create_input_fn(dataset_dir)\n\n    experiment = plx.experiments.Experiment(\n        estimator=plx.estimators.Estimator(model_fn=model_fn, model_dir=output_dir),\n        train_input_fn=train_input_fn,\n        eval_input_fn=eval_input_fn,\n        train_steps=10000,\n        eval_steps=10)\n\n    return experiment\n\n\ndef main(*args):\n    plx.experiments.run_experiment(experiment_fn=experiment_fn,\n                                   output_dir=\"/tmp/polyaxon_logs/lenet\",\n                                   schedule='continuous_train_and_eval')\n\n\nif __name__ == \"__main__\":\n    tf.logging.set_verbosity(tf.logging.INFO)\n    tf.app.run()\n```\n\n## A regression problem\n\n```python    \nfrom polyaxon_schemas.losses import MeanSquaredErrorConfig\nfrom polyaxon_schemas.metrics import (\n    RootMeanSquaredErrorConfig,\n    MeanAbsoluteErrorConfig,\n)\nfrom polyaxon_schemas.optimizers import AdagradConfig\n\nimport polyaxon_lib as plx\n\nNUM_RNN_LAYERS = 2\nNUM_RNN_UNITS = 2\n\ndef graph_fn(mode, features):\n    x = features['x']\n    for i in range(NUM_LAYERS):\n        x = plx.layers.LSTM(units=NUM_RNN_UNITS)(x)\n    return plx.layers.Dense(units=1)(x)\n\ndef model_fn(features, labels, mode):\n    return plx.models.Regressor(\n        mode=mode,\n        graph_fn=graph_fn,\n        loss=MeanSquaredErrorConfig(),\n        optimizer=AdagradConfig(learning_rate=0.1),\n        metrics=[\n            RootMeanSquaredErrorConfig(),\n            MeanAbsoluteErrorConfig()\n        ]\n    )(features=features, labels=labels)\n\nxp = plx.experiments.Experiment(\n        estimator=plx.estimators.Estimator(model_fn=model_fn, model_dir=output_dir),\n        train_input_fn=plx.processing.numpy_input_fn(\n            x={'x': x['train']}, y=y['train'], batch_size=64, num_epochs=None, shuffle=False),\n        eval_input_fn=plx.processing.numpy_input_fn(\n            x={'x': x['train']}, y=y['train'], batch_size=32, num_epochs=None, shuffle=False),\n        train_steps=train_steps,\n        eval_steps=10)\nxp.continuous_train_and_evaluate()\n```\n\n## Creating a distributed experiment\n\n```python\n\nimport numpy as np\nimport tensorflow as tf\nfrom polyaxon_schemas.settings import RunConfig, ClusterConfig\n\nimport polyaxon_lib as plx\n\nfrom polyaxon_schemas.losses import AbsoluteDifferenceConfig\nfrom polyaxon_schemas.optimizers import SGDConfig\n\ntf.logging.set_verbosity(tf.logging.INFO)\n\n\ndef create_experiment(task_type, task_id=0):\n    def graph_fn(mode, features):\n        x = plx.layers.Dense(units=32, activation='tanh')(features['X'])\n        return plx.layers.Dense(units=1, activation='sigmoid')(x)\n\n    def model_fn(features, labels, mode):\n        model = plx.models.Regressor(\n            mode, graph_fn=graph_fn,\n            loss=AbsoluteDifferenceConfig(),\n            optimizer=SGDConfig(learning_rate=0.5,\n                                decay_type='exponential_decay',\n                                decay_steps=10),\n            summaries='all', name='xor')\n        return model(features, labels)\n\n    config = RunConfig(cluster=ClusterConfig(master=['127.0.0.1:9000'],\n                                             worker=['127.0.0.1:9002'],\n                                             ps=['127.0.0.1:9001']))\n\n    config = plx.estimators.RunConfig.from_config(config)\n    config = config.replace(task_type=task_type, task_id=task_id)\n\n    est = plx.estimators.Estimator(model_fn=model_fn, model_dir=\"/tmp/polyaxon_logs/xor\",\n                                   config=config)\n\n    # Data\n    x = np.asarray([[0., 0.], [0., 1.], [1., 0.], [1., 1.]], dtype=np.float32)\n    y = np.asarray([[0], [1], [1], [0]], dtype=np.float32)\n\n    def input_fn(num_epochs=1):\n        return plx.processing.numpy_input_fn({'X': x}, y,\n                                             shuffle=False,\n                                             num_epochs=num_epochs,\n                                             batch_size=len(x))\n\n    return plx.experiments.Experiment(est, input_fn(10000), input_fn(100))\n\n\n# \u003e\u003e create_experiment('master').train_and_evaluate()\n# \u003e\u003e create_experiment('worker').train()\n# \u003e\u003e create_experiment('ps').run_std_server()\n```\n\n## Creating concurrent experiments in kubernetes clusters based on a yaml file\n\n```yaml\n\n---\nversion: 1\n\nproject:\n  name: conv_mnsit\n\nmatrix:\n  lr:\n    logspace: 0.01:0.1:2\n\nsettings:\n  logging:\n    level: INFO\n  run_type: kubernetes\n\nenvironment:\n  delay_workers_by_global_step: true\n  n_workers: 5\n  n_ps: 3\n  run_config:\n    save_summary_steps: 100\n    save_checkpoints_steps: 100\n\nmodel:\n  classifier:\n    loss:\n      SigmoidCrossEntropy:\n    optimizer:\n      Adam:\n        learning_rate: \"{{ lr }}\"\n    metrics:\n      - Accuracy\n      - Precision\n    one_hot_encode: true\n    n_classes: 10\n    graph:\n      input_layers: image\n      layers:\n        - Conv2D:\n            filters: 32\n            kernel_size: 3\n            strides: 1\n            activation: elu\n            regularizer:\n                L2:\n                  l: 0.02\n        - MaxPooling2D:\n            pool_size: 2\n        - Conv2D:\n            filters: 64\n            kernel_size: 3\n            activation: relu\n            regularizer:\n                L2:\n                  l: 0.02\n        - MaxPooling2D:\n            pool_size: 2\n        - Flatten:\n        - Dense:\n            units: 128\n            activation: tanh\n        - Dropout:\n            rate: 0.8\n        - Dense:\n            units: 256\n            activation: tanh\n        - Dropout:\n            rate: 0.8\n        - Dense:\n            units: 10\n\ntrain:\n  train_steps: 100\n  data_pipeline:\n    TFRecordImagePipeline:\n      batch_size: 64\n      num_epochs: 5\n      shuffle: true\n      data_files: [\"../data/mnist/mnist_train.tfrecord\"]\n      meta_data_file: \"../data/mnist/meta_data.json\"\n      feature_processors:\n        image:\n          input_layers: [image]\n          layers:\n            - Cast:\n                dtype: float32\n\neval:\n  data_pipeline:\n    TFRecordImagePipeline:\n      batch_size: 32\n      num_epochs: 1\n      shuffle: False\n      data_files: [\"../data/mnist/mnist_eval.tfrecord\"]\n      meta_data_file: \"../data/mnist/meta_data.json\"\n      feature_processors:\n        image:\n          input_layers: [image]\n          layers:\n            - Cast:\n                dtype: float32\n\n```\n\n\n# Installation\n\nTo install the latest version of Polyaxon: `pip install polyaxon-lib`\n\nAlternatively, you can also install from source by running (from source folder): `python setup.py install`\n\nOr you can just clone the repo `git clone https://github.com/polyaxon/polyaxon-lib.git`, and use the commands to do everything in docker:\n \n * `cmd/rebuild` to build the docker containers.\n * `cmd/py` to start a python3 shell with all requirements installed.\n * `cmd/jupyter` to start a jupyter notebook server.\n * `cmd/tensorboard` to start a tensorboard server.\n * `cmd/test` to run the tests.   \n\n# Examples\n\nSome examples are provided [here](examples), more examples and use cases will pushed, a contribution with an example is also appreciated.\n\n# Project status\n\nPolyaxon is in a pre-release \"alpha\" state. All interfaces, programming interfaces, and data structures may be changed without prior notice. \nWe'll do our best to communicate potentially disruptive changes.\n\n# Contributions\n\nPlease follow the contribution guide line: *[Contribute to Polyaxon](CONTRIBUTING.md)*.\n\n# License\n\n[![FOSSA Status](https://app.fossa.io/api/projects/git%2Bgithub.com%2Fpolyaxon%2Fpolyaxon-lib.svg?type=large)](https://app.fossa.io/projects/git%2Bgithub.com%2Fpolyaxon%2Fpolyaxon-lib?ref=badge_large)\n\n# Credit\n\nThis work is based and was inspired from different projects, `tensorflow.contrib.learn`, `keras`, `sonnet`, `seq2seq` and many other great open source projects, see [ACKNOWLEDGEMENTS](ACKNOWLEDGEMENTS).\n\nThe idea behind creating this library is to provide a tool that allow engineers and researchers to develop and experiment with end to end solutions.\n\nThe choice of creating a new library was very important to have a complete control over the apis and future design decisions.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpolyaxon%2Fpolyaxon-lib","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpolyaxon%2Fpolyaxon-lib","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpolyaxon%2Fpolyaxon-lib/lists"}