{"id":18439453,"url":"https://github.com/idiap/importance-sampling","last_synced_at":"2025-04-05T11:07:06.100Z","repository":{"id":49686558,"uuid":"97107549","full_name":"idiap/importance-sampling","owner":"idiap","description":"Code for experiments regarding importance sampling for training neural networks","archived":false,"fork":false,"pushed_at":"2021-11-03T06:22:48.000Z","size":1041,"stargazers_count":327,"open_issues_count":10,"forks_count":62,"subscribers_count":13,"default_branch":"master","last_synced_at":"2025-03-29T10:05:49.944Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/idiap.png","metadata":{"files":{"readme":"README.rst","changelog":null,"contributing":"docs/contributing.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2017-07-13T09:53:23.000Z","updated_at":"2025-03-07T01:22:58.000Z","dependencies_parsed_at":"2022-09-14T12:40:51.146Z","dependency_job_id":null,"html_url":"https://github.com/idiap/importance-sampling","commit_stats":null,"previous_names":[],"tags_count":11,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/idiap%2Fimportance-sampling","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/idiap%2Fimportance-sampling/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/idiap%2Fimportance-sampling/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/idiap%2Fimportance-sampling/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/idiap","download_url":"https://codeload.github.com/idiap/importance-sampling/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247325693,"owners_count":20920714,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-06T06:24:50.108Z","updated_at":"2025-04-05T11:07:06.071Z","avatar_url":"https://github.com/idiap.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"Importance Sampling\n====================\n\nThis python package provides a library that accelerates the training of\narbitrary neural networks created with `Keras \u003chttp://keras.io\u003e`__ using\n**importance sampling**.\n\n.. code:: python\n\n    # Keras imports\n\n    from importance_sampling.training import ImportanceTraining\n\n    x_train, y_train, x_val, y_val = load_data()\n    model = create_keras_model()\n    model.compile(\n        optimizer=\"adam\",\n        loss=\"categorical_crossentropy\",\n        metrics=[\"accuracy\"]\n    )\n\n    ImportanceTraining(model).fit(\n        x_train, y_train,\n        batch_size=32,\n        epochs=10,\n        verbose=1,\n        validation_data=(x_val, y_val)\n    )\n\n    model.evaluate(x_val, y_val)\n\nImportance sampling for Deep Learning is an active research field and this\nlibrary is undergoing development so your mileage may vary.\n\nRelevant Research\n-----------------\n\n**Ours**\n\n* Not All Samples Are Created Equal: Deep Learning with Importance Sampling [`preprint \u003chttps://arxiv.org/abs/1803.00942\u003e`__]\n* Biased Importance Sampling for Deep Neural Network Training [`preprint \u003chttps://arxiv.org/abs/1706.00043\u003e`__]\n\n**By others**\n\n* Stochastic optimization with importance sampling for regularized loss\n  minimization [`pdf \u003chttp://www.jmlr.org/proceedings/papers/v37/zhaoa15.pdf\u003e`__]\n* Variance reduction in SGD by distributed importance sampling [`pdf \u003chttps://arxiv.org/pdf/1511.06481\u003e`__]\n\nDependencies \u0026 Installation\n---------------------------\n\nNormally if you already have a functional Keras installation you just need to\n``pip install keras-importance-sampling``.\n\n* ``Keras`` \u003e 2\n* A Keras backend among *Tensorflow*, *Theano* and *CNTK*\n* ``blinker``\n* ``numpy``\n* ``matplotlib``, ``seaborn``, ``scikit-learn`` are optional (used by the plot\n  scripts)\n\nDocumentation\n-------------\n\nThe module has a dedicated `documentation site\n\u003chttp://idiap.ch/~katharas/importance-sampling/\u003e`__ but you can also read the\n`source code \u003chttps://github.com/idiap/importance-sampling\u003e`__ and the `examples\n\u003chttps://github.com/idiap/importance-sampling/tree/master/examples\u003e`__ to get an\nidea of how the library should be used and extended.\n\nExamples\n---------\n\nIn the ``examples`` folder you can find some Keras examples that have been edited\nto use importance sampling.\n\nCode examples\n*************\n\nIn this section we will showcase part of the API that can be used to train\nneural networks with importance sampling.\n\n.. code:: python\n\n    # Import what is needed to build the Keras model\n    from keras import backend as K\n    from keras.layers import Dense, Activation, Flatten\n    from keras.models import Sequential\n\n    # Import a toy dataset and the importance training\n    from importance_sampling.datasets import MNIST\n    from importance_sampling.training import ImportanceTraining\n\n\n    def create_nn():\n        \"\"\"Build a simple fully connected NN\"\"\"\n        model = Sequential([\n            Flatten(input_shape=(28, 28, 1)),\n            Dense(40, activation=\"tanh\"),\n            Dense(40, activation=\"tanh\"),\n            Dense(10),\n            Activation(\"softmax\") # Needs to be separate to automatically\n                                  # get the preactivation outputs\n        ])\n\n        model.compile(\n            optimizer=\"adam\",\n            loss=\"categorical_crossentropy\",\n            metrics=[\"accuracy\"]\n        )\n\n        return model\n\n\n    if __name__ == \"__main__\":\n        # Load the data\n        dataset = MNIST()\n        x_train, y_train = dataset.train_data[:]\n        x_test, y_test = dataset.test_data[:]\n\n        # Create the NN and keep the initial weights\n        model = create_nn()\n        weights = model.get_weights()\n\n        # Train with uniform sampling\n        K.set_value(model.optimizer.lr, 0.01)\n        model.fit(\n            x_train, y_train,\n            batch_size=64, epochs=10,\n            validation_data=(x_test, y_test)\n        )\n\n        # Train with importance sampling\n        model.set_weights(weights)\n        K.set_value(model.optimizer.lr, 0.01)\n        ImportanceTraining(model).fit(\n            x_train, y_train,\n            batch_size=64, epochs=2,\n            validation_data=(x_test, y_test)\n        )\n\nUsing the script\n****************\n\nThe following terminal commands train a small VGG-like network to ~0.65% error\non MNIST (the numbers are from a CPU).\n.. code::\n\n    $ # Train a small cnn with mnist for 500 mini-batches using importance\n    $ # sampling with bias to achieve ~ 0.65% error (on the CPU).\n    $ time ./importance_sampling.py \\\n    \u003e   small_cnn \\\n    \u003e   oracle-gnorm \\\n    \u003e   model \\\n    \u003e   predicted \\\n    \u003e   mnist \\\n    \u003e   /tmp/is \\\n    \u003e   --hyperparams 'batch_size=i128;lr=f0.003;lr_reductions=I10000' \\\n    \u003e   --train_for 500 --validate_every 500\n    real    1m41.985s\n    user    8m14.400s\n    sys     0m35.900s\n    $\n    $ # And with uniform sampling to achieve ~ 0.9% error.\n    $ time ./importance_sampling.py \\\n    \u003e   small_cnn \\\n    \u003e   oracle-loss \\\n    \u003e   uniform \\\n    \u003e   unweighted \\\n    \u003e   mnist \\\n    \u003e   /tmp/uniform \\\n    \u003e   --hyperparams 'batch_size=i128;lr=f0.003;lr_reductions=I10000' \\\n    \u003e   --train_for 3000 --validate_every 3000\n    real    9m23.971s\n    user    47m32.600s\n    sys     3m4.188s\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fidiap%2Fimportance-sampling","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fidiap%2Fimportance-sampling","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fidiap%2Fimportance-sampling/lists"}