{"id":23608674,"url":"https://github.com/hycis/pynet","last_synced_at":"2026-02-25T18:05:17.851Z","repository":{"id":70212422,"uuid":"20822324","full_name":"hycis/Pynet","owner":"hycis","description":"pynet is meant to be a flexible and modular deep learning framework base on theano.","archived":false,"fork":false,"pushed_at":"2015-08-04T16:21:42.000Z","size":852,"stargazers_count":8,"open_issues_count":1,"forks_count":2,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-05-07T12:35:23.765Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/hycis.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2014-06-14T01:20:44.000Z","updated_at":"2020-02-13T20:29:11.000Z","dependencies_parsed_at":"2023-03-01T08:30:20.855Z","dependency_job_id":null,"html_url":"https://github.com/hycis/Pynet","commit_stats":null,"previous_names":[],"tags_count":2,"template":false,"template_full_name":null,"purl":"pkg:github/hycis/Pynet","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hycis%2FPynet","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hycis%2FPynet/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hycis%2FPynet/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hycis%2FPynet/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/hycis","download_url":"https://codeload.github.com/hycis/Pynet/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hycis%2FPynet/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29833763,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-25T17:57:15.019Z","status":"ssl_error","status_checked_at":"2026-02-25T17:56:11.472Z","response_time":61,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-12-27T14:39:33.833Z","updated_at":"2026-02-25T18:05:17.834Z","avatar_url":"https://github.com/hycis.png","language":"Python","readme":"Pynet\n=====\n\nPynet is used to train an autoencoder to extract low dimensional features for speech synthesis.\nThe pynet is used to implement the results from the paper  \n[Deep Denoising Auto-encoder for Statistical Speech Synthesis](http://arxiv.org/abs/1506.05268)\n\n\n__1. Setting Environment Variables__\n\nIn pynet, there are three environment variables to be set.\n\n```python\nPYNET_DATA_PATH   # the directory for all the datasets\nPYNET_SAVE_PATH   # the directory to save the best models, the outputs logs and the hyperparameters\nPYNET_DATABASE_PATH # after training, the hyperparameters and training results from various\n                      # experiments is saved into a database for comparisions\n```\n\n__2. Model Script__\n\nIn order to build and run an AutoEncoder, we need to put together the various components\n(model, layer, dataset, learning_rule, log, cost function) into a train_object and run the\ntraining. The example model below is saved to the script [AE_example.py](../example/AE_example.py).\n\n```python\nimport theano\nimport theano.tensor as T\nimport numpy as np\n\nfrom pynet.model AutoEncoder\nfrom pynet.layer import RELU, Sigmoid, Softmax, Linear\nfrom pynet.datasets.spec import *\nfrom pynet.learning_rule import LearningRule\nfrom pynet.log import Log\nfrom pynet.train_object import TrainObject\nfrom pynet.cost import Cost\nfrom pynet.datasets.preprocessor import Standardize, GCN\n\ndef autoencoder():\n\n    # set environment\n    NNdir = os.path.dirname(os.path.realpath(__file__))\n    NNdir = os.path.dirname(NNdir)\n    NNdir = os.path.dirname(NNdir)\n\n    if not os.getenv('PYNET_DATA_PATH'):\n        os.environ['PYNET_DATA_PATH'] = NNdir + '/data'\n\n    if not os.getenv('PYNET_DATABASE_PATH'):\n        os.environ['PYNET_DATABASE_PATH'] = NNdir + '/database'\n        if not os.path.exists(os.environ['PYNET_DATABASE_PATH']):\n            os.mkdir(os.environ['PYNET_DATABASE_PATH'])\n\n    if not os.getenv('PYNET_SAVE_PATH'):\n        os.environ['PYNET_SAVE_PATH'] = NNdir + '/save'\n        if not os.path.exists(os.environ['PYNET_SAVE_PATH']):\n            os.mkdir(os.environ['PYNET_SAVE_PATH'])\n\n\n    # logging is optional, it is used to save the best trained model and records the training result to a database\n    log = Log(experiment_name = 'AE',\n            description = 'This experiment is about autoencoder',\n            save_outputs = True, # saves to outputs.log\n            save_learning_rule = True,\n            save_model = True,\n            save_to_database = {'name': 'Example.db',\n                                'records' : {'Dataset' : data.__class__.__name__,\n                                             'Weight_Init_Seed' : mlp.rand_seed,\n                                             'Dropout_Below' : str([layer.dropout_below for layer in mlp.layers]),\n                                             'Batch_Size' : data.batch_size,\n                                             'Layer_Size' : len(mlp.layers),\n                                             'Layer_Dim' : str([layer.dim for layer in mlp.layers]),\n                                             'Preprocessor' : data.preprocessor.__class__.__name__,\n                                             'Learning_Rate' : learning_rule.learning_rate,\n                                             'Momentum' : learning_rule.momentum}}\n            ) # end log\n\n\n    learning_rule = LearningRule(max_col_norm = 1, # max length of the weight vector from lower layer going into upper neuron\n                                learning_rate = 0.01,\n                                momentum = 0.1,\n                                momentum_type = 'normal',\n                                L1_lambda = None, # L1 regularization coefficient\n                                L2_lambda = None, # L2 regularization coefficient\n                                cost = Cost(type='mse'), # cost type use for backprop during training\n                                stopping_criteria = {'max_epoch' : 100, # maximum number of epochs for the training\n                                                    'cost' : Cost(type='mse'), # cost type use for testing the quality of the trained model\n                                                    'epoch_look_back' : 10, # number of epoch to look back for error improvement\n                                                    'percent_decrease' : 0.001} # requires at least 0.001 = 0.1% decrease in error when look back of 10 epochs\n                                )\n\n\n    # building dataset, batch_size and preprocessor\n    data = Laura_Blocks(train_valid_test_ratio=[8,1,1], batch_size=100, preprocessor=GCN())\n\n    # for AutoEncoder, the inputs and outputs must be the same\n    train = data.get_train()\n    data.set_train(train.X, train.X)\n\n    valid = data.get_valid()\n    data.set_valid(valid.X, valid.X)\n\n    test = data.get_test()\n    data.set_test(test.X, test.X)\n\n    # building autoencoder\n    ae = AutoEncoder(input_dim = data.feature_size(), rand_seed=123)\n    h1_layer = Tanh(dim=500, name='h1_layer', W=None, b=None)\n\n    # adding encoding layer\n    ae.add_encode_layer(h1_layer)\n\n    # mirror layer has W = h1_layer.W.T\n    h1_mirror = Tanh(name='h1_mirror', W=h1_layer.W.T, b=None)\n\n    # adding decoding mirror layer\n    ae.add_decode_layer(h1_mirror)\n\n    # put all the components into a TrainObject\n    train_object = TrainObject(model = ae,\n                                dataset = data,\n                                learning_rule = learning_rule,\n                                log = log)\n\n    # finally run the training\n    train_object.run()\n\n```\n\n__3. Hyperparams Search__\n\nIn order to do hyperparams search, run the script in [launch.py](../hps/launch.py) in [hps dir](../hps).\nTo do that, first log into helios\n\n```bash\ncd Pynet/hps\ncat model_config.py # this will show the configurations of different models\n```\n\nInside model_config.py, if the values is placed in a tuple for a variable,\nit means that during the sampling of values for a variable,\nthe value are sampled uniformly from the values in the tuple.\nFor example for\n```'learning_rate' : (1e-5, 1e-4, 1e-3, 1e-2, 1e-1, 0.5)```,\nlearning_rate is uniformly set as any of the 6 values in the tuple.\n\nBelow is the sample of model Laura from [model_config.py](../hps/model_config.py)  \n```python\n    'Laura' : DD({\n            'model' : DD({\n                    'rand_seed'             : None\n                    }), # end mlp\n\n            'log' : DD({\n                    'experiment_name'       : 'AE0918_Warp_Blocks_180_120_tanh_tanh_gpu_dropout', #helios\n                    # 'experiment_name'       : 'AE0914_Warp_Blocks_500_180_tanh_tanh_gpu_clean', #helios\n\n                    # 'experiment_name'       : 'AE0919_Blocks_180_120_tanh_tanh_gpu_dropout', #helios\n                    # 'experiment_name'       : 'AE0918_Blocks_180_120_tanh_tanh_gpu_clean', #helios\n\n                    # 'experiment_name'       : 'AE0916_Blocks_180_120_tanh_tanh_gpu_output_sig_dropout',\n                    # 'experiment_name'       : 'AE0916_Blocks_180_120_tanh_tanh_gpu_output_sig_clean',\n\n\n                    'description'           : '',\n                    'save_outputs'          : True,\n                    'save_learning_rule'      : True,\n                    'save_model'            : True,\n                    'save_to_database_name' : 'Laura.db'\n                    }), # end log\n\n\n            'learning_rule' : DD({\n                    'max_col_norm'          : (1, 10, 50),\n                    'learning_rate'         : (1e-5, 1e-4, 1e-3, 1e-2, 1e-1, 0.5),\n                    'momentum'              : (1e-3, 1e-2, 1e-1, 0.5, 0.9),\n                    'momentum_type'         : 'normal',\n                    'L1_lambda'             : None,\n                    'L2_lambda'             : None,\n                    'cost'                  : 'mse',\n                    'stopping_criteria'     : DD({\n                                                'max_epoch'         : 100,\n                                                'epoch_look_back'   : 10,\n                                                'cost'              : 'mse',\n                                                'percent_decrease'  : 0.05\n                                                }) # end stopping_criteria\n                    }), # end learning_rule\n\n            #===========================[ Dataset ]===========================#\n            'dataset' : DD({\n                    # 'type'                  : 'Laura_Warp_Blocks_500_Tanh',\n                    'type'                 : 'Laura_Warp_Blocks_180_Tanh_Dropout',\n                    # 'type'                  : 'Laura_Cut_Warp_Blocks_300',\n                    # 'type'                  : 'Laura_Blocks_180_Tanh_Tanh',\n                    # 'type'                  : 'Laura_Blocks_180_Tanh_Tanh_Dropout',\n                    # 'type'                  : 'Laura_Blocks_500_Tanh_Sigmoid',\n                    # 'type'                  : 'Laura_Blocks_500',\n                    # 'type'                  : 'Laura_Blocks',\n                    # 'type'                  : 'Laura_Warp_Blocks',\n                    # 'type'                  : 'Laura_Warp_Standardize_Blocks',\n                    # 'type'                  : 'Laura_Standardize_Blocks',\n                    # 'type'                  : 'Mnist',\n\n                    'feature_size'          : 180,\n                    'train_valid_test_ratio': [8, 1, 1],\n\n                    'preprocessor'          : None,\n                    # 'preprocessor'          : 'Scale',\n                    # 'preprocessor'          : 'GCN',\n                    # 'preprocessor'          : 'LogGCN',\n                    # 'preprocessor'          : 'Standardize',\n\n                    'batch_size'            : (50, 100, 150, 200),\n                    'num_batches'           : None,\n                    'iter_class'            : 'SequentialSubsetIterator',\n                    'rng'                   : None\n                    }), # end dataset\n\n            #============================[ Layers ]===========================#\n            'num_layers' : 1,\n\n            'hidden1' : DD({\n                    'name'                  : 'hidden1',\n                    'type'                  : 'Tanh',\n                    'dim'                   : 120,\n\n                    # 'dropout_below'         : None,\n                    'dropout_below'         : (0.1, 0.2, 0.3, 0.4, 0.5),\n                    # 'dropout_below'         : 0.5,\n\n                    }), # end hidden_layer\n\n            'hidden2' : DD({\n                    'name'                  : 'hidden2',\n                    'type'                  : 'RELU',\n                    'dim'                   : 100,\n                    'dropout_below'         : None,\n                    }), # end hidden_layer\n\n            'h2_mirror' : DD({\n                    'name'                  : 'h2_mirror',\n                    'type'                  : 'RELU',\n                    # 'dim'                   : 2049, # dim = input.dim\n                    'dropout_below'         : None,\n                    }), # end output_layer\n\n            'h1_mirror' : DD({\n                    'name'                  : 'h1_mirror',\n                    'type'                  : 'Tanh',\n                    # 'dim'                   : 2049, # dim = input.dim\n                    'dropout_below'         : None,\n                    }) # end output_layer\n\n            }), # end autoencoder\n```\n\n\nTo sample one set of hyperparams and run it locally, issue\n```bash\ncd Pynet/hps\npython launch.py --model Laura -c 1\n```\nTo submit 5 jobs to the gpu cluster, issue\n```bash\ncd Pynet/hps\npython launch.py --model Laura -n 5 -g\nshowq -u hycis\n```\n\nAfter finished running, you can checkout the results from the database\n```bash\ncdwu\nsqlite3 Pynet/database/Laura.db\n\u003e\u003e\u003e .header on\n\u003e\u003e\u003e .mode column\n\u003e\u003e\u003e .table\n\u003e\u003e\u003e select * from some_table order by test_error;\n```\n\nI have named the the experiment group in as way that is easier for understanding, for example\nfor an experiment group name of ```AE0912_Blocks_2049_500_tanh_tanh_gpu_clean```\nmeans AE0912 trained on Linear Blocks of autoencoder with 2049-500-2049 dims, and tanh-tanh units,\nit's run on gpu and it's a clean model without noise during training.\nThe best model for the experiment group is ```AE0912_Blocks_2049_500_tanh_tanh_gpu_clean_20140914_1242_27372903```\nwhere the last few numbers are the actual date_time_microsec in which the model is generated.\n\n\nI have saved the best results for each pretrain layer in the http://1drv.ms/1qSyrZI under the *combinations* section.\n\n__4. Reproduce Best Results__\n\nTo reproduce the results you can plug the hyperparams saved in the database into [AE_example.py](../example/AE_example.py)\nand run the job locally, or you can modify the [model_config.py](../hps/model_config.py) and\nset the hyperparams in the config file and run ```python launch.py --model Laura -c 1```\n\n*Stacking up Models*\n\nTo reproduce the stackup of trained model is very simple. Just put the name of the best\nmodel under 'hidden1' and 'hidden2' in the model_config.py and set the hyperparams, and issue\n```python launch.py --model Laura_Two_Layers -c 1``` to run the job locally.\n\n```python\n'Laura_Two_Layers' : DD({\n        'model' : DD({\n                'rand_seed'             : None\n                }), # end mlp\n\n        'log' : DD({\n                # 'experiment_name'       : 'AE0917_Blocks_2layers_finetune_2049_180_tanh_tanh_gpu_clean',\n                # 'experiment_name'       : 'AE0918_Blocks_2layers_finetune_2049_180_tanh_tanh_gpu_noisy',\n\n                # 'experiment_name'       : 'AE0918_Blocks_2layers_finetune_2049_180_tanh_sigmoid_gpu_clean',\n                # 'experiment_name'       : 'AE0917_Blocks_2layers_finetune_2049_180_tanh_sigmoid_gpu_noisy',\n\n                # 'experiment_name'       : 'AE0917_Warp_Blocks_2layers_finetune_2049_180_tanh_tanh_gpu_clean',\n                'experiment_name'       : 'AE0918_Warp_Blocks_2layers_finetune_2049_180_tanh_tanh_gpu_noisy',\n\n\n\n                'description'           : '',\n                'save_outputs'          : True,\n                'save_learning_rule'      : True,\n                'save_model'            : True,\n                'save_to_database_name' : 'Laura.db'\n                }), # end log\n\n\n        'learning_rule' : DD({\n                'max_col_norm'          : (1, 10, 50),\n                'learning_rate'         : (1e-5, 1e-4, 1e-3, 1e-2, 1e-1, 0.5),\n                # 'learning_rate'         : ((1e-5, 9e-1), float),\n                # 'learning_rate'         : 0.01,\n                'momentum'              : (1e-3, 1e-2, 1e-1, 0.5, 0.9),\n                # 'momentum'              : 0.05,\n                'momentum_type'         : 'normal',\n                'L1_lambda'             : None,\n                'L2_lambda'             : None,\n                'cost'                  : 'mse',\n                'stopping_criteria'     : DD({\n                                            'max_epoch'         : 100,\n                                            'epoch_look_back'   : 10,\n                                            'cost'              : 'mse',\n                                            'percent_decrease'  : 0.05\n                                            }) # end stopping_criteria\n                }), # end learning_rule\n\n        #===========================[ Dataset ]===========================#\n        'dataset' : DD({\n                # 'type'                  : 'Laura_Warp_Blocks_500',\n                # 'type'                  : 'Laura_Blocks_500',\n                # 'type'                  : 'Laura_Blocks',\n                'type'                  : 'Laura_Warp_Blocks',\n                # 'type'                  : 'Mnist_Blocks',\n                'feature_size'          : 2049,\n                'train_valid_test_ratio': [8, 1, 1],\n\n                # 'preprocessor'          : None,\n                # 'preprocessor'          : 'Scale',\n                'preprocessor'          : 'GCN',\n                # 'preprocessor'          : 'LogGCN',\n                # 'preprocessor'          : 'Standardize',\n\n                'batch_size'            : (50, 100, 150, 200),\n                'num_batches'           : None,\n                'iter_class'            : 'SequentialSubsetIterator',\n                'rng'                   : None\n                }), # end dataset\n\n        # #============================[ Layers ]===========================#\n\n        'hidden1' : DD({\n                'name'                  : 'hidden1',\n\n                # 'model'                 : 'AE0912_Blocks_2049_500_tanh_tanh_gpu_clean_20140914_1242_27372903',\n                # 'model'                 : 'AE0915_Blocks_2049_500_tanh_tanh_gpu_Dropout_20140915_1900_37160748',\n\n                # 'model'                 : 'AE0912_Blocks_2049_500_tanh_sigmoid_gpu_clean_20140913_1342_18300926',\n\n                # 'model'                 : 'AE0911_Warp_Blocks_2049_500_tanh_tanh_gpu_clean_20140912_2337_04263067',\n                'model'                 : 'AE0916_Warp_Blocks_2049_500_tanh_tanh_gpu_dropout_20140916_1705_29139505',\n\n\n                'dropout_below'         : None,\n                # 'dropout_below'         : 0.1,\n                }), # end hidden_layer\n\n        'hidden2' : DD({\n                'name'                  : 'hidden2',\n                # 'model'                 : 'AE0916_Blocks_500_180_tanh_tanh_gpu_clean_20140916_2255_06553688',\n                # 'model'                 : 'AE0914_Blocks_500_180_tanh_tanh_gpu_dropout_20140916_1059_59760060',\n                # 'model'                 : 'AE0918_Blocks_500_180_tanh_tanh_gpu_dropout_20140918_0920_42738052',\n\n                # 'model'                 : 'AE0916_Blocks_500_180_tanh_tanh_gpu_output_sig_clean_20140917_0301_44075773',\n\n                # 'model'                 : 'AE0914_Warp_Blocks_500_180_tanh_tanh_gpu_clean_20140915_0400_30113212',\n                # 'model'                 : 'AE0916_Warp_Blocks_500_180_tanh_tanh_gpu_dropout_20140916_1326_09742695',\n                'model'                 : 'AE0918_Warp_Blocks_500_180_tanh_tanh_gpu_dropout_20140918_1125_23612485',\n\n                'dropout_below'         : None,\n                }), # end hidden_layer\n\n\n        }), # end autoencoder\n```\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhycis%2Fpynet","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fhycis%2Fpynet","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhycis%2Fpynet/lists"}