{"id":38538292,"url":"https://github.com/aaltoml/periodicbnn","last_synced_at":"2026-01-17T07:05:04.223Z","repository":{"id":51491350,"uuid":"420075168","full_name":"AaltoML/PeriodicBNN","owner":"AaltoML","description":"Code for 'Periodic Activation Functions Induce Stationarity' (NeurIPS 2021)","archived":false,"fork":false,"pushed_at":"2021-10-27T14:50:24.000Z","size":5101,"stargazers_count":19,"open_issues_count":0,"forks_count":2,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-09-08T22:19:01.511Z","etag":null,"topics":["activation-functions","deep-learning","machine-learning","neural-network"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/AaltoML.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2021-10-22T11:36:06.000Z","updated_at":"2025-05-12T07:29:58.000Z","dependencies_parsed_at":"2022-08-22T00:21:24.509Z","dependency_job_id":null,"html_url":"https://github.com/AaltoML/PeriodicBNN","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/AaltoML/PeriodicBNN","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AaltoML%2FPeriodicBNN","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AaltoML%2FPeriodicBNN/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AaltoML%2FPeriodicBNN/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AaltoML%2FPeriodicBNN/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/AaltoML","download_url":"https://codeload.github.com/AaltoML/PeriodicBNN/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AaltoML%2FPeriodicBNN/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28503022,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-17T06:57:29.758Z","status":"ssl_error","status_checked_at":"2026-01-17T06:56:03.931Z","response_time":85,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["activation-functions","deep-learning","machine-learning","neural-network"],"created_at":"2026-01-17T07:05:03.790Z","updated_at":"2026-01-17T07:05:04.195Z","avatar_url":"https://github.com/AaltoML.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Periodic Activation Functions Induce Stationarity\n\nThis repository is the official implementation of the methods in the publication:\n\n* L. Meronen, M. Trapp, and A. Solin (2021). **Periodic Activation Functions Induce Stationarity**. *To appear at Advances in Neural Information Processing Systems (NeurIPS)*. [[arXiv]](https://arxiv.org/abs/2110.13572)\n\n\u003cimg src=\"src/fig/Illustration.png\" width=\"400\"\u003e\n\nThe paper's main result shows that periodic activation functions in Bayesian neural networks establish a direct connection between the prior on the network weights and the spectral density of the induced stationary (translation-invariant) Gaussian process prior. Moreover, this link goes beyond sinusoidal (Fourier) activations and also covers periodic functions such as the triangular wave and a novel periodic ReLU activation function. Thus, periodic activation functions induce conservative behaviour into Bayesian neural networks and allow principled prior specification.\n\nThe figure below illustates the different periodic activation discussed in our work.\n![activation functions](src/fig/activations.png)\n\nThe following Jupyter [notebook](notebook/PeriodicActivationFunctions.ipynb) illustrates the approach on a 1D toy regression data set.\n\n## Supplemental material\nStructure of the supplemental material folder:\n\n* `data` contains UCI and toy data sets\n* `notebook` contains a Jupyter notebook in Julia illustrating the proposed approach\n* `python_codes` contains Python codes implementing the approach in the paper using KFAC Laplace approximation and SWAG as approximate inference methods\n* `julia_codes` contains Julia codes implementing the proposed approach using dynamic HMC as approximate inference method\n\n## Python code requirements and usage instructions\n\nInstalling dependencies (recommended Python version 3.7.3 and pip version 20.1.1):\n```bash\npip install -r requirements.txt\n```\n\nAlternatively, using a conda environment:\n```bash\nconda create -n periodicBNN python=3.7.3 pip=20.1.1\nconda activate periodicBNN\npip install -r requirements.txt\n```\n\n### Pretrained CIFAR-10 model\n\nIf you wish to run the OOD detection experiment on CIFAR-10, CIFAR-100 and SVHN images, the pretrained GoogLeNet model that we used can be obtained from: https://github.com/huyvnphan/PyTorch_CIFAR10. The model file should be placed in path `./state_dicts/updated_googlenet.pt`\n\n### Running experiments\n\nTo running all Python experiments, first navigate to the following folder `python_codes/` inside the supplement folder on the terminal.\n\n#### Running UCI experiments:\n\nTrain and test the model:\n```bash\npython traintest_KFAC_uci.py 0 boston\n```\n\nwhere the first command line argument is the model setup index and the second one is the data set name. See the setups that different indexes use from the list below. To start multiple jobs for different setups running in parallel, you can create a shell script or use slurm. An example of such a script is shown here:\n```bash\n#!/bin/bash\nfor i in {0..3}\ndo\n  python traintest_KFAC_uci.py $i 'boston' \u0026\ndone\n```\n\nAfter calculating results for the models, you can create a LaTeX table of the results using the script `make_ucireg_tables.py` for regression results and using `make_uci_tables.py` for classification results. An example command of both of these python scripts are shown below:\n```bash\npython make_ucireg_tables.py full \u003e ./table_name.tex\npython make_uci_tables.py full NLPD_ACC \u003e ./table_name.tex\n```\n\nThe first argument is either `full` or `short` and determines whether the generated table contains entries for all possible models or only for a subset. The second argument in the classification script determines whether the script computes AUC numbers (use `AUC` as the argument) or both NLPD and accuracy numbers (use `NLPD_ACC` as the argument). The last argument defines the output path for saving the table.\n\n#### Running the MNIST experiment:\n\nTrain the model:\n```bash\npython train_KFAC_mnist.py 0\n```\nwhere the first command line argument is the model setup index. See the setups that different indexes use from the list below.\n\nTest the model:\n```bash\npython test_KFAC_mnist.py 0 standard\npython test_KFAC_mnist.py 0 rotated 0\n```\n\nwhere the first command line argument is the model setup index. See the setups that different indexes use from the end of this file. The second command line argument (`standard` or `rotated`) selects the type of MNIST test set. If the second command line argument is `rotated`, then the third command line argument is needed to select the test rotation angle (0 to 35 corresponding to rotation angles 10 to 360). Here you can again utilize a shell script or use slurm for example to run different rotation angles in parallel:\n```bash\n#!/bin/bash\nfor i in {0..35}\ndo\n  python test_KFAC_mnist.py 0 rotated $i \u0026\ndone\n```\n\nAfter calculating some results, you can use `visualize_MNIST_metrics.py` for plotting the results. The usage for this file is as follows:\n```bash\npython visualize_MNIST_metrics.py\n```\nOn line 22 of this file (`setup_ind_list = [0,1,2,10]`) you can define which setups are included into the plot. See the setups that different indexes use from the list below.\n\n#### Running the CIFAR-10 OOD detection experiment:\n\nTrain the model:\n```bash\npython train_SWAG_cifar.py 0\n```\nwhere the first command line argument is the model setup index. See the setups that different indexes use from the list below.\n\nTest the model:\n```bash\npython test_SWAG_cifar.py 0 CIFAR10_100\n```\nwhere the first command line argument is the model setup index. See the setups that different indexes use from the end of this file. The second command line argument is the OOD data set to test on, ether `CIFAR10_100` or `CIFAR_SVHN`.\n\nAfter calculating some results, you can use `visualize_CIFAR_uncertainty.py` for plotting the results, and `calculate_CIFAR_AUC_AUPR.py` for calculating AUC and AUPR numbers. The usage for these files is as follows:\n```bash\npython visualize_CIFAR_uncertainty.py 0\npython calculate_CIFAR_AUC_AUPR.py 0\n```\nwhere the first command line argument is the model setup index. See the setups that different indexes use from the list below.\n\n#### Model setups corresponding to different model setup indexes\n```\n0: ReLU\n1: local stationary RBF\n2: global stationary RBF (sinusoidal)\n3: global stationary RBF (triangle)\n4: local stationary matern52\n5: global stationary matern52 (sinusoidal)\n6: global stationary matern52 (triangle)\n7: local stationary matern32\n8: global stationary matern32 (sinusoidal)\n9: global stationary matern32 (triangle)\n10: global stationary RBF (sincos)\n11: global stationary matern52 (sincos)\n12: global stationary matern32 (sincos)\n13: global stationary RBF (prelu)\n14: global stationary matern52 (prelu)\n15: global stationary matern32 (prelu)\n```\n\n#### Creating your own task specific model using our implementation of periodic activation functions\nIf you wish to make your own model using a specific feature extractor network of your choice, you need to add it into the file `python_codes/model.py`. New models can be added at the bottom of the file among the already implemented ones, such as:\n```python\nclass my_model:\n    base = MLP\n    args = list()\n    kwargs = dict()\n    kwargs['K'] = 1000\n    kwargs['pipeline'] = MY_OWN_PIPELINE\n```\n\nHere you can name your new model and choose some keyword arguments to be used. `kwargs['pipeline']` determines which feature extractor your model is using, and it is a mandatory keyword argument. You can create your own feature extractor. As an example here we show the feature extractor for the MNIST model:\n```python\nclass MNIST_PIPELINE(nn.Module):\n\n    def __init__(self, D = 5, dropout = 0.25):\n        super(MNIST_PIPELINE, self).__init__()\n\n        self.O = 25\n        self.conv1 = nn.Conv2d(1, 32, 3, 1)\n        self.conv2 = nn.Conv2d(32, 64, 3, 1)\n        self.dropout = nn.Dropout(dropout)\n        self.linear = nn.Linear(9216, self.O)        \n\n    def forward(self, x):\n\n        x = self.conv1(x)\n        x = F.relu(x)\n        x = self.conv2(x)\n        x = F.relu(x)\n        x = F.max_pool2d(x, 2)\n        x = self.dropout(x)\n        x = torch.flatten(x, 1)\n        \n        #Additional bottleneck\n        x = self.linear(x)\n        x = F.relu(x)\n        \n        return x\n```\n\n#### Using our model for different data sets\nIf you wish to use our model for some other data set, you need to add the data set into the file `python_codes/dataset_maker.py`. There you need to configure your data set under the `load_dataset(name, datapath, seed):` function as an alternative `elif:` option. The implementation of the data set must specify the following variables: `train_set, test_set, num_classes, D`. After adding the data set here, you can use it through the model training and evaluation scripts.\n\n## Julia code requirements and usage instructions\nMake sure you have Julia installed on your system. If you do not have Julia, download it from https://julialang.org/downloads/.\n\nTo install the necessary dependencies for the Julia codes, run the following commands on the command line from the respective julia codes folder: \n\n```bash\njulia --project=. -e \"using Pkg; Pkg.instantiate();\"\n```\n\n### Running the experiment on the banana data set\n\nRun the following commands on the command line: \n```bash\njulia --project=. banana.jl [--nsamples NSAMPLES] [--nadapts NADAPTS] [--K K]\n                 [--kernel KERNEL] [--seed SEED] [--nu NU] [--ell ELL]\n                 [--ad AD] [--activation ACTIVATION] [--hideprogress]\n                 [--subsample SUBSAMPLE]\n                 [--subsampleseed SUBSAMPLESEED] [datapath] [outputpath]\n```\n\nExample to obtain 1000 samples using dynamic HMC for an BNN with 10 hidden units and priors equivalent to an RBF kernel: \n```bash\njulia --project=. banana.jl --nsamples 1000 --K 10 --kernel RBF --ad reverse ../data ./\n```\n\nAfter a short while, you will see a progress bar showing the sampling progress and an output showing the setup of the run. For example:\n```\n(K, n_samples, n_adapts, kernelstr, ad, seed, datapath, outputpath) = (10, 1000, 1000, \"RBF_SinActivation\", gradient_logjoint, 2021, \"../data\", \"./\")\n```\nDepending on the configuration, the sampling might result in divergencies of dynamic HMC shown as warnings, those samples will be discarded automatically.\nOnce the sampling is finished, you will see statistics on the sampling alongside with the `UID` and the `kernel string`. Both are used to identify the results for plotting.\n\nTo visualise the results, use the `banana_plot.jl` script, i.e.,\n```bash\njulia --project=. banana_plot.jl [datapath] [resultspath] [uid] [kernelstring]\n```\n\nFor example, to visualise the results calculated above (replace `8309399884939560691` with the uid shown in your run!), use:\n```bash\njulia --project=. banana_plot.jl ../data ./ 8309399884939560691 RBF_SinActivation\n```\n\nThe resulting visualisation will automatically be saved as a pdf in the current folder!\n\n### Notebook\nThe notebook can be run locally using:\n```bash\njulia --project -e 'using Pkg; Pkg.instantiate(); using IJulia; notebook(dir=pwd())'\n```\n\n\n## Citation\nIf you use the code in this repository for your research, please cite the paper as follows:\n```bibtex\n@inproceedings{meronen2021,\n  title={Periodic Activation Functions Induce Stationarity},\n  author={Meronen, Lassi and Trapp, Martin and Solin, Arno},\n  booktitle = {Advances in Neural Information Processing Systems (NeurIPS)},\n  year={2021}\n}\n```\n\n## Contributing\n\nFor all correspondence, please contact lassi.meronen@aalto.fi.\n\n## License\n\nThis software is provided under the [MIT license](LICENSE).\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faaltoml%2Fperiodicbnn","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Faaltoml%2Fperiodicbnn","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faaltoml%2Fperiodicbnn/lists"}