{"id":17815629,"url":"https://github.com/tiskw/random-fourier-features","last_synced_at":"2025-04-04T22:02:04.455Z","repository":{"id":41802327,"uuid":"151746223","full_name":"tiskw/random-fourier-features","owner":"tiskw","description":"Implementation of random Fourier features for kernel method, like support vector machine and Gaussian process model","archived":false,"fork":false,"pushed_at":"2024-10-29T10:33:06.000Z","size":4404,"stargazers_count":100,"open_issues_count":0,"forks_count":22,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-03-28T21:01:38.537Z","etag":null,"topics":["gaussian-processes","machine-learning","principal-component-analysis","python","pytorch","support-vector-machines"],"latest_commit_sha":null,"homepage":"https://tiskw.github.io/random-fourier-features/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/tiskw.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-10-05T16:05:12.000Z","updated_at":"2025-03-24T19:04:25.000Z","dependencies_parsed_at":"2023-11-23T01:45:08.296Z","dependency_job_id":"b1ee51ff-6b84-4101-aff7-4ec6721c1234","html_url":"https://github.com/tiskw/random-fourier-features","commit_stats":null,"previous_names":[],"tags_count":3,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tiskw%2Frandom-fourier-features","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tiskw%2Frandom-fourier-features/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tiskw%2Frandom-fourier-features/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tiskw%2Frandom-fourier-features/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/tiskw","download_url":"https://codeload.github.com/tiskw/random-fourier-features/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247256102,"owners_count":20909240,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["gaussian-processes","machine-learning","principal-component-analysis","python","pytorch","support-vector-machines"],"created_at":"2024-10-27T16:17:21.175Z","updated_at":"2025-04-04T22:02:04.428Z","avatar_url":"https://github.com/tiskw.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cdiv align=\"center\"\u003e\n  \u003cimg src=\"./figures/logo_long.svg\" width=\"480\" alt=\"rfflearn logo\" /\u003e\n\u003c/div\u003e\n\n\u003cdiv align=\"center\"\u003e\n  \u003cimg src=\"https://img.shields.io/badge/PYTHON-%3E%3D3.9-blue.svg?logo=python\u0026logoColor=white\"\u003e\n  \u0026nbsp;\n  \u003cimg src=\"https://img.shields.io/badge/LICENSE-MIT-orange.svg\"\u003e\n  \u0026nbsp;\n  \u003cimg src=\"https://img.shields.io/badge/COVERAGE-97%25-green.svg\"\u003e\n  \u0026nbsp;\n  \u003cimg src=\"https://img.shields.io/badge/QUALITY-9.96/10.0-yellow.svg\"\u003e\n\u003c/div\u003e\n\n\nRandom Fourier Features\n====================================================================================================\n\nThis repository provides the Python module `rfflearn` which is a Python library of random Fourier\nfeatures (hereinafter abbreviated as RFF) [1, 2] for kernel methods, like support vector\nmachine [3, 4] and Gaussian process model [5]. Features of this module are:\n\n- **User-friendly interfaces**: Interfaces of the `rfflearn` module are quite close to\n  the [scikit-learn](https://scikit-learn.org/) library,\n- **Example code first**: This repository provides plenty of [example code](./examples/) to\n  demonstrate that RFF is useful for actual machine learning tasks,\n- **GPU support**: Some classes in the `rfflearn` module provides both GPU training and inference\n  for faster computation,\n- **Wrapper to the other library**: Interface to [Optuna](https://optuna.org/) and\n  [SHAP](https://github.com/slundberg/shap) are provided for easier hyperparameter tuning and\n  feature importance analysis.\n\nNow, this module supports the following methods:\n\n| Method                          | CPU support                  | GPU support           |\n| ------------------------------- | ---------------------------- | --------------------- |\n| canonical correlation analysis  | `rfflearn.cpu.RFFCCA`        | -                     |\n| Gaussian process regression     | `rfflearn.cpu.RFFGPR`        | `rfflearn.gpu.RFFGPR` |\n| Gaussian process classification | `rfflearn.cpu.RFFGPC`        | `rfflearn.gpu.RFFGPC` |\n| principal component analysis    | `rfflearn.cpu.RFFPCA`        | `rfflearn.gpu.RFFPCA` |\n| regression                      | `rfflearn.cpu.RFFRegression` | -                     |\n| support vector classification   | `rfflearn.cpu.RFFSVC`        | `rfflearn.gpu.RFFSVC` |\n| support vector regression       | `rfflearn.cpu.RFFSVR`        | -                     |\n\nRFF can be applicable to many other machine learning algorithms than the above.\nThe author will provide implementations of the other algorithms soon.\n\n**NOTE**: The author confirmed that the `rfflearn` module works on PyTorch 2.0! :tada:\n\n\nInstallation\n----------------------------------------------------------------------------------------------------\n\nPlease install from PyPI.\n\n```shell\npip install rfflearn\n```\n\nIf you need GPU support, Optuna support or SHAP support,\nyou need to install the following packages.\n\n```shell\n# For GPU support.\npip install torch\n\n# For Optuna support.\npip install optuna\n\n# For SHAP support.\npip install matplotlib shap\n```\n\nThe author recommends using [venv](https://docs.python.org/3/library/venv.html) or\n[Docker](https://www.docker.com/) to avoid polluting your environment.\n\n\nMinimal example\n----------------------------------------------------------------------------------------------------\n\nInterfaces provided by the module `rfflearn` are quite close to scikit-learn.\nFor example, the following Python code is a sample usage of `RFFSVC`\n(support vector machine with random Fourier features) class.\n\n```python\n\u003e\u003e\u003e import numpy as np\n\u003e\u003e\u003e import rfflearn.cpu as rfflearn                     # Import module\n\u003e\u003e\u003e X = np.array([[-1, -1], [-2, -1], [1, 1], [2, 1]])  # Define input data\n\u003e\u003e\u003e y = np.array([1, 1, 2, 2])                          # Defile label data\n\u003e\u003e\u003e svc = rfflearn.RFFSVC().fit(X, y)                   # Training (on CPU)\n\u003e\u003e\u003e svc.score(X, y)                                     # Inference (on CPU)\n1.0\n\u003e\u003e\u003e svc.predict(np.array([[-0.8, -1]]))\narray([1])\n```\n\nThis module supports training/inference on GPU.\nFor example, the following Python code is a sample usage of `RFFGPC`\n(Gaussian process classifier with random Fourier features) on GPU.\nThe following code requires PyTorch (\u003e= 1.7.0).\n\n```python\n\u003e\u003e\u003e import numpy as np\n\u003e\u003e\u003e import rfflearn.gpu as rfflearn                     # Import module\n\u003e\u003e\u003e X = np.array([[-1, -1], [-2, -1], [1, 1], [2, 1]])  # Define input data\n\u003e\u003e\u003e y = np.array([1, 1, 2, 2])                          # Defile label data\n\u003e\u003e\u003e gpc = rfflearn.RFFGPC().fit(X, y)                   # Training on GPU\n\u003e\u003e\u003e gpc.score(X, y)                                     # Inference on GPU\n1.0\n\u003e\u003e\u003e gpc.predict(np.array([[-0.8, -1]]))\narray([1])\n```\n\nSee the [examples](./examples/) directory for more detailed examples.\n\n\nExample1: MNIST using random Fourier features\n----------------------------------------------------------------------------------------------------\n\nThe author tried SVC (support vector classifier) and GPC (Gaussian process classifier) with RFF to\nthe MNIST dataset which is one of the famous benchmark datasets on the image classification task,\nand the author has got better performance and much faster inference speed than kernel SVM.\nThe following table gives a brief comparison of kernel SVM, SVM with RFF, and GPC with RFF. \nSee the example of [RFF SVC module](./examples/svc_for_mnist/) and\n[RFF GP module](./examples/gpc_for_mnist/) for more details.\n\n| Method         | RFF dimension | Inference time [us/image] | Score [%] |\n|:--------------:|:-------------:|:-------------------------:|:---------:|\n| Kernel SVM     |      -        | 1312.6 us                 | 96.30 %   |\n| RFF SVC        |    640        |   33.6 us                 | 96.39 %   |\n| RFF SVC (GPU)  |    640        |   1.11 us                 | 96.39 %   |\n| RFF SVC        |  4,096        |  183.4 us                 | 98.14 %   |\n| RFF SVC (GPU)  |  4,096        |   2.62 us                 | 98.14 %   |\n| RFF GPC        | 20,000        |  517.9 us                 | 98.38 %   |\n| RFF GPC (GPU)  | 20,000        |  61.52 us                 | 98.38 %   |\n\n\u003cdiv align=\"center\"\u003e\n  \u003cimg src=\"./figures/Inference_time_and_acc_on_MNIST.svg\" width=\"640\" alt=\"Accuracy for each epochs in RFF SVC/GPC\" /\u003e\n\u003c/div\u003e\n\n\nExample2: Visualization of feature importance\n----------------------------------------------------------------------------------------------------\n\nThis module also has interfaces to some feature importance methods, like SHAP [6] and permutation\nimportance [7]. The author tried SHAP and permutation importance to `RFFGPR` trained on\nthe California housing dataset, and the followings are the visualization results obtained by\n`rfflearn.shap_feature_importance` and `rfflearn.permutation_feature_importance`.\n\n\u003cdiv align=\"center\"\u003e\n  \u003cimg src=\"./examples/feature_importances_for_california_housing/figures/figure_california_housing_shap_importance.png\" width=\"450\" alt=\"Permutation importances of Boston housing dataset\" /\u003e\n  \u003cimg src=\"./examples/feature_importances_for_california_housing/figures/figure_california_housing_permutation_importance.png\" width=\"450\" alt=\"SHAP importances of Boston housing dataset\" /\u003e\n\u003c/div\u003e\n\n\nQuick Tutorial\n----------------------------------------------------------------------------------------------------\n\nAt first, please clone the `random-fourier-features` repository from GitHub:\n\n```console\ngit clone https://github.com/tiskw/random-fourier-features.git\ncd random-fourier-features\n```\n\nIf you are using the docker image, enter into the docker container by the following command\n(not necessary to run thw following if you don't use docker):\n\n```console\ndocker run --rm -it --gpus all -v `pwd`:/work -w /work -u `id -u`:`id -g` tiskw/pytorch:latest bash\n```\n\nThen, launch python3 and try the following minimal code that runs support vector classification\nwith random Fourier features on an artificial tiny dataset.\n\n```python\n\u003e\u003e\u003e import numpy as np                                  # Import Numpy\n\u003e\u003e\u003e import rfflearn.cpu as rfflearn                     # Import our module\n\u003e\u003e\u003e X = np.array([[-1, -1], [-2, -1], [1, 1], [2, 1]])  # Define input data\n\u003e\u003e\u003e y = np.array([1, 1, 2, 2])                          # Defile label data\n\u003e\u003e\u003e svc = rfflearn.RFFSVC().fit(X, y)                   # Training\n\u003e\u003e\u003e svc.score(X, y)                                     # Inference (on CPU)\n1.0\n\u003e\u003e\u003e svc.predict(np.array([[-0.8, -1]]))\narray([1])\n```\n\n### Next Step\n\nNow you succeeded in installing the `rfflearn` module.\nThe author's recommendation for the next step is to see the [examples directory](/examples)\nand try a code you are interested in.\n\n\nNotes\n----------------------------------------------------------------------------------------------------\n\n- The name of this module is changed from `pyrff` to `rfflearn` on Oct 2020, because the package\n  name `pyrff` already exists in PyPI.\n- If the number of training data is huge, an error message like `RuntimeError: The task could not be\n  sent to the workers as it is too large for 'send_bytes'` will be raised from the joblib library.\n  The reason for this error is that the `sklearn.svm.LinearSVC` uses `joblib` as a multiprocessing\n  backend, but joblib cannot deal huge size of the array which cannot be managed with 32-bit\n  address space. In this case, please try `n_jobs = 1` option for the `RFFSVC` or `ORFSVC` function.\n  Default settings are `n_jobs = -1` which means automatically detecting available CPUs and using\n  them. (This bug information was reported by Mr. Katsuya Terahata @ Toyota Research Institute\n  Advanced Development. Thank you so much for the reporting!)\n- Application of RFF to the Gaussian process model is not straightforward.\n  See [this document](./articles/rff_for_gaussian_process.pdf) for mathematical details.\n\n\nLicense\n----------------------------------------------------------------------------------------------------\n\nThis software is distributed under the [MIT Licence](https://opensource.org/licenses/mit-license.php).\nSee [LICENSE](./LICENSE) for more information.\n\n\nReference\n----------------------------------------------------------------------------------------------------\n\n[1] A. Rahimi and B. Recht, \"Random Features for Large-Scale Kernel Machines\", NIPS, 2007.\n[PDF](https://papers.nips.cc/paper/3182-random-features-for-large-scale-kernel-machines.pdf)\n\n[2] F. X. Yu, A. T. Suresh, K. Choromanski, D. Holtmann-Rice and S. Kumar, \"Orthogonal Random Features\", NIPS, 2016.\n[PDF](https://papers.nips.cc/paper/6246-orthogonal-random-features.pdf)\n\n[3] V. Vapnik and A. Lerner, \"Pattern recognition using generalized portrait method\", Automation and Remote Control, vol. 24, 1963.\n\n[4] B. Boser, I. Guyon and V. Vapnik, \"A training algorithm for optimal margin classifiers\", COLT, pp. 144-152, 1992\n[URL](https://dl.acm.org/doi/10.1145/130385.130401)\n\n[5] C. Rasmussen and C. Williams, \"Gaussian Processes for Machine Learning\", MIT Press, 2006.\n\n[6] S. M. Lundberg and S. Lee, \"A Unified Approach to Interpreting Model Predictions\", NIPS, 2017.\n[PDF](https://proceedings.neurips.cc/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf)\n\n[7] L. Breiman, \"Random Forests\", Machine Learning, vol. 45, pp. 5-32, Springer, 2001.\n[Springer website](https://doi.org/10.1023/A:1010933404324).\n\n\nAuthor\n----------------------------------------------------------------------------------------------------\n\nTetsuya Ishikawa ([EMail](mailto:tiskw111@gmail.com), [Website](https://tiskw.github.io/about_en.html))\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftiskw%2Frandom-fourier-features","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftiskw%2Frandom-fourier-features","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftiskw%2Frandom-fourier-features/lists"}