{"id":26397375,"url":"https://github.com/thalesgroup/statistical-reliability-ml","last_synced_at":"2025-03-17T12:17:34.230Z","repository":{"id":248505293,"uuid":"813549638","full_name":"ThalesGroup/statistical-reliability-ml","owner":"ThalesGroup","description":"This package provides implementations of Monte Carlo methods to estimate the probability of failure of neural networks under noisy inputs.","archived":false,"fork":false,"pushed_at":"2024-07-15T08:58:21.000Z","size":44,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2024-07-15T12:01:51.422Z","etag":null,"topics":["monte-carlo-methods","neural-networks","reliability-engineering","statistical-analysis"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ThalesGroup.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":"SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-06-11T09:38:35.000Z","updated_at":"2024-07-15T12:02:01.098Z","dependencies_parsed_at":"2024-07-15T12:12:03.667Z","dependency_job_id":null,"html_url":"https://github.com/ThalesGroup/statistical-reliability-ml","commit_stats":null,"previous_names":["thalesgroup/statistical-reliability-ml"],"tags_count":0,"template":false,"template_full_name":"ThalesGroup/template-project","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ThalesGroup%2Fstatistical-reliability-ml","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ThalesGroup%2Fstatistical-reliability-ml/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ThalesGroup%2Fstatistical-reliability-ml/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ThalesGroup%2Fstatistical-reliability-ml/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ThalesGroup","download_url":"https://codeload.github.com/ThalesGroup/statistical-reliability-ml/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":244031130,"owners_count":20386534,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["monte-carlo-methods","neural-networks","reliability-engineering","statistical-analysis"],"created_at":"2025-03-17T12:17:33.640Z","updated_at":"2025-03-17T12:17:34.215Z","avatar_url":"https://github.com/ThalesGroup.png","language":"Python","readme":"# Statistical Reliability ML\n\nThis package provides implementations of Monte Carlo methods to estimate the probability of failure of neural networks under noisy inputs.\n\nIt takes as input a PyTorch model and samples from a dataset, and it outputs the probabilities of failure given the noise configuration.\n\nIt implements several estimators. Currently:\n\n* a basic Crude Monte Carlo,\n* the first-order reliability measure (FORM),\n* an importance sampling Monte Carlo estimation.\n\nThe last two estimators require searching for the Most Probable Failure Point (MPP).\nThe package provides several methods to find these points.\nOne of these methods uses [Foolbox](https://foolbox.jonasrauber.de) adversarial attacks, an optional dependecncy for the package.\n\n## Basic usage\n\nConsidering a PyTorch model ```model``` and a supervised datatet with a PyTorch tensor ```X_test``` of featrues and a tensor ```y_test``` of  true labels,\nwe configure a reliability experiment with the following steps.\ns\n### 1. Import the package\n\n```python\nimport statistical_reliability_ml as strml\n```\n\n### 2. Create an experiment\n\n```python\nexperiment = strml.StatsRelExperiment(model, X_test, y_test, x_min=0, x_max=1, n_rep=10, noise_dist='gaussian', epsilon_range=[0.3])\n```\n\nThe parameters provided here are:\n\n* ```x_min```, ```x_max```: the bounds for the samples features\n* ```n_rep```: the number of repetitions for each experiment. Setting this parameter greater than 1 will allow to estimate the variance of the estimations.\n* ```noise_dist```: the type of noise distribution (either Gaussian, uniform or real uniform noise).\n* ```epsilon_range```: a list of scale values for the noise. Each value will be evaluated in a separate experiment.\n\nOther parameters can be provided and are decribed in the description of the ```StatsRelExperiment``` class.\n\n### 3. Configure an estimator\n\nThe estimators classes are available in the package ```estimators```.\n\nFor instance the Crude Monte Carlo estimator is configured in the following manner:\n\n```python\nmethod = strml.estimators.CrudeMC(N_samples=[100,1000,1e4], track_advs=True)\n```\n\nThis estimator takes as parameter the number of samples to generate (```N_samples```). Such a parameter can either be a single value or a list of values. If it is list then several experiments will be performed, each using a different value for the parameter. If more parameters defined as a range of values, then the cross-product of all the parameters configurations will be tested.\n\nThe Boolean parameter ```track_advs``` set to True requires that random samples that are adversarial examples will be ouputed in the results of the experiment.\n\n#### Optionaly, configure a MPP search method\n\nThe other estimators (FORM and Importance Sampling) require as a mandatory parameter a method to search for the Most Probable Failure Point (MPP). The MPP search methods are classes in the package ```mpp_search```.\n\nFor instance the Newton search is configured in the following manner:\n\n```python\nsearch_method = strml.mpp_search.MPPSearchNewton(max_iter=[100, 1e4, 1e5], real_mpp=True)\n```\n\nAgain the parameter ```max_iter``` can either be a single value or a list of values.\n\nThen, the importance sampling estimator for instance is instanciated like this:\n\n```python\nmethod = strml.estimators.ImportanceSampling(search_method, N_samples=1000 track_advs=False, save_weights=False, save_mpp=False)\n```\n\n### 4. Run the experiments\n\nGiven an experiment object and an estimator, we run all experiments with the following command:\n\n```python\nresults_df, dict_out = experiment.run_estimation(method, test_indices=[0,1,5], verbose=2)\n```\n\nIn this command ```test_indices``` is the list of indices from ```X_test``` that will be analyzed.\n\nThe outputs of the command are:\n\n* ```results_df```: a Pandas DataFrame that store the results and the parameters of the experiments, with one line for each experiment.\n* ```dict_out```: a dictionnary with additional results that depend on the configuration of the experiment and the estimator.\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fthalesgroup%2Fstatistical-reliability-ml","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fthalesgroup%2Fstatistical-reliability-ml","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fthalesgroup%2Fstatistical-reliability-ml/lists"}