{"id":34017715,"url":"https://github.com/ai2es/miles-guess","last_synced_at":"2025-12-13T14:45:58.247Z","repository":{"id":193314922,"uuid":"685783156","full_name":"ai2es/miles-guess","owner":"ai2es","description":"Machine learning models for estimating aleatoric and epistemic uncertainty with evidential and ensemble methods.","archived":false,"fork":false,"pushed_at":"2025-07-09T00:22:06.000Z","size":25183,"stargazers_count":29,"open_issues_count":2,"forks_count":10,"subscribers_count":6,"default_branch":"main","last_synced_at":"2025-11-27T17:47:21.703Z","etag":null,"topics":["ai","bayesian","epistemic-uncertainty","evidential-deep-learning","machine-learning","neural-networks","uncertainty-quantification"],"latest_commit_sha":null,"homepage":"https://miles-guess.readthedocs.io/en/latest/","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"cc0-1.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ai2es.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2023-09-01T02:06:45.000Z","updated_at":"2025-10-01T16:14:00.000Z","dependencies_parsed_at":"2024-07-19T01:14:54.578Z","dependency_job_id":"bad1d2fc-2200-4c19-a129-5f5f7d6c678c","html_url":"https://github.com/ai2es/miles-guess","commit_stats":{"total_commits":164,"total_committers":8,"mean_commits":20.5,"dds":0.7073170731707317,"last_synced_commit":"cbb3dffcdba6505523302078b5dfa3bee46d0af8"},"previous_names":["ai2es/miles-guess"],"tags_count":4,"template":false,"template_full_name":null,"purl":"pkg:github/ai2es/miles-guess","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ai2es%2Fmiles-guess","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ai2es%2Fmiles-guess/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ai2es%2Fmiles-guess/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ai2es%2Fmiles-guess/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ai2es","download_url":"https://codeload.github.com/ai2es/miles-guess/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ai2es%2Fmiles-guess/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":27707424,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-12-13T02:00:09.769Z","response_time":147,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai","bayesian","epistemic-uncertainty","evidential-deep-learning","machine-learning","neural-networks","uncertainty-quantification"],"created_at":"2025-12-13T14:45:57.352Z","updated_at":"2025-12-13T14:45:58.242Z","avatar_url":"https://github.com/ai2es.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# MILES-Guess\nGeneralized Uncertainty for Earth System Science (GUESS)\n\nDeveloped by the Machine Ingetration and Learning for Earth Systems (MILES) group at the NSF National Center for Atmospheric Research (NCAR), Boulder CO, USA\n\n## Contributors \n* John Schreck\n* David John Gagne\n* Charlie Becker\n* Gabrielle Gantos\n* Dhamma Kimpara\n* Thomas Martin\n\n## Documentation\nFull documentation is [here](https://miles-guess.readthedocs.io/en/latest/).\n\n## Quick Setup\n\nInstall in your Python environment with the following command:\n```bash\npip install miles-guess\n```\nIf you want to install a particular backend (tensorflow, tensorflow_gpu, torch, jax): \n```bash\npip install miles-guess[\u003cbackend\u003e]\n```\n## Setup from Scratch\n\nInstall the Miniconda Python installer available\n[here](https://docs.conda.io/en/latest/miniconda.html).\n\nFirst clone the miles-guess repo from github.\n```bash\ngit clone https://github.com/ai2es/miles-guess.git`\ncd miles-guess\n```\n\nCreate a conda environment for non-Casper/Derecho users:\n```bash\nmamba env create -f environment.yml`\nconda activate guess`\n```\n\nCreate a conda environment for Casper/Derecho users including Tensorflow 2.15 with GPU support.\n```bash\nmamba env create -f environment_gpu.yml`\nconda activate guess\n```\n\n## Using miles-guess\n\nThe law of total variance for each model prediction target may be computed as\n\n$$LoTV = E[\\sigma^2] + Var[\\mu]$$ \n\nwhich is the sum of aleatoric and epistemic contributions, respectively. The MILES-GUESS package contains options for using either Keras or PyTorch for computing quantites according to the LoTV as well as utilizing Dempster-Shafer theory uncertainty in the classifier case. \n\nFor detailed information about training with Keras, refer to [the Keras training details README](docs/source/keras.md). There three scripts for training three regression models, and one for training categorical models. The regression examples are trained on our surface layer (\"SL\") dataset for predicting latent heat and other quantities, \nand the categorical example is trained on a precipitation dataset (\"p-type\").\n\nFor pyTorch, please visit the [the pyTorch training details README](docs/source/torch.md) where details on training scripts for both evidential standard classification tasks are detailed. Torch examples use the same datasets as the Keras models. The torch training code will also scale on GPUs, and is compatitible with DDP and FSDP.\n\n\u003c!--\n### 1a. Train/evaluate a deterministic multi-layer perceptrion (MLP) on the SL dataset:\n```bash\npython3 applications/train_mlp_SL.py -c config/model_mlp_SL.yml\n```\n\n### 1b. Train/evaluate a parametric \"Gaussian\" MLP on the SL dataset:\n```bash\n\npython applications/train_gaussian_SL.py -c config/model_gaussian_SL.yml\n```\n\n### 1c. Train/evaluate a parametric \"normal-inverse gamma\" (evidential) MLP on the SL dataset:\n```bash\npython applications/train_evidential_SL.py -c config/model_evidential_SL.yml\n```\n\n### 2a. Train a categorical MLP classifier on the p-type dataset:\n```bash\npython applications/train_classifier_ptype.py -c config/model_classifier_ptype.yml\n```\n\n### 2b. Train an evidential MLP classifier on the p-type dataset:\n```bash\npython applications/train_classifier_ptype.py -c config/model_evidential_ptype.yml\n```\n\n### 2c. Evaluate a categorical/evidential classifier on the p-type dataset:\n```bash\npython applications/evaluate_ptype.py -c config/model_classifier_ptype.yml\n```\n\n\n## 3. Ensembling modes for the deterministic model (1a)\n\nThere are four \"modes\" for training the deterministic MLP (1a) that are controlled using the \"ensemble\" field in a model configuration.\n```yaml\nensemble:\n    n_models: 100\n    n_splits: 20\n    monte_carlo_passes: 0\n```\nwhere n_models means the number of models to train using a fixed data split with variable initial weight initializations, n_splits means the number of models to train using variable training and validation splits (random initializations), and mc_carlo_passes means the number of MC-dropout evaluations performed on a given input to the model. \n\n### 3a. Single Mode\n```yaml\nensemble:\n    n_models: 1\n    n_splits: 1\n    monte_carlo_passes: 0\n```\nTrain a single deterministic model (no uncertainty evaluation). If MC passes \u003e 0, an ensemble is created after the model finishes training.\n\n### 3b. Data Mode\n```yaml\nensemble:\n    n_models: 1\n    n_splits: 10\n    monte_carlo_passes: 100\n```\nCreate an ensemble of models (random initialization) using cross-validation splits. If MC passes \u003e 0, an ensemble is created after each model finishes training on the test holdout. The LOTV may then be applied to the ensemble created from cross-validation. Otherwise a single ensemble is created but the LOTV is not applied. \n\n### 3c. Model Mode\n```yaml\nensemble:\n    n_models: 10\n    n_splits: 1\n    monte_carlo_passes: 100\n```\nCreate an ensemble of models using a fixed train/validation/test data split and variable model layer weight initializations. If MC passes \u003e 0, an ensemble is created after each model finishes training. The LOTV may then be applied to the ensemble created from variable weight initializations to obtain uncertainty estimations for each prediction target. Otherwise a single ensemble is created but the LOTV is not applied. \n\n### 3d. Ensemble Mode\n```yaml\nensemble:\n    n_models: 1\n    n_splits: 1\n    monte_carlo_passes: 0\n```\nCreate an ensemble of ensembles. The first ensemble is created using cross validation and a fixed weight initialization, from which a mean and variance may be obtained for each prediction target. The second ensemble is created by varying the weight initalization that can then be used with the LOTV to obtain uncertainty estimations for each prediction target. The MC steps field is ignored in ensemble mode. \n\n## 4. Ensembling modes for the Gaussian parametric model (1b)\nThere are three \"modes\" for training the Gaussian MLP (1b).\n\n### 4a. Single Mode\n```yaml\nensemble:\n    n_models: 1\n    n_splits: 1\n    monte_carlo_passes: 0\n```\nTrain a single deterministic model (no LOTV evaluation). If MC passes \u003e 0, an ensemble is created after the model finishes training (LOTV evaluation).\n\n### 4b. Data Mode\n```yaml\nensemble:\n    n_models: 1\n    n_splits: 10\n    monte_carlo_passes: 0\n```\nCreate an ensemble of models using cross-validation splits, and then LOTV evaluation.\n\n### 4c. Model Mode\n```yaml\nensemble:\n    n_models: 10\n    n_splits: 1\n    monte_carlo_passes: 0\n```\nCreate an ensemble of models using different random initializations and a fixed cross-validation split, and then LOTV evaluation.\n\n## Configuration files\nIn addition to the ensemble field, the other fields in the configuration file are \n\n```yaml\nseed: 1000\nsave_loc: \"/path/to/save/directory\"\ntraining_metric: \"val_mae\"\ndirection: \"min\"\n```\n\nwhere seed allows for reproducability, save_loc is where data will be saved, and training metric and direction are used as the validation metric (and direction).\n\nFor regression tasks, other fields in a configuration file are model and callbacks:\n```yaml\n\nmodel:\n    activation: relu\n    batch_size: 193\n    dropout_alpha: 0.2\n    epochs: 200\n    evidential_coef: 0.6654439861214466\n    hidden_layers: 3\n    hidden_neurons: 6088\n    kernel_reg: l2\n    l1_weight: 0.0\n    l2_weight: 7.908676527243475e-10\n    lr: 3.5779279071474884e-05\n    metrics: mae\n    optimizer: adam\n    uncertainties: true\n    use_dropout: true\n    use_noise: false\n    verbose: 2\n      \ncallbacks:\n  EarlyStopping:\n    monitor: \"val_loss\"\n    patience: 5\n    mode: \"min\"\n    verbose: 0\n  ReduceLROnPlateau: \n    monitor: \"val_loss\"\n    factor: 0.1\n    patience: 2\n    min_lr: 1.0e-12\n    min_delta: 1.0e-08\n    mode: \"min\"\n    verbose: 0\n  CSVLogger:\n    filename: \"training_log.csv\"\n    separator: \",\"\n    append: False\n  ModelCheckpoint:\n    filepath: \"model.h5\"\n    monitor: \"val_loss\"\n    save_weights: True\n    save_best_only: True\n    mode: \"min\"\n    verbose: 0\n```\n\nFor categorical tasks, the model field changes slightly:\n\n```yaml\nmodel:\n    activation: leaky\n    balanced_classes: 1\n    batch_size: 3097\n    dropout_alpha: 0.31256692323263807\n    epochs: 200\n    hidden_layers: 4\n    hidden_neurons: 6024\n    loss: categorical_crossentropy\n    loss_weights:\n    - 21.465788717561477\n    - 83.31367732936326\n    - 136.50944842077058\n    - 152.62042204485107\n    lr: 0.0004035503144482269\n    optimizer: adam\n    output_activation: softmax\n    use_dropout: 1\n    verbose: 0\n```\nwhere the user has two options:\n\n(1) A standard deterministic classifier is trained when \n```yaml \n    loss: cateogical_crossentropy\n    output_activation: softmax\n```\n(2) An evidential classifier is trained when \n```yaml \n    loss: dirichlet\n    output_activation: linear\n```\n\nCallbacks are not required in regression training, however a custom callback which tracks the current epoch is requried for the categorical model and is added automatically (the user does not need to specify it in the callbacks field). The user may add any other supported keras callback by adding the relevant fields to the callbacks field. \n\nDepending on the problem, a data field is customized and also present in the configuration files. See the examples for surface layer and p-type data sets for more details. \n\n\n## ECHO hyperparameter optimization \n\nConfiguration files are also supplied for use with the Earth Computing Hyperparameter Optimization (ECHO) package. See the echo package https://github.com/NCAR/echo-opt/tree/main/echo for more details on the configuration fields. --\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fai2es%2Fmiles-guess","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fai2es%2Fmiles-guess","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fai2es%2Fmiles-guess/lists"}