{"id":23100686,"url":"https://github.com/benkeser/nlpred","last_synced_at":"2025-08-16T14:31:56.430Z","repository":{"id":129606520,"uuid":"171500744","full_name":"benkeser/nlpred","owner":"benkeser","description":"Estimators of cross-validated prediction metrics with improved small sample performance","archived":false,"fork":false,"pushed_at":"2022-06-10T14:01:11.000Z","size":2701,"stargazers_count":3,"open_issues_count":0,"forks_count":3,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-04-04T12:52:46.602Z","etag":null,"topics":["auc","cross-validation","estimating-equations","machine-learning","tmle"],"latest_commit_sha":null,"homepage":"https://benkeser.github.io/nlpred/","language":"R","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/benkeser.png","metadata":{"files":{"readme":"README.Rmd","changelog":"NEWS.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-02-19T15:39:50.000Z","updated_at":"2022-06-10T14:01:15.000Z","dependencies_parsed_at":"2023-06-25T22:19:16.716Z","dependency_job_id":null,"html_url":"https://github.com/benkeser/nlpred","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/benkeser/nlpred","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/benkeser%2Fnlpred","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/benkeser%2Fnlpred/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/benkeser%2Fnlpred/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/benkeser%2Fnlpred/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/benkeser","download_url":"https://codeload.github.com/benkeser/nlpred/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/benkeser%2Fnlpred/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":270723211,"owners_count":24634339,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-16T02:00:11.002Z","response_time":91,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["auc","cross-validation","estimating-equations","machine-learning","tmle"],"created_at":"2024-12-16T23:33:39.391Z","updated_at":"2025-08-16T14:31:56.383Z","avatar_url":"https://github.com/benkeser.png","language":"R","funding_links":[],"categories":[],"sub_categories":[],"readme":"---\noutput: github_document\n---\n\n\u003c!-- README.md is generated from README.Rmd. Please edit that file --\u003e\n\n```{r, echo = FALSE}\nknitr::opts_chunk$set(\n  collapse = TRUE,\n  comment = \"#\u003e\",\n  fig.path = \"README-\"\n)\n```\n\n# R/`nlpred`\n\n[![Travis-CI Build Status](https://travis-ci.org/benkeser/nlpred.svg?branch=master)](https://travis-ci.org/benkeser/nlpred)\n[![AppVeyor Build  Status](https://ci.appveyor.com/api/projects/status/github/benkeser/nlpred?branch=master\u0026svg=true)](https://ci.appveyor.com/project/benkeser/nlpred)\n[![Coverage Status](https://img.shields.io/codecov/c/github/benkeser/nlpred/master.svg)](https://codecov.io/github/benkeser/nlpred?branch=master)\n[![Project Status: Active – The project has reached a stable, usable state and is being actively developed.](http://www.repostatus.org/badges/latest/active.svg)](http://www.repostatus.org/#active)\n[![MIT license](http://img.shields.io/badge/license-MIT-brightgreen.svg)](http://opensource.org/licenses/MIT)\n\u003c!-- [![CRAN](http://www.r-pkg.org/badges/version/nlpred)](http://www.r-pkg.org/pkg/nlpred) --\u003e\n\u003c!-- [![CRAN downloads](https://cranlogs.r-pkg.org/badges/nlpred)](https://CRAN.R-project.org/package=nlpred) --\u003e\n\u003c!-- [![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.835868.svg)](https://doi.org/10.5281/zenodo.835868) --\u003e\n\n\u003e Small-sample optimized estimators of cross-validated prediction metrics\n\n[David Benkeser](https://www.benkeserstatistics.com/) \n\n---\n\n## Description\n\n`nlpred` is an R package for computing estimates of cross-validated prediction metrics. These estimates are tailored for superior performance in small samples. Several estimators are available including ones based cross-validated targeted minimum loss-based estimation, estimating equations, and one-step estimation. \n\n---\n\n## Installation\n\nFor standard use, we recommend installing the package from\n[CRAN](https://cran.r-project.org/) via\n\n```{r cran-installation, eval = FALSE}\ninstall.packages(\"nlpred\")\n```\n\nYou can install the current release of `nlpred` from GitHub via\n[`devtools`](https://CRAN.R-project.org/package=devtools) with:\n\n```{r gh-installation, eval = FALSE}\ndevtools::install_github(\"benkeser/nlpred\")\n```\n---\n\n## Usage\n\nThe main functions in the package are `cv_auc` and `cv_scrnp`, which are used to \ncompute, respectively, the `K`-fold [cross-validated area under the receiver operating\ncharacteristics curve](http://projecteuclid.org/euclid.ejs/1437742107) (CVAUC) and \nthe `K`-fold cross-validated [sensitivity constrained rate of negative prediction](https://onlinelibrary.wiley.com/doi/abs/10.1002/sim.7296). However, rather than using\nstandard cross-validation estimators (where prediction algorithms are developed in a \ntraining sample and AUC/SCRNP estimated using the validation sample), we instead use\ntechniques from efficiency theory to estimate these quantities. This allows us to use\nthe training data both to develop the prediction algorithm, *as well as* key nuisance \nparameters needed to evaluate AUC/SCRNP. By reserving more data for estimation of these\nkey parameters, we obtain improved performance in small samples. \n\n```{r}\n# load package\nlibrary(nlpred)\n\n# turn off messages from np package\noptions(np.messages=FALSE)\n\n# simulate data\nn \u003c- 200\np \u003c- 10\nX \u003c- data.frame(matrix(rnorm(n*p), nrow = n, ncol = p))\nY \u003c- rbinom(n, 1, plogis(X[,1] + X[,10]))\n\n# get cv auc estimates for logistic regression\nlogistic_cv_auc_ests \u003c- cv_auc(Y = Y, X = X, K = 5, learner = \"glm_wrapper\")\nlogistic_cv_auc_ests\n\n# get cv auc estimates for random forest using nested \n# cross-validation for nuisance parameter estimation. nested\n# cross-validation is unfortunately necessary when aggressive learners \n# are used. \nrf_cv_auc_ests \u003c- cv_auc(Y = Y, X = X, K = 5, \n\t\t\t\t\t\t learner = \"randomforest_wrapper\", \n\t\t\t\t\t\t nested_cv = TRUE)\nrf_cv_auc_ests\n\n# same examples for scrnp\nlogistic_cv_scrnp_ests \u003c- cv_scrnp(Y = Y, X = X, K = 5, learner = \"glm_wrapper\")\nlogistic_cv_scrnp_ests\n\n\nrf_cv_scrnp_ests \u003c- cv_scrnp(Y = Y, X = X, K = 5, \n\t\t\t\t\t\t learner = \"randomforest_wrapper\", \n\t\t\t\t\t\t nested_cv = TRUE)\nrf_cv_scrnp_ests\n```\n\n---\n\n## Issues\n\nIf you encounter any bugs or have any specific feature requests, please [file an\nissue](https://github.com/benkeser/nlpred/issues).\n\n---\n\n## Contributions\n\nInterested contributors can consult our [`contribution guidelines`](https://github.com/benkeser/nlpred/blob/master/CONTRIBUTING.md) prior to submitting a pull request.\n\n---\n\n## Citation \n\nAfter using the `nlpred` package, please cite the following: \n\n\t@Manual{nlpredpackage,\n\t  title = {nlpred: Estimators of Non-Linear Cross-Validated Risks Optimized for Small Samples},\n\t  author = {David Benkeser},\n\t  note = {R package version 1.0.1}\n\t}\n\n\t@article{benkeser2019improved,\n\t  year  = {2019},\n\t  author = {Benkeser, David C and Petersen, Maya and van der Laan, Mark J},\n\t  title = {Improved Small-Sample Estimation of Nonlinear Cross-Validated Prediction Metrics},\n\t  journal = {Journal of the American Statistical Association},\n\t  doi = {10.1080/01621459.2019.1668794}\n\t}\n\n## License\n\n\u0026copy; 2019- David Benkeser\n\nThe contents of this repository are distributed under the MIT license. See\nbelow for details:\n```\nThe MIT License (MIT)\n\nCopyright (c) 2019- David C. Benkeser\n\nPermission is hereby granted, free of charge, to any person obtaining a copy\nof this software and associated documentation files (the \"Software\"), to deal\nin the Software without restriction, including without limitation the rights\nto use, copy, modify, merge, publish, distribute, sublicense, and/or sell\ncopies of the Software, and to permit persons to whom the Software is\nfurnished to do so, subject to the following conditions:\n\nThe above copyright notice and this permission notice shall be included in all\ncopies or substantial portions of the Software.\n\nTHE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\nIMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\nFITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\nAUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\nLIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\nOUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\nSOFTWARE.\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbenkeser%2Fnlpred","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbenkeser%2Fnlpred","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbenkeser%2Fnlpred/lists"}