{"id":32154384,"url":"https://github.com/rizalzaf/adversarialprediction.jl","last_synced_at":"2025-10-21T11:54:51.812Z","repository":{"id":61800710,"uuid":"225428428","full_name":"rizalzaf/AdversarialPrediction.jl","owner":"rizalzaf","description":"Easily optimize generic performance metrics in differentiable learning. ","archived":false,"fork":false,"pushed_at":"2020-06-06T18:06:52.000Z","size":513,"stargazers_count":18,"open_issues_count":0,"forks_count":3,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-10-21T11:54:38.104Z","etag":null,"topics":["adversarial-prediction","deep-learning","machine-learning","neural-networks"],"latest_commit_sha":null,"homepage":"","language":"Julia","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/rizalzaf.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2019-12-02T17:12:37.000Z","updated_at":"2025-10-04T14:10:46.000Z","dependencies_parsed_at":"2022-10-21T11:45:22.035Z","dependency_job_id":null,"html_url":"https://github.com/rizalzaf/AdversarialPrediction.jl","commit_stats":null,"previous_names":[],"tags_count":2,"template":false,"template_full_name":null,"purl":"pkg:github/rizalzaf/AdversarialPrediction.jl","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rizalzaf%2FAdversarialPrediction.jl","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rizalzaf%2FAdversarialPrediction.jl/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rizalzaf%2FAdversarialPrediction.jl/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rizalzaf%2FAdversarialPrediction.jl/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/rizalzaf","download_url":"https://codeload.github.com/rizalzaf/AdversarialPrediction.jl/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rizalzaf%2FAdversarialPrediction.jl/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":280256225,"owners_count":26299342,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-21T02:00:06.614Z","response_time":58,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["adversarial-prediction","deep-learning","machine-learning","neural-networks"],"created_at":"2025-10-21T11:54:50.664Z","updated_at":"2025-10-21T11:54:51.805Z","avatar_url":"https://github.com/rizalzaf.png","language":"Julia","funding_links":[],"categories":[],"sub_categories":[],"readme":"# AdversarialPrediction.jl\n\nThis package provides a way to easily optimize generic performance metrics in supervised learning settings using the [Adversarial Prediction](https://arxiv.org/abs/1812.07526) framework. \nThe method can be integrated easily into differentiable learning pipelines.\nThe package is a Julia implementation of an AISTATS 2020 paper, [\"AP-Perf: Incorporating Generic Performance Metrics in Differentiable Learning\"](https://arxiv.org/abs/1912.00965), by [Rizal Fathony](http://rizal.fathony.com) and [Zico Kolter](http://zicokolter.com). \nFor a Python implementation of the framework, please check  [ap_perf](https://github.com/rizalzaf/ap_perf).\n\n\n## Overview\n\nAdversarialPrediction.jl enables easy integration of generic performance metrics (including non-decomposable metrics) into our differentiable learning pipeline. It currently supports performance metrics that are defined over binary classification problems.\nBelow is a code example for incorporating the F-2 score metric into a convolutional neural network training pipeline of [FluxML](https://github.com/FluxML/Flux.jl). \n\n```julia\nusing Flux\nusing AdversarialPrediction\nimport AdversarialPrediction: define, constraint\n\nmodel = Chain(\n  Conv((5, 5), 1=\u003e20, relu), MaxPool((2,2)),\n  Conv((5, 5), 20=\u003e50, relu), MaxPool((2,2)),\n  x -\u003e reshape(x, :, size(x, 4)),\n  Dense(4*4*50, 500), Dense(500, 1), vec\n)      \n\n@metric FBeta beta\nfunction define(::Type{FBeta}, C::ConfusionMatrix, beta)\n    return ((1 + beta^2) * C.tp) / (beta^2 * C.ap + C.pp)  \nend   \nf2_score = FBeta(2)\nspecial_case_positive!(f2_score)\n\nobjective(x, y) = ap_objective(model(x), y, f2_score)\nFlux.train!(objective, params(model), train_set, ADAM(1e-3))\n```\n\nAs we can see from the code above, we can just write a function that calculates the F-2 score from the entities in the confusion matrix, and incorporate it into our learning pipeline using `ap_objective` function. \nThis is a straightforward modification from the standard cross entropy training by using the `ap_objective` function to replace the `logitbinarycrossentropy` objective.\n\n```julia\nusing Flux\n\nmodel = Chain(\n  Conv((5, 5), 1=\u003e20, relu), MaxPool((2,2)),\n  Conv((5, 5), 20=\u003e50, relu), MaxPool((2,2)),\n  x -\u003e reshape(x, :, size(x, 4)),\n  Dense(4*4*50, 500), Dense(500, 1), vec\n)     \n\nobjective(x, y) = mean(logitbinarycrossentropy(model(x), y))\nFlux.train!(objective, params(model), train_set, ADAM(1e-3))\n```\n\n\nNote that the equation for F-beta in general is:   \n\u003cdiv style=\"text-align:center\"\u003e\u003cimg src=\"assets/fbeta.gif\"\u003e\u003c/div\u003e\n\n\n\n## Installation\n\nAdversarialPrediction.jl can be installed from a Julia terminal:\n```\n]add AdversarialPrediction\n```\nSome pre-requisite packages will be installed automatically: `Zygote` (Flux's automatic differential engine), `Requires`, and `LBFGSB`. Please also install `Flux` separately. For GPU training, `CuArrays` package needs to be installed.\n\n\n## Performance Metrics\n\nDifferent tasks in machine learning  require different metrics that align  well with the tasks. For binary classification problems, many of the commonly used performance metrics are derived from the confusion matrix. \nA confusion matrix is a table that reports the values that relate the prediction of a classifier with the ground truth labels. The table below shows the anatomy of a confusion matrix.\n\n\u003cdiv style=\"text-align:center\"\u003e\u003cimg src=\"assets/confusion_matrix.png\" width=\"470\"\u003e\u003c/div\u003e\n\nSome of the metrics are decomposable, which means that it can be broken down to an independent sum of another metric that depends only on a single sample. However, most of the interesting performance metrics are non-decomposable, where we need to consider all samples at once. There are a wide variety of non-decomposable performance metrics, for example:\n\n\u003cdiv style=\"text-align:center\"\u003e\u003cimg src=\"assets/metrics.png\" width=\"500\"\u003e\u003c/div\u003e\n\nAdversarialPrediction.jl supports a family of performance metrics that can be expressed as a sum of fractions:\n\n\u003cdiv style=\"text-align:center\"\u003e\u003cimg src=\"assets/metric_construction.gif\"\u003e\u003c/div\u003e\n \nwhere a\u003csub\u003ej\u003c/sub\u003e and b\u003csub\u003ej\u003c/sub\u003e are constants, whereas f\u003csub\u003ej\u003c/sub\u003e and g\u003csub\u003ej\u003c/sub\u003e are functions over PP and AP.\nHence, the numerator is a linear function over true positive (TP) and true negative (TN) which may also depends on sum statistics, i.e., predicted and actual positive (PP and AP) as well as their negative counterparts (predicted and actual negative (PN and AN)) and all data (ALL). Note that PN, AN, and ALL can be derived form PP and AP since ALL is just a constant, PN = ALL - PP, and AN = ALL - AP.\nThe denominator depends only on the sum statistics (PP, AP, PN, AN, and ALL). This construction of performance metrics covers a vast range of commonly used metrics, including all metrics in the table above.\n\n\n## Defining Performance Metrics\n\nA performance metric can be defined in AdversarialPrediction.jl using the macro `@metric MetricName`.  We also need to write the definition of the metric by implementing a function that depends on the type of the metric and the confusion matrix: `define(::Type{MetricName}, C::ConfusionMatrix)`. Below is an example of the F-1 score metric definition.\n```julia\n@metric F1Score\nfunction define(::Type{F1Score}, C::ConfusionMatrix)\n    return (2 * C.tp) / (C.ap + C.pp)  \nend \n```\n\nSome performance metrics (e.g., precision, recall, F-score, sensitivity, and specificity) enforce special cases to avoid division by zero. For the metrics that contain true positive, the special case is usually defined when the prediction or the true label for every sample are all zero. In this case, the metric is usually defined as 1 if both the prediction and the true label are all zero; otherwise, the metric is 0. For the metrics that contain true negative, similar cases occur, but with the prediction or the true label for every sample are all one.\nTherefore, when instantiating a metric, we need to take into account these special cases, for example, in the case of F-1 score:\n```julia\nf1_score = F1Score()\nspecial_case_positive!(f1_score)\n```\n\nFor some performance metrics, we may want to define a parametric metric. For example, the F-beta score, which depends on the value of beta. In this case, we can write a macro with arguments, for example, `@metric MetricName arg1 arg2`. We also need to adjust the function definition to: `define(::Type{MetricName}, C::ConfusionMatrix, arg1, arg2)`. For the case of the F-beta score metric, the code is:\n```julia\n@metric FBeta beta\nfunction define(::Type{FBeta}, C::ConfusionMatrix, beta)\n    return ((1 + beta^2) * C.tp) / (beta^2 * C.ap + C.pp)  \nend   \n\nf1_score = FBeta(1)\nspecial_case_positive!(f1_score)\n\nf2_score = FBeta(2)\nspecial_case_positive!(f2_score)\n```\n\n\nWe can define arbitrary complex performance metrics inside the `define` function, so long as it follows the construction of metrics that the package support. We can also use intermediate variables to store partial expression of the metric. Below is a code example for Cohen's kappa score.\n```julia\n@metric Kappa\nfunction define(::Type{Kappa}, C::ConfusionMatrix)\n    pe = (C.ap * C.pp + C.an * C.pn) / C.all^2\n    num = (C.tp + C.tn) / C.all - pe\n    den = 1 - pe\n    return num / den\nend  \n\nkappa = Kappa()\nspecial_case_positive!(kappa)\nspecial_case_negative!(kappa)\n```\n\n## Performance Metric with Constraints \n\nIn some machine learning settings, we may want to optimize a performance metric subject to constraints on other metrics. This occurs in the case where there are trade-offs between different performance metrics. For example, a machine learning system may want to optimize the precision of the prediction; subject to its recall is greater than some threshold. We can define the constraints in the metric by implementing the function: `constraint(::Type{MetricName}, C::ConfusionMatrix)`. The code format for the constraints is `metric \u003e= th`, where `th` is a real-valued threshold. Below is an example:\n```julia\n# Precision given recall metric\n@metric PrecisionGvRecall th\nfunction define(::Type{PrecisionGvRecall}, C::ConfusionMatrix, th)\n    return C.tp / C.pp\nend   \nfunction constraint(::Type{PrecisionGvRecall}, C::ConfusionMatrix, th)\n    return C.tp / C.ap \u003e= th\nend   \n\nprecision_gv_recall_80 = PrecisionGvRecall(0.8)\nspecial_case_positive!(precision_gv_recall_80)\ncs_special_case_positive!(precision_gv_recall_80, true)\n\nprecision_gv_recall_60 = PrecisionGvRecall(0.6)\nspecial_case_positive!(precision_gv_recall_60)\ncs_special_case_positive!(precision_gv_recall_60, true)\n```\n\nNote that the function `special_case_positive!` enforces special cases for the precision metric, whereas `cs_special_case_positive!` enforces special cases for the metric in the constraint (recall metric).\n\nWe can also have two or more metrics in the constraints, for example:\n```julia\n# Precision given recall \u003e= th1 and specificity \u003e= th2\n@metric PrecisionGvRecallSpecificity th1 th2        \nfunction define(::Type{PrecisionGvRecallSpecificity}, C::ConfusionMatrix, th1, th2)\n    return C.tp / C.pp\nend   \nfunction constraint(::Type{PrecisionGvRecallSpecificity}, C::ConfusionMatrix, th1, th2)\n    return [C.tp / C.ap \u003e= th1,\n            C.tn / C.an \u003e= th2]\nend   \n\nprecision_gv_recall_spec = PrecisionGvRecallSpecificity(0.8, 0.8)\nspecial_case_positive!(precision_gv_recall_spec)\ncs_special_case_positive!(precision_gv_recall_spec, [true, false])\ncs_special_case_negative!(precision_gv_recall_spec, [false, true])\n```\n\nHere, we need to provide an array of boolean for the function `cs_special_case_positive!` and `cs_special_case_negative!`.\n\n## Computing the Values of the Metric\n\nGiven we have a prediction for each sample `yhat` and the true label `y`, we can call the function `compute_metric` to compute the value of the metric. Both `yhat` and `y` are vectors containing 0 or 1. \n```julia\njulia\u003e compute_metric(f1_score, yhat, y)\n0.8f0\n```  \n\nFor a metric with constraints, we can call the function `compute_constraints` to compute the value of every  metric in the constraints. For example:\n```julia\njulia\u003e compute_constraints(precision_gv_recall_spec, yhat, y)\n2-element Array{Float32,1}:\n 0.6\n 0.6\n```  \n\n## Incorporating the Metric into Differentiable Learning Pipeline\n\nAs we can see from the first code example, we can use the function `ap_objective` to incorporate the metrics we define into differentiable learning pipeline. \nThis function provides objective and gradient information from the adversarial prediction formulation, which then be propagated to the previous layers.\nThis serves as a replacement to the standard loss function like the binary cross-entropy loss, i.e.: \n```julia\nobjective(x, y) = mean(logitbinarycrossentropy(model(x), y))\nFlux.train!(objective, params(model), train_set, ADAM(1e-3))\n```\nWe can easily replace the existing codes that use binary cross-entropy by simply change the objective to `ap_objective`, i.e.:\n```julia\nobjective(x, y) = ap_objective(model(x), y, f1_score)\nFlux.train!(objective, params(model), train_set, ADAM(1e-3))\n```\n\n## Customizing Inner Optimization Solver\n\nFor solving the inner optimization problem, AdversarialPrediction.jl uses an ADMM based formulation. In the default setting, it will run 100 iterations of the ADMM optimization. We can also manually set the number of iteration using `max_iter` argument in the `ap_objective`.\n\n```julia\nobjective(x, y) = ap_objective(model(x), y, f1_score)            # 100 iterations\nobjective(x, y) = ap_objective(model(x), y, max_iter = 50)       # 50 iterations\nobjective(x, y) = ap_objective(model(x), y, max_iter = 200)      # 200 iterations\n```\n\n## Running Time and Batch Size\n\nThe adversarial prediction formulation inside the function `ap_objective` needs to solve a maximin problem with a quadratic size of variables using the ADMM solver. The complexity of solving the problem is O(m^3), where m is the number of samples in a minibatch.\nIn practice, for a batch size of 25, the ADMM solver takes around 20 - 30 milliseconds to solve on a PC with an Intel Core i7 processor. If we reduce the ADMM iterations to 50 iterations, it will take around 10 - 15 milliseconds.\n\n## Commonly Used Metrics\n\nThe package provides definitions of commonly use metrics including: `f1_score`, `f2_score`, `gpr`, `mcc`, `kappa`, etc. To load the metrics to the current Julia environment, please use `AdversarialPrediction.CommonMetrics`. Please check the detailed definition of the metrics in `src/common_metrics/common_metrics.jl`.\n\n```julia\nusing AdversarialPrediction\nusing AdversarialPrediction.CommonMetrics: f1_score, kappa\n\nobjective(x, y) = ap_objective(model(x), y, kappa)\n```\n\n\n## Code Examples\n\nFor working examples, please visit [AP-examples](https://github.com/rizalzaf/AP-examples) repository. The project contains examples of using AdversarialPrediction.jl for classification with tabular datasets, as well as for image classification with MNIST and FashionMNIST datasets.\n\n## Python Implementation and Interface\n\nWe also provides a Python implementation of the framework: [ap_perf](https://github.com/rizalzaf/ap_perf). This enables easy integration with Python codes as well as PyTorch deep learning framework. \n\nWe also provide a python interface to AdversarialPrediction.jl via [PyJulia](https://pyjulia.readthedocs.io/en/stable/) library in [ap_perf_py](https://github.com/rizalzaf/ap_perf_py). \n\n## Citation\n\nPlease cite the following paper if you use the AdversarialPrediction.jl for your research.\n```\n@article{ap-perf,\n  title={AP-Perf: Incorporating Generic Performance Metrics in Differentiable Learning},\n  author={Fathony, Rizal and Kolter, Zico},\n  journal={arXiv preprint arXiv:1912.00965},\n  year={2019}\n}\n```\n\n\n## Acknowledgements\n\nThis project is supported by a grant from the [Bosch Center for Artificial Intelligence](https://www.bosch-ai.com/).\n\nThis project is not possible without previous foundational research in Adversarial Prediction by [Prof. Brian Ziebart's](https://www.cs.uic.edu/Ziebart) and [Prof. Xinhua Zhang's](https://www.cs.uic.edu/~zhangx/) research groups at the [University of Illinois at Chicago](https://www.cs.uic.edu).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frizalzaf%2Fadversarialprediction.jl","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Frizalzaf%2Fadversarialprediction.jl","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frizalzaf%2Fadversarialprediction.jl/lists"}