{"id":20515718,"url":"https://github.com/ignf/roc4j","last_synced_at":"2025-04-14T00:33:12.294Z","repository":{"id":147293906,"uuid":"101413988","full_name":"IGNF/roc4j","owner":"IGNF","description":"The package roc4j is designed for estimating and handling Receiver Operating Characteristics (ROC) curves of binary classifiers in Java","archived":false,"fork":false,"pushed_at":"2018-08-13T18:30:09.000Z","size":5307,"stargazers_count":4,"open_issues_count":0,"forks_count":1,"subscribers_count":9,"default_branch":"master","last_synced_at":"2023-10-20T20:19:12.341Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/IGNF.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2017-08-25T14:50:33.000Z","updated_at":"2023-10-20T20:19:13.160Z","dependencies_parsed_at":null,"dependency_job_id":"bff273ad-cdf6-403b-bd6a-3d38838b3bb4","html_url":"https://github.com/IGNF/roc4j","commit_stats":null,"previous_names":[],"tags_count":0,"template":null,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/IGNF%2Froc4j","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/IGNF%2Froc4j/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/IGNF%2Froc4j/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/IGNF%2Froc4j/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/IGNF","download_url":"https://codeload.github.com/IGNF/roc4j/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":224846693,"owners_count":17379587,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-15T21:24:13.226Z","updated_at":"2024-11-15T21:24:13.911Z","avatar_url":"https://github.com/IGNF.png","language":"Java","readme":"\nHandling ROC curves with roc4j\n===================\n\nThe package roc4j is designed for estimating and handling Receiver Operating Characteristics (ROC) curves of binary classifiers in Java. Robust and accurate estimation of ROC curve is of utmost importance in statistical learning.\n\nAmong roc4j main features:\n\n- ROC curve computation\n- ROC curve filtering and smoothing\n- Confidence bands computation\n- Graphical plots (exportable in png, svg, jpg...)\n- Validation process handler\n- Optimal operating points computation\n\n\u003cbr\u003e\n\n### Screenshots\n\n\u003cimg src=\"https://github.com/IGNF/roc4j/blob/master/doc/images/binormalSmoothing.png\" width=\"200\"/\u003e \u003cimg src=\"https://github.com/IGNF/roc4j/blob/master/doc/images/ks5.png\" width=\"200\"/\u003e \u003cimg src=\"https://github.com/IGNF/roc4j/blob/master/doc/images/auc.png\" width=\"200\"/\u003e \u003cimg src=\"https://github.com/IGNF/roc4j/blob/master/doc/images/fixedwidthband.png\" width=\"200\"/\u003e\n\u003cimg src=\"https://github.com/IGNF/roc4j/blob/master/doc/images/bandsNonSmoothed.png\" width=\"200\"/\u003e  \u003cimg src=\"https://github.com/IGNF/roc4j/blob/master/doc/images/rainbow.png\" width=\"200\"/\u003e \u003cimg src=\"https://github.com/IGNF/roc4j/blob/master/doc/images/errorbars3.png\" width=\"200\"/\u003e   \u003cimg src=\"https://github.com/IGNF/roc4j/blob/master/doc/images/smoothing4.png\" width=\"200\"/\u003e   \n\n\u003cimg src=\"https://github.com/IGNF/roc4j/blob/master/doc/images/space.png\" width=\"5\"/\u003e \u003cimg src=\"https://github.com/IGNF/roc4j/blob/master/doc/images/scoringSpace.png\" height=\"185\"/\u003e \u003cimg src=\"https://github.com/IGNF/roc4j/blob/master/doc/images/space.png\" width=\"22\"/\u003e \u003cimg src=\"https://github.com/IGNF/roc4j/blob/master/doc/images/oscillo.png\" height=\"185\"/\u003e \u003cimg src=\"https://github.com/IGNF/roc4j/blob/master/doc/images/space.png\" width=\"23\"/\u003e \u003cimg src=\"https://github.com/IGNF/roc4j/blob/master/doc/images/customized.png\" height=\"185\"/\u003e\n\n\u003cimg src=\"https://github.com/IGNF/roc4j/blob/master/doc/images/multiple.png\" height=\"220\"/\u003e \u003cimg src=\"https://github.com/IGNF/roc4j/blob/master/doc/images/space.png\" width=\"20\"/\u003e \u003cimg src=\"https://github.com/IGNF/roc4j/blob/master/doc/images/scoring1.png\" height=\"230\"/\u003e \n\n\n\u003cbr\u003e\n\n### Installation\n------------------------\n\n#### Option 1: \n\nDownload roc4j jar file at:\n\nhttps://forge-cogit.ign.fr/nexus/content/repositories/snapshots/fr/ign/cogit/roc4j/1.0-SNAPSHOT/\n\n\n#### Option 2: \n\nInsert the following lines in your Maven pom.xml:\n\n```xml\n\u003cdependency\u003e\n\t\u003cgroupId\u003efr.ign.cogit\u003c/groupId\u003e\n\t\u003cartifactId\u003eroc4j\u003c/artifactId\u003e\n\t\u003cversion\u003e1.0-SNAPSHOT\u003c/version\u003e\n\u003c/dependency\u003e\n```\n\n```xml\n\u003crepository\u003e\n\t\u003cid\u003ecogit-snapshots\u003c/id\u003e\n\t\u003cname\u003eCogit Snapshots Repository\u003c/name\u003e\n\t\u003curl\u003ehttps://forge-cogit.ign.fr/nexus/content/repositories/snapshots/\u003c/url\u003e\n\u003c/repository\u003e\n```\n\n\n\u003cbr\u003e\n\n### Tutorial\n---------\n\nA complete tutorial on how to use roc4j may be found online at the following address:\n\nhttp://recherche.ign.fr/labos/cogit/demo/roc4j-doc/index.html\n\nOr in PDF version: https://github.com/IGNF/roc4j/blob/master/doc/roc4j-doc.pdf\n\n\n\u003cbr\u003e\n\n### Quick start\n----------\n\n```java\nimport java.util.Random;\nimport javax.swing.JFrame;\n\nimport fr.ign.cogit.roc4j.RocSpace;\nimport fr.ign.cogit.roc4j.RocSpaceStyle;\nimport fr.ign.cogit.roc4j.ConfidenceBands;\nimport fr.ign.cogit.roc4j.ReceiverOperatingCharacteristics;\n\n//-----------------------------------------------------------------\n// Program to compute a simple ROC curve on n simulated instances\n// Confidence bands are computed at 95% level (default parameter)\n// with Komogorov-Smirnov test statistic. ROC curve and its \n// associated confidence bands are then depicted in a ROC space \n//-----------------------------------------------------------------\n\npublic class Main {\n\n\tpublic static void main(String[] args) {\n\t\t\n\t\t// --------------------------------------------------------\n\t\t// Parameters\n\t\t// --------------------------------------------------------\n\t\tint n = 500;              // Number of validation instances\n\t\tdouble noise = 0.1;       // Standard deviation of noise\n\t\t// --------------------------------------------------------\n\t\t\n\t\t// Setting random seed\n\t\tRandom generator = new Random(123456789);\n\n\t\tint[] expected = new int[n];\n\t\tdouble[] score= new double[n];\n\n\t\t// Instances generation\n\t\tfor (int i=0; i\u003cn; i++){\n\n\t\t    double rand1 = generator.nextDouble();\n\t\t    double rand2 = generator.nextGaussian();\n\n\t\t    expected[i] = (int)(rand1+0.5);\n\t\t    score[i] = noise*rand2 + 0.2*expected[i] + 0.4;\n\n\t\t    score[i] = Math.max(score[i], 0);\n\t\t    score[i] = Math.min(score[i], 1);\n\n\t\t}\n\n\t\t// Roc curve computation\n\t\tReceiverOperatingCharacteristics roc = new ReceiverOperatingCharacteristics(expected, score);\n\t\t\n\t\t// Confidence bands computation with Kolmogorov-Smirnov method\n\t\tConfidenceBands bands = new ConfidenceBands(roc, ConfidenceBands.METHOD_KOLMOGOROV_SMIRNOV);\n\t\t\n\t\t// Creating ROC space\n\t\tRocSpace space = new RocSpace();\n\t\tspace.setStyle(RocSpaceStyle.STYLE_OSCILLO);\n\n\t\tspace.addRocCurve(roc);\t\t\n\t\tspace.addConfidenceBands(bands);\n\n\t\t// Graphical display\n\t\tJFrame fen = new JFrame();\n\t\tfen.setSize(700, 700);\n\t\tfen.setContentPane(space);\n\t\tfen.setLocationRelativeTo(null);\n\t\tfen.setDefaultCloseOperation(JFrame.EXIT_ON_CLOSE);\n\t\tfen.setVisible(true);\n\t\t\n\t}\n\n}\n```\n\n\u003cbr\u003e\n\n### Real Application - Castle detection from building database with Random Forest\n--------\n\n\u003cbr\u003e\n\n**Authors:** Marie-Dominique Van Damme and Yann Méneroux\n\n\u003cbr\u003e\n\nThe data used for this experimentation may be found on the same github repository:\n\nhttps://github.com/IGNF/roc4j/blob/master/sample/chateau.dat\n\n\u003cbr\u003e\n\n#### Goal\n\nThe objective of this experimentation is to discriminate castle and non-castle in a building database.\nThe dataset is an extract of the data we used, containing 600 buildings, 50% of them being castles. \n\nThe first line of the file contains header with parameter names. First columns contains boolean value (1 if the building described on the row is a castle, \n0 otherwise). \n\nEach row contains 13 parameters, which have been computed from BDTOPO\u0026copy; (IGN building database):\n\n\u003cimg align=\"right\" src=\"https://github.com/IGNF/roc4j/blob/master/doc/images/BatiParameters.png\" width=\"350\"/\u003e\n\n\u003cbr/\u003e\n\n1 - **hauteur** : height of the building\n\n2 - **nb_orientation_mur** : number of wall orientations\n\n3 - **orientation_generale** : general orientation of building\n\n4 - **orientation_principale_mur** : main orientation of building walls\n\n5 - **elongation** : length/width ratio of minimum bounding rectangle\n\n6 - **concavite** : area of footprints out of convex hull\n\n7 - **perimetre** : perimeter of footprint\n\n8 - **nb_convexe** : number of parts in convex decomposition\n\n9 - **compacite** : compacity index\n\n10 - **granularite** : shortest wall \n\n11 - **nb_concave** : number of concave parts\n\n12 - **nb_pt_squelette** : number of points in geometric skeletton\n\n13 - **long_squelette** : geometric skeletton length\n\n\u003cbr/\u003e\n\nClassification has been done with a Random Forest model (100 trees), using SMILE library.\n\nThe code provided below:  \n- reads the castle datafile (needs to be downloaded in local first)  \n- stores features and labels data in a DataSet object  \n- creates a ClassifierModel, wrapping SMILE RandomForest object  \n- designs a protocol for validation (15-fold cross validation)  \n- validates the classifier on the dataset and computes ROC curve  \n- performs boostrap to generate 20 replications of ROC curve  \n- computes smoothed version of the ROC curves with kernel estimation  \n- computes the average of all generated roc curves  \n- estimates ROC curve confidence bands at 95% with Fixed-Width Band method  \n- computes area under ROC curve (AUC) as a general performance index  \n- estimates the 95% confidence interval of AUC  \n- Displays ROC curves, confidence bands and numerical results in a plot  \n\nNote that the computation of confidence bands with Fixed-Width Band method require a few seconds (dependind upon the number of ROC curves generated by \nbootstrap sampling).\n\n\u003cbr\u003e\n\n```java\n\npackage \u003cpackage_name\u003e;\n\nimport javax.swing.JFrame;\n\nimport smile.classification.RandomForest;\n\nimport java.awt.Color;\nimport java.io.File;\nimport java.io.FileNotFoundException;\nimport java.util.ArrayList;\nimport java.util.Scanner;\nimport java.util.StringTokenizer;\n\nimport fr.ign.cogit.roc4j.core.ConfidenceBands;\nimport fr.ign.cogit.roc4j.core.ReceiverOperatingCharacteristics;\nimport fr.ign.cogit.roc4j.core.RocCurvesCollection;\nimport fr.ign.cogit.roc4j.graphics.RocSpace;\nimport fr.ign.cogit.roc4j.utils.Tools;\nimport fr.ign.cogit.roc4j.validation.Bootstrap;\nimport fr.ign.cogit.roc4j.validation.ClassifierModel;\nimport fr.ign.cogit.roc4j.validation.DataSet;\nimport fr.ign.cogit.roc4j.validation.ValidationProcess;\n\npublic class Main {\n\n\tpublic static void main(String[] args) {\n\n\t\t// Data labels and features\n\t\tDataSet dataset = new DataSet();\n\n\t\t// Data path in local\n\t\tString datafile_path = \"D:/workspace/roc4j/sample/chateau.dat\";\n\n\t\t // Number of features (max 13)\n\t\tint Nf = 13;  \n\t\t\n\t\t// Confidence level\n\t\tdouble level = 95.0;  \n\t\t\n\t\t// ----------------------------------------------------------------------\n\t\t// Reading building dataset\n\t\t// ----------------------------------------------------------------------\n\n\t\ttry {\n\n\t\t\tScanner scan = new Scanner(new File(datafile_path));\n\n\t\t\t// Headline\n\t\t\tscan.nextLine();\n\n\t\t\t// Number of features\n\n\t\t\twhile(scan.hasNextLine()){\n\n\t\t\t\t// Splitting line\n\t\t\t\tStringTokenizer splitter = new StringTokenizer(scan.nextLine(), \",\");\n\n\t\t\t\t// Data labels\n\t\t\t\tint y = Integer.parseInt(splitter.nextToken());\n\n\t\t\t\t// Data features\n\n\t\t\t\tdouble x[] = new double[Nf];\n\n\t\t\t\tfor (int i=0; i\u003cNf; i++){\n\n\t\t\t\t\tx[i] = Double.parseDouble(splitter.nextToken());\n\n\t\t\t\t}\n\n\t\t\t\tdataset.addData(x, y);\n\n\t\t\t}\n\n\t\t} catch (FileNotFoundException e) {\n\n\t\t\tSystem.out.println(\"No datafile found at the specified address\");\n\t\t\tSystem.exit(1);\n\n\t\t}\n\n\n\n\t\t// ----------------------------------------------------------------------\n\t\t// Training a classifier model\n\t\t// ----------------------------------------------------------------------\n\n\t\tClassifierModel classifier = new ClassifierModel() {\n\n\t\t\t@Override\n\t\t\tpublic void train(DataSet trainingData) {\n\n\t\t\t\tint n = trainingData.getSize();\n\n\t\t\t\tdouble[][] X = new double[n][4];\n\t\t\t\tint[] Y = new int[n];\n\n\t\t\t\tfor (int i=0; i\u003cX.length; i++){\n\n\t\t\t\t\tX[i] = (double[]) trainingData.getFeatures(i);\n\t\t\t\t\tY[i] = trainingData.getTarget(i);\n\n\t\t\t\t}\n\n\t\t\t\tthis.model = new RandomForest(X, Y, 100);\n\n\n\t\t\t}\n\n\t\t\t@Override\n\t\t\tpublic double posterior(Object dataFeatures) {\n\n\t\t\t\tRandomForest rf = (RandomForest) this.model;\n\n\t\t\t\tdouble[] posterior = new double[2];\n\n\t\t\t\trf.predict((double[]) dataFeatures, posterior);\n\n\t\t\t\treturn posterior[1];\n\n\t\t\t}\n\t\t\t\n\t\t};\n\n\n\t\t// ----------------------------------------------------------------------\n\t\t// Validation of the classifier model\n\t\t// ----------------------------------------------------------------------\n\n\t\t// 15-fold cross validation\n\t\tValidationProcess validation = new ValidationProcess(dataset, classifier, ValidationProcess.METHOD_CROSS_VALIDATION);\n\t\tvalidation.setNumberOfFolds(15);\n\t\t\n\t\tReceiverOperatingCharacteristics roc = validation.run();\n\n\t\t// Replication of 20 bootstrap samples\n\t\tRocCurvesCollection ROCS = Bootstrap.sample(roc, 20);\n\t\t\n\t\t// Processing generated curves\n\t\tROCS.smooth(ReceiverOperatingCharacteristics.SMOOTH_KERNEL);\n\t\tROCS.setThickness(0.2f);\n\t\t\t\n\t\t// Confidence bands computation at level % with FWB method\n\t\tConfidenceBands bands = new ConfidenceBands(ROCS, ConfidenceBands.METHOD_FIXED_WIDTH_BAND, level, 1);\n\t\tbands.getCentralROC().setThickness(2.f);\n\t\tbands.setBordersTransparency(0.5f);\n\t\t\n\n\t\t// ----------------------------------------------------------------------\n\t\t// Area Under Curve computation\n\t\t// ----------------------------------------------------------------------\n\t\t\n\t\t// Areas under bootstrapped curves computation\n\t\tArrayList\u003cDouble\u003e AUC = ROCS.computeAreasUnderCurves();\n\t\t\n\t\t// Average of area under curve\n\t\tdouble auc = Tools.round(100.0*Tools.computeMean(AUC), 1);\n\t\t\n\t\t// Confidence interval on area under curve\n\t\tdouble[] confidence_interval = Tools.computeConfidenceInterval(AUC, level);\n\t\tdouble conf_inf = Tools.round(100.0*confidence_interval[0], 1);\n\t\tdouble conf_sup = Tools.round(100.0*confidence_interval[1], 1);\n\t\t\n\n\t\t// ----------------------------------------------------------------------\n\t\t// Data representation\n\t\t// ----------------------------------------------------------------------\n\n\t\tRocSpace space = new RocSpace();\n\n\t\t// Adding data to graphics\n\t\tspace.addRocCurve(bands.getCentralROC());\n\t\tspace.addRocCurve(ROCS);\n\t\tspace.addConfidenceBands(bands);\n\t\t\n\t\t// Writing on graphics\n\t\tspace.setTitle(\"ROC curve of castle detection from building database\");\n\t\tspace.writeText(\"AUC = \"+auc+\" %\", 400, 500, 14, Color.BLACK);\n\t\tspace.writeText(\"IC @ 95% = [\"+conf_inf+\", \"+conf_sup+\"]\", 400, 520, 14, Color.BLACK);\n\n\t\t// Display\n\t\tJFrame fen = new JFrame();\n\t\tfen.setSize(700, 700);\n\t\tfen.setContentPane(space);\n\t\tfen.setLocationRelativeTo(null);\n\t\tfen.setDefaultCloseOperation(JFrame.EXIT_ON_CLOSE);\n\t\tfen.setVisible(true);\n\n\t}\n\n}\n\n```\n\nIt is possible to try the code above with different number of features, confidence level, bootstrap replication number, SMILE classifier model, \nsmoothing method, validation process... and observe the change in plot and results.\n\nFor the default code provided here, we got an AUC equal to 86.2 %. Confidence interval indicated that the true unknown AUC is somewhere in [84.6%, 88.2%].\nIncreasing the number of buildings may enable to decrease the uncertainty.\n\n\u003cp align=\"center\"\u003e \n\u003cimg src=\"https://github.com/IGNF/roc4j/blob/master/doc/images/castle.png\" width=\"600\"/\u003e\n\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e \n\u003ci\u003eReceiver Operating Characteristics curves (after bootstrap sampling) and 95% confidence bands for Random Forest classifier on castle detection problem\u003c/i\u003e\n\u003c/p\u003e\n\nNote that adding some extrinsic parameters (such as distance to the nearest road) enables to reach up to 94% classification performance (AUC). However, in order to get \naesthetic ROC curves, we did not provide all the features in the dataset.\n\n\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fignf%2Froc4j","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fignf%2Froc4j","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fignf%2Froc4j/lists"}