{"id":18609581,"url":"https://github.com/secure-software-engineering/achilles-benchmark-depscanners","last_synced_at":"2025-04-10T22:31:36.415Z","repository":{"id":39805015,"uuid":"244321091","full_name":"secure-software-engineering/achilles-benchmark-depscanners","owner":"secure-software-engineering","description":"Achilles - Benchmark for assessing OSS-Vulnerability Scanners 59","archived":false,"fork":false,"pushed_at":"2023-09-28T08:57:25.000Z","size":31998,"stargazers_count":7,"open_issues_count":11,"forks_count":4,"subscribers_count":2,"default_branch":"master","last_synced_at":"2024-05-21T06:10:15.388Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"lgpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/secure-software-engineering.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-03-02T08:44:30.000Z","updated_at":"2023-07-12T01:59:39.000Z","dependencies_parsed_at":"2023-01-29T13:45:50.540Z","dependency_job_id":null,"html_url":"https://github.com/secure-software-engineering/achilles-benchmark-depscanners","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/secure-software-engineering%2Fachilles-benchmark-depscanners","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/secure-software-engineering%2Fachilles-benchmark-depscanners/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/secure-software-engineering%2Fachilles-benchmark-depscanners/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/secure-software-engineering%2Fachilles-benchmark-depscanners/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/secure-software-engineering","download_url":"https://codeload.github.com/secure-software-engineering/achilles-benchmark-depscanners/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":223449847,"owners_count":17146984,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-07T03:06:30.887Z","updated_at":"2024-11-07T03:06:31.614Z","avatar_url":"https://github.com/secure-software-engineering.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Achilles - Test Suite for assessing OSS-Vulnerability Scanners\n\nAchilles is an open test suite for evaluating the performance of open-source vulnerability scanners specifically for *Java* and *Maven* applications. \n\n\nAchilles was created by Andreas Dann (1), Henrik Plate (2), Ben Hermann (3), Serena Elisa (2) Ponta,  and Eric Bodden (1) of the security research group at Paderborn University (1),  SAP Security Research Mougins, France (2), and the secure software engineering group at Technical University Dortmund (3).\n\n\n\n\n\n## Achilles is the right benchmark for you, if you are\n* Using open-source libraries and frameworks,\n* Using the build-automation system *Maven* to manage project dependencies,\n* Worried about (potential) vulnerabilities in the libraries you use,\n* Comparing the performance of depenendy scanners,\n* Deciding what tools to use for checking your open-source libraries for vulnerabilities,\n\n\n\n## What is included\n\n\n\n### Test Cases\n\nThe folder `detection` contains 2505 test fixtures to measure the performance of open-source vulnerability scanner for detecting the inclusion of known vulnerable dependencies.\nThe test fixtures are in the form of json files following the naming scheme: `\u003cCVE\u003e_\u003cGAV\u003e.json.`.\n\nThe test fixtures contain true and false positive warnings raised by the tools [Eclipse Steady](https://github.com/eclipse/steady), [OWASP Dependency-Check](https://github.com/jeremylong/DependencyCheck), and a commercial open-source dependency vulnerability scanner.\n\n\n\n#### Stats\nThe test cases cover *251* distinct CVEs, and *534* distinct \n\nAn overview of the included artifacts is given in the table below\n\n|  GroupId:ArtifactId                                            | #Num|\n|:---------------------------------------------------------------|----:|\n| ('axis', 'axis')                                               |   3 |\n| ('bouncycastle', 'bcprov-jdk14')                               |  13 |\n| ('com.fasterxml.jackson.core', 'jackson-databind')             | 606 |\n| ('com.fasterxml.jackson.dataformat', 'jackson-dataformat-xml') |  77 |\n| ('com.google.guava', 'guava')                                  |  25 |\n| ('com.google.guava', 'guava-bootstrap')                        |   2 |\n| ('com.google.protobuf', 'protobuf-java')                       |   1 |\n| ('com.itextpdf', 'itextpdf')                                   |   1 |\n| ('com.squareup.okhttp', 'okhttp')                              |   2 |\n| ('commons-beanutils', 'commons-beanutils')                     |   6 |\n| ('commons-collections', 'commons-collections')                 |  11 |\n| ('commons-fileupload', 'commons-fileupload')                   |  10 |\n| ('commons-httpclient', 'commons-httpclient')                   |   1 |\n| ('dom4j', 'dom4j')                                             |   1 |\n| ('io.netty', 'netty-all')                                      |  12 |\n| ('io.undertow', 'undertow-core')                               |   9 |\n| ('jasperreports', 'jasperreports')                             |   4 |\n| ('javax.el', 'el-api')                                         |   1 |\n| ('javax.el', 'javax.el-api')                                   |   2 |\n| ('javax.faces', 'javax.faces-api')                             |   2 |\n| ('javax.servlet.jsp', 'jsp-api')                               |   1 |\n| ('net.sf.jasperreports', 'jasperreports')                      |  14 |\n| ('ognl', 'ognl')                                               |   1 |\n| ('org.apache.axis', 'axis-saaj')                               |   3 |\n| ('org.apache.commons', 'commons-compress')                     |   1 |\n| ('org.apache.httpcomponents', 'httpasyncclient')               |   1 |\n| ('org.apache.httpcomponents', 'httpclient')                    |  32 |\n| ('org.apache.lucene', 'lucene-queryparser')                    |   1 |\n| ('org.apache.poi', 'poi')                                      |  37 |\n| ('org.apache.poi', 'poi-ooxml')                                |   3 |\n| ('org.apache.tomcat.embed', 'tomcat-embed-core')               |  29 |\n| ('org.beanshell', 'bsh')                                       |   2 |\n| ('org.bouncycastle', 'bcprov-jdk14')                           |   5 |\n| ('org.bouncycastle', 'bcprov-jdk15on')                         |   3 |\n| ('org.codehaus.castor', 'castor')                              |   1 |\n| ('org.codehaus.groovy', 'groovy')                              |   2 |\n| ('org.codehaus.groovy', 'groovy-all')                          |  17 |\n| ('org.codehaus.groovy', 'groovy-xml')                          |   1 |\n| ('org.eclipse.jetty', 'jetty-http')                            |  97 |\n| ('org.eclipse.jetty', 'jetty-security')                        |  15 |\n| ('org.eclipse.jetty', 'jetty-server')                          |  71 |\n| ('org.eclipse.jetty', 'jetty-servlet')                         |  19 |\n| ('org.eclipse.jetty', 'jetty-util')                            |  38 |\n| ('org.hibernate', 'com.springsource.org.hibernate.validator')  |   2 |\n| ('org.hibernate', 'hibernate-validator')                       |   7 |\n| ('org.jruby', 'jruby')                                         |   9 |\n| ('org.jruby', 'jruby-core')                                    |  11 |\n| ('org.jruby', 'openssl')                                       |  94 |\n| ('org.jruby', 'readline')                                      |   3 |\n| ('org.jruby', 'ripper')                                        |  11 |\n| ('org.jruby', 'yecht')                                         |   6 |\n| ('org.jruby.ext.posix', 'jnr-posix')                           |   3 |\n| ('org.jruby.extras', 'bytelist')                               |  12 |\n| ('org.jruby.extras', 'jffi')                                   |   3 |\n| ('org.jruby.util', 'bytelist')                                 |   3 |\n| ('org.jsoup', 'jsoup')                                         |   1 |\n| ('org.slf4j', 'slf4j-api')                                     |   1 |\n| ('org.springframework', 'spring-asm')                          |  62 |\n| ('org.springframework', 'spring-core')                         | 413 |\n| ('org.springframework', 'spring-expression')                   |  92 |\n| ('org.springframework', 'spring-jcl')                          |  36 |\n| ('org.springframework', 'spring-oxm')                          |  27 |\n| ('org.springframework', 'spring-tx')                           |   1 |\n| ('org.springframework', 'spring-web')                          | 104 |\n| ('org.springframework', 'spring-webmvc')                       |  33 |\n| ('p2.eclipse-plugin', 'org.apache.axis')                       |   1 |\n| ('rubygems', 'jruby-openssl')                                  | 438 |\n| ('stax', 'stax')                                               |   1 |\n| ('taglibs', 'standard')                                        |   1 |\n\n\n\n### Test Generator\nThe generator creates, based on the test fixtures, Maven/Java projects that can be used as an input for open-source vulnerability scanner.\nEach test fixtures specifies in the element `vulnerable` if the warnings is a true or false positive.\n\n\n\n\u003cp align=\"center\"\u003e\n  \u003cimg width=\"75%\" src=\"achilles_overview_testcases.png\"\u003e\n\u003c/p\u003e\n\n\n## Publications \n\nIf you want to read the details on how Achilles works, and the basis on which the test suite was created, our IEEE TSE paper [Identifying Challenges for OSS Vulnerability Scanners - A Study \u0026 Test Suite\n](https://ieeexplore.ieee.org/document/9506931) is a good place to start.\n\n\n\n\n## We welcome your contributions!\nYou are most welcome to contribute additional test cases to Achilles. \nTo do so, please fork the project, commit an appropriate test fixture, update this README and then send us a pull request.\n\n\n\n## License\nAchilles is licensed under the LGPLv3 license, see LICENSE file. This basically means that you are free to use the tool (even in commercial, closed-source projects). However, if you extend or modify the tool, you must make your changes available under the LGPLv3 as well. This ensures that we can continue to improve the tool as a community effort.\n\n\n## Contact\n\nIf you experience any issues, you can ask for help on GitHub issue board. You can also contact us at andreas.dann@uni-paderborn.de\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsecure-software-engineering%2Fachilles-benchmark-depscanners","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsecure-software-engineering%2Fachilles-benchmark-depscanners","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsecure-software-engineering%2Fachilles-benchmark-depscanners/lists"}