{"id":18546572,"url":"https://github.com/saveourtool/save-cli","last_synced_at":"2025-04-09T20:31:03.430Z","repository":{"id":36988353,"uuid":"340654529","full_name":"saveourtool/save-cli","owner":"saveourtool","description":"Universal test framework for cli tools [mainly for code analyzers and compilers]","archived":false,"fork":false,"pushed_at":"2025-04-08T14:38:14.000Z","size":2985,"stargazers_count":43,"open_issues_count":71,"forks_count":4,"subscribers_count":4,"default_branch":"main","last_synced_at":"2025-04-08T15:39:41.582Z","etag":null,"topics":["compiler","compilers","kotlin","programming-languages","static-analysis","static-analyzers"],"latest_commit_sha":null,"homepage":"","language":"Kotlin","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/saveourtool.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-02-20T12:55:34.000Z","updated_at":"2024-12-09T22:50:53.000Z","dependencies_parsed_at":"2023-02-13T20:00:47.973Z","dependency_job_id":"e7422826-2463-4bfb-98bd-8bf0cf81dbc5","html_url":"https://github.com/saveourtool/save-cli","commit_stats":null,"previous_names":["cqfn/save"],"tags_count":24,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/saveourtool%2Fsave-cli","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/saveourtool%2Fsave-cli/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/saveourtool%2Fsave-cli/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/saveourtool%2Fsave-cli/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/saveourtool","download_url":"https://codeload.github.com/saveourtool/save-cli/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248107220,"owners_count":21048882,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["compiler","compilers","kotlin","programming-languages","static-analysis","static-analyzers"],"created_at":"2024-11-06T20:25:59.330Z","updated_at":"2025-04-09T20:31:02.165Z","avatar_url":"https://github.com/saveourtool.png","language":"Kotlin","readme":"![GitHub release (latest SemVer)](https://img.shields.io/github/v/release/saveourtool/save-cli)\n![Maven Central](https://img.shields.io/maven-central/v/com.saveourtool.save/save-common)\n![Lines of code](https://img.shields.io/tokei/lines/github.com/saveourtool/save-cli)\n[![Hits-of-Code](https://hitsofcode.com/github/saveourtool/save-cli?branch=main)](https://hitsofcode.com/github/saveourtool/save-cli/view?branch=main)\n![GitHub repo size](https://img.shields.io/github/repo-size/saveourtool/save-cli)\n[![Run deteKT](https://github.com/saveourtool/save-cli/actions/workflows/detekt.yml/badge.svg)](https://github.com/saveourtool/save-cli/actions/workflows/detekt.yml)\n[![Run diKTat](https://github.com/saveourtool/save-cli/actions/workflows/diktat.yml/badge.svg)](https://github.com/saveourtool/save-cli/actions/workflows/diktat.yml)\n[![Build and test](https://github.com/saveourtool/save-cli/actions/workflows/build_and_test.yml/badge.svg)](https://github.com/saveourtool/save-cli/actions/workflows/build_and_test.yml)\n[![License](https://img.shields.io/github/license/saveourtool/save-cli)](https://github.com/saveourtool/save-cli/blob/main/LICENSE)\n[![FOSSA Status](https://app.fossa.com/api/projects/git%2Bgithub.com%2Fsaveourtool%2Fsave-cli.svg?type=shield)](https://app.fossa.com/projects/git%2Bgithub.com%2Fsaveourtool%2Fsave-cli?ref=badge_shield)\n[![codecov](https://codecov.io/gh/saveourtool/save-api/branch/master/graph/badge.svg)](https://codecov.io/gh/saveourtool/save-api)\n\n\u003c!--\n# ![codebeat badge] TO DO\n# ![maintainability] TO DO\n--\u003e\n\nSave is an all-purpose command-line test framework that can be used for testing tools that work with code, \nsuch as static analyzers and compilers. It is a fully native application, requiring no need to install any SDK.\n\n## What is SAVE?\nStatic Analysis Verification and Evaluation (SAVE) is an ecosystem (also see [save-cloud](https://github.com/saveourtool/save-cloud)) designed for the evaluation, \ntesting, and certification of static analyzers, compilers or any other software tools. Instead of developing your own test \nframework, you can utilize SAVE as a command-line test application. The only requirement is to prepare test \nresources in the appropriate format.\n\n## Contribution\nWe need your help! We will be glad if you will use, test or contribute to this project.\nIn case you don't have much time for this - at least **give us a star** to attract other contributors! \nThanks! 🙏 🥳\n\n## Quick start: User scenarios\n### 1. Static analysis, warnings, sequentially\n\u003e - My code analysis tool processes files **sequentially, one by one**;\n\u003e - It produces **warnings** and outputs them to **stdout**;\n\u003e - I want to compare actual warnings with expected warnings that are specified **within the test resource code**.\n\n### 2. Static analysis, warnings, processing whole project\n\u003e - I also have code analysis tool, but it processes **the entire project** at once and is aware of all the code **relations**.;\n\u003e - It produces **warnings** and outputs them to **stdout**;\n\u003e - I want to compare actual warnings with expected warnings that are specified **within the test resource code**.\n\n### 3. Automated code fixing or generation\n\u003e - My tool **manipulates** the original code, for example, by auto-fixing it;\n\u003e - I would like to check how my tool **fixes the code** by comparing it with expected result;\n\u003e - Additionally, it can be used by compilers to validate **code generation**, **transitioning** from the original source \n\u003e - code to **intermediate representation** (IR), another programming language, or even assembly.\n\n### 4. Expected warnings in a separated file\n\u003e - I do not want to specify my expected warnings in code; \n\u003e - I prefer to use **a separate file** in SARIF or any other format.\n\n## How to Run\n1. Download [the latest release](https://github.com/saveourtool/save-cli) suitable for your OS and architecture.\n2. Set up and configure your test base in the correct SAVE format. Refer to [test_detection](#test_detection) and [plugins](#plugins) for guidance.\n3. Execute the following command (modify it according to your architecture and OS): `save \"/my/path/to/tests\"`\n\nEnsure the `tests` directory contains the `save.toml` configuration file.\n\n## SAVE Logging\n\nTo debug SAVE execution, you can use the following argument:\n`--log=TYPE`, where `TYPE` can be one of the following:\n\n- `all` - Comprehensive logging that includes all information from SAVE execution, even more detailed than DEBUG (akin to a trace).\n- `debug` - Displays results, warnings, and debug information.\n- `warnings` - Shows results and critical warnings.\n- `results_only` - Displays only the results.\n\n## Plugins with examples\n\n\u003cimg src=\"/readme/static-analysis-process.png\" width=\"500px\"/\u003e\n\nHere is a list of standard plugins:\n* [warn plugin](save-plugins/warn-plugin/README.md): This is for testing tools that detect issues in the source code and produce warnings.\n* [fix plugin](save-plugins/fix-plugin/README.md): This is used for testing static analyzer tools that modify text.\n* [fix-and-warn plugin](save-plugins/fix-and-warn-plugin/README.md): An optimization for scenarios where you want to correct a file and subsequently check for warnings that the tool couldn't address in a single run.\n\nIf you want multiple plugins to operate in your directory using the same test files (resources), simply add them all to the `save.toml` configuration:\n\n```text\n[general]\n...\n\n[fix]\n...\n\n[warn]\n...\n\n[other plugin]\n...\n```\n\n## Save warnings DSL \n![save-cli](https://user-images.githubusercontent.com/58667063/146390474-71e4921d-416b-4922-b2ea-894f71e491c3.jpg)\nYou can read more about the `warn plugin` [here](save-plugins/warn-plugin/README.md)\n\n## How to Configure\n\nSAVE has a command-line interface that allows you to run both the framework and your executable. Your main task is to configure the output of your static analyzer so that SAVE can verify whether the appropriate error was flagged at the correct line of the test code.\n\nTo ensure the warning is accurate for SAVE, your static analyzer must output the result either to stderr/stdout or a designated log file (for example in Sarif format).\n\nYou can configure SAVE's general behavior using command-line arguments or by using a configuration file named `save.properties`. This file should be located in the same directory as the root test config, `save.toml`.\n\nFor a comprehensive list of options that can be passed to SAVE via the command line or the `save.properties` file, refer to the [options table](/OptionsTable.md) or execute the `save --help` command. Please be aware that options with choices are case-sensitive.\n\nThe SAVE framework will automatically **detect** your tests, run your analyzer on them, calculate the pass rate, and return test results in the expected format.\n\n## \u003ca name=\"test_detection\"\u003e\u003c/a\u003e Test Detection and save.toml File\nTo enable SAVE to detect your test suites, you must place a `save.toml` file in each directory containing **test suites**. It's important to note that these configuration files inherit configurations from parent directories.\n\nAlthough most fields can be left undefined at lower levels and can inherit values from top levels, you should be cautious. \nSome fields in the `[general]` section are mandatory for execution, so you need to specify them in at least one config file in the inheritance chain for tests that are meant to run. \n[Check which fields are mandatory](#save_toml_configuration_file).\n\nFor instance, with the following directory hierarchy:\n```text\n| A\n  | save.toml\n  | B\n    | save.toml\n```\nThe `save.toml` in directory B will inherit settings and properties from directory A.\n\nBear in mind that SAVE will detect all files with the 'Test' postfix and will automatically utilize configurations from the `save.toml` file present in the same directory (or inherited from parent). \nTests are named according to the test file's resource name, excluding the 'Test' suffix. \nIf SAVE detects a file with the 'Test' postfix in the test resources and cannot locate any `save.toml` configurations in the **directory hierarchy**, it will throw an error.\n\nFor instance, the scenario below is invalid and will trigger an error, as the SAVE framework cannot locate the `save.toml` configuration file:\n```text\n| A\n  | B\n  | myTest.java\n```\n\nAs previously mentioned, the `save.toml` is essential for configuring tests. \n_Ideally_, there should be one configuration file for each directory containing tests, establishing a one-to-many relationship. \nWe refer to these directories as `test suites`. \n\nThe rationale behind having a single configuration file for one test suite is to avoid redundant configurations within the same test suite.\n\n## \u003ca name=\"save_toml_configuration_file\"\u003e\u003c/a\u003e save.toml Configuration File\n\nThe save configuration uses the [TOML](https://toml.io/en/) format powered by [ktoml](https://github.com/akuleshov7/ktoml) project.\nAs mentioned [above](#test_detection), `save.toml` can be inherited from the directory hierarchy (parent directories).\n\nThe configuration file contains a `[general]` table and a `[plugins]` table. For more information about plugins, refer to the [plugins section](#plugins).\n\nIn this section, we will provide information only about the `[general]` table, which can be used across all plugins.\n\n```text\n[general]\n# Your custom tags that will be used to detect groups of tests (required)\ntags = [\"parsing\", \"null-pointer\", \"etc\"]\n\n# Custom free text that describes the test suite (required)\ndescription = \"My suite description\"\n\n# Simple suite name (required)\nsuiteName = \"DocsCheck\", \"CaseCheck\", \"NpeTests\", \"etc\" \n\n# Execution command (required at least once in the configuration hierarchy)\n# By the default these binaries should be in the same directory of where SAVE is run \n# or should have full or relational path (root - is the directory with save executable)\nexecCmd=\"./ktlint -R diktat-0.4.2.jar\"\n\n# Excluded tests in the suite (optional). Here, you can list the names of excluded tests, separated by commas. By default, no tests are excluded.\n# To exclude tests, use the relative path to the root of the test project (to the root directory of `save.toml`)\nexcludedTests = [\"warn/chapter1/GarbageTest.kt\", \"warn/otherDir/NewTest.kt\", \"etc\"]\n\n# Command execution time for one test (in milliseconds)\ntimeOutMillis = 10000\n\n# Language for tests\nlanguage = \"Kotlin\"\n```\n\n## Executing Specific Tests\n\nAt times, you might want to execute only a specific set of tests instead of running all the tests under a particular `save.toml` config. \nTo achieve this, pass the relative path to the test file after all configuration options (root - is directory with save binary):\n\n```bash\n$ save [options] /path/to/tests/Test1\n```\n\nYou can also provide a list of relative paths to test files (separated by spaces):\n\n```bash\n$ save [options] /path/to/tests/Test1 /path/to/tests/Test2\n```\n\nSAVE will automatically detect the nearest `save.toml` file and use the configuration from it.\n\n`Note:` On Windows, remember to use a double backslash `\\\\` as the path separator.\n\n## SAVE Output\nSAVE supports several formats for test report output:\n- `PLAIN`: A markdown-like table showing all test results.\n- `PLAIN_FAILED`: Similar to `PLAIN`, but only displays failed tests.\n- `JSON`: Structured representation of the execution result.\n\nThe desired format can be selected using the `--report-type=PLAIN` option.\n\n## Purpose of Static Analysis Verification and Evaluation (SAVE) project\n## Purpose of Static Analysis Verification and Evaluation (SAVE) Project\n\n\u003cdetails\u003e\n\u003csummary\u003ePurpose of SAVE\u003c/summary\u003e\n\n### Intro\n\nThe use of [static analyzers](https://en.wikipedia.org/wiki/Static_program_analysis) is an integral part of the development process for every \nsoftware product. While software developers may write various tests and achieve good test coverage, human error remains inevitable. \nSuch errors can result in significant financial losses for companies. Static program analysis assists in identifying and rectifying bugs \nand issues that might not be detectable through compiler validations alone.\n\nStatic analysis comes in various forms and serves different purposes. It might involve a simple analysis using an AST \n(abstract syntax tree) or delve into more complex procedures like CFA (control-flow analysis), interprocedural analysis, \nor context-sensitive analysis. Static analyzers can assess code style, pinpoint potential runtime issues in application logic, \ndetect code smells, and suggest best practices. However, there remains a lack of clarity about the core functions of static analyzers. \nHow can their efficacy be quantified? What criteria determine their acceptance? What functionalities are essential for developers creating \na new analyzer? Despite years of static analyzer development, these questions remain largely unanswered.\n\n### Problematics\n\nAt the onset of their development journey, every creator of a static analyzer begins with identifying the kinds of issues \ntheir tool will target. This often necessitates a search for existing lists of potential issues or test packages that can \nguide the development process, particularly if following a TDD (test-driven development) approach. While other domains in \nsystem programming have established benchmarks and test sets, such as the [SPEC.org](http://spec.org/benchmarks.html) benchmarks \nused globally to evaluate various software and hardware components, no such standards exist for identifying issues in popular \nprogramming languages. While guidelines for coding in C/C++ have been established by [MISRA](https://www.misra.org.uk/), \nthere are no equivalents for widely used languages like Python and\n[JVM-languages](https://stackoverflow.com/questions/6050618/is-there-a-java-equivalent-to-misra-c). \nThere are test suites available at [NIST](https://samate.nist.gov/SRD/testsuite.php), but their framework and ecosystem are somewhat restrictive.\n\nGiven this scenario, developers often find themselves recreating mechanisms for static analysis or developing new test frameworks,\nleading to repetitive work. Some might opt for existing guidelines such as the [Google code style](https://google.github.io/styleguide/javaguide.html) \nor [PMD rules](https://pmd.github.io/), but regardless of the approach, significant time is invariably spent on conceptualizing, writing, \nand debugging tests.\n\n\u003c/details\u003e\n\n## Development\n### Build\nThe project uses Gradle as its build system and can be built using the command `./gradlew build`.\n\nTo compile native artifacts, you must install the prerequisites as described in the Kotlin/Native documentation.\n\nTo access dependencies hosted on the GitHub Package Registry, add the following to either `gradle.properties` or `~/.gradle/gradle.properties`:\n```properties\ngprUser=\u003cGH username\u003e\ngprKey=\u003cGH personal access token\u003e\n```\nA Personal Access Token can be generated at https://github.com/settings/tokens/new. Ensure the token has a scope that includes `read:packages`.\n\nDue to the generated code, you need to **run the build once** to correctly import the project into an IDE with resolved imports.\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsaveourtool%2Fsave-cli","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsaveourtool%2Fsave-cli","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsaveourtool%2Fsave-cli/lists"}