{"id":42352700,"url":"https://github.com/cqframework/cql-tests-runner","last_synced_at":"2026-04-03T00:17:05.269Z","repository":{"id":233310587,"uuid":"786660274","full_name":"cqframework/cql-tests-runner","owner":"cqframework","description":"Test Runner for CQL Tests","archived":false,"fork":false,"pushed_at":"2026-01-23T20:40:30.000Z","size":694,"stargazers_count":4,"open_issues_count":23,"forks_count":2,"subscribers_count":7,"default_branch":"main","last_synced_at":"2026-01-24T09:41:08.784Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/cqframework.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2024-04-15T04:03:11.000Z","updated_at":"2026-01-23T20:40:31.000Z","dependencies_parsed_at":"2024-04-15T18:26:16.988Z","dependency_job_id":"553a1e70-1039-46ec-8646-9eb88b319d04","html_url":"https://github.com/cqframework/cql-tests-runner","commit_stats":null,"previous_names":["cqframework/cql-tests-runner"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/cqframework/cql-tests-runner","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cqframework%2Fcql-tests-runner","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cqframework%2Fcql-tests-runner/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cqframework%2Fcql-tests-runner/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cqframework%2Fcql-tests-runner/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/cqframework","download_url":"https://codeload.github.com/cqframework/cql-tests-runner/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cqframework%2Fcql-tests-runner/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28816557,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-27T12:25:15.069Z","status":"ssl_error","status_checked_at":"2026-01-27T12:25:05.297Z","response_time":168,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-01-27T16:09:23.216Z","updated_at":"2026-04-03T00:17:05.260Z","avatar_url":"https://github.com/cqframework.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# cql-tests-runner\n\n[![Website](https://shields.foundry.hl7.org/website?url=https%3A%2F%2Fcql-tests-runner.quality.hl7.org\u0026logo=fireship\u0026label=try%20it%20now)](https://cql-tests-runner.quality.hl7.org)\n[![GitHub contributors](https://shields.foundry.hl7.org/github/contributors/cqframework/cql-tests-runner?logo=github)](https://github.com/cqframework/cql-tests-runner/graphs/contributors)\n[![GitHub last commit](https://shields.foundry.hl7.org/github/last-commit/cqframework/cql-tests-runner?logo=github)](https://github.com/cqframework/cql-tests-runner/graphs/commit-activity)\n[![GitHub top language](https://shields.foundry.hl7.org/github/languages/top/cqframework/cql-tests-runner?logo=github)](https://github.com/cqframework/cql-tests-runner)\n[![Docker automated build](https://shields.foundry.hl7.org/docker/automated/hlseven/quality-cql-tests-runner?logo=docker)](https://hub.docker.com/r/hlseven/quality-cql-tests-runner)\n[![Docker pulls](https://shields.foundry.hl7.org/docker/pulls/hlseven/quality-cql-tests-runner?logo=docker)](https://hub.docker.com/r/hlseven/quality-cql-tests-runner)\n[![Docker image size](https://shields.foundry.hl7.org/docker/image-size/hlseven/quality-cql-tests-runner?logo=docker)](https://hub.docker.com/r/hlseven/quality-cql-tests-runner)\n\n\nTest Runner for the [CQL Tests](https://github.com/cqframework/cql-tests) repository. This node application allows you to run the tests in the CQL Tests repository against a server of your choice using the [$cql](https://hl7.org/fhir/uv/cql/OperationDefinition-cql-cql.html) operation. The runner in its current state uses only this operation, and there is no expectation of any other FHIR server capability made by this runner. Additional capabilities may be required in the future as we expand the runner to support full Library/$evaluate as well. None of the tests in the repository have any expectation of being able to access data (i.e. the tests have no retrieve expressions).\n\nThe application runs all the tests in the repository and outputs the results as a JSON file in the `results` directory. If the output directory does not exist, it will be created.\n\nResults output from running these tests can be posted to the [CQL Tests Results](https://github.com/cqframework/cql-tests-results) repository.\n\n## Setting up the Environment\n\nThis application requires Node v25 and makes use of the [Axios](https://axios-http.com/docs/intro) framework for HTTP request/response processing. [Node Download](https://nodejs.org/en/download)\n\nInstall application dependencies using\n\n```sh\nnpm install\n```\n\n### CQL Tests Submodule\n\nThe cql-tests folder has been added as a submodule. After pulling, you'll find a cql-tests folder inside cql-tests-runner. However, when you peek inside that folder, depending on your Git version, you might see nothing. Newer versions of Git will handle this automatically, but older versions may require you to explicitly instruct Git to download the contents of cql-tests.\n\n```bash\ngit submodule update --init --recursive\n```\n\n### Configuration Settings\n\nConfiguration settings are set in a JSON configuration file. The file `conf/localhost.json` provides a sample configuration.\n\n```json\n{\n  \"FhirServer\": {\n    \"BaseUrl\": \"https://fhirServerBaseUrl\",\n    \"CqlOperation\": \"$cql\"\n  },\n  \"Build\": {\n    \"CqlFileVersion\": \"1.0.000\",\n    \"CqlOutputPath\": \"./cql\",\n    \"testsRunDescription\": '',\n    \"testsRunDescription\": \"Local host test run\",\n    \"cqlTranslator\": \"Java CQFramework Translator\",\n    \"cqlTranslatorVersion\": \"Unknown\",\n    \"cqlEngine\": \"Java CQFramework Engine\",\n    \"cqlEngineVersion\": \"4.1.0\"\n  },\n  \"Tests\": {\n    \"ResultsPath\": \"./results\",\n    \"SkipList\": []\n  },\n  \"Debug\": {\n    \"QuickTest\": true\n  }\n}\n```\n\nTo skip tests, add entries to the `SkipList` with the corresponding `testsName`, `groupName`, `testName`, and `reason`.\n\n```jsonc\n{\n  \"FhirServer\": {/* omitted */},\n  \"Build\": {/* omitted */},\n  \"Tests\": {\n    \"ResultsPath\": \"./results\",\n    \"SkipList\": [\n      {\n        \"testsName\": \"CqlAggregateTest\",\n        \"groupName\": \"AggregateTests\",\n        \"testName\": \"RolledOutIntervals\",\n        \"reason\": \"CQLtoELM - Could not resolve identifier MedicationRequestIntervals in the current library\"\n      },\n      // add more tests to skip as necessary...\n    ]\n  },\n  \"Debug\": {/* ommitted */}\n}\n```\n\nTo run only a specified set of tests (and skip all others), add entries to the `OnlyList` with the corresponding `testsName`, `groupName`, and `testName`.\n\n```jsonc\n{\n  \"FhirServer\": {/* omitted */},\n  \"Build\": {/* omitted */},\n  \"Tests\": {\n    \"ResultsPath\": \"./results\",\n    \"SkipList\": [],\n    \"OnlyList\": [\n      {\n        \"testsName\": \"CqlAggregateTest\",\n        \"groupName\": \"AggregateTests\",\n        \"testName\": \"RolledOutIntervals\"\n      },\n      // add more tests to only run as necessary...\n    ]\n  },\n  \"Debug\": {/* ommitted */}\n}\n```\n\nCreate your own configuration file and reference it when running the commands. You can use `conf/localhost.json` as a template for a new configuration with your own settings.\n\n### Running the tests\n\nThe CLI now requires a configuration file path as an argument. Run the tests with the following commands:\n\n#\n\n### Running from Source Code\n\nTo run the application directly from source:\n\n```bash\n# Install dependencies\nnpm install\n\n# Initialize the cql-tests submodule\ngit submodule update --init --recursive\n\n# Run commands directly from TypeScript source\nnpx tsx src/bin/cql-tests.ts run-tests conf/localhost.json ./results # Run CQL tests\nnpx tsx src/bin/cql-tests.ts run-tests conf/localhost.json ./results --quick # Run with quick test mode enabled\nnpx tsx src/bin/cql-tests.ts server                               # Run in server API mode\nnpx tsx src/bin/cql-tests.ts help                               # Hetailed command help\n\nnpx tsx src/bin/cql-tests.ts build-cql conf/localhost.json ./cql    # Unused legacy tool\n\n```\n\n### Running from Pre-Built OCI/Docker Image\n\nThe application is available as the pre-built image tag `hlseven/quality-cql-tests-runner:latest`.\n\n#### Using the Docker Image\n\nBy default, the image runs the CLI. When you bind in any local directories (such as configuration and results directories) you may use it as you would any other command line utility.\n\n```bash\n# Run CQL tests with a configuration file\ndocker run --rm -v $(pwd)/conf:/app/conf -v $(pwd)/results:/app/results \\\n  hlseven/quality-cql-tests-runner:latest run-tests conf/localhost.json ./results\n\n# Run CQL tests with quick test mode enabled\ndocker run --rm -v $(pwd)/conf:/app/conf -v $(pwd)/results:/app/results \\\n  hlseven/quality-cql-tests-runner:latest run-tests conf/localhost.json ./results --quick\n\n# Build CQL libraries (Unused)\ndocker run --rm -v $(pwd)/conf:/app/conf -v $(pwd)/cql:/app/cql \\\n  hlseven/quality-cql-tests-runner:latest build-cql conf/localhost.json ./cql\n\n# Start in REST server mode listening on port 3000.\ndocker run --rm -p 3000:3000 -v $(pwd)/conf:/app/conf \\\n  hlseven/quality-cql-tests-runner:latest server\n\n# Using host networking to test against a server running on the host machine\ndocker run --rm --network host -v $(pwd)/conf:/app/conf -v $(pwd)/results:/app/results \\\n  hlseven/quality-cql-tests-runner:latest run-tests conf/localhost.json ./results\n```\n\n#### Building the Docker Image\n\n```bash\n# Build the Docker image locally\ndocker build -t cql-tests-runner .\n\n# Build multi-platform image for distribution\ndocker buildx build --platform linux/arm64,linux/amd64 -t hlseven/quality-cql-tests-runner:latest .\n\n# Run with built image\ndocker run --rm -v $(pwd)/conf:/app/conf -v $(pwd)/results:/app/results hlseven/quality-cql-tests-runner:latest run-tests conf/localhost.json ./results\n\n# Using host networking with built image\ndocker run --rm --network host -v $(pwd)/conf:/app/conf -v $(pwd)/results:/app/results hlseven/quality-cql-tests-runner:latest run-tests conf/localhost.json ./results\n```\n\n#### Environment Variable Overrides\n\nYou can still override specific settings using environment variables:\n\n```sh\nexport SERVER_BASE_URL=http://fhirServerBaseEndpoint\nexport CQL_OPERATION=$cql\n```\n\n### Development Environment\n\nIf using vscode for development, below are some examples for running the tests with configuration files:\n\n```json\n{\n  \"type\": \"node\",\n  \"request\": \"launch\",\n  \"name\": \"Launch Build Command\",\n  \"skipFiles\": [\"\u003cnode_internals\u003e/**\"],\n  \"program\": \"${workspaceFolder}/src/bin/cql-tests.ts\",\n  \"args\": [\"build-cql\", \"conf/localhost.json\"],\n  \"runtimeArgs\": [\"--import\", \"tsx\"]\n},\n{\n  \"type\": \"node\",\n  \"request\": \"launch\",\n  \"name\": \"Launch Run Command\",\n  \"skipFiles\": [\"\u003cnode_internals\u003e/**\"],\n  \"program\": \"${workspaceFolder}/src/bin/cql-tests.ts\",\n  \"args\": [\"run-tests\", \"conf/localhost.json\"],\n  \"runtimeArgs\": [\"--import\", \"tsx\"],\n  \"env\": {\n    \"SERVER_BASE_URL\": \"http://localhost:3000\"\n  }\n}\n```\n\n### Server Command\n\nThe server command starts an HTTP server that provides a REST API for running CQL tests. This is mainly intended to be used by [CQL Tests UI](https://github.com/cqframework/cql-tests-ui)\n\n#### Starting the Server\n\n```bash\n# Using tsx (development mode)\nnpx tsx src/bin/cql-tests.ts server\n\n# Using Docker\ndocker run --rm -p 3000:3000 -v $(pwd)/conf:/app/conf \\\n  hlseven/quality-cql-tests-runner:latest server\n\n# Using Docker with host networking\ndocker run --rm --network host -v $(pwd)/conf:/app/conf \\\n  hlseven/quality-cql-tests-runner:latest server\n```\n\n#### Using the Server API\n\nThe server provides the following endpoints:\n\n- **GET /** - Server information and available endpoints\n- **POST /** - Run CQL tests with configuration in request body (synchronous)\n- **POST /jobs** - Create a new job to run CQL tests asynchronously\n- **GET /jobs/:id** - Get job status and results by job ID\n- **GET /health** - Health check endpoint\n\n#### Asynchronous Job Processing\n\nFor long-running test suites, the server supports asynchronous job processing:\n\n```bash\n# Create a job (returns immediately with job ID)\ncurl -X POST http://localhost:3000/jobs \\\n  -H \"Content-Type: application/json\" \\\n  -d @conf/localhost.json\n\n# Poll job status and results\ncurl http://localhost:3000/jobs/{job-id}\n```\n\nJobs support progress tracking and can be polled for status updates. The original synchronous endpoint (`POST /`) remains available for quick tests.\n\n#### Example Usage\n\n```bash\n# Start the server\nnpx tsx src/bin/cql-tests server --port 3000\n\n# In another terminal, run tests via synchronous execution API\ncurl -X POST http://localhost:3000/ \\\n  -H \"Content-Type: application/json\" \\\n  -d @conf/localhost.json \\\n  -o results.json\n\n# Check server health\ncurl http://localhost:3000/health\n```\n\nThe server accepts a configuration object in the request body and returns the test results as JSON.\n\n### MCP (Model Context Protocol) Support for AI and Agentic Clients\n\nThe server exposes MCP endpoints using Streamable HTTP transport, enabling AI agents to discover and interact with the CQL tests runner autonomously.\n\n#### Using the MCP Inspector\n\nTo explore the MCP features of your running server, use the [MCP Inspector](https://modelcontextprotocol.io/docs/tools/inspector) with Streamable HTTP transport:\n\n```bash\n# Run the inspector\nnpx @modelcontextprotocol/inspector --server-url http://localhost:3000/mcp --transport http\n```\n\nConnection Type should be set to \"Via Proxy\".\n\nThe inspector provides an interactive UI to browse test resources, view schemas, and invoke tools for running tests and managing jobs.\n\n### Unit Testing\n\nUnit testing is implemented with [Vitest](https://vitest.dev/). _This is for only testing of the cql-test-runner logic, and not for testing FHIR operations._\n\nTest cases are stored in the `test/` folder.\n\n##### Executing Unit Tests\n\nUnit tests can be run from the command line using the following\n\n```bash\nnpm run unit-tests\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcqframework%2Fcql-tests-runner","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcqframework%2Fcql-tests-runner","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcqframework%2Fcql-tests-runner/lists"}