{"id":18965980,"url":"https://github.com/wso2/contract-test-runner","last_synced_at":"2025-04-19T14:10:55.801Z","repository":{"id":47478323,"uuid":"384128187","full_name":"wso2/contract-test-runner","owner":"wso2","description":"A library for contract testing","archived":false,"fork":false,"pushed_at":"2021-08-30T17:12:15.000Z","size":39,"stargazers_count":34,"open_issues_count":1,"forks_count":5,"subscribers_count":121,"default_branch":"main","last_synced_at":"2025-03-29T08:32:42.371Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/wso2.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2021-07-08T13:09:26.000Z","updated_at":"2024-03-31T14:25:05.000Z","dependencies_parsed_at":"2022-07-21T06:32:20.233Z","dependency_job_id":null,"html_url":"https://github.com/wso2/contract-test-runner","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wso2%2Fcontract-test-runner","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wso2%2Fcontract-test-runner/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wso2%2Fcontract-test-runner/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wso2%2Fcontract-test-runner/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/wso2","download_url":"https://codeload.github.com/wso2/contract-test-runner/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":249213731,"owners_count":21231096,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-08T14:34:28.546Z","updated_at":"2025-04-16T07:32:48.155Z","avatar_url":"https://github.com/wso2.png","language":"Java","funding_links":[],"categories":["测试"],"sub_categories":[],"readme":"### What are we going to do\n\nThis API testing library is based on the concept named consumer-based contract testing. With this test framework, we\nwill invoke some rest api calls to some live API endpoints and get the response to compare the attributes inside that\nresponse with a predefined response (The response we define inside the contract).\n\n1. We will create the contract file including a sample response from the API endpoint. Let’s name that response as R1\n2. We will invoke a REST API call to a live endpoint and get the response. Let’s name that response as R2\n3. Tests will be executed to compare and assert the attributes of R1 and R2\n\n*Note: To make the readers' life easy, we will name the sample response we define in the contract mentioned above in the\nfirst step as R1 and the real response from the API endpoint mentioned above in the second step as R2 in the below\nsections in this document.*\n\n### Steps to execute a test\n\nThere are two main steps to execute a test\n\n1. Create the contract files and config files\n2. Place them in a proper order and run the docker image\n\n### 1. Create the contract file\n\nLet’s have a look at a sample contract file first.\n\n```\n{\n \"name\": \"POST_CreateApp_Expect_Success\",\n \"request\": {\n   \"method\": \"POST\",\n   \"url\": \"/orgs/${ORG_NAME}/apps\",\n   \"body\": {\n     \"name\":\"${APP_NAME}\",\n     \"displayName\": \"${APP_NAME}\"\n   },\n   \"headers\": {\n     \"Authorization\": \"Bearer ${BEARER_TOKEN}\",\n     \"Content-Type\": \"application/json\",\n     \"Cookie\": \"cwatf=${CWATF}\"\n   }\n },\n \"response\": {\n   \"status\": 200,\n   \"body\": {\n     \"id\": \"29095\",\n     \"name\": \"${APP_NAME}\",\n     \"organization\": {\n       \"id\": \"115\"\n     },\n     \"org\": \"${ORG_NAME}\",\n     \"displayName\": \"${APP_NAME}\",\n     \"createdAt\": \"2021-05-04T04:02:03Z\",\n     \"observability\": {}\n   },\n   \"headers\": {\n     \"Content-type\": \"application/json\"\n   }\n },\n \"assertions\": {\n   \"statusCodeCheck\": true,\n   \"headersPathCheck\": [\n     {\n       \"jsonPath\": \"Content-type\",\n       \"type\": \"EXACT\"\n     }\n   ],\n   \"bodyPathCheck\": [\n     {\n       \"jsonPath\": \"name\",\n       \"type\": \"EXACT\"\n     },\n     {\n       \"jsonPath\": \"displayName\",\n       \"type\": \"EXACT\"\n     },\n     {\n       \"jsonPath\": \"organization.id\",\n       \"value\": \"^\\\\d{5}$\",\n       \"type\": \"REGEX\"\n     },\n     {\n       \"jsonPath\": \"\",\n       \"type\": \"STRUCTURE\"\n     }\n   ]\n },\n \"postConditions\": {\n   \"setEnvs\": [\n     {\n       \"key\": \"APP_ID\",\n       \"jsonPath\": \"id\"\n     }\n   ],\n   \"setGlobalEnvs\": [\n     {\n       \"key\": \"ORG_ID\",\n       \"jsonPath\": \"organization.id\"\n     }\n   ],\n   \"waitTime\": \"5\"\n }\n}\n```\n\nAs we can see, there are 5 major parts in the contract json file as below\n\n1. name\n\n   This is just the testname which can be used to identify the test which is being running\n\n\n2. request\n\n   This section include all the information to execute a rest api call to the relevant API endpoint which is going to be\n   tested\n\n\n3. response\n\n   This is the sample response (R1) we should get from the endpoint. We will not need to define everything in the\n   response, but we should define everything we are going to test or compare. As an example, let’s consider the above\n   example, Let’s have a look on the assertions section,\n\n        \"assertions\": {\n           \"statusCodeCheck\": true,\n           \"headersPathCheck\": [\n             {\n               \"jsonPath\": \"Content-type\",\n               \"type\": \"EXACT\"\n             }\n           ],\n           \"bodyPathCheck\": [\n             {\n               \"jsonPath\": \"name\",\n               \"type\": \"EXACT\"\n             },\n             {\n               \"jsonPath\": \"displayName\",\n               \"type\": \"EXACT\"\n             },\n             {\n               \"jsonPath\": \"organization.id\",\n               \"value\": \"^\\\\d{5}$\",\n               \"type\": \"REGEX\"\n             }\n           ]\n         },\n\n   We can see that there are only four assertions. But if we take a look at the response part we can see there are more\n   fields there. That is not necessary. So we can reduce the unwanted parts in the response and improve that as below.\n\n        \"response\": {\n           \"status\": 200,\n           \"body\": {\n             \"id\": \"29095\",\n             \"name\": \"${APP_NAME}\",\n             \"organization\": {\n               \"id\": \"115\"\n             },\n             \"org\": \"${ORG_NAME}\",\n             \"displayName\": \"${APP_NAME}\",\n             \"createdAt\": \"2021-05-04T04:02:03Z\",\n             \"observability\": {}\n           },\n           \"headers\": {\n             \"Content-type\": \"application/json\"\n           }\n         },\n\n\n4. assertions\n\n   This section includes all the tests which should be executed. In other words, all the assertions we should do with\n   the real response we get from the endpoint. There are three types of assertions for now as below,\n    * StatusCodeCheck - This is a boolean variable. If this attribute is not there in the contract, this will be set to\n      true automatically. In other words, if this is null or true, the status code of the real response (R2) will be\n      checked with the status code in the contract (in R1).\n    * BodyPathCheck - This is a list of PathChecks which includes the assertions which should be done between the body\n      of the actual response (R2) and the sample response in the contract (R1). This will be a json array with json\n      objects of PathChecks as below.\n\n            {\n                   \"jsonPath\": \"organization.id\",\n                   \"type\": \"REGEX\",\n                   \"typeValue\": \"^\\\\d{5}$\"\n             }\n\n        * JsonPath - This is the path to the relevant attribute inside the body of the response which is going to be\n          tested or asserted. In the above example, there should be an json object called organization inside the\n          response body and inside that organization json object there should be an attribute named id as below\n\n               \"body\": {\n                   \"organization\": {\n                      \"id\": \"115\"\n                    }\n                },\n\n          type - There are 3 types of assertions types as below\n            * EXACT - check whether the relevant attribute in sample response in the contract (R1) and the real\n              response (R2) are exactly the same\n            * REGEX - check whether the relevant attribute in the real response (R2) is matched with a regular\n              expression\n            * IS_EMPTY - check whether the relevant attribute in the real response (R2) is empty or not\n            * IS_NOT_EMPTY - check whether the relevant attribute in the real response (R2) is empty or not\n            * STRUCTURE - check whether the keys (only the keys, not the values) in the relevant json object in real\n              response (R2) are equal to the keys of the relevant json object in the sample response (R1) in the\n              contract. In other words, this compares the structure of two json objects.\n            * IS_ARRAY - Check whether the given Json path contains a Json array. Test will fail if it is not a Json\n              array\n            * IS_OBJECT - Check whether the given Json path contains a Json object. Test will fail if it is not a Json\n              object\n\n          typeValue - This should contain the value for the type as below\n            * EXACT - there is no need of a typeValue for this\n            * REGEX - the regex pattern as a string\n            * IS_EMPTY - there is no need of a typeValue for this\n            * IS_NOT_EMPTY - there is no need of a typeValue for this\n            * STRUCTURE - there is no need of a typeValue for this\n            * IS_ARRAY - there is no need of a typeValue for this\n            * IS_OBJECT - there is no need of a typeValue for this\n\n    * HeaderPathCheck - This is a list of PathChecks which includes the assertions which should be done between the\n      headers of the actual response (R2) and the sample response in the contract (R2). This is also a json array with\n      PathCheck json objects as mentioned above. But here, the json path should include the path inside the header\n      except the body\n\n\n5. postConditions\n\n   Actions in the post conditions section include the things we can do after the assertions are over. There are three\n   actions we can execute for now as below,\n    1. setEnvs\n\n       As we know, there may be dependencies between some API endpoints. Some attribute from an API endpoint will be\n       needed to execute another API call. Hence, those kinds of attributes can be saved as envs and can be injected to\n       relevant contracts at the runtime. The envs we set from this action, can only be used within the same suite\n\n    2. setGlobalEnvs\n\n       This is as same as the above setEnvs action, but the envs we set from this action can be accessed from any suite,\n       In other words, these envs are set as global envs\n    3. waitTime\n\n       We can add a waiting time after executing a contract, In other words, the framework will sleep the specified\n       amount of seconds after executing the relevant contract\n\n### 2. Create config files, place them in a proper order and run the docker image\n\nThis is how we should place the files inside the volume mount\n\n    ├── configs\n    │   ├── .env\n    │   ├── 1_config.json\n    │   ├── 2_config.json\n    │   ├── certificate.jks\n    │   └── client-truststore.jks\n    └── contracts\n        ├── DELETE_RemoveApp_Expect_Success.json\n        ├── GET_AppStatus_Expect_CorrectData.json\n        ├── GET_CheckBackEndHealth_Expect_CorrectData.json\n        ├── GET_ConnectionConfig_Expect_CorrectData.json\n        ├── GET_FetchApisOfOrgThroughProxy_Expect_CorrectData.json\n        ├── GET_OrgStatus_Expect_CorrectData.json\n        ├── POST_AppPing_Expect_200OK.json\n        ├── POST_CreateApp_Expect_Success.json\n        ├── POST_GetApisOfOrg_Expect_CorrectData.json\n        └── PUT_AddApiToApp_Expect_CorrectData.json\n\nHere are some sample files\n\nsuite1config.json\n\n    {\n         \"beforeSuiteConfigs\": {\n            \"contracts\": [\n              \"console/POST_CreateApp_Expect_Success.json\"\n            ]\n         },\n         \"afterSuiteConfigs\": {\n           \"contracts\": [\n             \"DELETE_RemoveApp_Expect_Success.json\"\n           ]\n         },\n         \"tests\": [\n            \"console/parallel_tests\"\n        ]\n    }\n\nThe envs we set in that postConditions section will be available for all the tests in the suite. As an example, if we\nneed to test the following API endpoint,\n\n\t\"url\": \"/app/${APP_ID}/documents/${DOCUMENT_ID}\"\n\nAs we can see, there are two environmental variables in this url, APP_ID and DOCUMENT_ID. In order to test this endpoint\nwe will have to create an app and a document inside that app. Hence, we will have to execute some API calls (execute\ncontracts) to create the app and the document before running this test. Let’s consider that, we have two contracts\ncalled POST_CreateApp_Expect_Success.json and POST_CreateDocument_Expect_Success.json to do this task, we can include\nthat within the beforeSuiteConfigs as below,\n\n    \"beforeSuiteConfigs\": {\n       \"contracts\": [\n         \"POST_CreateApp_Expect_Success.json\",\n         \"POST_CreateDocument_Expect_Success.json\"\n       ]\n     },\n\nThese contracts will be executed in the beforeSuite method in testng. In other words, These contracts will be executed\nsynchronously one by one and execute the post condition section in each contract. Then all the tests in the relevant\nsuite file will be executed parallely and they can use this APP_ID and DOCUMENT_ID set by those contracts in the\nbeforeSuiteConfigs. Note that the assertions inside these contracts in the beforeSuiteMethod will also be executed. If\nwe want not to execute those tests in the contracts, we can include them in the preContracts attribute. Imagine that, we\nare creating the app with a unique static name. Hence, if there is an app with the same name already, the\nPOST_CreateApp_Expect_Success will be failed. Then the entire test suite will not be executed. Hence we can do a cleanup\nbefore the tests. We can execute the DELETE_RemoveApp_Expect_Success.json contract to delete the app with the same name\nif it exists. But, we can’t predict the response of that contract hence the response will depend on the existence of the\nparticular app. Hence, we have to include that in the preContracts section as below,\n\n    \"beforeSuiteConfigs\": {\n       \"preContracts\": [\n         \"DELETE_RemoveApp_Expect_Success.json\"\n       ]\n       \"contracts\": [\n         \"POST_CreateApp_Expect_Success.json\",\n         \"POST_CreateDocument_Expect_Success.json\"\n       ]\n     },\n\n### 3. Run the docker image\n\nNow we can execute the docker image using the following command\n\n    docker run -e RESOURCES_PATH='{PATH}' -d -v {path_to_file_mount}:{Path} org/image\n\n### Todos\n\n* Implement XML paths","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwso2%2Fcontract-test-runner","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fwso2%2Fcontract-test-runner","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwso2%2Fcontract-test-runner/lists"}