{"id":13557579,"url":"https://github.com/ONSdigital/eq-survey-runner","last_synced_at":"2025-04-03T12:30:50.161Z","repository":{"id":5303088,"uuid":"49205589","full_name":"ONSdigital/eq-survey-runner","owner":"ONSdigital","description":"eQ Survey Runner","archived":true,"fork":false,"pushed_at":"2023-11-15T11:12:02.000Z","size":24060,"stargazers_count":21,"open_issues_count":0,"forks_count":14,"subscribers_count":19,"default_branch":"master","last_synced_at":"2024-11-04T07:36:07.243Z","etag":null,"topics":["docker","forms","hacktoberfest","jwt","npm","python","questionnaire","webdriverio","wtforms","yarn"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ONSdigital.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2016-01-07T13:15:26.000Z","updated_at":"2024-09-15T04:13:22.000Z","dependencies_parsed_at":"2023-11-15T12:26:21.792Z","dependency_job_id":"4e50a758-6267-49a9-91b7-71d842dd72c5","html_url":"https://github.com/ONSdigital/eq-survey-runner","commit_stats":null,"previous_names":[],"tags_count":446,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ONSdigital%2Feq-survey-runner","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ONSdigital%2Feq-survey-runner/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ONSdigital%2Feq-survey-runner/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ONSdigital%2Feq-survey-runner/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ONSdigital","download_url":"https://codeload.github.com/ONSdigital/eq-survey-runner/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247002122,"owners_count":20867403,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["docker","forms","hacktoberfest","jwt","npm","python","questionnaire","webdriverio","wtforms","yarn"],"created_at":"2024-08-01T12:04:25.922Z","updated_at":"2025-04-03T12:30:45.717Z","avatar_url":"https://github.com/ONSdigital.png","language":"Python","funding_links":[],"categories":["Python","docker"],"sub_categories":[],"readme":"# eQ Survey Runner\n![EQ functional tests](https://github.com/ONSdigital/eq-survey-runner/workflows/EQ%20functional%20tests/badge.svg)\n\n**IMPORTANT! This repository is archived and is no longer actively maintained.**\n\nIf you're looking for the latest version, please check out the successor project: [eQ Questionnaire Runner](https://github.com/ONSdigital/eq-questionnaire-runner).\n\n## Run with Docker\nInstall Docker for your system: https://www.docker.com/\n\nTo get eq-survey-runner running the following command will build and run the containers\n```\ndocker-compose up -d\n\n```\n\nTo launch a survey, navigate to `http://localhost:8000/`\n\nWhen the containers are running you are able to access the application as normal, and code changes will be reflected in the running application.\nHowever, any new dependencies that are added would require a re-build.\n\nTo rebuild the eq-survey-runner container, the following command can be used.\n```\ndocker-compose build\n```\n\nIf you need to rebuild the container from scratch to re-load any dependencies then you can run the following\n```\ndocker-compose build --no-cache\n```\n\nTo run just the unit tests inside Docker:\n```\ndocker build -t onsdigital/eq-survey-runner .\ndocker build -t onsdigital/eq-survey-runner-unit-tests -f Dockerfile.test .\ndocker run onsdigital/eq-survey-runner-unit-tests\n```\n\nTo run the unit tests locally:\n```\npipenv run scripts/run_tests_unit.sh\n```\n\n\n## Pre-Requisites\nIn order to run locally you'll need PostgreSQL, Node.js, sqlite3, snappy and pyenv installed\n\n```\nbrew install postgres snappy npm sqlite3 pyenv\n```\n\nNote that npm currently requires Python 2.x for some of the setup steps,\nit doesn't work with Python 3.\n\n## Setup\nIt is preferable to use the version of Python locally that matches that\nused on deployment. This project has a `.python_version` file for this\npurpose.\n\n\nUpgrade pip and install dependencies:\n\n```\npyenv install\npip install --upgrade pip setuptools pipenv\npipenv install --dev\n```\n\nRun the server inside the virtual env created by Pipenv with:\n\n```\npipenv run ./scripts/run_app.sh\n```\n\nNote, you will also need to run an upstream tool (eg, https://github.com/ONSDigital/go-launch-a-survey) to launch a survey.\n\n```\ndocker run -e SURVEY_RUNNER_SCHEMA_URL=http://docker.for.mac.host.internal:5000 -it -p 8000:8000 onsdigital/go-launch-a-survey:latest\n```\nThis will generate a JWT for you to log into the application.\n\nTo submit data you will also need to run an additional upstream tool (eg, https://github.com/ONSDigital/eq-docker-dynamodb) to launch a dynamoDB container.\n\n```\ndocker run -it -p 6060:8000 onsdigital/eq-docker-dynamodb:latest\n```\n\nPlease note that currently the SQL backend only supports questionnaire state\n\n---\n\n### Front-end Toolkit\n\nThe front-end toolkit uses nodejs, yarn and gulp.\n\nCurrently, in order to build the front-end toolkit, you will need to have node version 8.X.  To do this, do the following commands:\n```\nbrew install nvm\nnvm install 8\nnvm use 8\n```\n\nInstall yarn with:\n\n```\nnpm install yarn --global\n```\n\nFetch npm dependencies (Note that this overrides the python version defined in `.python-version`):\n\n```\nPYENV_VERSION=system yarn\n```\n\nCompile the project with\n```\nyarn compile\n```\n\nThere are a few additional npm tasks:\n\nCommand                                    | Task\n-------------------------------------------|----------------------\n`yarn compile`                          | Build the assets (js, css, img) into `/static`.\n`yarn dev`                              | Build assets and watch for changes. Runs Browsersync.\n`yarn test`                             | Runs the unit tests through Karma and the functional tests through a local Selenium instance.\n`yarn test_unit`                        | Watches the unit tests via Karma.\n`yarn test_functional`                  | Runs the functional tests through ChimpJS (requires app running on localhost:5000 and generated pages).\n`yarn generate_pages`                   | Generates the functional test pages.\n`yarn lint`                             | Lints the JS, reporting errors/warnings.\n`yarn format`                           | Format the json schemas.\n\n---\n\n###\nUpgrade usage of the pattern library\n(Currently) To make an upgrade to the pattern library you'll need to change the short-hand commit hash in the following files:\n* app/assets/favicons/browserconfig.xml `\u003csquare150x150logo src=\"https://cdn.ons.gov.uk/sdc/[COMMIT HASH HERE]/favicons/mstile-150x150.png\"/\u003e`\n* app/assets/styles/partials/vars/_vars.scss.xml `$cdn-url-root: \"https://cdn.ons.gov.uk/sdc/[COMMIT HASH HERE]\";`\n* app/templates/layouts/base.html `{% set cdn_hash = \"[COMMIT HASH HERE]\" %}`\n\n## Functional test options\n\nThe functional tests use a set of selectors that are generated from each of the test schemas. These make it quick to add new functional tests.\n\nTo run the functional tests use the script:\n\n`./scripts/run_tests_functional.sh`\n\nThis will delete the `tests/functional/generated_pages` directory and regenerate all the files in it from the schemas.\n\nYou can also individually run the `generate_pages` and `test_functional` yarn scripts:\n\n`yarn generate_pages; yarn test_functional`\n\nTo generate the pages manually you can run the `generate_pages` scripts with the schema directory. Run it from the `tests/functional` directory as follows:\n\n`./generate_pages.py ../../data/en/ ./generated_pages -r \"../../base_pages\"`\n\nTo generate a spec file with the imports included, you can use the `generate_pages.py` script on a single schema with the `-s` argument.\n\n`./generate_pages.py ../../data/en/test_multiple_piping.json ./temp_directory -r \"../../base_pages\" -s spec/test_multiple_piping.spec.js`\n\nIf you have already built the generated pages, then the functional tests can be executed with:\n\n`yarn test_functional`\n\nThis can be limited to tests under a directory with:\n\n`yarn test_functional --path tests/functional/spec/components/`\n\nTo run a single test, add `@watch` into the name of any `describe` or `it` function and run:\n\n`yarn test_functional --watch`\n\nAn example of adding `@watch` looks like this:\n`describe('@watch Skip Conditions', function() {...}` or\n`it('@watch Given this is a test', function() {...}`\n\nTo run the tests against a remote deployment you will need to specify the environment variable of EQ_FUNCTIONAL_TEST_ENV eg:\n\n`EQ_FUNCTIONAL_TEST_ENV=https://staging-new-surveys.dev.eq.ons.digital/ yarn test_functional`\n\n## Deployment with elastic beanstalk\n\nYou will need to install the EB CLI tools using PIP.\n\n```\npip install --user awsebcli        # install the eb cli tools\n```\n\nThe Elastic Beanstalk CLI requires the presence of a requirements.txt file. To generate one with Pipenv use the following:\n```\npipenv lock -r \u003e requirements.txt\n```\n\nInitialise the project using the command\n\n```\neb init --region eu-west-1\n```\n\nThis will launch a wizard asking for the AWS credentials and some questions about the environment to create.\n\n`eu-west-1` is the name for Ireland.  I chose the default application name.\n\nOnce completed, you can then deploy the application using the following command:\n\n```\neb create\n```\n\nThis will create the environment and spin up the application . Once the application has deployed you can use the following command to open it in a browser\n\n```\neb open\n```\n\n## Internationalisation\n\nWe use flask-babel to do internationalisation.  To extract messages from source, in the project root run the following command.\n\n```\npipenv run pybabel extract -F babel.cfg -o app/translations/messages.pot .\n```\n\nThis will extract messages and place them in the translations/messages.pot file ready for translation.\n\nYou should only need to create the language files once.\n\nTo create Welsh language files, run the following command\n\n```\npipenv run pybabel init -i app/translations/messages.pot -d app/translations -l cy\n```\n\nTo create the gaelic language files, use the following:\n\n```\npipenv run pybabel init -i app/translations/messages.pot -d app/translations -l gd\n```\n\n### Getting text translated\n\nOur current language translation service requires a .csv rather than a .po file. To convert a .po file to a .csv you'll need to install the Python translate-toolkit:\n```\nbrew install translate-toolkit\n```\n\nTo generate the .csv file:\n```\npo2csv app/translations/cy/LC_MESSAGES/messages.po app.translations/static-cy.csv\n```\n\nTo convert back to a .po file:\n```\ncsv2po app.translations/static-cy.csv app/translations/cy/LC_MESSAGES/messages.po\n```\n\n*Important:* There are some encoding issues when opening the .csv file in Excel. Opening in Google sheets and saving as a .xslx file resolves this.\n\n### Compiling the translations\n\nTo compile the language files for use in the application, use the following:\n\n```\npipenv run pybabel compile -d app/translations\n```\n\nAs strings are added to the application, you will need to update but not overwrite the translations for the various languages.\nTo update the language strings, use:\n\n```\npipenv run pybabel update -i app/translations/messages.pot -d app/translations\n```\n\n## Environment Variables\n\nThe following env variables can be used\n```\nEQ_USER_AUTHENTICATION_RRM_PUBLIC_KEY - the RRM public key for JWT user authentication\nEQ_USER_AUTHENTICATION_SR_PRIVATE_KEY - the SR private key for JWT user authentication\nEQ_USER_AUTHENTICATION_SR_PRIVATE_KEY_PASSWORD - password of the SR private key for JWT user authentication\nEQ_SUBMISSION_SDX_PUBLIC_KEY - the SDX public key for encryption of Submission data\nEQ_SUBMISSION_SR_PRIVATE_SIGNING_KEY - the SR private key for signing of submission data\nEQ_SUBMISSION_SR_PRIVATE_SIGNING_KEY_PASSWORD - the password to the SR private key\nEQ_RABBITMQ_URL - the RabbitMQ connection string\nEQ_RABBITMQ_QUEUE_NAME - the name of the submission queue\nEQ_SERVER_SIDE_STORAGE_DATABASE_URL - url of the database to connect to, e.g. 'sqlite:////tmp/questionnaire.db')\nEQ_SERVER_SIDE_STORAGE_DATABASE_SETUP_RETRY_COUNT - Number of times to retry setting up the database (connection/creation) if it fails\nEQ_SERVER_SIDE_STORAGE_DATABASE_SETUP_RETRY_DELAY_SECONDS - Number of seconds to wait between retry attempts to setup the database\nEQ_LOG_LEVEL - The default logging level (defaults to 'INFO' for local development)\nEQ_WERKZEUG_LOG_LEVEL - The default logging level for werkzeug (defaults to 'INFO' for local development)\nEQ_SCHEMA_DIRECTORY - The directory that contains the schema files\nEQ_SESSION_TIMEOUT_SECONDS - The duration of the flask session\nEQ_SECRET_KEY - The Flask secret key for signing cookies\nEQ_PROFILING - Enables or disables profiling (True/False) Default False/Disabled\nEQ_GTM_ID - The Google Tag Manager ID\nEQ_GTM_ENV_ID - The Google Tag Manager Environment\nEQ_SCHEMA_BUCKET - The name of the bucket in S3 where to look to find schemas\nSAUCE_USERNAME - Sauce Labs username\nSAUCE_ACCESS_KEY - Sauce Labs private key\nEQ_DEV_MODE - Enable dev mode\nEQ_ENABLE_FLASK_DEBUG_TOOLBAR - Enable the flask debug toolbar\nEQ_ENABLE_CACHE - Enable caching of the schema\nEQ_ENABLE_SECURE_SESSION_COOKIE - Set secure session cookies\nEQ_MAX_HTTP_POST_CONTENT_LENGTH - The maximum http post content length that the system wil accept\nEQ_MAX_NUM_REPEATS - The maximum number of repeats the system will allow\nEQ_DEVELOPER_LOGGING - Enable developer style logging described here http://structlog.readthedocs.io/en/stable/development.html\nEQ_ENABLE_LIVE_RELOAD - Enable livereload of browser when scripts, styles or templates are updated\n\nEQ_NEW_RELIC_ENABLED - Enable New Relic monitoring\nNEW_RELIC_LICENSE_KEY - Enable new relic monitoring by supplying a New Relic licence key\nNEW_RELIC_APP_NAME - The name to display for the application in New Relic\n```\n\nThe following env variables can be used when running tests\n```\nEQ_FUNCTIONAL_TEST_ENV - the pre-configured environment [local, docker, preprod] or the url of the environment that should be targeted\n```\n\n## JWT Integration\nIntegration with the survey runner requires the use of a signed JWT using public and private key pair (see https://jwt.io,\nhttps://tools.ietf.org/html/rfc7519, https://tools.ietf.org/html/rfc7515).\n\nOnce signed the JWT must be encrypted using JWE (see https://tools.ietf.org/html/rfc7516).\n\nThe JWT payload must contain the following claims:\n- exp - expiration time\n- iat - issued at time\n\nThe header of the JWT must include the following:\n- alg - the signing algorithm (must be RS256)\n- type - the token type (must be JWT)\n- kid - key identification  (must be EDCRRM)\n\nThe JOSE header of the final JWE must include:\n- alg - the key encryption algorithm (must be RSA-OAEP)\n- enc - the key encryption encoding (must be A256GCM)\n\nTo access the application you must provide a valid JWT. To do this browse to the /session url and append a token parameter.\nThis parameter must be set to a valid JWE encrypted JWT token. Only encrypted tokens are allowed.\n\nThere is a python script for generating tokens for use in development, to run:\n```\npython token_generator.py\n```\n\n## profiling\n\nSetting the `EQ_PROFILING` environment variable to `True` will enable profiling of the application.  Profiling information\nwill be collected per-request in the `profiling` directory where it can be examined using the Pstats Interactive Browser.\n\n`$ python -m pstats \u003cfilename\u003e`\n\nwill load the file into the interactive browser where it can be sorted and queried as required.\n\nTo visualise the profile, `snakeviz` can be used. This provides a nice interface with an 'icicle' graph:\n\n```\n# First combine all the profiles in the 'profiling' directory.\n# Ensure you delete all the files in this directory before starting your profiling session\n# This will create a file called `combined_profile.prof`\npipenv run python scripts/merge_profiles.py\n\nsnakeviz combined_profile.prof\n```\n\n## Updating / Installing dependencies\n\nTo add a new dependency, use `pipenv install [package-name]`, which not only installs the package but Pipenv will also go to the trouble of updating the Pipfile as well.\n\nNB: both the Pipfile and Pipfile.lock files are required in source control to accurately pin dependencies.\n\n## Alpha Survey Runner\nIf you're looking for the Survey Runner code from the Alpha then it has been renamed to: alpha-eq-survey-runner\n- https://github.com/ONSdigital/alpha-eq-survey-runner\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FONSdigital%2Feq-survey-runner","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FONSdigital%2Feq-survey-runner","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FONSdigital%2Feq-survey-runner/lists"}