{"id":19440388,"url":"https://github.com/devinit/ddw-analyst-ui","last_synced_at":"2025-10-24T16:07:38.516Z","repository":{"id":37450985,"uuid":"170649425","full_name":"devinit/ddw-analyst-ui","owner":"devinit","description":"The Development Data Warehouse","archived":false,"fork":false,"pushed_at":"2024-06-18T03:16:26.000Z","size":49728,"stargazers_count":2,"open_issues_count":26,"forks_count":1,"subscribers_count":6,"default_branch":"develop","last_synced_at":"2024-06-18T04:46:37.783Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"http://ddw.devinit.org/","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/devinit.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-02-14T07:46:40.000Z","updated_at":"2024-06-18T04:46:43.248Z","dependencies_parsed_at":"2023-02-13T18:00:54.229Z","dependency_job_id":"c324cabe-a2ed-4f20-91f7-fea77aa02651","html_url":"https://github.com/devinit/ddw-analyst-ui","commit_stats":null,"previous_names":[],"tags_count":100,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/devinit%2Fddw-analyst-ui","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/devinit%2Fddw-analyst-ui/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/devinit%2Fddw-analyst-ui/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/devinit%2Fddw-analyst-ui/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/devinit","download_url":"https://codeload.github.com/devinit/ddw-analyst-ui/tar.gz/refs/heads/develop","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":223971544,"owners_count":17234077,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-10T15:29:11.801Z","updated_at":"2025-10-24T16:07:33.463Z","avatar_url":"https://github.com/devinit.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# DDW Analyst UI\nThe new and improved DDW Analyst UI interface\n\n## Setup\n\n\n### Docker Deployment\n\n1. Make sure you're starting with a clean DB volume, so Docker knows to create the new User\n\n        docker-compose down` `docker volume rm metadata2\n\n2. Create a persistent dev volume\n\n        docker volume create --name=metadata2\n\n3. Create a self-signed certificate\n\n        mkdir -p ssl \u0026\u0026 openssl req -newkey rsa:2048 -new -nodes -x509 -days 3650 -keyout ssl/privkey.pem -out ssl/fullchain.pem\n\n4. Build your app\n\n        docker-compose up --build -d\n\n5. Fetch CSV files from GitHub\n\n        docker-compose exec web python manage.py update_csv_files\n\n6. Migrate the database.\n\n        docker-compose exec web python manage.py migrate\n\n7. Load test data\n\n        docker-compose exec web python manage.py loaddata test_data` `docker-compose exec web python manage.py loaddata --database=datasets test_datasets\n\n8. Alternatively, load the real data\n\n        export FTSUSER=X` `export FTSPASS=Y` `docker-compose exec web data_updates/completed_scripts.sh\n\n9.  Create a superuser.\n\n        docker-compose exec web python manage.py createsuperuser\n\n10. Add the bit registry to npm config to install bit dependencies\n\n        npm config set @bit:registry https://node.bitsrc.io\n\n11. Install frontend dependencies\n\n        npm install\n\n12. Bundle frontend code and collect static files\n\n        npm run build\n\n13. Restart the app.\n\n        docker-compose restart\n\n\n### Import CSV Files\n\n1. Manually run management command to download csv files:\n\n        docker-compose exec web python manage.py update_csv_files\n\n2. Create a scheduled event to periodically download updates from the git repo. The bash script is `update_csv_files.sh`\n\n3. The git hub repo with csv files can be found [here](https://github.com/devinit/ddw-data-update-configs)\n\n\n### Development Database\n\nTo create a test development DB, for local development (e.g. virtualenv steps below)\n\n1. Ensure the line that normally appears as `local   all             postgres                                peer` in `pg_hba.conf` instead reads `local   all             postgres                                trust`\n2. Run script `./dev-db-setup.sh`\n3. A database **analyst_ui** will be created in your local postgres instance.\n4. Access sample database through default postgres user using\n\n    `psql -d postgres`\n\n    `\\c analyst_ui`\n\n5. For additional users, edit script **analyst_ui_users.sql** adding the username that you need\n6. Run script to grant permissions to all the schemas and tables of analyst_ui\n7. Follow the steps under the virtualenv section below to intergrate with your local environment.\n\n### virtualenv - development\n\n**Prerequisites**\n\n  - python\n  - virtualenv\n  - NodeJS\n  - postgreSQL\n\n1. Initialise a virtual environment `virtualenv env`\n2. Activate \u0026 enter virtualenv environment `source env/bin/activate`\n3. Install python dependencies `pip install -r requirements.txt`\n4. Fetch CSV files from GitHub `python manage.py update_csv_files`\n5. Migrate the database. `python manage.py migrate`\n6. Load test data from a fixture like so `python manage.py loaddata test_data` `python manage.py loaddata --database=datasets test_datasets`\n7. Create a superuser. `python manage.py createsuperuser`\n8. Add the bit registry to npm config to install bit dependencies ``npm config set @bit:registry https://node.bitsrc.io\n9. Install frontend dependencies `npm install`\n10. Bundle frontend code and collect static files `npm run dev` NB: is set to watch for changes and recompile\n11. Run the app. `export DJANGO_DEV='True' \u0026\u0026 python manage.py runserver`\n\n### Docker Development\n\n1. Make sure you're starting with a clean DB volume, so Docker knows to create the new User:\n\n        docker-compose down\n\n        docker volume rm metadata2\n2. Create a persistent dev volume:\n\n        docker volume create --name=metadata2\n\n3. Create a self-signed certificate:\n\n        mkdir -p ssl \u0026\u0026 openssl req -newkey rsa:2048 -new -nodes -x509 -days 3650 -keyout ssl/privkey.pem -out ssl/fullchain.pem\n\n4. Build \u0026 run your app with the dev docker config:\n\n        docker-compose -f docker-compose.dev.yml up --build\n\n5. Fetch CSV files from GitHub\n\n        docker-compose -f docker-compose.dev.yml exec web python manage.py update_csv_files\n\n6. Migrate the database:\n\n        docker-compose -f docker-compose.dev.yml exec web python manage.py migrate\n\n7. Load test data:\n\n        docker-compose -f docker-compose.dev.yml exec web python manage.py loaddata test_data\n\n        docker-compose -f docker-compose.dev.yml exec web python manage.py loaddata --database=datasets test_datasets\n\n8. Alternatively, you can acquire a db dump of the live data (binary) and import it into your database:\n\n        docker-compose -f docker-compose.dev.yml exec db psql -U analyst_ui_user -d analyst_ui -c 'drop schema public CASCADE;'\n        docker-compose -f docker-compose.dev.yml exec db psql -U analyst_ui_user -d analyst_ui -c 'create schema public;'\n        docker cp [DB BUMP FILE NAME].backup ddw-analyst-ui_db_1:/var/lib/postgresql/data\n        docker exec ddw-analyst-ui_db_1 pg_restore -U analyst_ui_user -d analyst_ui /var/lib/postgresql/data/[DB BUMP FILE NAME].backup\n        docker-compose -f docker-compose.dev.yml exec web python manage.py update_csv_files\n        docker-compose -f docker-compose.dev.yml exec web python3 manage.py migrate\n\n9.  Create a superuser:\n\n        docker-compose -f docker-compose.dev.yml exec web python manage.py createsuperuser\n\n10. Add the bit registry to npm config to install bit dependencies:\n\n        npm config set @bit:registry https://node.bitsrc.io\n\n11. Install frontend dependencies:\n\n        npm install\n\n12. Dynamic API base URL setting\n\n    Add an `API_BASE_URL` in the `.env` file and assign it either a localhost, staging, or production url. If not set, this defaults to the url of the current environment in which the application is running.\n\n13. Start frontend dev environment which watches and collects static files:\n\n        npm start\n\n### Scheduled Events\n\nConfigure a cronjob to run the `run-schedules.sh` script which in turn runs the command that checks for scheduled events every minute\n\n    * * * * * /root/run-schedules.sh \u003e/root/cron-logs.txt 2\u003e\u00261\n\n### Freeze/Unfreeze and Save Queryset features\nMake sure the following schemas are created;\n\n- `archives` i.e `CREATE SCHEMA archives;`\n- `dataset` i.e `CREATE SCHEMA dataset;`\n\n### FTS Precode feature set up and use\nOn first run (i.e if you have used a database dump without FTS Precoded tables included or a clean DB set up) run the following scripts;\n1. `docker-compose exec web data_updates/manual_data.sh`\n2. `docker-compose exec web data_updates/manual_data_fts.sh` which adds the FTS metadata into the DB\n\nThe above should only be run once, on initial deployment of the feature, and only if using a clean and fresh DB set up with no data or using a DB dump that does not include the FTS metadata\n\nTo pull the latest FTS updates from the APIs, we shall run\n3. `docker-compose exec web data_updates/fts.sh`. Note that at this point the analyst may download the updated codelists and edit them, then re-upload them using the https://ddw.devinit.org/update/ feature\n\nTo precode and join the dependency tables, we shall run:\n4. `docker-compose exec web data_updates/fts_precode.sh`. This should be run everytime there is a change made to the codelist entries or everytime the script in 3 above is run.\n\nTo update the manual FTS tables with missing codelist items, we shall finally run the below script\n5. `docker-compose exec web data_updates/fts_diff.sh`\n\nWe can run 4 and 5 above in one step by using:\n`docker-compose exec web data_updates/finalise_precode.sh`\nThis will be the preferred way of running them from the front end as a scheduled event.\n\nNOTE:\n1. All the above scripts should be run in that exact order on first run / deployment.\n2. After any update (i.e after running the script in 3 above, or an analyst using the https://ddw.devinit.org/update/ feature) is made to the codelists, run `docker-compose exec web data_updates/finalise_precode.sh` which combines 4 and 5 into one step. This can be run from the `Scheduled Events` on the front end.\n\n\n### End-To-End Testing\n\nThis is set up to run with [Cypress](https://www.cypress.io/).\n\nTo test locally:\n\n1. Update the `baseUrl` option in the `cypress.json` file to one that suits your current need\n2. Setup test users as specified in the `frontend/cypress/fixtures/users.json` file\n3. Run `npm run cy:run` for headless tests and `npm run cy:open` for interactive tests in a browser.\n\n### Postman Setup for API Testing\n\nIf you're using Postman for testing the REST api, you can use the following setup:\n\n1. Make sure you have an environment set for your collection.\n2. `POST` to `http://localhost:8000/api/auth/login/` with `Basic Auth` and the Username and Password.\n3. Paste this code in `Tests` which will save the token to the environment.\n```\nvar jsonData = JSON.parse(responseBody);\npostman.setEnvironmentVariable(\"token\", jsonData.token);\n```\n4. In the Headers of your subsequent posts, send the Header `Authorization: Token {{token}}`\n\n\n### Letsencrypt certificates generation\n\n1. If certbot has not been installed already, install certbot by following commands\n  ```\n  sudo add-apt-repository ppa:certbot/certbot\n  sudo apt-get install certbot\n  ```\n\n2. Run below script to generate certificates\n`certbot renew --dry-run  --webroot -w /root/ddw-analyst-ui/static/letsencrypt`\n\n3. If the command above is run successfully copy certificates to the ssl folder of ddw app\n  ```\n  cp -f  /etc/letsencrypt/live/ddw.devinit.org/privkey.pem /root/ddw-analyst-ui/ssl/\n  cp -f /etc/letsencrypt/live/ddw.devinit.org/fullchain.pem /root/ddw-analyst-ui/ssl/\n\n  ```\n  From ddw-analyst-ui root folder, reload nginx so that certificates are picked\n  `docker-compose exec ddw-analyst-ui_nginx_1 nginx reload`\n\n4. Check if there is a cron job set to renew certificates. If there is non add the cron task below. This will try to renew the certificate twice a day every day\n\n`0 */12 * * * /root/ddw-analyst-ui/certbot.sh \u003e/dev/null 2\u003e\u00261`\n\n\n### API Documentation\n\nRead more [here](./core/docs/API.md)\n\n\n### Management Commands\n\n\n### Deployment\n\n1. Create a release branch with a name format `release/v[VERSION NUMBER]`\n2. Update version number in `package.json`\n3. Create a GitHub release with a name format `v[VERSION NUMBER]` - this will deploy to production\n4. Create PR from release branch to master/main\n5. Merge in release branch to master/main\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdevinit%2Fddw-analyst-ui","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdevinit%2Fddw-analyst-ui","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdevinit%2Fddw-analyst-ui/lists"}