{"id":23768570,"url":"https://github.com/nci-gdc/gdc-ng-models","last_synced_at":"2026-03-27T19:30:18.600Z","repository":{"id":41107122,"uuid":"192352546","full_name":"NCI-GDC/gdc-ng-models","owner":"NCI-GDC","description":null,"archived":false,"fork":false,"pushed_at":"2024-04-13T01:03:56.000Z","size":346,"stargazers_count":2,"open_issues_count":2,"forks_count":1,"subscribers_count":24,"default_branch":"develop","last_synced_at":"2024-04-14T12:44:40.499Z","etag":null,"topics":["core","library"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/NCI-GDC.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2019-06-17T13:28:32.000Z","updated_at":"2024-04-16T11:41:55.888Z","dependencies_parsed_at":"2024-02-16T13:25:34.519Z","dependency_job_id":"0a81e655-6259-4f12-b8e4-bd0bdd785203","html_url":"https://github.com/NCI-GDC/gdc-ng-models","commit_stats":null,"previous_names":[],"tags_count":34,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NCI-GDC%2Fgdc-ng-models","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NCI-GDC%2Fgdc-ng-models/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NCI-GDC%2Fgdc-ng-models/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NCI-GDC%2Fgdc-ng-models/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/NCI-GDC","download_url":"https://codeload.github.com/NCI-GDC/gdc-ng-models/tar.gz/refs/heads/develop","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":239946915,"owners_count":19723018,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["core","library"],"created_at":"2025-01-01T01:37:36.510Z","updated_at":"2026-03-27T19:30:18.543Z","avatar_url":"https://github.com/NCI-GDC.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"[![pre-commit](https://img.shields.io/badge/pre--commit-enabled-brightgreen?logo=pre-commitlogoColor=white)](https://github.com/pre-commit/pre-commit)\n\n---\n# gdc-ng-models\n\nRepository for GDC models that do not have a dependency on [psqlgraph](https://github.com/NCI-GDC/psqlgraph).\n\nThe `ng` in `gdc-ng-models` stands for _non-graph_.\n\n- [gdc-ng-models](#gdc-ng-models)\n  - [Testing](#testing)\n  - [Command-Line Scripts](#command-line-scripts)\n  - [Setup pre-commit hook to check for secrets](#setup-pre-commit-hook-to-check-for-secrets)\n\n\n## Testing\n\n```sh\npsql -c \"create user gdc_test with superuser password 'gdc_test';\" -U postgres\npsql -c 'create database automated_test with owner \"gdc_test\";' -U postgres\npytest tests/\n```\n\nOr we can use [tox](https://tox.readthedocs.io/en/latest/) for testing\n\n```\npip install tox\ntox\n```\n\n## Resources\nAt the time of this writing, a database migration of ng-models had not occurred in awhile. There\nare multiple references for running the migrations.\n* This README.md\n* [Confluence Wiki](https://gdc-ctds.atlassian.net/wiki/spaces/GDC/pages/19402891/Database+migration)\n* [Cohort API](https://github.com/NCI-GDC/cohortapi?tab=readme-ov-file#cohort-database-tables)\n\n\n## Command-Line Scripts\n\nThis repository supplies the `ng-models` script which allows you to create the databases and tables in development and production environments.\n\nTo use, supply the `-m` parameter specifying the module (and related models/tables to be created). See example usage below:\n\n```sh\nng-models -m misc --host 127.0.0.1 -d automated_test -u postgres -p postgres\n```\n\n### Setup the execution environment\nIf you would like to run alembic directly.\n```sh\n# datatools.service.consul or repl.service.consul\nsudo -i -u ubuntu # Ubuntu Pro\nsudo -i -u ec2-user # Amazon Linux\n\nexport http_proxy=http://cloud-proxy:3128 \u0026\u0026 export https_proxy=http://cloud-proxy:3128\ngit clone https://github.com/NCI-GDC/gdc-ng-models.git\ncd gdc-ng-models/\npython3 -m venv venv\nsource venv/bin/activate\npython --version  # verify the python version, expects python 3.9\npip install -e \".[alembic]\" --index-url https://nexus.osdc.io/repository/pypi-all/simple\n\n```\n\u003e Update the alembic.ini file for local execution, if running on the datatools VM this\n\u003e  will already be configured in /var/tungsten/services/datatools/alembic.ini\n\u003e  `sqlalchemy.url = postgres://dev_admin:\u003cpass\u003e@postgres.service.consul/dev_gdc`\n\n\n### Option: Install database models\n```bash\n################################################\n# Uncomment the target environment for execution\n################################################\n# export ENV_PREFIX=prod PASS=$(cat __path_to_password_file__/prod_admin) # PROD\n# export ENV_PREFIX=qa PASS=$(cat __path_to_password_file__/qa_admin) # QA\n# export ENV_PREFIX=dev PASS=$(cat __path_to_password_file__/dev_admin) # DEV\nexport MODEL=\u003cmodel_name\u003e # e.g., MODEL=cohort\n\n# Create tables\nng-models \\\n  -m $MODEL \\\n  --host postgres.service.consul \\\n  -d ${ENV_PREFIX}_gdc \\\n  -u $USER \\\n  -p \"$PASS\" create\n\n# Add write permission\nng-models \\\n  -m $MODEL \\\n  --host postgres.service.consul \\\n  -d ${ENV_PREFIX}_gdc \\\n  -u $USER \\\n  -p \"$PASS\" \\\n  grant -r ${ENV_PREFIX}_graph_readwrite -P write\n\n# Add READ permission\nng-models \\\n  -m $MODEL \\\n  --host postgres.service.consul \\\n  -d ${ENV_PREFIX}_gdc \\\n  -u $USER \\\n  -p \"$PASS\" \\\n  grant -r ${ENV_PREFIX}_graph_readonly -P read\n```\n\n### Option: Migrate database models\n\n\u003e WARNING: That '+' sign in front of '+1' is _REALLY_ important.\n\nUsing [Alembic](https://alembic.sqlalchemy.org/en/latest/index.html) migrate existing data models\n\n\u003e NOTE: offline mode - alembic will print out the sql it will execute for `upgrade` and `downgrade`\n\u003e   commands if you supply the `--sql` flag. It does not accept relative identifiers.\n\n\n```bash\n############################################\n# `alembic_migration` is a script held in tungsten and deployed to the datatools VM\n# https://github.com/NCI-GDC/tungsten/blob/develop/salt/srv/services/datatools/wrappers/alembic_migration\n#\n# datatools uses the alembic from gdc-ng-models.\n#   alembic==1.13.0\n#      # via gdc-ng-models\n############################################\n\n# make sure the migration scripts you want to run are located in /var/tungsten/services/datatools/deploy/current/venv/lib/python3.9/site-packages/gdc_ng_models/alembic/versions\n\n# check to see what version is currently in postgres (if its empty, also fine)\nsudo bash alembic_migration current  # with wrapper on datatools\n  # or\n  alembic --config alembic.ini current # on repl (need to update the sqlalchemy.url in alembic.ini)\n\n# apply the necessary migrations (if you want to apply all of them, do the following)\nsudo bash alembic_migration upgrade head\n  # or upgrade to version\n  alembic --config alembic.ini upgrade +1\n\n# if you just wanted to apply 1 revision from the current HEAD, do the following\nsudo bash alembic_migration upgrade +1\n  # or upgrade to version\n  alembic --config alembic.ini upgrade +1\n  # or get the SQL to run manually\n  alembic --config alembic.ini upgrade 12dbbcac7a1d:38eab5dd7b0d --sql\n\n# if you want to rollback 1 revision from the current HEAD, do the following\nsudo bash alembic_migration downgrade -1\n  # or downgrade to version\n  alembic --config alembic.ini downgrade e9d53a640d5d\n  # or get the SQL to run manually\n  alembic --config alembic.ini downgrade 12dbbcac7a1d:e9d53a640d5d --sql\n```\n\n## Setup pre-commit hook to check for secrets\n\nWe use [pre-commit](https://pre-commit.com/) to setup pre-commit hooks for this repo.\nWe use [detect-secrets](https://github.com/Yelp/detect-secrets) to search for secrets being committed into the repo.\n\nTo install the pre-commit hook, run\n```\npre-commit install\n```\n\nTo update the .secrets.baseline file run\n```\ndetect-secrets scan --update .secrets.baseline\n```\n\n`.secrets.baseline` contains all the string that were caught by detect-secrets but are not stored in plain text. Audit the baseline to view the secrets .\n\n```\ndetect-secrets audit .secrets.baseline\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnci-gdc%2Fgdc-ng-models","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnci-gdc%2Fgdc-ng-models","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnci-gdc%2Fgdc-ng-models/lists"}