{"id":20186923,"url":"https://github.com/finndersen/python-db-project-template","last_synced_at":"2025-09-21T19:31:31.531Z","repository":{"id":259431142,"uuid":"876280048","full_name":"Finndersen/python-db-project-template","owner":"Finndersen","description":"A template for Python projects that need to use a relational database, including tooling for managing schema migrations and testing against a containerised DB instance.","archived":false,"fork":false,"pushed_at":"2024-12-12T17:08:30.000Z","size":47,"stargazers_count":3,"open_issues_count":0,"forks_count":1,"subscribers_count":1,"default_branch":"main","last_synced_at":"2024-12-12T17:25:55.535Z","etag":null,"topics":["atlas","databases","python","sql","sqlalchemy"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Finndersen.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-10-21T17:45:56.000Z","updated_at":"2024-12-12T17:08:34.000Z","dependencies_parsed_at":"2024-10-25T10:23:06.488Z","dependency_job_id":"20483ed0-01b6-42d2-b8f4-48bd2cb0b864","html_url":"https://github.com/Finndersen/python-db-project-template","commit_stats":null,"previous_names":["finndersen/atlas-project-template"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Finndersen%2Fpython-db-project-template","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Finndersen%2Fpython-db-project-template/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Finndersen%2Fpython-db-project-template/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Finndersen%2Fpython-db-project-template/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Finndersen","download_url":"https://codeload.github.com/Finndersen/python-db-project-template/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":233787189,"owners_count":18730124,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["atlas","databases","python","sql","sqlalchemy"],"created_at":"2024-11-14T03:19:11.374Z","updated_at":"2025-09-21T19:31:26.245Z","avatar_url":"https://github.com/Finndersen.png","language":"Python","readme":"# Python DB Project Template\nA project template for Python applications that need to use a relational database, helping to simplify the process for managing schema migrations and setting up a containerised DB instance for testing. \n\nInvolves an implementation of a Python service running on AWS with a PostgreSQL database, but can be adapted to other contexts.\n\nIntended for use on a Mac or Linux development machine.\n\n## Features\n \n* A collection of commands/scripts and configurations leveraging the [Atlas](https://atlasgo.io/) tool to make the process of creating, validating and applying schema migrations simple, safe and automated:\n  * `create-migrations` command for generating migration files from changes to DB schema\n  * `check-migrations` command for validating schema \u0026 migration state \u0026 integrity, and showing migration-related warnings (can be used as a pre-commit hook and in CI)\n  * AWS Lambda-backed custom resource for applying migrations upon CloudFormation stack deployment (With a custom parameter that causes the CR to only be triggered when migration state changes, and a Lambda Layer to contain the Atlas binary). \n* Automatically installs Atlas and a license-free docker runtime for Mac machines ([colima](https://github.com/abiosoft/colima))\n* Pytest fixtures for managing a local containerised PostgreSQL database for testing, and providing a SQLAlchemy session wih per-test isolation.\n\n## Installation\n\nClone this repository, adapt it to your existing project if required.\n\nRun `make install` to install project dependencies, including:\n- Docker engine runtime (colima)\n- Atlas DB migration tool\n- Python project dependencies\n- Downloads the `postgres:16-alpine` docker image for local testing and migration verification\n\n## Usage\n### Configure Database Schema Source\nThis template comes with an example configuration using SQLAlchemy as a DB schema source. Check the [Atlas docs](https://atlasgo.io/guides) for how to update the `atlas.hcl` configuration file to use a different schema source.\n\n### Add your app project code\nAdd your application code to the `src/app/` directory. Your application could be a Lambda function, a Flask/Fast API app running on AWS, or any other Python application that needs to interact with a relational database. You will also need to add appropriate resources to `cloud-formation.yaml`, or another solution to deploy your application.\n\n### Create Migrations\nDelete the existing files in the `src/db/migrations/` directory, then run `make create-migrations` from the project root to create initial migration and hash files. \n\nThe `MigrationHash` property of the AWS Custom Resource will be automatically updated to the new migration hash value, so that it will be considered updated and trigger the Lambda to run migrations when deployed. \n\n### Validate Migration files\nRunning `make check-migrations` will:\n* Verify that the `MigrationHash` property of the AWS Custom Resource is set appropriately\n* Verify hash integrity of migration files\n* Exit early if there are no changes to DB schema or migrations\n* Verify that DB schema and migrations are in sync (this involves running the migrations against a temporary database, which will raise any SQL semantic issues)\n* Run [Atlas linting](https://atlasgo.io/versioned/lint) which generates warnings for common migration issues\n\nThe `make lint` command will run `check-migrations` along with other project linting and type checking.\nThis command can be added to a Git pre-commit hook, and should be used in your automated CI process as part of PR validation. Linting warning outputs from the command should be included in the PR description for review. \n\n### Set configuration in AWS SAM template\n\nThe `cloud-formation.yaml` AWS SAM template file needs to be updated with the correct parameters for your project. You can use references to outputs from the database infrastructure resources, which should be defined in a separate stack. \n\n### Testing\nUse the `db_session_maker` fixture in your tests to get a SQLAlchemy session maker with a connection to an automatically created PostgreSQL DB container. The DB tables will automatically be created, and content cleared between tests. \n\nExample usage:\n```python\ndef test_my_function_that_needs_a_session(db_session: Session):\n      # Set up test data in the DB\n      ...\n      db_session.flush()\n  \n      # Test your function that requires a SQLAlchemy session\n      result = my_function(db_session)\n      db_session.flush()\n      \n      # Verify state of the DB after the function has run\n      ...\n\ndef test_my_function_that_needs_a_session_maker(db_session_maker):\n    with db_session_maker.begin() as session:\n        # Set up test data in the DB\n        ...\n    \n    # Test your function that requires a SQLAlchemy session maker\n    result = my_function(db_session_maker)\n    \n    with db_session_maker.begin() as session:\n        # Verify state of the DB after the function has run\n        ...\n```\n\nRun `make test` to run tests. \n\n### Apply Migrations\nConfigure your deployment pipeline to run the following commands (if using Lambda-backed custom resource to apply migrations):\n```bash\n# Download atlas binary to be packaged into the Lambda Layer\n./scripts/download_atlas.sh 'atlas_layer/bin/'\n# Deploy the SAM template\nsam deploy --template-file cloud-formation.yaml\n```\n\nIf using AWS with the provided Lambda-backed custom resource, migrations will automatically be applied when the SAM template is deployed. The Lambda logs will contain details of the outcome.\n\nTo manually apply migrations in other contexts, run:\n\n```bash\natlas migrate apply -u \u003cDB_CONNECTION_STRING\u003e --dir file://src/db/migrations/\n```\n\n## Notes\n* Uses the Atlas [versioned workflow](https://atlasgo.io/versioned/intro) for migrations, and only free features (no Atlas Cloud).\n* Any other services or application code that needs to reference the DB schema (e.g. SQLAlchemy models) should be included within this same project, so that they are kept in sync and deployed together.\n* The mechanism for applying migrations (Lambda-backed custom resource) can be changed to suit your project needs, while retaining the commands used for local development and in CI.\n* Any additional AWS resources relating to this project/services should have a `DependsOn: DBSchemaMigration` set, so that they are only deployed after migrations have successfully completed. \n\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffinndersen%2Fpython-db-project-template","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ffinndersen%2Fpython-db-project-template","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffinndersen%2Fpython-db-project-template/lists"}