{"id":13557491,"url":"https://github.com/mara/mara-db","last_synced_at":"2025-06-13T01:33:45.775Z","repository":{"id":37759300,"uuid":"84362545","full_name":"mara/mara-db","owner":"mara","description":"Lightweight configuration and access to multiple databases in a single project","archived":false,"fork":false,"pushed_at":"2023-12-15T14:26:00.000Z","size":777,"stargazers_count":38,"open_issues_count":8,"forks_count":17,"subscribers_count":5,"default_branch":"main","last_synced_at":"2025-04-22T16:53:10.047Z","etag":null,"topics":["backend","flask","mara","sqlalchemy"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mara.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2017-03-08T20:20:43.000Z","updated_at":"2024-08-17T06:11:12.000Z","dependencies_parsed_at":"2023-01-22T12:35:16.571Z","dependency_job_id":"91efed0f-8216-4727-a679-bdff3d0074d3","html_url":"https://github.com/mara/mara-db","commit_stats":{"total_commits":153,"total_committers":16,"mean_commits":9.5625,"dds":0.5032679738562091,"last_synced_commit":"ae69f7bc6809d5ac82e610894c9a27bc648afe4b"},"previous_names":[],"tags_count":48,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mara%2Fmara-db","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mara%2Fmara-db/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mara%2Fmara-db/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mara%2Fmara-db/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mara","download_url":"https://codeload.github.com/mara/mara-db/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":251691620,"owners_count":21628357,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["backend","flask","mara","sqlalchemy"],"created_at":"2024-08-01T12:04:23.012Z","updated_at":"2025-04-30T11:33:25.216Z","avatar_url":"https://github.com/mara.png","language":"Python","funding_links":[],"categories":["Python","flask"],"sub_categories":[],"readme":"# Mara DB\n\n[![Build Status](https://github.com/mara/mara-db/actions/workflows/build.yml/badge.svg)](https://github.com/mara/mara-db/actions/workflows/build.yml)\n[![PyPI - License](https://img.shields.io/pypi/l/mara-db.svg)](https://github.com/mara/mara-db/blob/main/LICENSE)\n[![PyPI version](https://badge.fury.io/py/mara-db.svg)](https://badge.fury.io/py/mara-db)\n[![Slack Status](https://img.shields.io/badge/slack-join_chat-white.svg?logo=slack\u0026style=social)](https://communityinviter.com/apps/mara-users/public-invite)\n\nMini package for configuring and accessing multiple databases in a single project. Decouples the use of databases and their configuration by using \"aliases\" for databases.\n\nThe file [mara_db/dbs.py](https://github.com/mara/mara-db/blob/main/mara_db/dbs.py) contains abstract database configurations for PostgreSQL, Mysql, SQL Server, Oracle, SQLite and Big Query. The database connections of a project are configured by overwriting the `databases` function in [mara_db/config.py](https://github.com/mara/mara-db/blob/main/mara_db/config.py):\n\n```python\nimport mara_db.config\nimport mara_db.dbs\n\n## configure database connections for different aliases\nmara_db.config.databases = lambda: {\n    'mara': mara_db.dbs.PostgreSQLDB(host='localhost', user='root', database='mara'),\n    'dwh': mara_db.dbs.PostgreSQLDB(database='dwh'),\n    'source-1': mara_db.dbs.MysqlDB(host='some-localhost', database='my_app', user='dwh'),\n    'source-2': mara_db.dbs.SQLServerDB(user='dwh_read', password='123abc', database='db1', host='some-sql-server')\n}\n\n## access individual database configurations with `dbs.db`:\nprint(mara_db.dbs.db('mara'))\n# -\u003e \u003cPostgreSQLDB: host=localhost, database=mara\u003e\n```\n\n\u0026nbsp;\n\n\n## Visualization of (PostgreSQL, MySQL, SQL Server) database schemas\n\n[mara_db/views.py](https://github.com/mara/mara-db/blob/main/mara_db/views.py) contains a schema visualization for all configured databases using graphviz (currently PostgreSQL, Mysql and SQL Server only). It basically show tables of selected schemas together with the foreign key relations between them.\n\n\n![Schema visualization](https://github.com/mara/mara-db/blob/main/docs/_static/schema-visualization.png)\n\nFor finding missing foreign key constraints, columns that follow a specific naming pattern (configurable via `config.schema_ui_foreign_key_column_regex`, default `*_fk`) and that are not part of foreign key constraints are drawn in pink.\n\n\u0026nbsp;\n\n\n## Fast batch processing: Accessing databases with shell commands\n\nThe file [mara_db/shell.py](https://github.com/mara/mara-db/blob/main/mara_db/shell.py) contains functions that create commands for accessing databases via their command line clients.\n\nFor example, the `query_command` function creates a shell command that can receive an SQL query from stdin and execute it:\n\n```python\nimport mara_db.shell\n\nprint(mara_db.shell.query_command('source-1'))\n# -\u003e mysql --default-character-set=utf8mb4 --user=dwh --host=some-localhost my_app\n\nprint(mara_db.shell.query_command('dwh', timezone='Europe/Lisbon', echo_queries=False))\n# -\u003e PGTZ=Europe/Lisbon PGOPTIONS=--client-min-messages=warning psql  --no-psqlrc --set ON_ERROR_STOP=on dwh\n```\n\nThe function `copy_to_stdout_command` creates a shell command that receives a query on stdin and writes the result to stdout in tabular form:\n\n```python\nprint(mara_db.shell.copy_to_stdout_command('source-1'))\n# -\u003e mysql --default-character-set=utf8mb4 --user=dwh --host=some-localhost my_app --skip-column-names\n```\n\nSimilarly, `copy_from_stdin_command` creates a client command that receives tabular data from stdin and and writes it to a target table:\n\n```python\nprint(mara_db.shell.copy_from_stdin_command('dwh', target_table='some_table', delimiter_char=';'))\n# -\u003e PGTZ=Europe/Berlin PGOPTIONS=--client-min-messages=warning psql --echo-all --no-psqlrc --set ON_ERROR_STOP=on dwh \\\n#      --command=\"COPY some_table FROM STDIN WITH DELIMITER AS ';'\"\n```\n\nFinally, `copy_command` creates a shell command that receives a sql query from stdin, executes the query in `source_db` and then writes the result of to `target_table` in `target_db`:\n\n```python\nprint(mara_db.shell.copy_command('source-2', 'dwh', target_table='some_table'))\n# -\u003e sed 's/\\\\\\\\$/\\$/g;s/\\$/\\\\\\\\$/g' \\\n#   | sqsh  -U dwh_read -P 123abc -S some-sql-server -D db1 -m csv \\\n#   | PGTZ=Europe/Berlin PGOPTIONS=--client-min-messages=warning psql --echo-all --no-psqlrc --set ON_ERROR_STOP=on dwh \\\n#         --command = \"COPY some_table FROM STDIN WITH CSV HEADER\"\n```\n\n\u0026nbsp;\n\n\nThe following **command line clients** are used to access the various databases:\n\n| Database | Client binary | Comments |\n| --- | --- | --- |\n| Postgresql / Redshift | `psql` | Included in standard distributions. |\n| MariaDB / Mysql | `mysql` | Included in standard distributions. |\n| SQL Server | `sqsh`\u003cbr\u003e- or -\u003cbr\u003e`sqlcmd` | **sqsh**: From [https://sourceforge.net/projects/sqsh/](https://sourceforge.net/projects/sqsh/), usually messy to get working. On ubuntu, use [http://ppa.launchpad.net/jasc/sqsh/ubuntu/](http://ppa.launchpad.net/jasc/sqsh/ubuntu/) backport. On Mac, try the homebrew version or install from source.\u003cbr\u003e**sqlcmd**: Official Microsoft Utility for SQL Server. See [sqlcmd Utility](https://docs.microsoft.com/en-us/sql/tools/sqlcmd-utility) |\n| Oracle | `sqlplus64` | See the [Oracle Instant Client](https://www.oracle.com/technetwork/database/database-technologies/instant-client/overview/index.html) homepage for details. On Mac, follow [these instructions](https://vanwollingen.nl/install-oracle-instant-client-and-sqlplus-using-homebrew-a233ce224bf). Then ` sudo ln -s /usr/local/bin/sqlplus /usr/local/bin/sqlplus64` to make the binary accessible as `sqlplus64`. |\n| SQLite | `sqlite3` | Available in standard distributions. Version \u003e3.20.x required (not the case on Ubuntu 14.04). |\n| Big Query | `bq` | See the [Google Cloud SDK](https://cloud.google.com/sdk/docs/quickstarts) page for details. |\n| Snowflake | `snowsql` | See [SnowSQL (CLI Client)](https://docs.snowflake.com/en/user-guide/snowsql.html) |\n| Databricks | `dbsqlcli` | Included when using package extra `databricks` via package [databricks-sql-cli](https://pypi.org/project/databricks-sql-cli/). See [Databricks SQL CLI](https://docs.databricks.com/dev-tools/databricks-sql-cli.html#) |\n\n\u0026nbsp;\n\n\n## Make it so! Auto-migration of SQLAlchemy models\n\n[Alembic has a feature](http://alembic.zzzcomputing.com/en/latest/autogenerate.html) that can create a diff between the state of a database and the ORM models of an application. This feature is used in [mara_db/auto_migrate.py](https://github.com/mara/mara-db/blob/main/mara_db/auto_migrate.py) to automatically perform all necessary database transformations, without intermediate migration files:\n\n```python\n# define a model / table\nclass MyTable(sqlalchemy.ext.declarative.declarative_base()):\n    __tablename__ = 'my_table'\n    my_table_id = sqlalchemy.Column(sqlalchemy.Integer, primary_key=True)\n    column_1 = sqlalchemy.Column(sqlalchemy.TEXT, nullable=False, index=True)\n\n\ndb = mara_db.dbs.SQLiteDB(file_name='/tmp/test.sqlite')\n\n# create database and table\nmara_db.auto_migration.auto_migrate(engine=mara_db.auto_migration.engine(db), models=[MyTable])\n# -\u003e\n# Created database \"sqlite:////tmp/test.sqlite\"\n#\n# CREATE TABLE my_table (\n#     my_table_id SERIAL NOT NULL,\n#     column_1 TEXT NOT NULL,\n#     PRIMARY KEY (my_table_id)\n# );\n#\n# CREATE INDEX ix_my_table_column_1 ON my_table (column_1);\n```\n\nWhen the model is changed later, then `auto_migrate` creates a diff against the existing database and applies it:\n\n```python\n# remove index and add another column\nclass MyTable(sqlalchemy.ext.declarative.declarative_base()):\n    __tablename__ = 'my_table'\n    my_table_id = sqlalchemy.Column(sqlalchemy.Integer, primary_key=True)\n    column_1 = sqlalchemy.Column(sqlalchemy.TEXT, nullable=False)\n    column_2 = sqlalchemy.Column(sqlalchemy.Integer)\n\nauto_migrate(engine=engine(db), models=[MyTable])\n# -\u003e\n# ALTER TABLE my_table ADD COLUMN column_2 INTEGER;\n#\n# DROP INDEX ix_my_table_text_column_1;\n```\n\n**Use with care**! The are lot of changes [that alembic auto-generate can not detect](http://alembic.zzzcomputing.com/en/latest/autogenerate.html#what-does-autogenerate-detect-and-what-does-it-not-detect). We recommend testing each aut-migration on a staging system first before deploying to production. Sometimes manual migration scripts will be necessary.\n\n\n\n## Installation\n\n```bash\npip install mara-db\n```\n\nor\n\n```bash\npip install git+https://github.com/mara/mara-db.git\n```\n\n### Optional: Installation of requirements for SQL Server\n\nFor usage with SQL Server, the python module pyodbc and a odbc driver (e.g. Microsoft ODBC Driver 17 for SQL Server) is required which is not included in the general requirement.\n\nTo see how to install pyodbc, take a look into [this install guide](https://github.com/mkleehammer/pyodbc/wiki/Install).\nTo see how to install ODBC 17, take a look into [Installing the Microsoft ODBC Driver for SQL Server on Linux and macOS](https://docs.microsoft.com/en-us/sql/connect/odbc/linux-mac/installing-the-microsoft-odbc-driver-for-sql-server?view=sql-server-ver15).\n\nOn Linux, you most likely will have to deal with an SSL issue, see [this issue](https://github.com/microsoft/msphpsql/issues/1023). A quick, dirty option in a test/development environment could be to [disable the requirement for TLS 1.2](https://github.com/microsoft/msphpsql/issues/1023#issuecomment-523214695).\n\n### Optional: Installation of requirements for BigQuery\n\nFor usage with BigQuery, the official `bq` and `gcloud` clients are required.\nSee the [Google Cloud SDK](https://cloud.google.com/sdk/docs/quickstarts) page for installation details.\n\nEnabling the BigQuery API and Service account JSON credentials are also required as listed\nin the official documentation [here](https://cloud.google.com/bigquery/docs/quickstarts/quickstart-client-libraries#before-you-begin).\n\nOne time authentication of the service-account used:\n```cmd\ngcloud auth activate-service-account --key-file='path-to/service-account.json'\n```\n\nOptionally, for loading data from files into BigQuery, the `gcloud_gcs_bucket_name` can be specified in the database initialization.\nThis will use the Google Cloud Storage bucket specified as cache for loading data and over-coming potential limitations.\nFor more see [loading-data](https://cloud.google.com/bigquery/docs/bq-command-line-tool#loading_data).\nBy default, files will directly loaded locally as described in [loading-local-data](https://cloud.google.com/bigquery/docs/loading-data-local#loading_data_from_a_local_data_source).\n\nA BigQuery context with a python cursor is also available on demand for easy access to BigQuery databases.\nIn order to use, install the official Google python client library: [google-cloud-bigquery](https://cloud.google.com/bigquery/docs/reference/libraries#client-libraries-install-python).\n\n## Links\n\n* Documentation: https://mara-db.readthedocs.io/\n* Changes: https://mara-db.readthedocs.io/en/latest/changes.html\n* PyPI Releases: https://pypi.org/project/mara-db/\n* Source Code: https://github.com/mara/mara-db\n* Issue Tracker: https://github.com/mara/mara-db/issues\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmara%2Fmara-db","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmara%2Fmara-db","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmara%2Fmara-db/lists"}