https://github.com/schemamap/schemamap

The Postgres-native data movement platform for multi-tenant SaaS apps
https://github.com/schemamap/schemamap

domain-driven-design etl postgresql spreadsheets

Last synced: 4 months ago
JSON representation

The Postgres-native data movement platform for multi-tenant SaaS apps

• Host: GitHub
• URL: https://github.com/schemamap/schemamap
• Owner: schemamap
• License: mit
• Created: 2023-05-10T09:17:07.000Z (almost 2 years ago)
• Default Branch: main
• Last Pushed: 2024-04-25T10:01:57.000Z (10 months ago)
• Last Synced: 2024-04-26T09:31:33.866Z (10 months ago)
• Topics: domain-driven-design, etl, postgresql, spreadsheets
• Language: Clojure
• Homepage: https://schemamap.io
• Size: 13.7 MB
• Stars: 5
• Watchers: 1
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

• jimsghstars - schemamap/schemamap - Instant batch data import for Postgres (Rust)

README

        
# Schemamap.io

This repository contains the open-source SQL schema and SDKs of [Schemamap.io](https://schemamap.io).

## When to use Schemamap.io

Comparison to similar tools:

| Need | Solutions |

|--------------------|-------------------------------------------------------|

| REST API           | [Postgrest](https://postgrest.org/en/stable/)         |

| GraphQL API        | [Hasura](https://hasura.io/), [Supabase](https://github.com/supabase/pg_graphql) |

| Table-based API    | :tada: **Schemamap.io** :tada:                            |

## Overview

Schemamap.io provides a common SQL and backend interface for your Postgres-based multi-tenant application, regardless of your framework.

Instead of maintaining CSV/Excel imports and exports by hand, generate them using industry-standard patterns via a rule engine.

As your schema evolves (new columns/tables) so do your table-based interfaces, along with import/export SQL scripts.

## Formats supported

- CSV

- Google Sheets

- Excel (`.xls` and `.xlsx`) (Work in progress)

- Salesforce API (Work in progress)

## Features

- **Schema as a View**: To support analysis, schema metadata is collected into a column-wise materialized view.

  - Query and analyze your schema with simple SQL queries, get holistic view of anomalies/exceptions to the patterns in your DB.

  - Great for onboarding new developers without needing an ER diagram

  - Schedule refresh at your convenience by just calling a SQL function:

    - periodically, using [pg_cron](https://github.com/citusdata/pg_cron)

    - after SQL migrations

    - from application code

- **Security**: Robust handling for application Postgres DB roles / RLS

  - [Principle of least privilege](https://en.wikipedia.org/wiki/Principle_of_least_privilege)

  - Multiple roles with incrementally more privileges

    - Schema read access

    - Read-only access (to the minimum subset of tables, for exporting data)

    - Write access (to the minimum subset of tables, to import data without writing application layer code)

  - Give access as much or as little, given your security & threat model.

  - Support tenant-scoped roles that guarantee safe access via row-level-security

- **TCP Port-Forwarding**: Connect your local/Docker Postgres securely for a local-first developer experience.

  - No need to setup firewall rules/Ngrok/bastion hosts to try out the feature-set.

  - Use with local mock/seed data or with an empty DB.

  - Get setup/usage help interactively from the app, while coding along in your favorite SQL client/psql.

## SDKs Supported

Integrate `schemamap` seamlessly with language-specific SDKs.

Currently supported:

- **[CLI (Rust)](./rust)**

- **[Clojure (library)](./clojure)**:

[![Clojars Project](https://img.shields.io/clojars/v/io.schemamap/schemamap-clj.svg)](https://clojars.org/io.schemamap/schemamap-clj)

_Watch out for more languages coming soon! Have a request? [Open an issue](https://github.com/schemamap/schemamap/issues/new)._

## Developing

### First-time setup

1. Install direnv and hook into your shell: https://direnv.net/#basic-installation

2. Install the Nix package manager: https://github.com/DeterminateSystems/nix-installer#readme

3. Run `direnv allow` (will prompt you to install https://devenv.sh/getting-started/#2-install-cachix, from Step 2.)

4. To make sure the LFS files (db dumps) are present, run: `git lfs pull`

### Day-to-day operations

1. `process-compose` to bring up the development environment services

2. `ci-test` to run the integration test suite locally (shut down `process-compose` beforehand)

3. `devenv info` to see what packages and scripts are available

## Feedback and Contributions

We'd love to hear from you! Whether it's a bug report, feature request, or general feedback - feel free to [raise an issue](https://github.com/schemamap/schemamap/issues/new).

## Security Policy

Security is at the core Schemamap.io.

If you discover any issue regarding security, please disclose the information responsibly by sending an email to [email protected] and not by creating a GitHub issue.

We'll get back to you ASAP and work with you to confirm and plan a fix for the issue.

Please note that we do not currently offer a bug bounty program.

## Acknowledgements

This software uses the following third-party libraries:

- [Rathole](https://github.com/rapiz1/rathole) licensed under Apache-2.0. See [LICENSES/rathole.txt](LICENSES/rathole.txt).

## License

Copyright © 2023-2024 Schemamap.io Kft.

This project is distributed under the MIT License. For more details, refer to the [LICENSE](./LICENSE) file.