{"id":29256314,"url":"https://github.com/allient/shem","last_synced_at":"2025-07-04T03:07:44.891Z","repository":{"id":298961446,"uuid":"1001682831","full_name":"allient/shem","owner":"allient","description":null,"archived":false,"fork":false,"pushed_at":"2025-06-20T23:19:36.000Z","size":352,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-06-20T23:29:20.001Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/allient.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-06-13T20:11:50.000Z","updated_at":"2025-06-20T23:19:39.000Z","dependencies_parsed_at":"2025-06-13T21:34:56.210Z","dependency_job_id":null,"html_url":"https://github.com/allient/shem","commit_stats":null,"previous_names":["allient/shem"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/allient/shem","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/allient%2Fshem","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/allient%2Fshem/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/allient%2Fshem/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/allient%2Fshem/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/allient","download_url":"https://codeload.github.com/allient/shem/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/allient%2Fshem/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":263437346,"owners_count":23466369,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-07-04T03:07:40.321Z","updated_at":"2025-07-04T03:07:44.882Z","avatar_url":"https://github.com/allient.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Shem CLI\n\nDeclarative Database Schema Management for PostgreSQL\n\n---\n\n## Overview\n\n**Shem** is a CLI tool for managing your PostgreSQL database schema declaratively. Instead of writing imperative migration scripts, you declare your desired schema state in SQL files, and Shem generates versioned migrations for you. This approach is inspired by modern tools like Supabase CLI and aims to make schema management safer, more reproducible, and developer-friendly.\n\n---\n\n## Features\n\n- **Declarative schema files**: Describe your database in SQL, not migration scripts\n- **Automatic migration generation**: Generate migrations by diffing your schema files against the current database\n- **Shadow database diffing**: Safe, isolated schema comparison using a temporary database\n- **Glob pattern support**: Organize your schema files flexibly\n- **Safety checks**: Warnings for destructive operations (e.g., DROP statements)\n- **Migration history tracking**: Reliable, versioned migrations\n\n---\n\n## Installation\n\n```sh\n# Clone the repository\n$ git clone https://github.com/yourusername/shem.git\n$ cd shem\n\n# Build the CLI (requires Rust toolchain)\n$ cargo build --release\n\n# Optionally, add to your PATH\n$ cp target/release/shem /usr/local/bin/\n```\n\n## Quick Start: Step-by-Step Example\n\n### 1. Initialize a New Project\n\n```sh\n$ shem init my_project\n$ cd my_project\n```\n\nThis creates:\n- `schema/` directory for your declarative SQL files\n- `migrations/` directory for generated migration scripts\n- `shem.yaml` or `shem.toml` config file\n\n### 2. Declare Your Schema\n\nCreate a file `schema/01_employees.sql`:\n\n```sql\nCREATE TABLE employees (\n  id SERIAL PRIMARY KEY,\n  name TEXT NOT NULL\n);\n```\n\nYou can split your schema into multiple files. Files are processed in lexicographic order.\n\n### 3. Configure Your Database Connection\n\nYou have two options to configure your database connection:\n\n#### Option A: Configure in shem.toml (Recommended)\n\nEdit `shem.toml` and uncomment the database URL line:\n\n```toml\n# shem.toml\n[database]\nurl = \"postgresql://user:password@localhost:5432/mydb\"\n\n[declarative]\nenabled = true\nschema_paths = [\"./schema/*.sql\"]\nshadow_port = 5432\n```\n\n**Important**: Remove the `#` comment symbol from the beginning of the line.\n\n#### Option B: Use Command Line Parameter\n\nProvide the database URL directly when running commands:\n\n```sh\ncargo run --bin shem -- migrate --database-url \"postgresql://user:password@localhost:5432/mydb\"\n```\n\n#### Database URL Format\n\n```\npostgresql://username:password@host:port/database_name\n```\n\n**Examples:**\n- Local PostgreSQL: `postgresql://postgres:password@localhost:5432/myapp_dev`\n- No password: `postgresql://postgres@localhost:5432/myapp_dev`\n- Remote database: `postgresql://user:pass@db.example.com:5432/production`\n\n#### Quick PostgreSQL Setup\n\nIf you don't have PostgreSQL running, you can start it with Docker:\n\n```sh\n# Start PostgreSQL container\ndocker run --name postgres-dev \\\n  -e POSTGRES_PASSWORD=postgres \\\n  -e POSTGRES_DB=myapp_dev \\\n  -p 5432:5432 \\\n  -d postgres:15\n\n# Then use this URL in shem.toml:\n# url = \"postgresql://postgres:postgres@localhost:5432/myapp_dev\"\n```\n\n### 4. Generate a Migration (Diff)\n\n```sh\n$ shem diff\n```\n\n- Shem will spin up a shadow database, apply existing migrations, and diff it against your declared schema.\n- A new migration file will be created in `migrations/` (e.g., `20241004112233_create_employees_table.sql`).\n- If destructive changes (e.g., DROP) are detected, you will be warned.\n\n### 5. Apply the Migration\n\n```sh\n$ shem migrate\n```\n\n- Applies all pending migrations to your target database.\n- Migration history is tracked for safety and reproducibility.\n\n### 6. Update Your Schema\n\nEdit `schema/01_employees.sql` to add a new column:\n\n```sql\nCREATE TABLE employees (\n  id SERIAL PRIMARY KEY,\n  name TEXT NOT NULL,\n  age SMALLINT NOT NULL\n);\n```\n\nGenerate and apply the migration:\n\n```sh\n$ shem diff -m \"add_age_column\"\n$ shem migrate\n```\n\n### 7. Rollback (Optional)\n\nTo rollback to a previous migration version:\n\n```sh\n$ shem reset --version 20241004112233\n```\n\n---\n\n## Safety Features\n\n- **DROP statement detection**: Shem warns you if a migration contains potentially destructive operations.\n- **Shadow database**: All diffs are performed in an isolated environment, never against production data.\n- **Migration history**: All applied migrations are tracked in a dedicated table.\n\n---\n\n## Advanced Usage\n\n- **Custom schema file order**: Use numeric prefixes or configure `schema_paths` in your config file for precise control.\n- **Multiple environments**: Use different config files for dev, staging, and production.\n- **Glob patterns**: Organize your schema files by feature or domain.\n\n---\n\n## Development\n\nTo test the CLI in development mode without installing it system-wide:\n\n```sh\n# Run CLI directly with Cargo (dev mode)\n$ cargo run --bin shem -- --help\n\n# Initialize a new project\n$ cargo run --bin shem -- init my_project\n\n# 2. Change to the project directory\n$ cd my_project\n\n# Generate a migration diff\n$ cargo run --bin shem -- diff\n\n# Run tests\n$ cargo test\n```\n\n**Note**: The `--` after `cargo run --bin shem` is important—it tells Cargo to pass the following arguments to your CLI, not to Cargo itself.\n\n---\n\n## Troubleshooting\n\n### Database Connection Issues\n\n**Error: \"No database URL provided\"**\n- Make sure you've uncommented the `url` line in `shem.toml`\n- Or provide the URL with `--database-url` parameter\n- Verify your PostgreSQL server is running\n\n**Error: \"Connection refused\"**\n- Check if PostgreSQL is running: `pg_isready -h localhost -p 5432`\n- Verify the host, port, and credentials in your database URL\n- For Docker: ensure the container is running with `docker ps`\n\n**Error: \"Authentication failed\"**\n- Double-check username and password in the database URL\n- Verify the user has access to the specified database\n- For local PostgreSQL: try `psql -U postgres -d myapp_dev` to test connection\n\n### Common Issues\n\n**\"Schema path does not exist\"**\n- Run `shem init` first to create the project structure\n- Make sure you're in the correct directory\n- Check that `schema/` directory exists\n\n**\"TOML parse error\"**\n- Ensure your `shem.toml` file has the correct structure\n- Check for missing or extra brackets `[]`\n- Verify all required fields are present\n\n**\"No migrations found\"**\n- Run `shem diff` first to generate migration files\n- Check that `migrations/` directory contains `.sql` files\n- Verify migration files have the correct timestamp format\n\n### Verbose Output\n\nFor detailed debugging information, use the `--verbose` flag:\n\n```sh\ncargo run --bin shem -- --verbose diff\ncargo run --bin shem -- --verbose migrate\n```\n\n---\n\n## Contributing\n\nContributions are welcome! Please open issues or pull requests for bug fixes, features, or documentation improvements.\n\n---\n\n## License\n\nMIT\n\n\n\n\n## Postgres objects\n🟦 Data Structures\n| Object              | Description                                            |\n| ------------------- | ------------------------------------------------------ |\n| `Table`             | Stores rows of data in columns.                        |\n| `View`              | Virtual table defined by a SQL query.                  |\n| `Materialized View` | View with stored results for fast access.              |\n| `Composite Type`    | User-defined record/struct type.                       |\n| `Enum Type`         | Type with fixed set of text values.                    |\n| `Domain`            | Base type with constraints.                            |\n| `Range Type`        | Type that stores a range of values (e.g. `int4range`). |\n\n\n🟧 Constraints \u0026 Rules\n| Object                 | Description                                   |\n| ---------------------- | --------------------------------------------- |\n| `Primary Key`          | Unique, not-null identifier.                  |\n| `Foreign Key`          | Enforces link to another table's key.         |\n| `Unique`               | Disallows duplicate values.                   |\n| `Check`                | Enforces condition (e.g. `age \u003e 0`).          |\n| `Not Null`             | Column must not be NULL.                      |\n| `Exclusion Constraint` | Prevents overlapping data (e.g. time ranges). |\n| `Rule`                 | Query rewriting mechanism (rarely used).      |\n\n🟨 Logic \u0026 Computation\n| Object          | Description                                            |\n| --------------- | ------------------------------------------------------ |\n| `Function`      | Returns a value, used in SQL.                          |\n| `Procedure`     | Executed via `CALL`, no return.                        |\n| `Trigger`       | Executes on row events (INSERT/UPDATE/DELETE).         |\n| `Event Trigger` | Executes on schema-level events (e.g. `CREATE TABLE`). |\n\n\n🟩 Security \u0026 Access Control\n| Object         | Description                |\n| -------------- | -------------------------- |\n| `Role/User`    | Database user or group.    |\n| `GRANT/REVOKE` | Access control.            |\n| `Policy`       | Row-Level Security filter. |\n\n🟪 Storage \u0026 Generation\n| Object       | Description                        |\n| ------------ | ---------------------------------- |\n| `Sequence`   | Auto-incrementing ID generator.    |\n| `Index`      | Speed up query performance.        |\n| `Tablespace` | Defines physical storage location. |\n| `Partition`  | Subtable for partitioned table.    |\n\n⬛ Extensions \u0026 Foreign Systems\n| Object           | Description                                       |\n| ---------------- | ------------------------------------------------- |\n| `Extension`      | Plugin package (e.g. `pgcrypto`, `uuid-ossp`).    |\n| `Foreign Table`  | Represents external data as a table.              |\n| `Foreign Server` | Configuration for an external database.           |\n| `FDW`            | Foreign Data Wrapper, interface for foreign data. |\n\n🟥 Miscellaneous\n| Object           | Description                                      |\n| ---------------- | ------------------------------------------------ |\n| `Schema`         | Logical namespace for grouping objects.          |\n| `Collation`      | Rules for text comparison/sorting.               |\n| `Cast`           | Conversion rules between types.                  |\n| `Operator`       | Custom operation (e.g., `#\u003e`, `+=`).             |\n| `Operator Class` | Index behavior definition.                       |\n| `Aggregate`      | Custom aggregation logic (e.g. `avg`, `sum`).    |\n| `Language`       | PL/pgSQL, SQL, C, etc. for functions/procedures. |\n\n\nFor your Shem declarative PostgreSQL migration tool, focus on supporting it\n\n\u003c!-- markdown-table-sort-disable --\u003e\n| Rank | Object Type | Short Description | Usage Freq. | Declarative Complexity | Category | Status |\n| :--- | :--- | :--- | :--- | :--- | :--- | :--- |\n| 1 | **Table** | Fundamental data storage unit containing rows and columns. | Very High | **High** | Data Structure | ✅ Complete |\n| 2 | **Index** | A data structure that improves the speed of data retrieval. | Very High | **Medium** | Performance | ✅ Complete |\n| 3 | **Constraint** | Rules that enforce data integrity (e.g., PK, FK, UNIQUE, CHECK). | Very High | **Medium** | Integrity | ✅ Complete |\n| 4 | **Schema** | A namespace or folder that contains a collection of database objects. | Very High | **Simple** | Organization | 🔶 **Missing** |\n| 5 | **Function** | A reusable block of code that performs an action and returns a value. | High | **High** | Logic | ✅ Complete |\n| 6 | **View** | A virtual table based on the result-set of a SQL statement. | High | Low | Logic / Abstraction| ✅ Complete |\n| 7 | **Sequence** | A generator for producing unique integer sequences (e.g., for IDs). | High | Low | Data Structure | ✅ Complete |\n| 8 | **Extension** | A software package that adds new functionality to PostgreSQL. | High | Low | Administration | ✅ Complete |\n| 9 | **Enum Type** | A custom data type that consists of a static, ordered set of values. | Medium | Low | Data Type | ✅ Complete |\n| 10 | **Trigger** | Executes a function automatically when a DML event occurs on a table. | Medium | Medium | Logic | ✅ Complete |\n| 11 | **Materialized View** | A physically stored (cached) view for performance-intensive queries. | Medium | Medium | Performance | ✅ Complete |\n| 12 | **Procedure** | A block of code to perform actions, can include transaction control. | Medium | **High** | Logic | ✅ Complete |\n| 13 | **Domain** | A user-defined data type with custom constraints for reusability. | Medium | Low | Data Type | ✅ Complete |\n| 14 | **Policy** | A rule for Row-Level Security (RLS) defining row visibility. | Medium | Medium | Security | ✅ Complete |\n| 15 | **Composite Type**| A custom data type that groups multiple fields, like a `struct`. | Low | Low | Data Type | ✅ Complete |\n| 16 | **Foreign Server** | A definition of a connection to an external data source (FDW). | Low | Low | Integration | 🔶 **Missing** |\n| 17 | **Range Type** | A data type for representing a range of values (e.g., a time range). | Low | Medium | Data Type | ✅ Complete |\n| 18 | **Collation** | Defines the rules for sorting and comparing strings. | Low | Low | Data Type | 🔶 **Missing** |\n| 19 | **Rule** | A legacy query-rewrite system. (Largely superseded by Triggers). | Very Low | **High** | Logic (Legacy) | ✅ Complete |\n| 20 | **Event Trigger** | A trigger that fires in response to DDL events (e.g., `CREATE TABLE`). | Very Low | Medium | Administration | ✅ Complete |\n| 21 | **Constraint Trigger**| A special trigger tied to a constraint, often for deferrability. | Very Low | **High** | Logic | ✅ Complete |\n| 22 | **Cast** | Defines how to convert one data type into another. | Very Low | Medium | Data Type | 🔶 Missing |\n| 23 | **Aggregate** | A custom aggregation function (like `SUM` or `AVG`). | Very Low | **High** | Logic | 🔶 Missing |\n\u003c!-- markdown-table-sort-enable --\u003e\n\n## Inspiration\nTool/Crate\tNotes\nrefinery\tFor migration management, not schema diffing or introspection.\npostgres-parser\tCan parse PostgreSQL SQL into AST; useful for reverse-engineering SQL.\nAtlas (Go)\tBest-in-class for schema diffing and migration but not in Rust.\npg-schema-diff GO\n\n## Sample data\n-- ========== EXTENSIONS ==========\nCREATE EXTENSION IF NOT EXISTS \"uuid-ossp\";\nCREATE EXTENSION IF NOT EXISTS \"btree_gist\";\n\n-- ========== DOMAIN ==========\nCREATE DOMAIN email AS TEXT\n  CHECK (VALUE ~* '^[A-Z0-9._%+-]+@[A-Z0-9.-]+\\.[A-Z]{2,}$');\n\n-- ========== ENUM ==========\nCREATE TYPE user_role AS ENUM ('admin', 'user', 'guest');\n\n-- ========== COMPOSITE TYPE ==========\nCREATE TYPE address_type AS (\n    street TEXT,\n    city TEXT,\n    zip TEXT\n);\n\n-- ========== RANGE TYPE ==========\nCREATE TYPE int4range_custom AS RANGE (subtype = integer);\n\n-- ========== COLLATION ==========\nCREATE COLLATION german_ci (provider = icu, locale = 'de-DE-u-co-phonebk', deterministic = false);\n\n-- ========== SEQUENCE ==========\nCREATE SEQUENCE user_seq START 1000;\n\n-- ========== TABLES, COLUMNS, CHECK, UNIQUE, FK ==========\nCREATE TABLE departments (\n    id UUID PRIMARY KEY DEFAULT uuid_generate_v4(),\n    name TEXT UNIQUE NOT NULL,\n    budget NUMERIC CHECK (budget \u003e 0)\n);\n\nCREATE TABLE employees (\n    id UUID PRIMARY KEY DEFAULT uuid_generate_v4(),\n    email email NOT NULL UNIQUE,\n    role user_role NOT NULL DEFAULT 'user',\n    salary NUMERIC CHECK (salary \u003e= 0),\n    department_id UUID REFERENCES departments(id),\n    address address_type,\n    created_at TIMESTAMP DEFAULT now(),\n    CONSTRAINT salary_nonzero CHECK (salary \u003c\u003e 0)\n);\n\n-- ========== INDEX ==========\nCREATE INDEX idx_employees_salary ON employees(salary);\n\n-- ========== EXCLUSION CONSTRAINT ==========\nCREATE TABLE meeting_rooms (\n    id SERIAL PRIMARY KEY,\n    room_name TEXT,\n    during TSRANGE,\n    EXCLUDE USING gist (room_name WITH =, during WITH \u0026\u0026)\n);\n\n-- ========== VIEW ==========\nCREATE VIEW active_employees AS\nSELECT id, email FROM employees WHERE role \u003c\u003e 'guest';\n\n-- ========== MATERIALIZED VIEW ==========\nCREATE MATERIALIZED VIEW department_budgets AS\nSELECT d.name, SUM(e.salary) AS total_salary\nFROM departments d\nJOIN employees e ON e.department_id = d.id\nGROUP BY d.name;\n\n-- ========== FUNCTION ==========\nCREATE FUNCTION get_department_budget(dept_id UUID) RETURNS NUMERIC AS $$\n    SELECT COALESCE(SUM(salary), 0)\n    FROM employees\n    WHERE department_id = dept_id;\n$$ LANGUAGE SQL;\n\n-- ========== PROCEDURE ==========\nCREATE PROCEDURE increase_salary_all(pct NUMERIC)\nLANGUAGE plpgsql\nAS $$\nBEGIN\n    UPDATE employees SET salary = salary + (salary * pct);\nEND;\n$$;\n\n-- ========== FOREIGN SERVER ==========\nCREATE SERVER foreign_pg_server\n  FOREIGN DATA WRAPPER postgres_fdw\n  OPTIONS (host 'localhost', dbname 'other_db', port '5432');\n\n-- ========== ROW-LEVEL SECURITY ==========\nALTER TABLE employees ENABLE ROW LEVEL SECURITY;\n\nCREATE POLICY only_admins ON employees\n  FOR SELECT USING (role = 'admin');\n\n-- ========== TRIGGER FUNCTION \u0026 TRIGGER ==========\nCREATE FUNCTION log_insert() RETURNS trigger AS $$\nBEGIN\n    RAISE NOTICE 'Inserted row with ID: %', NEW.id;\n    RETURN NEW;\nEND;\n$$ LANGUAGE plpgsql;\n\nCREATE TRIGGER trg_log_insert\nAFTER INSERT ON employees\nFOR EACH ROW\nEXECUTE FUNCTION log_insert();\n\n-- ========== CONSTRAINT TRIGGER ==========\nCREATE FUNCTION validate_department_budget() RETURNS trigger AS $$\nBEGIN\n    IF (SELECT SUM(salary) FROM employees WHERE department_id = NEW.department_id) \u003e 100000 THEN\n        RAISE EXCEPTION 'Department budget exceeded';\n    END IF;\n    RETURN NEW;\nEND;\n$$ LANGUAGE plpgsql;\n\nCREATE CONSTRAINT TRIGGER trg_budget_check\nAFTER INSERT OR UPDATE ON employees\nDEFERRABLE INITIALLY DEFERRED\nFOR EACH ROW\nEXECUTE FUNCTION validate_department_budget();\n\n-- ========== EVENT TRIGGER ==========\nCREATE FUNCTION on_ddl_command() RETURNS event_trigger AS $$\nBEGIN\n    RAISE NOTICE 'DDL Command: %', tg_tag;\nEND;\n$$ LANGUAGE plpgsql;\n\nCREATE EVENT TRIGGER ddl_logger\nON ddl_command_start\nEXECUTE FUNCTION on_ddl_command();\n\n-- ========== COMMENTS ==========\nCOMMENT ON TABLE employees IS 'Stores employee data';\nCOMMENT ON COLUMN employees.email IS 'Work email of the employee';\nCOMMENT ON FUNCTION get_department_budget IS 'Returns total budget per department';\n\n\n\n\n\ncargo check\ncargo build\n\n\ncargo upgrade\n\n\n\n\ncargo build --workspace --release\ncargo install --path crates/cli\n\n\n\n\ncargo run -- init example2\nshem diff --database-url \"postgresql://postgres:postgres@localhost:5432/myapp_dev\"\n\n\ncargo run -- diff --database-url \"postgresql://postgres:postgres@localhost:5432/myapp_dev\"\n\n\n\n\n\ncargo run --bin shem introspect\n\n\ncargo test test_extensions_serialized_content_standalone -- --nocapture","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fallient%2Fshem","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fallient%2Fshem","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fallient%2Fshem/lists"}