{"id":14531116,"url":"https://github.com/GClunies/Reflekt","last_synced_at":"2025-09-02T03:31:59.797Z","repository":{"id":36989847,"uuid":"481292100","full_name":"GClunies/reflekt","owner":"GClunies","description":"Define, govern, and model event data for warehouse-first product analytics.","archived":false,"fork":false,"pushed_at":"2024-06-29T15:03:13.000Z","size":6120,"stargazers_count":83,"open_issues_count":3,"forks_count":4,"subscribers_count":5,"default_branch":"main","last_synced_at":"2025-08-26T05:53:24.221Z","etag":null,"topics":["avo","customer-data-platform","data-modeling","data-quality","data-warehouse","dbt","dbt-package","events","governance","product-analytics","schema-registry","segment","segment-protocols"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/GClunies.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-04-13T16:27:47.000Z","updated_at":"2025-03-14T06:52:01.000Z","dependencies_parsed_at":"2023-01-17T12:00:47.597Z","dependency_job_id":"d8fa6a17-8c76-4f85-8992-82f55dafb633","html_url":"https://github.com/GClunies/reflekt","commit_stats":{"total_commits":263,"total_committers":2,"mean_commits":131.5,"dds":0.05323193916349811,"last_synced_commit":"9b3f0ff9e22da1b9f9ac669715c616f72d064b0f"},"previous_names":[],"tags_count":5,"template":false,"template_full_name":null,"purl":"pkg:github/GClunies/reflekt","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/GClunies%2Freflekt","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/GClunies%2Freflekt/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/GClunies%2Freflekt/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/GClunies%2Freflekt/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/GClunies","download_url":"https://codeload.github.com/GClunies/reflekt/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/GClunies%2Freflekt/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":273225001,"owners_count":25067207,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-09-02T02:00:09.530Z","response_time":77,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["avo","customer-data-platform","data-modeling","data-quality","data-warehouse","dbt","dbt-package","events","governance","product-analytics","schema-registry","segment","segment-protocols"],"created_at":"2024-09-05T00:01:12.166Z","updated_at":"2025-09-02T03:31:58.683Z","avatar_url":"https://github.com/GClunies.png","language":"Python","readme":"\u003c!--\nSPDX-FileCopyrightText: 2022 Gregory Clunies \u003cgreg@reflekt-ci.com\u003e\n\nSPDX-License-Identifier: Apache-2.0\n--\u003e\n\n# Reflekt\n\u003e /rəˈflek(t)/ - _to embody or represent in a faithful or appropriate way_\n\n![PyPI](https://img.shields.io/pypi/v/reflekt?style=for-the-badge)\n![PyPI - Python Version](https://img.shields.io/pypi/pyversions/reflekt?style=for-the-badge)\n![GitHub](https://img.shields.io/github/license/gclunies/reflekt?style=for-the-badge)\n\nReflekt helps teams define, govern, and model event data for warehouse-first product analytics. It integrates with [Customer Data Platforms](#customer-data-platform-cdp), [Schema Registries](#schema-registry), [Data Warehouses](#data-warehouse), and [dbt](#dbt). Events are defined using [JSON schema](https://json-schema.org/), version controlled, and updated by pull requests (PRs), enabling:\n  - Branches for parallel development and testing.\n  - Reviews and discussion amongst teams and stakeholders.\n  - CI/CD suites to:\n    - `reflekt lint` schemas against naming and metadata rules.\n    - `reflekt push` schemas to deploy them to a [schema registry](#schema-registry) for event data validation.\n\nStop writing SQL \u0026 YAML. `reflekt build` a [dbt](#dbt) package to model, document, and test events in the warehouse.\n\n![reflekt build](docs/reflekt_build.gif)\n\nTo learn more about `reflekt`, checkout:\n- [Getting Started](#getting-started)\n- [CLI Command Reference](#cli-commands)\n- [Integrations](#integrations)\n- [reflekt-jaffle-shop](https://github.com/GClunies/reflekt-jaffle-shop/) and this [demo](https://www.loom.com/share/75b60cfc2b3549edafde4eedcb3c9631?sid=fb610521-c651-40f9-9de5-8f07a2534302)\n\n## Getting Started\n### Install\nReflekt is available on [PyPI](https://pypi.org/project/reflekt/). Install it with `pip` or package manager, preferably in a virtual environment:\n```bash\n❯ source /path/to/venv/bin/activate  # Activate virtual environment\n❯ pip install reflekt                # Install Reflekt\n❯ reflekt --version                  # Confirm installation\nReflekt CLI Version: 0.6.0\n```\n\u003cbr\u003e\n\n### Create a Reflekt Project\nTo create a new Reflekt project, make a directory, initialize a Git repo, and run `reflekt init`.\n```bash\n❯ mkdir ~/Repos/my-reflekt-project  # Create a new directory for the project\n❯ cd ~/Repos/my-reflekt-project     # Navigate to the project directory\n❯ git init                          # Initialize a new Git repo (REQUIRED)\n❯ reflekt init                      # Initialize a new Reflekt project in the current directory\n\n# Follow the prompts to configure the project\n```\n\nYou now have a Reflekt project with the structure:\n```bash\nmy-reflekt-project\n├── .logs/                # CLI command logs\n├── .reflekt_cache/       # Local cache used by reflekt CLI\n├── artifacts/            # Data artifacts created by `reflekt build`\n├── schemas/              # Event schema definitions\n├── .gitignore\n├── README.md\n└── reflekt_project.yml   # Project configuration\n```\n\u003cbr\u003e\n\n### Configure a Reflekt Project\nReflekt uses 3 files to define and configure a Reflekt project.\n\n#### `reflekt_project.yml`\nDefines project settings, event and metadata conventions, data artifact generation, and optional registry config (Avo only).\n\n\u003cdetails\u003e\n\u003csummary\u003eExample: \u003ccode\u003ereflekt_project.yml\u003c/code\u003e (click to expand)\u003c/summary\u003e\n\u003cbr\u003e\n\n```yaml\n# Example reflekt_project.yml\n# GENERAL CONFIG ----------------------------------------------------------------------\nversion: 1.0\n\nname: reflekt_demo              # Project name\nvendor: com.company_name        # Default vendor for schemas in reflekt project\ndefault_profile: dev_reflekt    # Default profile to use from reflekt_profiles.yml\nprofiles_path: ~/.reflekt/reflekt_profiles.yml  # Path to reflekt_profiles.yml\n\n# SCHEMAS CONFIG ----------------------------------------------------------------------\nschemas:                        # Define schema conventions\n  conventions:\n    event:\n      casing: title             # title | snake | camel | pascal | any\n      numbers: false            # Allow numbers in event names\n      reserved: []              # Reserved event names\n    property:\n      casing: snake             # title | snake | camel | pascal | any\n      numbers: false            # Allow numbers in property names\n      reserved: []              # Reserved property names\n    data_types: [               # Allowed data types\n        string, integer, number, boolean, object, array, any, 'null'\n    ]\n\n# REGISTRY CONFIG ---------------------------------------------------------------------\nregistry:                       # Additional config for schema registry if needed\n  avo:                          # Avo specific config\n    branches:                   # Provide ID for Avo branches for `reflekt pull` to work\n      staging: AbC12dEfG        # Safe to version control (See Avo docs to find branch ID: https://bit.ly/avo-docs-branch-id)\n      main: main                # 'main' always refers to the main branch\n\n# ARTIFACTS CONFIG -----------------------------------------------------------------------\nartifacts:                      # Configure how data artifacts are built\n  dbt:                          # dbt package config\n    sources:\n      prefix: __src_            # Source files will start with this prefix\n    models:\n      prefix: stg_              # Model files will start with this prefix\n    docs:\n      prefix: _stg_             # Docs files will start with this prefix\n      in_folder: false          # Docs files in separate folder?\n      tests:                    # dbt tests to add based on column name (can be empty dict {})\n        id: [unique, not_null]\n```\n\u003c/details\u003e\n\n#### `reflekt_profiles.yml`\nThis file defines connections to schema registries and data warehouse connections.\n\n\u003cdetails\u003e\n\u003csummary\u003eExample: \u003ccode\u003ereflekt_profiles.yml\u003c/code\u003e (click to expand)\u003c/summary\u003e\n\u003cbr\u003e\n\n```yaml\n# Example reflekt_profiles.yml\nversion: 1.0\n\ndev_reflekt:                                         # Profile name (multiple allowed)\n  registry:                                          # Schema registry connection details (multiple allowed)\n    - type: segment\n      api_token: segment_api_token                   # https://docs.segmentapis.com/tag/Getting-Started#section/Get-an-API-token\n\n    - type: avo\n      workspace_id: avo_workspace_id                 # https://www.avo.app/docs/public-api/export-tracking-plan#endpoint\n      service_account_name: avo_service_account_name # https://www.avo.app/docs/public-api/authentication#creating-service-accounts\n      service_account_secret: avo_service_account_secret\n\n  source:                          # Data warehouse connection details (multiple allowed)\n    - id: snowflake                # ID must be unique per profile\n      type: snowflake              # Specify details where raw event data is stored\n      account: abc12345\n      database: raw\n      warehouse: transforming\n      role: transformer\n      user: reflekt_user           # Create reflekt_user with access to raw data (permissions: USAGE, SELECT)\n      password: reflekt_user_password\n\n    - id: redshift                 # ID must be unique per profile\n      type: redshift               # Specify details where raw event data is stored\n      host: example-redshift-cluster-1.abc123.us-west-1.redshift.amazonaws.com\n      database: analytics\n      port: 5439\n      user: reflekt_user           # Create reflekt_user with access to raw data (permissions: USAGE, SELECT)\n      password: reflekt_user_password\n\n    - id: bigquery                 # ID must be unique per profile\n      type: bigquery               # Specify details where raw event data is stored\n      project: raw-data\n      dataset: jaffle_shop_segment\n      keyfile_json:                # Create a reflekt-user service account with access to BigQuery project where raw event data lands (permissions: BigQuery Data Viewer, BigQuery Job User). Include JSON keyfile fields below.\n        type: \"service_account\"\n        project_id: \"foo-bar-123456\"\n        private_key_id: \"abc123def456ghi789\"\n        private_key: \"-----BEGIN PRIVATE KEY-----\\nmy-very-long-private-keyF\\n\\n-----END PRIVATE KEY-----\\n\"\n        client_email: \"reflekt-user@foo-bar-123456.iam.gserviceaccount.com\"\n        client_id: \"123456789101112131415161718\"\n        auth_uri: \"https://accounts.google.com/o/oauth2/auth\"\n        token_uri: \"https://oauth2.googleapis.com/token\"\n        auth_provider_x509_cert_url: \"https://www.googleapis.com/oauth2/v1/certs\"\n        client_x509_cert_url: \"https://www.googleapis.com/robot/v1/metadata/x509/reflekt-user%40foo-bar-123456.iam.gserviceaccount.com\"\n```\n\u003c/details\u003e\n\n#### `schemas/.reflekt/meta/1-0.json`\nA meta-schema used by `reflekt lint` to ensure all events in `schemas/` follow the Reflekt format. Can also be used to define gloablly required metadata for all event schemas.\n\n\u003cdetails\u003e\n\u003csummary\u003eExample: \u003ccode\u003eschemas/.reflekt/meta/1-0.json\u003c/code\u003e (click to expand)\u003c/summary\u003e\n\u003cbr\u003e\n\n```json\n{\n    \"$schema\": \"http://json-schema.org/draft-07/schema#\",\n    \"$id\": \".reflekt/meta/1-0.json\",\n    \"description\": \"Meta-schema for all Reflekt events\",\n    \"self\": {\n        \"vendor\": \"reflekt\",\n        \"name\": \"meta\",\n        \"format\": \"jsonschema\",\n        \"version\": \"1-0\"\n    },\n    \"type\": \"object\",\n    \"allOf\": [\n        {\n            \"$ref\": \"http://json-schema.org/draft-07/schema#\"\n        },\n        {\n            \"properties\": {\n                \"self\": {\n                    \"type\": \"object\",\n                    \"properties\": {\n                        \"vendor\": {\n                            \"type\": \"string\",\n                            \"description\": \"The company, application, team, or system that authored the schema (e.g., com.company, com.company.android, com.company.marketing)\"\n                        },\n                        \"name\": {\n                            \"type\": \"string\",\n                            \"description\": \"The schema name. Describes what the schema is meant to capture (e.g., pageViewed, clickedLink)\"\n                        },\n                        \"format\": {\n                            \"type\": \"string\",\n                            \"description\": \"The format of the schema\",\n                            \"const\": \"jsonschema\"\n                        },\n                        \"version\": {\n                            \"type\": \"string\",\n                            \"description\": \"The schema version, in MODEL-ADDITION format (e.g., 1-0, 1-1, 2-3, etc.)\",\n                            \"pattern\": \"^[1-9][0-9]*-(0|[1-9][0-9]*)$\"\n                        },\n                        \"metadata\": {  // EXAMPLE: Defining required metadata (code_owner, product_owner)\n                            \"type\": \"object\",\n                            \"description\": \"Required metadata for all event schemas\",\n                            \"properties\": {\n                                \"code_owner\": {\"type\": \"string\"},\n                                \"product_owner\": {\"type\": \"string\"}\n                            },\n                            \"required\": [\"code_owner\", \"product_owner\"],\n                            \"additionalProperties\": false\n                        },\n                    },\n                    \"required\": [\"vendor\", \"name\", \"format\", \"version\"],\n                    \"additionalProperties\": false\n                },\n                \"properties\": {},\n                \"tests\": {},\n            },\n            \"required\": [\"self\", \"metadata\", \"properties\"]\n        }\n    ]\n}\n\n```\n\u003c/details\u003e\n\u003cbr\u003e\n\n### Defining Event Schemas\nEvents in a Reflekt project are defined using the [JSON schema](https://json-schema.org/) specification and are stored in the `schemas/` directory of the project. Click to expand the `Order Completed` example below.\n\n\u003cdetails\u003e\n\u003csummary\u003eExample: \u003ccode\u003emy-reflekt-project/schemas/jaffle_shop/Order_Completed/1-0.json\u003c/code\u003e (click to expand)\u003c/summary\u003e\n\u003cbr\u003e\n\n```json\n{\n    \"$schema\": \"http://json-schema.org/draft-07/schema#\",\n    \"$id\": \"jaffle_shop/Order_Completed/1-0.json\",  // Unique ID for schema (relative to `schemas/` dir)\n    \"description\": \"User completed an order.\",      // Event description (REQUIRED)\n    \"self\": {\n        \"vendor\": \"com.thejaffleshop\",                         // Company, application, or system that authored the schema\n        \"name\": \"Order Completed\",                             // Name of the event\n        \"format\": \"jsonschema\",                                // Format of the schema\n        \"version\": \"1-0\",                                      // Version of the schema\n        \"metadata\": {                                          // Metadata for the event\n            \"code_owner\": \"@the-jaffle-shop/frontend-guild\",\n            \"product_owner\": \"pmanager@thejaffleshop.com\",\n        }\n    },\n    \"type\": \"object\",\n    \"properties\": {                                            // Event properties (REQUIRED, but can be empty)\n        \"coupon\": {\n            \"description\": \"Coupon code used for the order.\",  // Property description (REQUIRED)\n            \"type\": [\n                \"string\",\n                \"null\"                                         // Allow null values\n            ]\n        },\n        \"currency\": {\n            \"description\": \"Currency for the order.\",\n            \"type\": \"string\"\n        },\n        \"discount\": {\n            \"description\": \"Total discount for the order.\",\n            \"type\": \"number\"\n        },\n        \"order_id\": {\n            \"description\": \"Unique identifier for the order.\",\n            \"type\": \"string\"\n        },\n        \"products\": {\n            \"description\": \"List of products in the cart.\",\n            \"type\": \"array\",                                    // Array type\n            \"items\": {                                          // Items in the array\n                \"type\": \"object\",\n                \"properties\": {                                 // Properties of the items\n                    \"category\": {\n                        \"description\": \"Category of the product.\",\n                        \"type\": \"string\"\n                    },\n                    \"name\": {\n                        \"description\": \"Name of the product.\",\n                        \"type\": \"string\"\n                    },\n                    \"price\": {\n                        \"description\": \"Price of the product.\",\n                        \"type\": \"number\"\n                    },\n                    \"product_id\": {\n                        \"description\": \"Unique identifier for the product.\",\n                        \"type\": \"string\"\n                    },\n                    \"quantity\": {\n                        \"description\": \"Quantity of the product in the cart.\",\n                        \"type\": \"integer\"\n                    },\n                    \"sku\": {\n                        \"description\": \"Stock keeping unit for the product.\",\n                        \"type\": \"string\"\n                    }\n                },\n                \"required\": [                                  // Required properties for the items\n                    \"product_id\",\n                    \"sku\",\n                    \"category\",\n                    \"name\",\n                    \"price\",\n                    \"quantity\"\n                ],\n                \"additionalProperties\": false,                // Are additional item properties allowed?\n            }\n        },\n        \"revenue\": {\n            \"description\": \"Total revenue for the order.\",\n            \"type\": \"number\"\n        },\n        \"session_id\": {\n            \"description\": \"Unique identifier for the session.\",\n            \"type\": \"string\"\n        },\n        \"shipping\": {\n            \"description\": \"Shipping cost for the order.\",\n            \"type\": \"number\"\n        },\n        \"subtotal\": {\n            \"description\": \"Subtotal for the order (revenue - discount).\",\n            \"type\": \"number\"\n        },\n        \"tax\": {\n            \"description\": \"Tax for the order.\",\n            \"type\": \"number\"\n        },\n        \"total\": {\n            \"description\": \"Total cost for the order (revenue - discount + shipping + tax = subtotal + shipping + tax).\",\n            \"type\": \"number\"\n        }\n    },\n    \"required\": [                  // Required properties (can be empty)\n        \"session_id\",\n        \"order_id\",\n        \"revenue\",\n        \"coupon\",\n        \"discount\",\n        \"subtotal\",\n        \"shipping\",\n        \"tax\",\n        \"total\",\n        \"currency\",\n        \"products\"\n    ],\n    \"additionalProperties\": false  // Are additional properties allowed?\n}\n```\n\u003c/details\u003e\n\n\n#### Schema `$id` and `version`\nSchemas in a Reflekt project are identified and `--select`ed by their `$id`, which is their path relative to the `schemas/` directory. For example:\n| File Path to Schema                                                   | Schema `$id`                       |\n|-----------------------------------------------------------------------|------------------------------------|\n| `~/repos/my-reflekt-project/schemas/jaffle_shop/Cart_Viewed/1-0.json` | `jaffle_shop/Cart_Viewed/1-0.json` |\n| `~/repos/my-reflekt-project/schemas/jaffle_shop/Cart_Viewed/2-1.json` | `jaffle_shop/Cart_Viewed/2-1.json` |\n\nEach schema has a `version` (e.g., `1-0`, `2-1`), used to indicate changes to data collection requirements. New event schemas start at `1-0` and follow a `MAJOR-MINOR` version spec, as shown in the table below.\n| Type  | Description | Example | Use Case |\n|-------|---------------------------------------------------|---------|----------|\n| MAJOR | Breaking change incompatible with previous data. | `1-0`, `2-0`\u003cbr\u003e(ends in `-0) | - Add/remove/rename a *required* property\u003cbr\u003e - Change a property from *optional to required*\u003cbr\u003e - Change a property's type |\n| MINOR | Non-breaking change compatible with previous data. | `1-1`, `2-3` | - Add/remove/rename an *optional* property\u003cbr\u003e - Change a property from *required to optional* |\n\n\u003e [!NOTE]\n\u003e For `MINOR` schema versions (non-breaking changes), you can either:\n\u003e - Update the existing schema and increment the MINOR version number.\n\u003e - Create a new `.json` file with the updated schema and increment the MINOR version number.\n\u003e\n\u003e For `MAJOR` schema versions (breaking changes), you MUST:\n\u003e - **Create a new `.json` file** with the updated schema. This way, an application/product/feature can begin using the new schema while others continue to use the old schema (migrating later).\n\n\u003cbr\u003e\n\n### Linting Event Schemas\nSchemas can be linted to test if they follow the naming conventions in your [`reflekt_project.yml`] and metadata conventions in `schemas/.reflekt/meta/1-0.json`.\n\n```bash\n❯ reflekt lint --select schemas/jaffle_shop\n[18:31:12] INFO     Running with reflekt=0.5.0\n[18:31:12] INFO     Searching for JSON schemas in: /Users/gclunies/Repos/reflekt/schemas/jaffle_shop\n[18:31:12] INFO     Found 9 schema(s) to lint\n[18:31:12] INFO     1 of 9 Linting /Users/gclunies/Repos/reflekt/schemas/jaffle_shop/Order_Completed/1-0.json\n[18:31:19] INFO     2 of 9 Linting /Users/gclunies/Repos/reflekt/schemas/jaffle_shop/Identify/1-0.json\n[18:31:20] INFO     3 of 9 Linting /Users/gclunies/Repos/reflekt/schemas/jaffle_shop/Product_Clicked/1-0.json\n[18:31:24] INFO     4 of 9 Linting /Users/gclunies/Repos/reflekt/schemas/jaffle_shop/Cart_Viewed/1-0.json\n[18:31:26] INFO     5 of 9 Linting /Users/gclunies/Repos/reflekt/schemas/jaffle_shop/Product_Removed/1-0.json\n[18:31:32] INFO     6 of 9 Linting /Users/gclunies/Repos/reflekt/schemas/jaffle_shop/Product_Added/1-0.json\n[18:31:37] INFO     7 of 9 Linting /Users/gclunies/Repos/reflekt/schemas/jaffle_shop/Checkout_Step_Viewed/1-0.json\n[18:31:40] INFO     8 of 9 Linting /Users/gclunies/Repos/reflekt/schemas/jaffle_shop/Checkout_Step_Completed/1-0.json\n[18:31:44] INFO     9 of 9 Linting /Users/gclunies/Repos/reflekt/schemas/jaffle_shop/Page_Viewed/1-0.json\n[18:31:51] INFO     Completed successfully\n```\n\u003cbr\u003e\n\n### Sending Event Schemas to a Schema Registries\nIn order to validate events as they flow from **Application -\u003e Registry -\u003e Customer Data Platform (CDP) -\u003e Data Warehouse**, we need to send a copy of our event schemas to a schema registry (see [supported registries](#schema-registry)). This is done with the `reflekt push` command.\n\n```bash\n❯ reflekt push --registry segment --select schemas/jaffle_shop\n[18:41:05] INFO     Running with reflekt=0.5.0\n[18:41:06] INFO     Searching for JSON schemas in: /Users/gclunies/Repos/reflekt/schemas/jaffle_shop\n[18:41:06] INFO     Found 9 schemas to push\n[18:41:06] INFO     1 of 9 Pushing /Users/gclunies/Repos/reflekt/schemas/jaffle_shop/Order_Completed/1-0.json\n[18:41:06] INFO     2 of 9 Pushing /Users/gclunies/Repos/reflekt/schemas/jaffle_shop/Identify/1-0.json\n[18:41:06] INFO     3 of 9 Pushing /Users/gclunies/Repos/reflekt/schemas/jaffle_shop/Product_Clicked/1-0.json\n[18:41:06] INFO     4 of 9 Pushing /Users/gclunies/Repos/reflekt/schemas/jaffle_shop/Cart_Viewed/1-0.json\n[18:41:06] INFO     5 of 9 Pushing /Users/gclunies/Repos/reflekt/schemas/jaffle_shop/Product_Removed/1-0.json\n[18:41:06] INFO     6 of 9 Pushing /Users/gclunies/Repos/reflekt/schemas/jaffle_shop/Product_Added/1-0.json\n[18:41:06] INFO     7 of 9 Pushing /Users/gclunies/Repos/reflekt/schemas/jaffle_shop/Checkout_Step_Viewed/1-0.json\n[18:41:06] INFO     8 of 9 Pushing /Users/gclunies/Repos/reflekt/schemas/jaffle_shop/Checkout_Step_Completed/1-0.json\n[18:41:06] INFO     9 of 9 Pushing /Users/gclunies/Repos/reflekt/schemas/jaffle_shop/Page_Viewed/1-0.json\n[18:41:08] INFO     Completed successfully\n```\n\u003cbr\u003e\n\n### Building `dbt` Packages to Model Event Data\nModeling event data in `dbt` is a lot of work. Everyone wants staging models that are clean, documented, and tested. But who wants to write and maintain SQL and YAML for hundreds of events?\n\nYou don't have to choose. Put `reflekt build` to work for you - staging models, documentation, even tests - all in a dbt package ready for you to use in your dbt project.\n\n```bash\n❯ reflekt build --artifact dbt --select schemas/jaffle_shop --source snowflake.raw.jaffle_shop_segment --sdk segment\n[18:56:25] INFO     Running with reflekt=0.5.0\n[18:56:26] INFO     Searching for JSON schemas in: /Users/gclunies/Repos/reflekt/schemas/jaffle_shop\n[18:56:26] INFO     Found 9 schemas to build\n[18:56:27] INFO     Building dbt package:\n                        name: jaffle_shop\n                        dir: /Users/gclunies/Repos/reflekt/artifacts/dbt/jaffle_shop\n                        --select: jaffle_shop\n                        --sdk_arg: segment\n                        --source: snowflake.raw.jaffle_shop_segment\n[18:56:27] INFO     Building dbt source 'jaffle_shop_segment'\n[18:56:27] INFO     Building dbt artifacts for schema: /Users/gclunies/Repos/reflekt/schemas/jaffle_shop/Order_Completed/1-0.json\n[18:56:28] INFO     Building dbt table 'order_completed' in source 'jaffle_shop_segment'\n[18:56:28] INFO     Building staging model 'stg_jaffle_shop_segment__order_completed.sql'\n[18:56:28] INFO     Building dbt documentation '_stg_jaffle_shop_segment__order_completed.yml'\n[18:56:28] INFO     Building dbt artifacts for schema: /Users/gclunies/Repos/reflekt/schemas/jaffle_shop/Identify/1-0.json\n[18:56:29] INFO     Building dbt table 'identifies' in source 'jaffle_shop_segment'\n[18:56:29] INFO     Building staging model 'stg_jaffle_shop_segment__identifies.sql'\n[18:56:29] INFO     Building dbt documentation '_stg_jaffle_shop_segment__identifies.yml'\n[18:56:29] INFO     Building dbt artifacts for schema: Segment 'users' table\n[18:56:29] INFO     Building dbt table 'users' in source 'jaffle_shop_segment'\n[18:56:29] INFO     Building staging model 'stg_jaffle_shop_segment__users.sql'\n[18:56:29] INFO     Building dbt documentation '_stg_jaffle_shop_segment__users.yml'\n[18:56:29] INFO     Building dbt artifacts for schema: /Users/gclunies/Repos/reflekt/schemas/jaffle_shop/Product_Clicked/1-0.json\n[18:56:29] INFO     Building dbt table 'product_clicked' in source 'jaffle_shop_segment'\n[18:56:29] INFO     Building staging model 'stg_jaffle_shop_segment__product_clicked.sql'\n[18:56:29] INFO     Building dbt documentation '_stg_jaffle_shop_segment__product_clicked.yml'\n[18:56:29] INFO     Building dbt artifacts for schema: /Users/gclunies/Repos/reflekt/schemas/jaffle_shop/Cart_Viewed/1-0.json\n[18:56:29] INFO     Building dbt table 'cart_viewed' in source 'jaffle_shop_segment'\n[18:56:29] INFO     Building staging model 'stg_jaffle_shop_segment__cart_viewed.sql'\n[18:56:29] INFO     Building dbt documentation '_stg_jaffle_shop_segment__cart_viewed.yml'\n[18:56:29] INFO     Building dbt artifacts for schema: /Users/gclunies/Repos/reflekt/schemas/jaffle_shop/Product_Removed/1-0.json\n[18:56:30] INFO     Building dbt table 'product_removed' in source 'jaffle_shop_segment'\n[18:56:30] INFO     Building staging model 'stg_jaffle_shop_segment__product_removed.sql'\n[18:56:30] INFO     Building dbt documentation '_stg_jaffle_shop_segment__product_removed.yml'\n[18:56:30] INFO     Building dbt artifacts for schema: /Users/gclunies/Repos/reflekt/schemas/jaffle_shop/Product_Added/1-0.json\n[18:56:30] INFO     Building dbt table 'product_added' in source 'jaffle_shop_segment'\n[18:56:30] INFO     Building staging model 'stg_jaffle_shop_segment__product_added.sql'\n[18:56:30] INFO     Building dbt documentation '_stg_jaffle_shop_segment__product_added.yml'\n[18:56:30] INFO     Building dbt artifacts for schema: /Users/gclunies/Repos/reflekt/schemas/jaffle_shop/Checkout_Step_Viewed/1-0.json\n[18:56:30] INFO     Building dbt table 'checkout_step_viewed' in source 'jaffle_shop_segment'\n[18:56:30] INFO     Building staging model 'stg_jaffle_shop_segment__checkout_step_viewed.sql'\n[18:56:30] INFO     Building dbt documentation '_stg_jaffle_shop_segment__checkout_step_viewed.yml'\n[18:56:30] INFO     Building dbt artifacts for schema: /Users/gclunies/Repos/reflekt/schemas/jaffle_shop/Checkout_Step_Completed/1-0.json\n[18:56:30] INFO     Building dbt table 'checkout_step_completed' in source 'jaffle_shop_segment'\n[18:56:30] INFO     Building staging model 'stg_jaffle_shop_segment__checkout_step_completed.sql'\n[18:56:30] INFO     Building dbt documentation '_stg_jaffle_shop_segment__checkout_step_completed.yml'\n[18:56:30] INFO     Building dbt artifacts for schema: /Users/gclunies/Repos/reflekt/schemas/jaffle_shop/Page_Viewed/1-0.json\n[18:56:30] INFO     Building dbt table 'pages' in source 'jaffle_shop_segment'\n[18:56:30] INFO     Building staging model 'stg_jaffle_shop_segment__pages.sql'\n[18:56:30] INFO     Building dbt documentation '_stg_jaffle_shop_segment__pages.yml'\n[18:56:30] INFO     Building dbt artifacts for schema: Segment 'tracks' table\n[18:56:31] INFO     Building dbt table 'tracks' in source 'jaffle_shop_segment'\n[18:56:31] INFO     Building staging model 'stg_jaffle_shop_segment__tracks.sql'\n[18:56:31] INFO     Building dbt documentation '_stg_jaffle_shop_segment__tracks.yml'\n[18:56:31] INFO     Copying dbt package from temporary path /Users/gclunies/Repos/reflekt/.reflekt_cache/artifacts/dbt/jaffle_shop to /Users/gclunies/Repos/reflekt/artifacts/dbt/jaffle_shop\n[18:56:31] INFO     Successfully built dbt package\n```\n---\n\u003cbr\u003e\n\n## CLI Commands\nA description of commands can be seen by running `reflekt --help`. The help page for each CLI command is shown below.\n\n### `reflekt init`\n```bash\n❯ reflekt init --help\n[11:17:16] INFO     Running with reflekt=0.6.0\n\n Usage: reflekt init [OPTIONS]\n\n Initialize a Reflekt project.\n\n╭─ Options ──────────────────────────────────────────────────────╮\n│ --dir                        TEXT  [default: .]                │\n│ --verbose    --no-verbose          [default: no-verbose]       │\n│ --help                             Show this message and exit. │\n╰────────────────────────────────────────────────────────────────╯\n```\n\n### `reflekt debug`\n```bash\n❯ reflekt debug --help\n[11:18:07] INFO     Running with reflekt=0.6.0\n\n Usage: reflekt debug [OPTIONS]\n\n Check Reflekt project configuration.\n\n╭─ Options ───────────────────────────────────╮\n│ --help          Show this message and exit. │\n╰─────────────────────────────────────────────╯\n```\n\n### `reflekt lint`\n```bash\n❯ reflekt lint --help\n[11:20:29] INFO     Running with reflekt=0.6.0\n\n Usage: reflekt lint [OPTIONS]\n\n Lint schema(s) to test for naming and metadata conventions.\n\n╭─ Options ───────────────────────────────────────────────────────────────────────────────────────────────────────────╮\n│ *  --select   -s      TEXT  Schema(s) to lint. Starting with 'schemas/' is optional. [default: None] [required] │\n│    --verbose  -v            Verbose logging.                                                                        │\n│    --help                   Show this message and exit.                                                             │\n╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯\n```\n\n### `reflekt push`\n```bash\n❯ reflekt push --help\n[11:22:37] INFO     Running with reflekt=0.6.0\n\n Usage: reflekt push [OPTIONS]\n\n Push schema(s) to a schema registry.\n\n╭─ Options ─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮\n│ *  --registry  -r      [avo|segment]  Schema registry to push to. [default: None] [required]                                          │\n│ *  --select    -s      TEXT           Schema(s) to push to schema registry. Starting with 'schemas/' is optional. [default: None] │\n│    --delete    -D                     Delete schema(s) from schema registry. Prompts for confirmation                                 │\n│    --force     -F                     Force command to run without confirmation.                                                      │\n│    --profile   -p      TEXT           Profile in reflekt_profiles.yml to use for schema registry connection.                          │\n│    --verbose   -v                     Verbose logging.                                                                                │\n│    --help                             Show this message and exit.                                                                     │\n╰───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯\n```\n\n### `reflekt pull`\n```bash\n❯ reflekt pull --help\n[11:25:18] INFO     Running with reflekt=0.6.0\n\n Usage: reflekt pull [OPTIONS]\n\n Pull schema(s) from a schema registry.\n\n╭─ Options ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮\n│ *  --registry  -r      [avo|segment]  Schema registry to pull from. [default: None] [required]                                                                         │\n│ *  --select    -s      TEXT           Schema(s) to pull from schema registry. If registry uses tracking plans, starting with the plan name. [default: None] [required] │\n│    --profile   -p      TEXT           Profile in reflekt_profiles.yml to use for schema registry connection.                                                           │\n│    --verbose   -v                     Verbose logging.                                                                                                                 │\n│    --help                             Show this message and exit.                                                                                                      │\n╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯\n```\n\n### `reflekt build`\n```bash\n❯ reflekt build --help\n[11:31:36] INFO     Running with reflekt=0.6.0\n\n Usage: reflekt build [OPTIONS]\n\n Build data artifacts based on schemas.\n\n╭─ Options ────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮\n│ *  --artifact  -a      [dbt]      Type of data artifact to build. [default: None] [required]                                                                                                         │\n│ *  --select    -s      TEXT       Schema(s) to build data artifacts for. Starting with 'schemas/' is optional. [default: None] [required]                                                            │\n│ *  --sdk               [segment]  The SDK used to collect the event data. [default: None] [required]                                                                                                 │\n│ *  --source            TEXT       The \u003csource_id\u003e.\u003cdatabase\u003e.\u003cschema\u003e storing raw event data. \u003csource_id\u003e must be a data warehouse source defined in reflekt_profiles.yml [default: None] [required] │\n│    --profile   -p      TEXT       Profile in reflekt_profiles.yml to look for the data source specified by the --source option. Defaults to default_profile in reflekt_project.yml                   │\n│    --verbose   -v                 Verbose logging.                                                                                                                                                   │\n│    --help                         Show this message and exit.                                                                                                                                        │\n╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯\n```\n\n### `reflekt report`\n```bash\n❯ reflekt report --help\n[08:20:09] INFO     Running with reflekt=0.6.0\n\n Usage: reflekt report [OPTIONS]\n\n Generate Markdown report(s) for schema(s).\n\n╭─ Options ───────────────────────────────────────────────────────────────────────────────────────╮\n│ *  --select   -s      TEXT  Schema(s) to generate Markdown report(s) for. Starting with         │\n│                             'schemas/' is optional.                                             │\n│                             [default: None]                                                     │\n│                             [required]                                                          │\n│    --to-file  -f            Write report(s) to file instead of terminal.                        │\n│    --verbose  -v            Verbose logging.                                                    │\n│    --help                   Show this message and exit.                                         │\n╰─────────────────────────────────────────────────────────────────────────────────────────────────╯\n```\n\n---\n\u003cbr\u003e\n\n## Integrations\n\n### Customer Data Platform (CDP)\nReflekt understands how Customer Data Platforms (CDPs) collect event data and load them into data warehouses, allowing it to:\n  - Parse the schemas in a Reflekt project.\n  - Find matching tables for events and columns for properties in the data warehouse.\n  - Build a `dbt` package with sources, staging models, and documentation for event data.\n\n| CDP | Supported |\n|-----|-----------|\n| [Segment](https://segment.com/) | ✅ |\n| [Rudderstack](https://www.rudderstack.com/) | 🚧 Coming Soon 🚧 |\n| [Amplitude](https://amplitude.com/) | 🚧 Coming Soon 🚧  |\n\n\n### Schema Registry\nSchema registries store and serve schemas. When a schema is registered in a regsitry, it can be used to validate events as they flow through your data collection infrastructure. Reflekt works with schema registries from CDPs, SaaS vendors, and open-source projects, letting teams to decide between managed and self-hosted solutions.\n\n| Registry |  Open Source  | Schema Versions | Recommended Workflow |\n|----------|:-------------:|-----------------|----------------------|\n| [Segment Protocols](https://segment.com/docs/protocols/) | ❌ | `MAJOR` only | Manage schemas in Reflekt.\u003cbr\u003e `reflekt push` to Protocols for event validation.\u003cbr\u003e `reflekt build --artifact dbt` to build dbt package. |\n| [Avo](https://www.avo.app/) | ❌ | `MAJOR` only | Manage schemas in Avo.\u003cbr\u003e `reflekt pull` to get schemas.\u003cbr\u003e  `reflekt build --artifact dbt` to build dbt package. |\n| [reflekt-registry](https://github.com/GClunies/reflekt-registry)\u003cbr\u003e 🚧 Coming Soon 🚧 | ✅ | `MAJOR` \u0026 `MINOR` |  Manage schemas in Reflekt.\u003cbr\u003e `reflekt push` to reflekt-registry.\u003cbr\u003e `reflekt build --artifact dbt` to build dbt package. |\n\n### Data Warehouse\nIn order to build dbt packages, Reflekt needs to connect to a cloud data warehouse where raw event data is stored.\n\n| Data Warehouse | Supported |\n|----------------|-----------|\n| [Snowflake](https://www.snowflake.com/) | ✅ |\n| [Redshift](https://aws.amazon.com/redshift/) | ✅ |\n| [BigQuery](https://cloud.google.com/bigquery) | ✅ |\n\nReflekt **NEVER** copies, moves, or modifies events in the data warehouse. It ONLY reads table and column names for templating.\n\n### dbt\n[dbt](https://www.getdbt.com/) enables anyone that knows SQL to transform data in a cloud data warehouse. When modeling in dbt, it is [best practice](https://docs.getdbt.com/guides/best-practices/how-we-structure/1-guide-overview) to:\n- Define sources pointing to the raw data.\n- Write staging models that [rename, recast, or usefully reconsider](https://discourse.getdbt.com/t/how-we-used-to-structure-our-dbt-projects/355#data-transformation-101-1) columns into a consistent format.\n- Document and test the staging models.\n\nBut as the number of events intsrumented in your product grow or change as the product evolves, mainting this best practice can be **burdensome and boring.** This is where [`reflekt build`](#reflekt-build) steps in.\n\n## Contribute\n- Source Code: [github.com/GClunies/reflekt](https://github.com/GClunies/reflekt)\n- Issue Tracker: [github.com/GClunies/reflekt/issues](https://github.com/GClunies/reflekt/issues)\n- Pull Requests: [github.com/GClunies/reflekt/pulls](https://github.com/GClunies/reflekt/pulls)\n\n## License\nThis project is [licensed](LICENSE) under the Apache-2.0 License.\n","funding_links":[],"categories":["Tools"],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FGClunies%2FReflekt","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FGClunies%2FReflekt","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FGClunies%2FReflekt/lists"}