{"id":20936903,"url":"https://github.com/datafold/demo","last_synced_at":"2025-05-13T21:31:15.083Z","repository":{"id":61696452,"uuid":"551595718","full_name":"datafold/demo","owner":"datafold","description":null,"archived":false,"fork":false,"pushed_at":"2025-05-11T00:39:19.000Z","size":4271,"stargazers_count":8,"open_issues_count":18,"forks_count":18,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-05-11T01:24:26.170Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/datafold.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2022-10-14T17:53:15.000Z","updated_at":"2025-05-11T00:23:32.000Z","dependencies_parsed_at":"2024-01-16T22:38:20.328Z","dependency_job_id":"71d776d3-69a4-4c7d-b46c-b3995c21f592","html_url":"https://github.com/datafold/demo","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/datafold%2Fdemo","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/datafold%2Fdemo/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/datafold%2Fdemo/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/datafold%2Fdemo/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/datafold","download_url":"https://codeload.github.com/datafold/demo/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254030963,"owners_count":22002681,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-18T22:29:32.822Z","updated_at":"2025-05-13T21:31:15.034Z","avatar_url":"https://github.com/datafold.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Datafold Demo Project\n\nThis repo contains a demo project suited to leveraging Datafold:\n- dbt project that includes\n  - raw data (implemented via [seed CSV files](https://docs.getdbt.com/docs/building-a-dbt-project/seeds)) from a fictional app\n  - a few downstream models, as shown in the project DAG below\n- several 'master' branches, corresponding to the various supported cloud data platforms\n  - `master` - 'primary' master branch, runs in Snowflake\n  - `master-databricks` - 'secondary' master branch, runs in Databricks, is reset to the `master` branch daily or manually when needed via the `branch_replication.yml` workflow\n  - `master-bigquery` - 'secondary' master branch, runs in BigQuery, is reset to the `master` branch daily or manually when needed via the `branch_replication.yml` workflow\n  - `master-dremio` - 'secondary' master branch, runs in Dremio, is reset to the `master` branch daily or manually when needed via the `branch_replication.yml` workflow\n- several GitHub Actions workflows illustrating CI/CD best practices for dbt Core\n  - dbt PR job - is triggered on PRs targeting the `master` branch, runs dbt project in Snowflake\n  - dbt prod - is triggered on pushes into the `master` branch, runs dbt project in Snowflake\n  - dbt PR job (Databricks) - is triggered on PRs targeting the `master-databricks` branch, runs dbt project in Databricks\n  - dbt prod (Databricks) - is triggered on pushes into the `master-databricks` branch, runs dbt project in Databricks\n  - dbt PR job (BigQuery) - is triggered on PRs targeting the `master-bigquery` branch, runs dbt project in BigQuery\n  - dbt prod (BigQuery) - is triggered on pushes into the `master-bigquery` branch, runs dbt project in BigQuery\n  - dbt PR job (Dremio) - is triggered on PRs targeting the `master-dremio` branch, runs dbt project in BigQuery\n  - dbt prod (Dremio) - is triggered on pushes into the `master-dremio` branch, runs dbt project in BigQuery\n  - Apply monitors.yaml configuration to Datafold app - applies monitor-as-code configuration to Datafold application\n  - raw data generation tool to simulate a data flow typical for real existing projects\n\n\u003cp align=\"center\"\u003e\n    \u003cimg src=\"img/demo_project_dag.png\" width=\"750\"\u003e\n\u003c/p\u003e\n\n## Running this project in the pre-configured Datafold environment\n\n### Code management\nAll actual changes should be commited to the `master` branch, other `master-...` branches are supposed to be reset to the `master` branch daily.\n\n### CI demo\n! To ensure the integrity and isolation of GitHub Actions workflows, it is advisable to create pull requests (PRs) for different 'master' branches from distinct commits. This practice helps prevent cross-PR leakage and ensures that workflows run independently.\n\n#### Snowflake\nTo demonstrate Datafold experience in CI on Snowflake - one needs to create PRs targeting the `master` branch.\n- production schema in Snowflake: `demo.core`\n- PR schemas: `demo.pr_num_\u003cpr_number\u003e`\n\n#### Databricks\nTo demonstrate Datafold experience in CI on Databricks - one needs to create PRs targeting the `master-databricks` branch.\n- production schema in Databricks: `demo.default`\n- PR schemas: `demo.pr_num_\u003cpr_number\u003e`\n\n#### BigQuery\nTo demonstrate Datafold experience in CI on BigQuery - one needs to create PRs targeting the `master-bigquery` branch.\n- production schema in BigQuery: `datafold-demo-429713.prod`\n- PR schemas: `datafold-demo-429713.pr_num_\u003cpr_number\u003e`\n\n#### Dremio\nTo demonstrate Datafold experience in CI on Dremio - one needs to create PRs targeting the `master-dremio` branch.\n- production schema in Dremio: `\"Alexey S3\".alexeydremiobucket.prod`\n- PR schemas: `\"Alexey S3\".alexeydremiobucket.pr_num_\u003cpr_number\u003e`\n\n### Data replication demo\n\nTo demonstrate Datafold functionality for data replication monitoring, a pre-configured Postgres instance (simulates transactional database) is populated with 'correct raw data' (`analytics.data_source.subscription_created` table); the `subscription__created` seed CSV file contains 'corrupted raw data'.\n\n### BI apps demo\n- Looker view, explore, and dashboard are connected to the `fct__monthly__financials` model in Snowflake, Databricks, and BigQuery.\n  - Snowflake\n    - `fct__monthly__financials` view\n    - `fct__monthly__financials` explore\n    - `Monthly Financials (Demo, Snowflake)` dashboard\n  - Databricks\n    - `fct__monthly__financials_databricks` view\n    - `fct__monthly__financials_databricks` explore\n    - `Monthly Financials (Demo, Databricks)` dashboard\n  - BigQuery\n    - `fct__monthly__financials_bigquery` view\n    - `fct__monthly__financials_bigquery` explore\n    - `Monthly Financials (Demo, BigQuery)` dashboard\n\n- Tableau data source, workbook, and dashboard are connected to the `fct__yearly__financials` model in Snowflake, Databricks, and BigQuery.\n  - Snowflake\n    - `FCT__YEARLY__FINANCIALS (DEMO.FCT__YEARLY__FINANCIALS) (CORE)` data source\n    - `Yearly Financials (Snowflake)` workbook\n    - `Yearly Financials Dashboard (Snowflake)` dashboard\n  - Databricks\n    - `fct__yearly__financials (demo.default.fct__yearly__financials) (default)` data source\n    - `Yearly Financials (Databricks)` workbook\n    - `Yearly Financials Dashboard (Databricks)` dashboard\n  - BigQuery\n    - `fct__yearly__financials (prod)` data source\n    - `Yearly Financials (BigQuery)` workbook\n    - `Yearly Financials Dashboard (BigQuery)` dashboard\n\n- Power BI table, report, and dashboard are connected to the `fct__monthly__financials` model in Snowflake, Databricks, and BigQuery.\n  - Snowflake\n    - `FCT__MONTHLY__FINANCIALS` table\n    - `Monthly Financials Snowflake` report\n    - `Monthly Financials Snowflake` dashboard\n  - Databricks\n    - `fct__monthly__financials` table\n    - `fact-monthly-financials-databricks` report\n    - `Fact Monthly Financials Databricks` dashboard\n  - BigQuery\n    - `fct__monthly__financials` table\n    - `Monthly Financials BigQuery` report\n    - `Monthly Financials BigQuery` dashboard\n\n\n### Datafold Demo Org structure\nThe corresponding Datafold Demo Org contains the following integrations:\n- Common\n  - `datafold/demo` repository integration\n  - `Postgres` data connection for Cross-DB data diff monitors\n  - `Looker Public Demo` BI app integration\n  - `Power BI` BI app integration\n  - `Tableau Public Demo` BI app integration\n- Snowflake specific\n  - `Snowflake` data connection\n  - `Coalesce-Demo` CI integration for the `Snowflake` data connection and the `master` branch\n- Databricks specific\n  - `Databricks-Demo` data connection\n  - `Coalesce-Demo-Databricks` CI integration for the `Databricks-Demo` data connection and the `master-databricks` branch\n- BigQuery specific\n  - `BigQuery - Demo` data connection\n  - `Coalesce-Demo-BigQuery` CI integration for the `BigQuery - Demo` data connection and the `master-bigquery` branch\n- Dremio specific\n  - `Dremio-Demo` data connection\n  - `Coalesce-Demo-Dremio` CI integration for the `Dremio-Demo` data connection and the `master-dremio` branch\n\n## Running this project in a custom environment\nTo get up and running with this project:\n1. Install dbt using [these instructions](https://docs.getdbt.com/docs/installation).\n\n2. Fork this repository.\n\n3. Set up a profile called `demo` to connect to a data warehouse by following [these instructions](https://docs.getdbt.com/docs/configure-your-profile). You'll need `dev` and `prod` targets in your profile.\n\n4. Ensure your profile is setup correctly from the command line:\n```bash\n$ dbt debug\n```\n\n5. Create your `prod` models:\n```bash\n$ dbt build --profile demo --target prod\n```\n\nWith `prod` models created, you're clear to develop and diff changes between your `dev` and `prod` targets.\n\n### Using Datafold with this project\n\nFollow the [quickstart guide](https://docs.datafold.com/quickstart_guide) to integrate this project with Datafold.\n\n## Generated data\n\n### Generated files\n- `datagen/feature_used_broken.csv` - copied to `seeds/feature__used.csv`\n- `datagen/feature_used.csv`\n- `datagen/org_created_broken.csv` - copied to `seeds/org__created.csv.csv`\n- `datagen/org_created.csv`\n- `datagen/signed_in_broken.csv` - copied to `seeds/signed__in.csv.csv`\n- `datagen/signed_in.csv`\n- `datagen/subscription_created_broken.csv` - copied to `seeds/subscription__created.csv.csv`\n- `datagen/subscription_created.csv` - pushed to Postgres (`analytics.data_source.subscription_created` table)\n- `datagen/user_created_broken.csv` - copied to `seeds/user__created.csv.csv`\n- `datagen/user_created.csv`\n- `datagen/persons_pool.csv` - pool of persons used for user/org generation\n\n### Data generation scripts\n\n- `datagen/data_generate.py` - main data generation script\n- `datagen/data_to_postgres.sh` - pushes generated data to Postgres \n- `datagen/persons_pool_replenish.py` - replenishes the pool of persons using ChatGPT\n- `datagen/data_delete.sh` - deletes data for further re-generation\n- `datagen/dremio__upload_seeds.py` - uploads seed files to Dremio (due to limitations in the starndard dbt-dremio connector)\n\n### Data anomaly types\n- zero on negative prices in the `subscription__created` seed\n- corrupted emails in the `user__created` seed (user$somecompany.com)\n- irregular spikes in the workday seasonal daily number of sign-ins in the `signed__in` seed\n- `null` spikes in the `feature__used` seed\n- schema change: a 'wandering' column appears ~weekly in the `signed__in` seed\n\n## Other\n\n### Known issues\n- PR job fails when the 2nd commit is pushed to a PR branch targeting Databricks. Most likely related to: https://github.com/databricks/dbt-databricks/issues/691.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdatafold%2Fdemo","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdatafold%2Fdemo","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdatafold%2Fdemo/lists"}