{"id":13808042,"url":"https://github.com/grai-io/grai-core","last_synced_at":"2025-05-15T05:05:19.460Z","repository":{"id":46591534,"uuid":"501725302","full_name":"grai-io/grai-core","owner":"grai-io","description":null,"archived":false,"fork":false,"pushed_at":"2025-05-08T15:11:47.000Z","size":127240,"stargazers_count":304,"open_issues_count":17,"forks_count":21,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-05-15T05:05:00.220Z","etag":null,"topics":["data","data-lineage","data-science","dataengineering","datalineage","dbt","django","fivetran","hacktoberfest","mssql","mysql","open-source","parquet","postgresql","python","redshift","snowflake"],"latest_commit_sha":null,"homepage":"https://www.grai.io","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit-0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/grai-io.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":".github/CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":"SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2022-06-09T16:16:22.000Z","updated_at":"2025-05-01T04:31:59.000Z","dependencies_parsed_at":"2022-08-12T13:00:34.556Z","dependency_job_id":"fba72494-8588-466b-99d0-9abf67700673","html_url":"https://github.com/grai-io/grai-core","commit_stats":null,"previous_names":[],"tags_count":49,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/grai-io%2Fgrai-core","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/grai-io%2Fgrai-core/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/grai-io%2Fgrai-core/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/grai-io%2Fgrai-core/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/grai-io","download_url":"https://codeload.github.com/grai-io/grai-core/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254276446,"owners_count":22043866,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["data","data-lineage","data-science","dataengineering","datalineage","dbt","django","fivetran","hacktoberfest","mssql","mysql","open-source","parquet","postgresql","python","redshift","snowflake"],"created_at":"2024-08-04T01:01:34.110Z","updated_at":"2025-05-15T05:05:19.442Z","avatar_url":"https://github.com/grai-io.png","language":"Python","funding_links":[],"categories":["Observability \u0026 Monitoring","Testing","GenAI Readiness Features"],"sub_categories":["Data Profiler","Data Quality, Observability \u0026 Governance"],"readme":"\u003cdiv align=\"center\"\u003e\n  \u003cimg src=\"docs/assets/Grai-Logo-Horizontal-2.png\" width=\"375px\"/\u003e\u003cbr/\u003e\n\u003c/div\u003e\n\n\u003cp align=\"center\"\u003e\n  \u003ca href=\"https://join.slack.com/t/graicommunity/shared_invite/zt-1il70kfeb-TaCm5fwHg_quWCpKNYyj6w\"\u003e\n    \u003cimg src=\"https://img.shields.io/badge/slack-Grai-FFB567.svg?logo=slack\u0026labelColor=351D36\" alt=\"Join Grai on Slack\" height=\"22\"/\u003e\n  \u003c/a\u003e\n   \u003ca href=\"https://github.com/grai-io/grai-core\"\u003e\n    \u003cimg src=\"https://img.shields.io/github/issues/grai-io/grai-core?style=flat-square\" height=\"22\" alt=\"Open Issues\"/\u003e\n  \u003c/a\u003e\n  \u003ca href=\"https://github.com/grai-io/grai-core\"\u003e\n    \u003cimg src=\"https://img.shields.io/github/languages/top/grai-io/grai-core?style=flat-square\" height=\"22\" alt=\"Python fraction of codebase\"/\u003e\n  \u003c/a\u003e\n  \u003ca href=\"https://github.com/grai-io/grai-core\"\u003e\n    \u003cimg src=\"https://img.shields.io/pypi/pyversions/grai-graph?style=for-the-badge\" height=\"22\" alt=\"Supported python versions\"/\u003e\n  \u003c/a\u003e\n  \u003ca href=https://www.ycombinator.com/launches/H8h-grai-open-source-version-control-for-metadata target='_blank'\u003e\n    \u003cimg alt=\"Launch YC: ✨ Grai - Open-source version control for metadata\" src=\"https://www.ycombinator.com/launches/H8h-grai-open-source-version-control-for-metadata/upvote_embed.svg\" height=\"22\"/\u003e\n  \u003c/a\u003e\n\u003c/p\u003e\n\n\u003ch3 align=\"center\"\u003e\n  \u003ca href=\"https://docs.grai.io/\"\u003e\u003cb\u003eDocumentation\u003c/b\u003e\u003c/a\u003e \u0026bull;\n  \u003ca href=\"https://www.grai.io\" title=\"Grai Homepage\"\u003e Website \u003c/a\u003e \u0026bull;\n  \u003ca href=\"https://join.slack.com/t/graicommunity/shared_invite/zt-1il70kfeb-TaCm5fwHg_quWCpKNYyj6w\"\u003e\u003cb\u003eSlack\u003c/b\u003e\u003c/a\u003e \u0026bull;\n  \u003ca href=\"https://github.com/orgs/grai-io/discussions/\" title=\"Community Page\"\u003e Discussion \u003c/a\u003e \u0026bull;\n  \u003ca href=\"https://calendly.com/grai-founders\"\u003e\u003cb\u003eWant to Chat?\u003c/b\u003e\u003c/a\u003e\n\u003c/h3\u003e\n\n\n## Introduction\n\n**Data lineage made simple.**\nGrai makes it easy to understand and test how your data relates across databases, warehouses, APIs, and dashboards.\n\n- **Pre-built connectors.** Automatically synchronize lineage from across the stack so your metadata is never out of date.\n- **Centralized data tests.** Write data validation tests that run whenever upstream data sources change.\n- **Integrated with GitHub.** Run data validation tasks as part of your CI/CD process to test changes everywhere your data is used.\n- **Your data, your cloud.**  is fully open source and self-hosted. You maintain full control over your data and hosting environment.\n\n## How it works\n\n- Automatically build column level lineage spanning your warehouse and production services with connectors for `dbt`, `Snowflake`, `Fivetran`, and more (see below).\n- Get alerts in your CI/CD workflows whenever changes to a production system will impact your warehouse or dbt projects with [GitHub Actions](https://github.com/-io/grai-actions-server).\n- Self-host the project or run it in the [Grai Cloud](https://app.grai.io) for free.\n\n\n### Connectors\n\n|                                                                   | integration   | install                             |\n|-------------------------------------------------------------------|---------------|-------------------------------------|\n| \u003cimg src=\"resources/snowflake.png\" alt=“” width=\"30\" height=\"30\"\u003e | Snowflake     | `pip install grai-source-snowflake` |\n| \u003cimg src=\"resources/bigquery.svg\" alt=“” width=\"30\" height=\"30\"\u003e  | BigQuery      | `pip install grai-source-bigquery`  |\n| \u003cimg src=\"resources/redshift.png\" alt=“” width=\"30\" height=\"30\"\u003e  | Redshift      | `pip install grai-source-redshift`  |\n| \u003cimg src=\"resources/postgres.png\" alt=“” width=\"30\" height=\"30\"\u003e  | Postgres      | `pip install grai-source-postgres`  |\n| \u003cimg src=\"resources/mysql.png\" alt=“” width=\"30\" height=\"30\"\u003e     | MySQL         | `pip install grai-source-mysql`     |\n| \u003cimg src=\"resources/mssql.png\" alt=“” width=\"30\" height=\"30\"\u003e     | SQL Server    | `pip install grai-source-mssql`     |\n| \u003cimg src=\"resources/dbt.png\" alt=“” width=\"30\" height=\"30\"\u003e       | dbt           | `pip install grai-source-dbt`       |\n| \u003cimg src=\"resources/fivetran.png\" alt=“” width=\"30\" height=\"30\"\u003e  | Fivetran      | `pip install grai-source-fivetran`  |\n| \u003cimg src=\"resources/csv.png\" alt=“” width=\"30\" height=\"30\"\u003e       | csv, parquet, feather | `pip install grai-source-flat-file` |\n| \u003cimg src=\"resources/metabase.png\" alt=“” width=\"30\" height=\"30\"\u003e  | Metabase      | `pip install grai-source-metabase`  |\n| \u003cimg src=\"resources/looker.svg\" alt=“” width=\"25\" height=\"40\"\u003e    | Looker (alpha) | `pip install grai-source-looker`    |\n\n\n## Quickstart\n\nYou can find a full quickstart guide in the [documentation](https://docs.grai.io/quick-start) which covers deploying your own instance of Grai and getting set up with your first connector in Python.\nThe fastest way to get started is through the Grai CLI but you can also run the project locally with docker compose.\n\nDefault login credentials:\n\n```\nusername: null@grai.io\npassword: super_secret\n```\n\n### CLI\n\n```bash\npip install grai-cli\ngrai demo start\n```\n\n### Running Locally\n\nYou can always find pre-built images of the backend server at `ghcr.io/grai-io/grai-core/grai-server:latest` and the frontend at `ghcr.io/grai-io/grai-core/grai-frontend:latest`, however, if you prefer to build from source, you can do so with docker compose.\n\n```bash\ngit clone https://github.com/grai-io/grai-core\ncp examples/deployment/docker-compose/docker-compose.yml\ndocker compose up\n```\n\nThe backend server will be available at [http://localhost:8000/](http://localhost:8000/) and the frontend is now here [http://localhost:3000/](http://localhost:3000/).\n\nAfter logging in and connecting a data source you'll be greeted with a lineage graph looking something like this\n\n![Frontend](resources/frontend.png)\n\nFor more information about using the web application check out the [getting started guide](https://docs.grai.io/web-app/getting-started).\n\n### Other Deployment Mechanisms\n\nYou can find example configurations for docker compose and Kubernetes in the [examples](/examples/deployment) folder.\n\n#### Helm\n\nWe also publish a set of [Helm charts](https://charts.grai.io/) which are available if you prefer.\n\n```\nhelm repo add grai https://charts.grai.io\nhelm install grai grai/grai\n```\n\n## Component Services\n\n* [grai-server](https://github.com/grai-io/grai-core/tree/master/grai-server): The backend metadata service built on Postgres and Django as the Metadata persistence layer.\n* [grai-frontend](https://github.com/grai-io/grai-core/tree/master/grai-frontend): The frontend web application built on React.\n* [grai-cli](https://github.com/grai-io/grai-core/tree/master/grai-client): Python client library for interacting with the Grai server.\n* [grai-schemas](https://github.com/grai-io/grai-core/tree/master/grai-schemas): The python metadata schema implementation library of Grai. It provides a standardized view of all Grai objects used to ensure compatibility between the server, integrations, and the client.\n* [grai-graph](https://github.com/grai-io/grai-core/tree/master/grai-graph): A python utility library for working with the Grai metadata graph.\n* [grai-actions](https://github.com/grai-io/grai-actions): A library of GitHub Actions implementations to integrate Grai tests into your CI/CD pipelines.\n* [integrations](https://github.com/grai-io/grai-core/tree/master/grai-integrations): A collection of integration libraries to extract metadata and persist their results to Grai.\n\n\n## Community Roadmap\n\nCommunity Feedback drives our roadmap. Please let us know what you'd like to see next by asking questions and upvoting feature requests!\n\n* [Feature Requests](https://github.com/orgs/grai-io/discussions/categories/feature-requests)\n* [Documentation Requests](https://github.com/orgs/grai-io/discussions/categories/documentation-requests)\n* [Bug Reports](https://github.com/grai-io/grai-core/issues)\n* [FAQ](https://github.com/orgs/grai-io/discussions/categories/q-a)\n\n## Repo Activity\n\n![Repo activity](https://repobeats.axiom.co/api/embed/31e89b7eda9ea0ebad3005fff55589496f79dc2d.svg \"Repobeats analytics image\")\n\n## Community\n\nEmail us: founders@grai.io\n\nJoin us on Slack:  \u003ca href=\"https://join.slack.com/t/graicommunity/shared_invite/zt-1il70kfeb-TaCm5fwHg_quWCpKNYyj6w\"\u003e\u003cimg src=\"https://img.shields.io/badge/slack-Grai-FFB567.svg?logo=slack\u0026labelColor=351D36\" alt=\"Join Grai on Slack\"\u003e\u003c/a\u003e\n\nCheck us out at www.grai.io\n\nSign up for our Newsletter `Grai Matters` [email list](https://blog.grai.io/#/portal/signup).\n\n\u003ca href=\"https://github.com/grai-io/grai-core/graphs/contributors\"\u003e\n  \u003cimg src=\"https://contrib.rocks/image?repo=grai-io/grai-core\" /\u003e\n\u003c/a\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgrai-io%2Fgrai-core","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgrai-io%2Fgrai-core","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgrai-io%2Fgrai-core/lists"}