{"id":20261039,"url":"https://github.com/datarootsio/notion-dbs-data-quality","last_synced_at":"2025-04-11T01:45:17.724Z","repository":{"id":45946909,"uuid":"431458472","full_name":"datarootsio/notion-dbs-data-quality","owner":"datarootsio","description":"Using Great Expectations and Notion's API, this repo aims to provide data quality for our databases in Notion.","archived":false,"fork":false,"pushed_at":"2021-11-30T16:53:16.000Z","size":58994,"stargazers_count":9,"open_issues_count":1,"forks_count":0,"subscribers_count":7,"default_branch":"main","last_synced_at":"2025-04-01T15:53:47.879Z","etag":null,"topics":["data-engineering-pipeline","data-quality","great-expectations","notion","notion-api","notion-database"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/datarootsio.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2021-11-24T11:30:39.000Z","updated_at":"2023-10-16T17:01:51.000Z","dependencies_parsed_at":"2022-09-23T09:30:27.671Z","dependency_job_id":null,"html_url":"https://github.com/datarootsio/notion-dbs-data-quality","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/datarootsio%2Fnotion-dbs-data-quality","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/datarootsio%2Fnotion-dbs-data-quality/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/datarootsio%2Fnotion-dbs-data-quality/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/datarootsio%2Fnotion-dbs-data-quality/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/datarootsio","download_url":"https://codeload.github.com/datarootsio/notion-dbs-data-quality/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248327863,"owners_count":21085258,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["data-engineering-pipeline","data-quality","great-expectations","notion","notion-api","notion-database"],"created_at":"2024-11-14T11:23:38.960Z","updated_at":"2025-04-11T01:45:17.704Z","avatar_url":"https://github.com/datarootsio.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\n[![maintained by dataroots](https://img.shields.io/badge/maintained%20by-dataroots-%2300b189)](https://dataroots.io) [![PythonVersion](https://img.shields.io/badge/python-3.9-blue)](https://img.shields.io/badge/python-3.9-blue) [![PRs Welcome](https://img.shields.io/badge/PRs-welcome-brightgreen.svg?style=flat-square)](http://makeapullrequest.com) ![](https://media-exp1.licdn.com/dms/image/C4D1BAQFJFecNiY6xNA/company-background_10000/0/1606894615032?e=1628604000\u0026v=beta\u0026t=hNYzs9y3EA-620Ck8ip1QaZc77eXlH1ZUl-E-sLI6wo \"Logo\")\u003ca href=\"https://github.com/psf/black\"\u003e\u003cimg alt=\"Code style: black\" src=\"https://img.shields.io/badge/code%20style-black-000000.svg\"\u003e\u003c/a\u003e [![Imports: isort](https://img.shields.io/badge/%20imports-isort-%231674b1?style=flat\u0026labelColor=ef8336)](https://pycqa.github.io/isort/)\n\n# Data Quality for Notion Databases 🚀\nUsing [Notion's API](https://developers.notion.com/) + [Great Expectations](https://greatexpectations.io/), you'll be able to easily set up expectations for your Notion's database, while taking advantage of their web-based Data Docs to explore your results\n\n![languages result](./resources/new_data_docs_validation_results.png)\n\n## 👶 ELI5: *'expectations'*\nExpectations, as the name implies are qualities you expect from your data. The most common are column-scoped, but there are also row and table-scoped.\n\nExpectations can be as simple as [\"I want to be sure that this column is never null\"](https://greatexpectations.io/expectations/expect_column_values_to_not_be_null) or [\"I want to make sure the row count is always X\".](https://greatexpectations.io/expectations/expect_table_row_count_to_equal) If you want to dig deeper or find a list of possible expectations, you can do so at [Great Expectation's official site](https://greatexpectations.io/expectations)\n\n🚨Be aware that if you don't find an expectation you need, you can always make it yourself. Find the tutorial for that [here](https://docs.greatexpectations.io/docs/guides/expectations/creating_custom_expectations/how_to_create_custom_expectations/).\n\n# What's in here? 📑\nThis repo contains the following:\n- Minimal, lightweight Notion API focused on databases\n- Minimal setup for a Great Expectations project\n- Scripts to connect your databases to expectation suites, check your Notion API token, etc...\n\n# Installing dependencies\n\nRequirements:\n\n- Python ≥3.9\n- Poetry ([official instructions](https://python-poetry.org/docs/)).\n  - For linux, run:  `curl -sSL https://raw.githubusercontent.com/python-poetry/poetry/master/get-poetry.py | python -`\n\n1. Clone the repo\n2. Run `poetry install`\n\n# Example / Demo\nWe've made available a public database so you can go through the whole process without having to have a database yourself.\n\n### Step 1: Open and copy our public database [here](https://dataroots.notion.site/Notion-DB-Data-Quality-2753e01d5f694e43b9455c74743395ae)\nIt should look something like the image below. Be aware this may change in the future.\n\n![public db](./resources/copy_public_db.png)\n\nNow you can duplicate it into your workspace by clicking on \"Duplicate\" on the top-right corner.\n\nOnce you've successfully duplicated it, the databse is yours. You are free to edit it or keep it for the rest of the demo.\n\n# Step 2: Authentication 🔐\n**First things first**: before you can do anything, you need to have a token to authenticate to Notion's API. Notion does this via 'Integrations'.\n\n## Authenticating (creating a Notion integration) 🤫\n\nFor the most up to date, follow the official guide for authentication [here](https://developers.notion.com/docs/authorization). Choose \"internal integration\".\n\nGo to [https://www.notion.so/my-integrations](https://www.notion.so/my-integrations) and create your internal integration.\n\n![create integration](./resources/create_integration.gif)\n\nOnce you have your integration's token (it starts with '**secret_**'), be sure that it is accessible as the environment variable `NOTION_API_KEY` as both the notebook and scripts will look for it there.\n\nTo do so, in linux do `export NOTION_API_KEY=secret_XXXXX`.\n\nTo check that your token is valid and the script can access it, you can run:  `poetry run python ./src/check_connection.py`\n\n\n## Allowing your integration to access your db 🔓\n\nCreating the integration is not enough. The integration must have access to each of the databases (this also applies to databases to which one of your original databases 'relate to'.\n\nTo give access to a page or db to your integration. Open the page/db and on the top right click \"Share\". Then look for the name of your integration as if it were an user.\n\n![allow integration image](./resources/allow_integration_on_page.gif)\n\n# Step 3: Your first Expectation Suite\n\n**🚨There are a pair of expectation suite examples included. Feel free to skip to step #4 if you just want to run the demo ASAP 🚨**\n\nAn *expectation suite* is Great Expectations' way of calling a group of expectations.\n\n## Creating an expectation suite\nIncluded is a jupyter notebook to interactively create your expectations. This is useful as it lets you see how your data looks like and if your expectations will run or not with your current data.\n\nrun `poetry run jupyter notebook --script src/build_expectations.ipynb` and follow the instructions inside the notebook.\n\n### Step 3.1 Getting your db's url\nFor this and further steps you'll need your db's url. This is not the page itself where the db lives, but the db is a page of itself.\n\nSimply click on the \"...\" on the db and choose \"open as page\". Then copy (including the https) url.\n![getting link](./resources/new_get_db_link.gif)\n\nInside the jupyter notebook, you'll be able to interactively create expectations while seeing results, like so:\n\n![notebook interactive gif](./resources/expectation_notebook_interactive_loop.gif)\n\n⚠️ **Be sure to run the last cell to save your expectations!** ⚠️\n\n## Where are my expectations? 🕵\nSo you've created your expectation suite and saved it using the jupyter notebook we talked about above. Now you want to see where are your expectations saved, right?\n\n### Option 1: In Data Docs (Web)\nAnother great way of seeing your available expectation suites is through the Data Docs, which are explained below. In short, run `poetry run great_expectations docs build`, answer yes and the your Data Docs website should open automatically.\n\n![data docs expectations](./resources/new_expectation_suite_data_docs.png)\n\n### Option 2: In terminal\nIf you only want to see a list of your created expectation suites in the terminal, run:  `poetry run great_expectations suite list`\n\n### Option 3: JSON files\nYou can find all your expectation suites as .json files under [./great_expectations/expectations/](./great_expectations/expectations/)\n\n# Step 4: Running your expectations ✅\n\nNow you have your expectations and you want to start running some validations, right? Or maybe you skipped step #3 and will use the provided examples, that's ok, too.\n\nThe script [./src/validate.py](./src/validate.py) is there to help you run your validations by running an Expectation Suite against a Notion DB.      \nRun `poetry run python src/validate.py -h` to see an explanation of each argument.\n\nIn the simplest case, you'll need the following 3 arguments:\n\n- **run_name** (string in between quotes)\n  - Example: \"Running provided example_3_columns_and_2_languages\"\n- **expectation_suite** (string, expectation suite)\n  - Example: example_3_columns\n- **db** (string, https link of the database in any view)\n  - Example: https://www.notion.so/7510946dd2d64d94925c4389291f71c7?v=8fa5e1afced34272b250f69699a21df9\n  - 🚨 This url varies by workspace so it will be different for you. Refer to step 3.1 above to learn how to get yours.\n\nTo run this example: `poetry run python src/validate.py --run_name \"Running provided example_3_columns_and_2_languages\" --expectation_suite example_3_columns --db https://www.notion.so/7510946dd2d64d94925c4389291f71c7?v=8fa5e1afced34272b250f69699a21df9`\n\n# Data Docs (seeing your results, expectation suites and history) 📊\nOne of the great things from Great Expectations is their Data Docs. Data Docs are these HTML pages that Great Expectations compile from your expectation suites and validation runs. To learn more, here is the original [website](https://docs.greatexpectations.io/docs/reference/data_docs/).\n\nHere you can see a log of all your previous runs with information on a per-expectation level.\n\n![languages result](./resources/new_data_docs_validation_results.png)\n\nTo see your Data Docs, run: `poetry run great_expectations docs build`. This will prompt a yes/no question, answer 'y' and the website should automatically open in a web browser.\n## Result Example\nIn Data Docs you can see the results of all your expectations.\nIn the image below, you can see a subset of the results, for column 'Favorite Country' and 'Favorite Foods'\n- ✅ there are no empty entries on the column 'Favorite Country'\n- ❌ there is 1 empty/null entry on the column 'Favorite Foods'\n\n![languages result](./resources/new_example_validation_data_doc.png)\n\n\n# Known problems / Future features\n\n- **Problem**: When running validation.py with a validation suite that contains validations of type *expect_column_values_to_match_regex*, GE will complain and throw an error about the profiler not being able to render something. Nevertheless, everything works as expected.\n\n- **Future nice-to-have**: Columns of type \"relate_to\", now return the page id of the entry (see column 'Favorite Foods', not the name of the entry itself. This is perfectly ok to run expectations that check a minimum number of entries (e.g. expect_column_values_to_match_regex), but for further string parsing, the name could be more useful.\n\n![relate to type column image](./resources/new_relate_to_type_column.png)\n\n- **Future nice-to-have:** the Data Docs parser should be able to be edited to show more information (e.g. which row made the expectation fail)\n\n\n## Questions/improvements? 🙋\nFeel free to open up a PR or drop me a message @ ricardo@dataroots.io\n\n## Curious about other projects? 🤔\nFor more cool stuff regarding anything data, check us out:      \n[Website](https://dataroots.io/) / \n[Github](https://github.com/datarootsio)","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdatarootsio%2Fnotion-dbs-data-quality","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdatarootsio%2Fnotion-dbs-data-quality","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdatarootsio%2Fnotion-dbs-data-quality/lists"}