{"id":13703086,"url":"https://github.com/sungchun12/fst","last_synced_at":"2025-03-21T05:30:28.264Z","repository":{"id":143506395,"uuid":"613133208","full_name":"sungchun12/fst","owner":"sungchun12","description":"fst: flow state tool | smooth where you want it, friction where you need it when data engineering","archived":false,"fork":false,"pushed_at":"2023-06-13T16:27:13.000Z","size":504,"stargazers_count":24,"open_issues_count":15,"forks_count":1,"subscribers_count":4,"default_branch":"main","last_synced_at":"2024-01-28T23:08:15.193Z","etag":null,"topics":["dbt","fast","flowstate","hot-reload","workflow"],"latest_commit_sha":null,"homepage":"https://www.loom.com/share/ecfbdfb981e4443d94d2c95f16176118","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/sungchun12.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2023-03-13T00:30:00.000Z","updated_at":"2024-01-23T06:56:21.000Z","dependencies_parsed_at":"2024-01-06T14:03:10.620Z","dependency_job_id":null,"html_url":"https://github.com/sungchun12/fst","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sungchun12%2Ffst","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sungchun12%2Ffst/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sungchun12%2Ffst/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sungchun12%2Ffst/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/sungchun12","download_url":"https://codeload.github.com/sungchun12/fst/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":244745615,"owners_count":20503042,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["dbt","fast","flowstate","hot-reload","workflow"],"created_at":"2024-08-02T21:00:50.114Z","updated_at":"2025-03-21T05:30:27.955Z","avatar_url":"https://github.com/sungchun12.png","language":"Python","funding_links":[],"categories":["Utilities"],"sub_categories":[],"readme":"\u003cp align=\"center\"\u003e\n  \u003cimg src=\"./images/fst_logo.png\" alt=\"fst: flow state tool]\"\u003e\n\u003c/p\u003e\n\nfst(flow state tool): A tool to help you stay in flow state while developing dbt models.\n\nLet's make it the overwhelming normal that these questions are answered in seconds or less when engineering data(think: you don't need 10+ tabs and 40+ mouse clicks to do your jobs)\n\n\u003cdetails\u003e\n  \u003csummary\u003eQuestions to Answer\u003c/summary\u003e\n  \n- Who else is touching this precious file of mine? I’m tired of pull request clashing\n\n- What’s historical performance on this and am I beating it?\n\n- How often does this fail in production?\n\n- Who uses this model and how often?\n\n- What dashboards will this help vs. hurt?\n\n- What’s a data preview based on my updates look like? (e.g. 5 rows)\n\n- How many scheduled data pipelines are tied to this model?\n\n- How much does this cost to run in production and am I helping vs. hurting?\n\n- What are existing database permissions on this model?\n\n- Anyone working on pull requests in real time that rely on my work?\n\n- What’s a data diff compared to current production data?\n  \n\u003c/details\u003e\n\n\u003e **Note:** This tool is still in development. Please feel free to contribute to this project.\n\n## Description\n\nThis is a file watcher for a dbt project using duckdb as the database. It runs `dbt build` and a query preview on a SQL file when it detects a modification. It also generates a test file for the modified SQL file if tests are not detected.\n\nIt works with any SQL file within the `models/` directory of the dbt project. You must run this tool from the root directory of the dbt project.\n\nYou'll notice for the sake of MVP, I am running nested git clones to get this working. I'll release to pypi soon.\n\n\n```bash\n# my command to run this tool in an infinite loop in a split terminal\ngit clone https://github.com/sungchun12/fst.git\ncd fst\ngit clone https://github.com/dbt-labs/jaffle_shop_duckdb.git\ncd jaffle_shop_duckdb\npython -m venv venv\nsource venv/bin/activate\npython -m pip install --upgrade pip setuptools wheel\nsource venv/bin/activate\npip install -r requirements.txt # installs the dbt dependencies\npip install -e ../ # installs the fst package locally\ndbt build # Create the duckb database file and get commands working\n```\n\n```bash\n# open up your IDE or another terminal to start the fst workbench\nfst start\n```\n\n```shell\n# example of running this tool on each modification to any SQL file within the `models/` directory\n# pro tip: open up the compiled query in a split IDE window for hot reloading as you develop\n2023-04-12 11:30:15 - INFO - Running `dbt build` with the modified SQL file (/Users/sung/fst/jaffle_shop_duckdb/models/new_file.sql)...\n2023-04-12 11:30:21 - INFO - `dbt build` was successful.\n2023-04-12 11:30:21 - INFO - 18:30:20  Running with dbt=1.4.5\n18:30:20  Found 7 models, 22 tests, 0 snapshots, 0 analyses, 297 macros, 0 operations, 3 seed files, 0 sources, 0 exposures, 0 metrics\n18:30:20  \n18:30:20  Concurrency: 24 threads (target='dev')\n18:30:20  \n18:30:20  1 of 1 START sql table model main.new_file ..................................... [RUN]\n18:30:21  1 of 1 OK created sql table model main.new_file ................................ [OK in 0.26s]\n18:30:21  \n18:30:21  Finished running 1 table model in 0 hours 0 minutes and 0.53 seconds (0.53s).\n18:30:21  \n18:30:21  Completed successfully\n18:30:21  \n18:30:21  Done. PASS=1 WARN=0 ERROR=0 SKIP=0 TOTAL=1\n\n2023-04-12 11:30:21 - WARNING - Warning: No tests were run with the `dbt build` command. Consider adding tests to your project.\n2023-04-12 11:30:21 - WARNING - Generated test YAML file: /Users/sung/fst/jaffle_shop_duckdb/models/new_file.yml\n2023-04-12 11:30:21 - WARNING - Running `dbt test` with the generated test YAML file...\n2023-04-12 11:30:28 - INFO - `dbt test` with generated tests was successful.\n2023-04-12 11:30:28 - INFO - 18:30:26  Running with dbt=1.4.5\n18:30:27  Found 7 models, 24 tests, 0 snapshots, 0 analyses, 297 macros, 0 operations, 3 seed files, 0 sources, 0 exposures, 0 metrics\n18:30:27  \n18:30:27  Concurrency: 24 threads (target='dev')\n18:30:27  \n18:30:27  1 of 2 START test not_null_new_file_customer_id ................................ [RUN]\n18:30:27  2 of 2 START test unique_new_file_customer_id .................................. [RUN]\n18:30:27  1 of 2 PASS not_null_new_file_customer_id ...................................... [PASS in 0.20s]\n18:30:27  2 of 2 PASS unique_new_file_customer_id ........................................ [PASS in 0.21s]\n18:30:27  \n18:30:27  Finished running 2 tests in 0 hours 0 minutes and 0.44 seconds (0.44s).\n18:30:27  \n18:30:27  Completed successfully\n18:30:27  \n18:30:27  Done. PASS=2 WARN=0 ERROR=0 SKIP=0 TOTAL=2\n\n2023-04-12 11:30:28 - INFO - Executing compiled query from: /Users/sung/fst/jaffle_shop_duckdb/target/compiled/jaffle_shop/models/new_file.sql\n2023-04-12 11:30:28 - INFO - Using DuckDB file: jaffle_shop.duckdb\n2023-04-12 11:30:28 - INFO - `dbt build` time: 6.76 seconds\n2023-04-12 11:30:28 - INFO - Query time: 0.00 seconds\n2023-04-12 11:30:28 - INFO - Result Preview\n+---------------+--------------+-------------+---------------+---------------------+--------------------+---------------------------+\n|   customer_id | first_name   | last_name   | first_order   | most_recent_order   |   number_of_orders |   customer_lifetime_value |\n+===============+==============+=============+===============+=====================+====================+===========================+\n|             1 | Michael      | P.          | 2018-01-01    | 2018-02-10          |                  2 |                        33 |\n+---------------+--------------+-------------+---------------+---------------------+--------------------+---------------------------+\n|             2 | Shawn        | M.          | 2018-01-11    | 2018-01-11          |                  1 |                        23 |\n+---------------+--------------+-------------+---------------+---------------------+--------------------+---------------------------+\n|             3 | Kathleen     | P.          | 2018-01-02    | 2018-03-11          |                  3 |                        65 |\n+---------------+--------------+-------------+---------------+---------------------+--------------------+---------------------------+\n|             6 | Sarah        | R.          | 2018-02-19    | 2018-02-19          |                  1 |                         8 |\n+---------------+--------------+-------------+---------------+---------------------+--------------------+---------------------------+\n|             7 | Martin       | M.          | 2018-01-14    | 2018-01-14          |                  1 |                        26 |\n+---------------+--------------+-------------+---------------+---------------------+--------------------+---------------------------+\n2023-04-12 11:30:28 - INFO - fst metrics saved to the database: fst_metrics.duckdb\n```\n\n\u003e Note: Tested with python version: 3.8.9 on MacOs Intel\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsungchun12%2Ffst","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsungchun12%2Ffst","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsungchun12%2Ffst/lists"}