https://github.com/ashish10alex/vscode-dataform-tools

Dataform tools - a vscode extension
https://github.com/ashish10alex/vscode-dataform-tools

data-engineering dataform gcp vscode vscode-extension

Last synced: 12 months ago
JSON representation

Dataform tools - a vscode extension

Host: GitHub
URL: https://github.com/ashish10alex/vscode-dataform-tools
Owner: ashish10alex
License: mit
Created: 2024-05-23T00:36:30.000Z (over 1 year ago)
Default Branch: main
Last Pushed: 2024-06-12T23:53:17.000Z (over 1 year ago)
Last Synced: 2024-06-13T04:28:08.471Z (over 1 year ago)
Topics: data-engineering, dataform, gcp, vscode, vscode-extension
Language: TypeScript
Homepage: https://marketplace.visualstudio.com/items?itemName=ashishalex.dataform-lsp-vscode
Size: 4.6 MB
Stars: 2
Watchers: 1
Forks: 1
Open Issues: 0
Metadata Files:
- Readme: README.md
- Changelog: CHANGELOG.md
- License: LICENSE

Awesome Lists containing this project

README

          # Dataform tools - a vscode extension

🎥 [Click here for YouTube video with installation steps and demo →](https://www.youtube.com/watch?v=nb_OFh6YgOc)

[![Version](https://img.shields.io/github/v/release/ashish10alex/vscode-dataform-tools)](https://github.com/ashish10alex/vscode-dataform-tools/releases)

![Installs](https://img.shields.io/vscode-marketplace/i/ashishalex.dataform-lsp-vscode.svg)

![Linux](https://img.shields.io/badge/Linux-supported-success)

![macOS](https://img.shields.io/badge/macOS-supported-success)

![Windows](https://img.shields.io/badge/windows-supported-success)

[!["Buy Me A Coffee"](https://www.buymeacoffee.com/assets/img/custom_images/yellow_img.png)](https://buymeacoffee.com/ashishalexj)

🎉 **Officially recommended by Google[^1]**. Though not an officially supported Google product.

[VS Code extension](https://marketplace.visualstudio.com/items?itemName=ashishalex.dataform-lsp-vscode) for [Dataform](https://github.com/dataform-co/dataform). Supports both Dataform version 2.x and 3.x

| Feature | Description |

|---------|-------------|

| [Compiled Query & Dry run stats](#compilation) | Compiled query with dry run stats in a vertical split |

| [Dependancy graph](#depgraph) | Interative dependancy graph with external sources higlighted in distinct colors |

| [Inline diagnostics on `.sqlx` file](#diagnostics) 🚨 | Native LSP like experience with diagnostics being directly put on sqlx file |

| [Preview query results](#preview_query_results) | Preview query results in a table by running the file |

| [Cost estimator](#cost_estimator) 💸 | Estimate the cost of running a Tag|

| [Go to definition](#definition) | Go to definition for source in `$ref{("my_source")}` and javascript blocks in `.sqlx` files  |

| [Auto-completion](#autocomplete) | - Column names of current model 
 - Dependencies and declarations in `${ref("..")}` trigger when `$` character is typed 
 - Dependencies when `"` or `'` is typed inside the config block which has `dependencies` keyword is in the line prefix 
 - `tags` when `"` or `'` is typed inside the config block which has `tags` keyword is in the line prefix |

| [Code actions](#codeactions) | Apply dry run suggestions at the speed of thought |

| [Run file(s)/tag(s)](#filetagruns) | Run file(s)/tag(s), optionally with dependencies/dependents/full refresh using vscode command pallet / menu icons |

| [Format using Sqlfluff](#formatting) 🪄 | Fromat `.sqlx` files using [sqlfluff](https://github.com/sqlfluff/sqlfluff)|

| [BigQuery snippets](#snippets) | Code snippets for generic BigQuery functions taken from [vscode-langauge-sql-bigquery](https://github.com/shinichi-takii/vscode-language-sql-bigquery) extension |

## Requirements

1. [Dataform cli](https://cloud.google.com/dataform/docs/use-dataform-cli)

   ```bash

   # requires nodejs

   npm i -g @dataform/cli

   ```

   Run `dataform compile` from the root of your Dataform project to ensure that you are able to use the cli

2. [Install gcloud cli](https://cloud.google.com/sdk/docs/install) and run

   ```bash

   gcloud init

   ```

   ```bash

   gcloud auth application-default login

   ```

   ```bash

    gcloud config set project  #replace with your gcp project id

   ```

3. To enable formatting using [sqlfluff](https://github.com/sqlfluff/sqlfluff) install [sqlfluff](https://github.com/sqlfluff/sqlfluff)

   ```bash

   # install python and run

   pip install sqlfluff

   ```

4. To enable prettier diagnostics install [Error Lens](https://marketplace.visualstudio.com/items?itemName=usernamehw.errorlens) extension [ **optional** ]

> [!NOTE]

Trouble installing or looking for a specific customization ? Please see [FAQ section](FAQ.md), if you are still stuck, please [raise an issue here](https://github.com/ashish10alex/vscode-dataform-tools/issues)

* 🎥 [Installation on Windows](https://www.youtube.com/watch?v=8AsSwzmzhV4)

* 🎥 [Installation and demo on Ubuntu](https://www.youtube.com/watch?v=nb_OFh6YgOc)

## Features

### Compiled query & Dry run stats

![compilation](media/images/compiled_query_preview.png)

### Dependency graph

![depgraph](/media/images/dependancy_tree.png)

### Inline diagnostics errors on `.sqlx` files ❗

![diagnostics](media/images/diagnostics.png)

### Preview query results

![preview_query_results](/media/images/preview_query_results.png)

### Estimate cost of running a Tag

![cost_estimator](/media/images/tag_cost_estimator.png)

### Go to definition

Go to definition for source in `$ref{("my_source")}`. Takes you to `my_source.sqlx` or `sources.js` at the line where `my_source` is defined. There is also support for go to definiton

from a javascript variable/module from a `.sqlx` file to `js` block or `.js` file where the virable or module declaration exsists

![go-to-definition](media/images/go_to_definition.gif)

### Autocomplete model, tags, dependencies

Auto completion support for `dependencies` when `"` or `'` is typed inside the config block which has `dependencies` keyword is in the line prefix

![auto-completion](media/images/dependencies_autocompletion.gif)

Declarations in `${ref("..")}` trigger when $ character is typed

![auto-completion](media/images/sources_autocompletion.gif)

Auto completion support for `tags` when `"` or `'` is typed inside the config block which has `tags` keyword is in the line prefix

![auto-completion](media/images/tags_autocompletion.gif)

### Formatting using sqlfluff

![formatting](media/images/formatting.gif)

### Run file/tag with dependencies/dependents

Open vscode command pallet by pressing CTLR + SHIFT + p or CMD + SHIFT + p on mac and run one of the required commands

| Commands                                               |

|------------------------------------------------------  |

| `Dataform: Run current file`                           |

| `Dataform: Run current file with dependencies`         |

| `Dataform: Run current file with dependents`           |

| `Dataform: Run current tag`                            |

| `Dataform: Run current tag with dependencies`          |

| `Dataform: Run current tag with dependents`            |

| `Dataform: Format current file`                        |

| `Dataform: Run file(s) / tag(s) with options`          |

## Known Issues

* [ ] Features such as go to definition / dependancy graph might not work with consistantly with `${ref("dataset", "table")}` or when it is multiline or a different format works best with `${ref('table_name')}` format

## TODO

* [ ] Add option to include dependents / dependencies when running cost estimator for tag

* [ ] Add hover docs for config block elements. e.g. Assertions, type etc

* [ ] Handle case where user is not connected to internet or on vpn where network request for dry run cannot be made

[^1]: Link to confirmation of official recommendation by Google:  [link](https://github.com/dataform-co/dataform/blob/main/vscode/README.md)

ecosyste.ms

Data

Tools

Indexes

Applications

Experiments

Awesome

https://github.com/ashish10alex/vscode-dataform-tools

Awesome Lists containing this project

README