Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/dataform-co/dataform

Dataform is a framework for managing SQL based data operations in BigQuery
https://github.com/dataform-co/dataform

analytics business-intelligence data-engineering data-pipelines elt etl hacktoberfest

Last synced: 2 days ago
JSON representation

Dataform is a framework for managing SQL based data operations in BigQuery

Host: GitHub
URL: https://github.com/dataform-co/dataform
Owner: dataform-co
License: apache-2.0
Created: 2018-09-03T14:36:28.000Z (over 6 years ago)
Default Branch: main
Last Pushed: 2024-12-01T21:28:39.000Z (11 days ago)
Last Synced: 2024-12-03T05:01:55.530Z (9 days ago)
Topics: analytics, business-intelligence, data-engineering, data-pipelines, elt, etl, hacktoberfest
Language: TypeScript
Homepage: https://cloud.google.com/dataform/docs
Size: 16.1 MB
Stars: 856
Watchers: 27
Forks: 166
Open Issues: 78
Metadata Files:
- Readme: readme.md
- Contributing: contributing.md
- License: LICENSE

Awesome Lists containing this project

awesome-starred - dataform-co/dataform - Dataform is a framework for managing SQL based data operations in BigQuery, Snowflake, and Redshift (hacktoberfest)
jimsghstars - dataform-co/dataform - Dataform is a framework for managing SQL based data operations in BigQuery (TypeScript)

README

        # Dataform Core

Dataform Core is an open source meta-language to create SQL tables and workflows in BigQuery. Dataform Core extends SQL by providing a dependency management system, automated data quality testing, and data documentation.

Using Dataform Core, data teams can build scalable SQL data transformation pipelines following software engineering best practices, like version control and testing.

For more details, see [how Dataform works](https://cloud.google.com/dataform/docs/overview).

![Data collections and integrations feed into Dataform, which exports this data to BI and analytics tools.](static/images/single-source-of-truth.png?raw=true)

## Get started

### In Google Cloud Platform

Dataform in Google Cloud Platform provides a fully managed experience to build scalable data transformations pipelines in **BigQuery** using SQL. It includes:

- A cloud development environment to develop data assets with SQL and Dataform Core and version control code with GitHub, GitLab, and other Git providers.

- A fully managed, serverless orchestration environment for data pipelines, fully integrated in Google Cloud Platform.

Follow the [quickstart guide](https://cloud.google.com/dataform/docs/quickstart)!

### With the CLI

You can run Dataform locally using the Dataform CLI tool, which can be installed using the following command line. Follow the [CLI guide](https://cloud.google.com/dataform/docs/use-dataform-cli) to get started.

```

npm i -g @dataform/cli

```

## Useful Links

- [Documentation home page](https://cloud.google.com/dataform).

- [Create tables and views](https://cloud.google.com/dataform/docs/tables).

- [Configure dependencies](https://cloud.google.com/dataform/docs/define-table#define_table_structure_and_dependencies).

- Write [data quality checks](https://cloud.google.com/dataform/docs/assertions).

- Enable [scripting](https://cloud.google.com/dataform/docs/develop-workflows-js) and code re-use with a JavaScript API.

- Import [pre-defined packages](https://dataform-co.github.io/dataform/docs/packages), or create your own.

- View the [Dataform Core reference](https://cloud.google.com/dataform/docs/reference/dataform-core-reference).

- View the [Dataform configs reference](https://dataform-co.github.io/dataform/docs/configs-reference).

_Note: this readme can also be viewed on https://dataform-co.github.io/dataform._

## Example Projects

- [https://github.com/GoogleCloudPlatform/marketing-data-engine-dataform](https://github.com/GoogleCloudPlatform/marketing-data-engine-dataform).

- [https://github.com/wintermi/movielens-dataform](https://github.com/wintermi/movielens-dataform).

- [https://github.com/wintermi/bqe-dataform](https://github.com/wintermi/bqe-dataform).

- [https://github.com/wintermi/imdb-dataform](https://github.com/wintermi/imdb-dataform).

- [https://github.com/wintermi/fashion-dataform](https://github.com/wintermi/fashion-dataform).

- [https://github.com/G2H/dataform-stackoverflow](https://github.com/G2H/dataform-stackoverflow).

- [https://github.com/karcot1/dataform_deployment_sample](https://github.com/karcot1/dataform_deployment_sample).

## Want to report a bug or request a feature?

- For Dataform Core / open source requests, you can open an [issue](https://github.com/dataform-co/dataform/issues) in GitHub.

- For Dataform in Google Cloud Platform, you can file a bug [here](https://issuetracker.google.com/issues/new?component=1193995&template=1698201), and file feature requests [here](https://issuetracker.google.com/issues/new?component=1193995&template=1713836).

## Want to contribute?

Check out our [contributors guide](https://github.com/dataform-co/dataform/blob/main/contributing.md) to get started with setting up the repo.