{"id":19370026,"url":"https://github.com/bustle/manifold","last_synced_at":"2026-02-26T21:35:32.882Z","repository":{"id":236981803,"uuid":"793559886","full_name":"bustle/manifold","owner":"bustle","description":"An opinionated BigQuery data framework built atop Terraform","archived":false,"fork":false,"pushed_at":"2025-04-19T18:16:12.000Z","size":230,"stargazers_count":0,"open_issues_count":1,"forks_count":0,"subscribers_count":6,"default_branch":"master","last_synced_at":"2025-04-22T23:47:01.116Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Ruby","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/bustle.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"docs/CONTRIBUTING.md","funding":null,"license":"LICENSE.txt","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-04-29T13:00:25.000Z","updated_at":"2025-04-19T18:16:15.000Z","dependencies_parsed_at":"2024-05-07T21:23:07.970Z","dependency_job_id":"f1fa7734-dee1-4ee1-a146-fb2043d368f3","html_url":"https://github.com/bustle/manifold","commit_stats":null,"previous_names":["bustle/manifolds","bustle/manifold"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bustle%2Fmanifold","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bustle%2Fmanifold/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bustle%2Fmanifold/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bustle%2Fmanifold/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/bustle","download_url":"https://codeload.github.com/bustle/manifold/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":250460471,"owners_count":21434253,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-10T08:14:02.114Z","updated_at":"2026-02-26T21:35:27.844Z","avatar_url":"https://github.com/bustle.png","language":"Ruby","funding_links":[],"categories":[],"sub_categories":[],"readme":"![manifold](https://github.com/bustle/manifold/assets/8030836/be7f24f6-27b9-4924-8e1a-ae2899c7b272)\n\n# Manifold\n\n![CI](https://github.com/bustle/manifold/actions/workflows/ci.yml/badge.svg)\n\nManifold is a framework designed to streamline the process of generating and managing data infrastructures in Google BigQuery using Terraform. By leveraging Manifold, teams can efficiently create complex views that join dimensional data with multiple metrics tables, enabling more dynamic data interactions and fewer sandcastles.\n\n## Philosophy\n\nAt the heart of Manifold, our philosophy is to simplify the complexity inherent in managing large-scale data architectures. We aim to provide a tool that not only facilitates the easy setup of data structures but also adheres to best practices in scalability, maintainability, and performance. Manifold is built for data engineers, by data engineers, ensuring that the nuances and common challenges in data management are well-addressed.\n\n## Features\n\n- **Unified Data Modeling**: Manifold introduces a standardized way to model dimensions and metrics, ensuring consistency and reliability in data reporting and analysis.\n- **Scalability**: Designed to handle large volumes of data, supporting a variety of data types and structures.\n- **Flexibility**: Easily adapt to different kinds of metric groupings such as by device type (e.g., desktop, tablet, mobile) with identical metric structures beneath these segmentations.\n\n## Getting Started\n\n### Prerequisites\n\n- Ruby\n- Terraform\n- Google Cloud SDK (gcloud)\n\n### Installation\n\n1. **Install the Manifold Gem**:\n   `manifold` is distributed as a Ruby gem. To install it, run:\n\n```bash\ngem install manifold\n```\n\n2. **Setup Terraform**: Ensure that Terraform is installed and configured to interact with your Google Cloud Platform account.\n\n3. **Configure Your Environment**: Set up your environment variables and credentials to access Google BigQuery and other necessary services.\n\n## Usage\n\n1. **Initialize a New Umbrella Project**\n\nSet up a new umbrella project directory with the necessary structure for managing multiple data projects.\n\n```bash\nmanifold init \u003cproject_name\u003e\n```\n\n2. **Add a New Data Project**\n\nAdd a new data project under the umbrella. This setup includes creating a directory for the data project and initializing with a template `manifold.yml` file.\n\n```bash\ncd \u003cproject_name\u003e\nmanifold add \u003cdata_project_name\u003e\n```\n\n3. **Generate BigQuery Resource Definitions and Terraform Configuration**\n\nAfter you fill out the manifold.yml file, this command generates the necessary BigQuery schema files and Terraform configurations based on the specified dimensions and metrics.\n\n```bash\n# Generate with submodule configuration (default)\nmanifold generate\n\n# Generate with provider configuration (if needed)\nmanifold generate --no-submodule\n```\n\nThis will create:\n\n- A root `main.tf.json` file that sets up workspace modules (by default) or includes provider configuration (with --no-submodule)\n- Individual workspace configurations in `workspaces/\u003cworkspace_name\u003e/main.tf.json`\n- Dataset and table definitions that reference your generated BigQuery schemas\n\nBy default, the generated Terraform configurations are designed to be included as a module in your larger Terraform configuration:\n\n```hcl\nmodule \"my_manifold_project\" {\n  source = \"./path/to/manifold/project\"\n  project_id = var.project_id\n}\n```\n\nIf you need to use the configuration standalone (with `--no-submodule`), you can apply it directly:\n\n```bash\nterraform init\nterraform plan -var=\"project_id=your-project-id\"\nterraform apply -var=\"project_id=your-project-id\"\n```\n\n## Manifold Configuration\n\n### Vectors\n\nVectors are the entities you can roll up data for. Each vector has a set of dimensions defined in its `vectors/\u003cvector_name\u003e.yml` configuration file.\n\n```yaml\nvectors:\n  - page\n```\n\n#### Add a vector to your project\n\n```bash\nmanifold vectors add page\n```\n\n### Metrics\n\nMetrics are fields that contain numerical data that can be aggregated. They are typically used to measure performance or other quantitative data.\n\n#### Example\n\n```yaml\nmetrics:\n  - name: Pageviews\n    id:\n      field: pageId\n      type: STRING\n    interval:\n      type: TIMESTAMP\n      expression: TIMESTAMP_TRUNC(timestamp, HOUR)\n    aggregations:\n      - name: pageviews\n        method: count\n      - name: sessions\n        method: distinct\n        field: sessionid\n    source:\n      type: bigquery\n      name: Events.Requests\n      breakouts:\n        - name: us\n          condition: CountryId = 2840\n```\n\n- _Name_: The name of the metric.\n- _ID_: The field that uniquely identifies the metric, along with its type\n- _Interval_: The time interval over which the metric is aggregated\n- _Aggregations_: The distinct used to aggregate the metric\n- _Source_: The source table from which the metric is derived\n- _Breakouts_: Custom segmentations of the metric\n\n## Contributing\n\nWe welcome contributions from the community! Please check out our [contribution guidelines](docs/CONTRIBUTING.md) for more information on how to get involved.\n\n## License\n\nDistributed under the MIT License. See LICENSE for more information\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbustle%2Fmanifold","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbustle%2Fmanifold","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbustle%2Fmanifold/lists"}