{"id":15294910,"url":"https://github.com/toucantoco/weaverbird","last_synced_at":"2025-04-12T21:25:08.926Z","repository":{"id":37405964,"uuid":"167165685","full_name":"ToucanToco/weaverbird","owner":"ToucanToco","description":"A visual data pipeline builder with various backends","archived":false,"fork":false,"pushed_at":"2025-04-07T01:41:33.000Z","size":48237,"stargazers_count":99,"open_issues_count":8,"forks_count":17,"subscribers_count":13,"default_branch":"master","last_synced_at":"2025-04-07T04:02:29.533Z","etag":null,"topics":["data-transformation","mongodb","mysql","pandas","postgresql","redshift","snowflake","sql","vuejs"],"latest_commit_sha":null,"homepage":"https://weaverbird.toucantoco.dev","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-3-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ToucanToco.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":".github/CODEOWNERS","security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-01-23T10:40:17.000Z","updated_at":"2025-04-02T15:21:26.000Z","dependencies_parsed_at":"2022-07-08T17:47:58.334Z","dependency_job_id":"3928c4ba-d4a2-4b7b-9311-d130a3374ee3","html_url":"https://github.com/ToucanToco/weaverbird","commit_stats":{"total_commits":2900,"total_committers":36,"mean_commits":80.55555555555556,"dds":0.8451724137931035,"last_synced_commit":"7aaee05619bb6239f62c6926cf50488e2e8500fb"},"previous_names":["toucantoco/vue-query-builder"],"tags_count":206,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ToucanToco%2Fweaverbird","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ToucanToco%2Fweaverbird/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ToucanToco%2Fweaverbird/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ToucanToco%2Fweaverbird/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ToucanToco","download_url":"https://codeload.github.com/ToucanToco/weaverbird/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248633244,"owners_count":21136837,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["data-transformation","mongodb","mysql","pandas","postgresql","redshift","snowflake","sql","vuejs"],"created_at":"2024-09-30T17:07:51.293Z","updated_at":"2025-04-12T21:25:08.901Z","avatar_url":"https://github.com/ToucanToco.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# \u003cimg src=\"docs/img/logonav.png\" alt=\"GitHub Logo\" width=\"30\"/\u003e Weaverbird\n\n![\"Weaverbird Screenshot](docs/img/readme_screenshot.png)\n\nWeaverbird is [Toucan Toco](https://toucantoco.com)'s data pipelines toolkit, it contains :\n\n- a pipeline **Data Model**, currently supporting more than 40 transformation steps\n- a friendly **User Interface** for building those pipelines without writing any code,\n  made with TypeScript, VueJS \u0026 VueX\n- a set of **BackEnds** to use those pipelines :\n  - the MongoDB Translator that generate Mongo Queries, written in TypeScript\n  - the Pandas Executor that compute the result using Pandas dataframes, written in Python\n  - the Snowflake SQL translator, written in Python\n\nFor in depth user \u0026 technical documentation, have a look at [weaverbird.toucantoco.dev](https://weaverbird.toucantoco.dev)\nor at the documentation's source files in the `docs` directory.\n\n## Badges\n\n### UI\n\n[![npm](https://img.shields.io/npm/v/weaverbird)](https://www.npmjs.com/package/weaverbird)\n![CI UI](https://github.com/ToucanToco/weaverbird/actions/workflows/ci-ui.yml/badge.svg)\n[![Coverage](https://sonarcloud.io/api/project_badges/measure?project=ToucanToco_weaverbird_ui\u0026metric=coverage)](https://sonarcloud.io/dashboard?id=ToucanToco_weaverbird)\n[![Maintainability Rating](https://sonarcloud.io/api/project_badges/measure?project=ToucanToco_weaverbird_ui\u0026metric=sqale_rating)](https://sonarcloud.io/dashboard?id=ToucanToco_weaverbird)\n[![Lines of Code](https://sonarcloud.io/api/project_badges/measure?project=ToucanToco_weaverbird_ui\u0026metric=ncloc)](https://sonarcloud.io/dashboard?id=ToucanToco_weaverbird)\n\n\u003c!-- [![Codecov Coverage](https://img.shields.io/codecov/c/github/ToucanToco/weaverbird.svg?style=flat-square)](https://codecov.io/gh/ToucanToco/weaverbird/) --\u003e\n\n### Server\n\n[![pypi](https://img.shields.io/pypi/v/weaverbird)](https://pypi.org/project/weaverbird/)\n![CI server](https://github.com/ToucanToco/weaverbird/actions/workflows/ci-server.yml/badge.svg)\n\n## Project setup\n\n```bash\nyarn install\n```\n\n\u003e See Dockerfile for supported node version\n\n### Compiles target library\n\n```bash\nyarn build-bundle\n```\n\nThis will generate an importable JS `weaverbird` library in the `dist` directory.\n\n**Important note**: While we do our best to embrace [semantic versioning](https://semver.org/),\nwe do not guarantee full backward compatibility until version 1.0.0 is released.\n\n### Run your tests\n\nThe basic command to run all tests is:\n\n```bash\nyarn test\n```\n\n### Lints and fixes files\n\n```bash\nyarn format:fix\nyarn lint --fix\n```\n\n### Build the API documentation\n\n```bash\nyarn build-doc\n```\n\nThis will run [typedoc](https://typedoc.org/) on the `src/` directory and\ngenerate the corresponding documentation in the `dist/docs` directory.\n\n### Build and run the documentation website\n\nThe web documentation is powered by [Jekyll](https://jekyllrb.com).\nAll sources can be found in the `docs` folder.\nTo build and run the documentation with docker:\n\n```bash\ncd docs/\ndocker buildx build -t weaverbird-jekyll .\ndocker container run --rm -p 4000:4000 -v $PWD:/jekyll weaverbird-jekyll\n```\n\nOnce the docs are be built, they'll be available on `http://localhost:4000`. Any change to a `.md` source\nfile will trigger a rebuild.\n\n#### Enrich it!\n\n\u003e put your `.md` file into the `docs` folder. You can add a folder as well to better organization\n\n\u003e into your `.md` file don't forget to declare this at the beginning of the file :\n\n```\n---\ntitle: your title doc name\npermalink: /docs/your-page-doc-name/\n---\n```\n\n\u003e to finish to get your page into the doc navigation you have to add it in `\\_data/docs.yml``\n\nexample :\n\n```\n- title: Technical documentation\n  docs:\n  - steps\n  - stepforms\n  - your-page-doc-name\n```\n\n### Run the storybook\n\n\u003e Storybook uses the bundled lib, so all showcased components must be in the public API.\n\n```bash\nyarn storybook\n```\n\nThis will run [storybook](https://storybook.js.org/), displaying the stories\n(use cases) of UI components.\n\nStories are defined in the `stories/` directory.\n\n## Publication\n\nThis library is published on npm under the name `weaverbird` automatically each time a release is created in GitHub.\n\n### Create a release (frontend)\n\n- Define new version using [semantic versioning](https://semver.org/)\n\n- Create a new local branch `release/X.Y.Z` from master\n\n  ex: `release/0.20.0`\n\n- Update the `version` property in `package.json` and in `sonar-project.properties`\n\n- Check differences between last release and current and fill `CHANGELOG.md` with updates\n\n  - Delete the `##changes` title at start of the `CHANGELOG.md` if provided\n  - Add the date and version at start of `CHANGELOG.md` following this convention\n\n    ```\n    [X.Y.Z] - YYYY-MM-DD\n    ```\n\n    ex: `[0.20.0] - 2020-08-03`\n\n  - Add link to the `CHANGELOG.md` from this version to the previous one at the end of the `CHANGELOG.md`\n\n    ```\n    [X.Y.Z]: https://github.com/ToucanToco/weaverbird/compare/voldX.oldY.oldZ...vX.Y.Z\n    ```\n\n    ex: [0.20.0]: https://github.com/ToucanToco/weaverbird/compare/v0.19.2...v0.20.0\n\n- Commit changes with version number\n\n  ex: `v0.20.0`\n\n- Push branch\n\n- Create a pull request into master from your branch\n\n- When pull request is merged, [create a release](https://github.com/ToucanToco/weaverbird/releases) with the version number in tag version and title (no description needed)\n\n  ex: `v0.20.0`\n\n- Hit the release \"publish release\" button (this will automatically create a tag and trigger the [package publication](https://github.com/ToucanToco/weaverbird/actions?query=workflow%3A%22npm+publish%22) )\n\n### Create a release (backend)\n\n- Create a new local branch `chore/bump-server-version-x-x-x`\n\n- Edit `server/pyproject.toml` \u0026 increment the version in `[tool.poetry]` section\n\n- Push branch\n\n- Create a pull request into master from your branch\n\n- Once the PR is approved \u0026 merged in master publish the release in Pypi with `make build` \u0026 `make upload`\n\n## Usage as library\n\n### Without any module bundler\n\n```html\n\u003c!-- Import styles --\u003e\n\u003clink rel=\"stylesheet\" href=\"weaverbird/dist/weaverbird.umd.min.js\" /\u003e\n\n\u003c!-- Import scripts --\u003e\n\u003cscript src=\"vue.js\"\u003e\u003c/script\u003e\n\u003cscript src=\"weaverbird/dist/weaverbird.umd.min.js\"\u003e\u003c/script\u003e\n```\n\n### With an ES module bundler (typically webpack, vite or rollup)\n\n```js\nimport { Pipeline } from \"weaverbird\";\n```\n\n\u003e By default, the CommonJS module is imported. If you prefer the ES module\n\u003e version, import `dist/weaverbird.esm.js`.\n\n## API\n\n### Modules\n\nSee the documentation generated in `dist/docs` directory\n\n### Styles\n\nTODO: document here sass variables that can be overriden\n\n## Playground\n\nThe `/playground` directory hosts a demo application with a small server that\nshowcases how to integrate the exported components and API. To run it, use the provided `Dockerfile`:\n\n```bash\ndocker build -t weaverbird-playground .\ndocker run -p 5000:5000 --rm -d weaverbird-playground\n```\n\nwhich is basically a shortcut for the following steps:\n\n```bash\ncd ui\n# install front-end dependencies\nyarn\n# build the front-end bundle\nyarn build-bundle\n\ncd ../server\n# install the backend dependencies\npip install -e \".[playground,all]\"\n# run the server\nQUART_APP=playground QUART_ENV=development quart run\n# note: in the dockerfile, a production-ready webserver is used instead of a development one\n```\n\nOnce the server is started, you should be able to open the\n`http://localhost:5000` in your favorite browser and enjoy!\n\n\u003e When developing the UI, use `yarn vite` that provides its own dev server.\n\n### Mongo back-end\n\nThe default back-end for the playground is a small server passing queries to MongoDB.\nConnect the playground to a running MongoDB instance with the environment variables:\n\n- MONGODB_CONNECTION_STRING (default to localhost:27017)\n- MONGODB_DATABASE_NAME (default to 'data')\n\n#### Run Weaverbird + MongoDB or PostgreSQL with docker-compose\n\nIf you want to test the playground with a populated MongoDB or PostgreSQL instance, you can use `docker-compose`:\n\n```\n# At the directory root\n# For MongoDB\ndocker-compose up -d weaverbird mongodb\n# For PostgreSQL\ndocker-compose up -d weaverbird postgres\n```\n\n### Pandas back-end\n\nAn alternative back-end for the playground is a small server running in python, executing pipelines with pandas.\nAdd `?backend=pandas` to the URL to see it in action.\n\n### Athena back-end\n\nIn order to run the playground with AWS Athena, make sure the following environment variables are set before starting the\ndocker-compose stack:\n\n- `ATHENA_REGION`\n- `ATHENA_SECRET_ACCESS_KEY`\n- `ATHENA_ACCESS_KEY_ID`\n- `ATHENA_DATABASE`\n- `ATHENA_OUTPUT`\n\n### BigQuery back-end\n\nIn order to run the playground with Google BigQuery, download the JSON file containing the credentials for your service account, place\nit at the root of the `weaverbird` repo and name it `bigquery-credentials.json`. It will be mounted inside of the playground container.\n\n#### Use your own data files\n\nCSVs from `playground/datastore` are available to use in the playground with pandas.\nYou can override this folder when running the container using by adding a volume parameter:\n`-v /path/to/your/folder/with/csv:/weaverbird/playground/datastore`.\n\n### Use another back-end\n\nYou can point a front-end to another API by using a query parameter: `?api=http://localhost:5000`.\nThis is particularly useful for front-end development, with `yarn vite`.\n\n\u003e To avoid CORS issues when front-end and back-end are on different domains, whitelist your front-end domain using `ALLOW_ORIGIN=\u003cfront-end domain\u003e` or `ALLOW_ORIGIN=*`.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftoucantoco%2Fweaverbird","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftoucantoco%2Fweaverbird","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftoucantoco%2Fweaverbird/lists"}