{"id":22725426,"url":"https://github.com/ovotech/bit-node-tools","last_synced_at":"2025-06-26T01:03:21.711Z","repository":{"id":41957254,"uuid":"161492262","full_name":"ovotech/bit-node-tools","owner":"ovotech","description":"Misc node tools from the Boost Internal Tools team","archived":false,"fork":false,"pushed_at":"2025-01-14T17:38:36.000Z","size":1972,"stargazers_count":9,"open_issues_count":6,"forks_count":5,"subscribers_count":33,"default_branch":"master","last_synced_at":"2025-04-13T20:40:24.138Z","etag":null,"topics":["boost-bit","boostpower","company-ovo"],"latest_commit_sha":null,"homepage":"","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ovotech.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":".github/CODEOWNERS","security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-12-12T13:27:35.000Z","updated_at":"2025-01-14T17:38:40.000Z","dependencies_parsed_at":"2025-01-14T12:34:14.886Z","dependency_job_id":"8f87db92-1101-4c59-98cc-465ccb1804ca","html_url":"https://github.com/ovotech/bit-node-tools","commit_stats":null,"previous_names":[],"tags_count":6,"template":false,"template_full_name":null,"purl":"pkg:github/ovotech/bit-node-tools","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ovotech%2Fbit-node-tools","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ovotech%2Fbit-node-tools/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ovotech%2Fbit-node-tools/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ovotech%2Fbit-node-tools/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ovotech","download_url":"https://codeload.github.com/ovotech/bit-node-tools/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ovotech%2Fbit-node-tools/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":261978909,"owners_count":23239417,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["boost-bit","boostpower","company-ovo"],"created_at":"2024-12-10T16:10:42.375Z","updated_at":"2025-06-26T01:03:21.669Z","avatar_url":"https://github.com/ovotech.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# BIT Kafka Tools\n\n[![CircleCI](https://circleci.com/gh/ovotech/bit-node-tools.svg?style=svg\u0026circle-token=ae40b0f9ff7943343688a0319478e70091e37fbe)](https://circleci.com/gh/ovotech/bit-node-tools)\n\nBIT Team tools for working with Kafka, Avro and other misc stuff. They are split into several independent packages that can be imported separately.\n\n## Working with node-kafka, avro and our confluence schema registry\n\n| Package                                                                | Status       | Description                                                                                                                                                                                                                                                          |\n| ---------------------------------------------------------------------- | ------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |\n| [@ovotech/kafka-avro-cli](packages/kafka-avro-cli/README.md)           | Used in prod | A CLI for inspecting the [confluent schema-registry](https://docs.confluent.io/current/schema-registry/docs/index.html), produce and consume avro kafka events.                                                                                                      |\n| [@ovotech/avro-stream](packages/avro-stream/README.md)                 | Used in prod | Serialize/deserialize [kafka-node](https://github.com/SOHU-Co/kafka-node) streams with [avro](https://avro.apache.org/docs/current/) data, using [confluent schema-registry](https://docs.confluent.io/current/schema-registry/docs/index.html) to hold the schemas. |\n| [@ovotech/kafka-pg-sink](packages/kafka-pg-sink/README.md)             | Used in prod | Store [kafka-node](https://github.com/SOHU-Co/kafka-node) events into a postgres database.                                                                                                                                                                           |\n| [@ovotech/schema-registry-api](packages/schema-registry-api/README.md) | Used in prod | A simple typescript [node-fetch](https://github.com/bitinn/node-fetch) wrapper on the [confluent schema-registry](https://docs.confluent.io/current/schema-registry/docs/index.html) api.                                                                            |\n| [@ovotech/re-pipeline](packages/re-pipeline/README.md)                 | Used in prod | A node streams pipeline implementation, that reconnects the pipes on error, once the error has been handled.                                                                                                                                                         |\n| [@ovotech/kafka-consumer](packages/kafka-consumer/README.md)           | Used in prod | A generic kafka consumer                                                                                                                                                                                                                                             |\n\n## Misc repos\n\n| Package                                                                        | Status       | Description                                                                                                                                                               |\n| ------------------------------------------------------------------------------ | ------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |\n| [@ovotech/winston-logger](packages/winston-logger/README.md)                   | Used in prod | Wrap winston logger to hide graylog semantics and implement safe static meta contexts with PII sanitisers                                                                 |\n| [@ovotech/apollo-datasource-axios](packages/apollo-datasource-axios/README.md) | Used in prod | A rest datasource that uses axios under the hood. This allows adding generic interceptors, adapters etc. Integrates with cache and cache policies. Supports Interceptors. |\n| [@ovotech/bigquery-pg-sink](packages/bigquery-pg-sink/README.md)               | Used in prod | Stream the results of query made by [nodejs-bigquery](https://github.com/googleapis/nodejs-bigquery) into a [postgres database](https://www.postgresql.org/).             |\n| [@ovotech/influx-metrics-tracker](packages/influx-metrics-tracker/README.md)   | Used in prod | Track metrics and store them in an Influx database, with secondary [logging](packages/winston-logger/README.md) if Influx is unavailable.                                 |\n\n## Getting Started\n\nThese instructions will get you a copy of the project up and running on your local machine for development and testing purposes.\n\n### Prerequisites\n\nTo get up and running, you will need the following tools. Use [Homebrew](https://docs.brew.sh/Installation) for installation where possible.\n\n#### Code\n\nTo install dependencies and run the code you will need [node v12.4.1+](https://nodejs.org/en/) and [yarn](https://yarnpkg.com/lang/en/) installed. It has been tested with node version 12.4.1.\n\n```shell\nbrew install node@12\nbrew install yarn\n```\n\n### Installing\n\nTo get a development environment running:\n\nChange to the root directory and install required packages.\n\n```\n# Install required packages\nyarn\n```\n\n```\n# Build packages that are dependencies of other packages\nyarn build\n```\n\n## Running the tests\n\nThe tests require a running schema registry service, and we're using docker compose to start it, alongside kafka, zookeeper and postgres.\n\nSo in the project's root directory run:\n\n```bash\ndocker-compose up\n```\n\nThen you can run the tests with:\n\n```bash\nyarn test\n```\n\n### Coding Style Tests\n\nCode style is enforced by using a linter ([tslint](https://palantir.github.io/tslint/)) and [Prettier](https://prettier.io/).\n\n```\nyarn lint\n```\n\n## Deployment\n\nDeployment is performed by lerna automatically on merge / push to master. You'll need to bump the package version numbers yourself. Only updated packages with newer versions will be pushed to the npm registry.\n\n## Built With\n\n### Languages / Core Tools\n\n- [Typescript](http://www.typescriptlang.org/) - The primary language\n- [NPM](https://www.npmjs.com/) - Node package registry\n\n### Secondary Tooling\n\n- [jest](https://jestjs.io/) - Testing framework used for unit and integration tests\n- [yarn](https://yarnpkg.com/lang/en/) - Typescript package management\n- [lerna](https://www.npmjs.com/package/lerna) - Tool for managing JavaScript projects with multiple packages\n- [docker](https://www.docker.com/) - Deployable containers for code\n\n## Versioning\n\nYou'll need to bump the package version numbers yourself. Only updated packages with newer versions will be pushed to the npm registry.\n\n## Contributing\n\nHave a bug? File an issue with a simple example that reproduces this so we can take a look \u0026 confirm.\n\nWant to make a change? Submit a PR, explain why it's useful, and make sure you've updated the docs (this file) and the tests.\n\n## License\n\nThis project is licensed under Apache 2 - see the [LICENSE](LICENSE) file for details\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fovotech%2Fbit-node-tools","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fovotech%2Fbit-node-tools","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fovotech%2Fbit-node-tools/lists"}