{"id":22572090,"url":"https://github.com/sdairs/bluebird","last_synced_at":"2025-07-18T01:35:36.869Z","repository":{"id":263188316,"uuid":"889547161","full_name":"sdairs/bluebird","owner":"sdairs","description":"Pushes the Bluesky Firehose to various data services","archived":false,"fork":false,"pushed_at":"2025-02-03T11:47:26.000Z","size":404,"stargazers_count":8,"open_issues_count":0,"forks_count":3,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-03-24T13:02:22.723Z","etag":null,"topics":["bluesky","kafka","redpanda","tinybird"],"latest_commit_sha":null,"homepage":"","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/sdairs.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-11-16T15:54:24.000Z","updated_at":"2025-03-13T05:35:03.000Z","dependencies_parsed_at":"2024-12-16T20:24:50.489Z","dependency_job_id":"2a8ee275-26c9-4dd8-b3ae-cf487d1f07cb","html_url":"https://github.com/sdairs/bluebird","commit_stats":null,"previous_names":["sdairs/bluebird"],"tags_count":11,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sdairs%2Fbluebird","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sdairs%2Fbluebird/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sdairs%2Fbluebird/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sdairs%2Fbluebird/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/sdairs","download_url":"https://codeload.github.com/sdairs/bluebird/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248233936,"owners_count":21069493,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bluesky","kafka","redpanda","tinybird"],"created_at":"2024-12-08T02:07:43.637Z","updated_at":"2025-04-10T14:20:46.291Z","avatar_url":"https://github.com/sdairs.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Bluebird\n\nBluebird is a CLI that consumes the Bluesky firehose and sends it to a downstream destination.\n\n## Destinations\n\n- Tinybird\n- Kafka\n- ClickHouse\n- Timeplus\n\n## Usage\n\nYou can use `npx` (or `pnpm dlx`) to run the CLI without installing it. \n\nOr use `npm install -g @sdairs/bluebird` to install it globally.\n\nAlternatively you can use the provided docker image `ghcr.io/sdairs/bluebird:latest` to run it. You can either pass arguments on the command line (shown below) or use environment variables to control the behavior. See [documentation here](USING_DOCKER.md) for more details.\n\n### Tinybird\n\n```\nnpx @sdairs/bluebird start tinybird --token e.XXX --endpoint https://api.tinybird.co --datasource bluebird_feed\n```\n\n```\ndocker run --rm ghcr.io/sdairs/bluebird:latest start tinybird --token e.XXX --endpoint https://api.tinybird.co --datasource bluebird_feed\n```\n\n### Kafka\n\n```\nnpx @sdairs/bluebird start kafka --brokers broker:9092 --topic bluebird --username user --password pass --sasl-mechanism scram-sha-512 --batch-size 819200\n```\n\n```\ndocker run --rm ghcr.io/sdairs/bluebird:latest start kafka --brokers broker:9092 --topic bluebird --username user --password pass --sasl-mechanism scram-sha-512 --batch-size 819200\n```\n\n### ClickHouse\n\n```\nnpx @sdairs/bluebird start clickhouse --url http://localhost:8123 --database default --table bluebird\n```\n\n```\ndocker run --rm ghcr.io/sdairs/bluebird:latest start clickhouse --url http://localhost:8123 --database default --table bluebird\n```\n\n### Timeplus\n\nYou can create a free account at https://us-west-2.timeplus.cloud, then follow the guide to create the API token: https://docs.timeplus.com/apikey. The `stream` will be created automatically if not exists. This also works with self-hosting Timeplus Enterprise, please set the token as `username:password`.\n\n```\nnpx @sdairs/bluebird start timeplus --token XXX --endpoint https://us-west-2.timeplus.cloud/ws_id --stream bluebird\n```\n\n```\ndocker run --rm ghcr.io/sdairs/bluebird:latest start timeplus --token XXX --endpoint https://us-west-2.timeplus.cloud/ws_id --stream bluebird\n```\n\n## CLI development\n\nThe CLI is built with [oclif](https://oclif.io).\n\n### Writing a new destination\n\nAdd a new directory under `src/destinations`, e.g., `src/destinations/my_destination`.\n\nCreate your destination class, e.g., `src/destinations/my_destination/my_destination.ts`. This must export a class that extends `Destination` from `src/destinations/base.ts`.\n\nThere are two methods you can override, `init` and `send`.\n\n- `init`: **optional** - called once when the destination is first created\n- `send`: **required** - called every time a batch is ready to be processed. This is where you should handle sending events to the downstream destination.\n\n\nHere's the template to start from:\n\n```\nimport { Destination } from '../base.js';\nimport { Event } from '../../lib/types.js';\n\ninterface MyDestinationConfig {\n  someConfig: string\n}\n\nexport class MyDestination extends Destination {\n  private config: MyDestinationConfig;\n\n  constructor(config: MyDestinationConfig) {\n    super();\n    this.config = config;\n  }\n\n  async init(): Promise\u003cvoid\u003e {\n\n  }\n\n  async send(events: Event[]): Promise\u003cvoid\u003e {\n\n  }\n}\n```\n\n### Adding the destination to the CLI start command\n\nAdd the new destination as a subcommandto the `start` topic. Create a new file in `src/commands/start/` named after your destination, e.g., `src/commands/start/my_destination.ts`.\n\nHere's the template to start from:\n\n```\nimport { Flags } from '@oclif/core';\nimport { BaseStartCommand } from './base.js';\nimport { MyDestination } from '../../destinations/my_destination/my_destination.js';\nimport { Destination } from '../../destinations/base.js';\n\nexport default class StartMyDestination extends BaseStartCommand\u003ctypeof StartMyDestination\u003e {\n  static description = 'Send the Bluebird feed to My Destination';\n\n  static examples = [\n    '\u003c%= config.bin %\u003e \u003c%= command.id %\u003e --some-config XXX',\n  ];\n\n  static flags = {\n    someConfig: Flags.string({\n      description: 'Some flag',\n      char: 'c',\n      required: true,\n    }),\n  };\n\n  protected createDestination(flags: Record\u003cstring, any\u003e): Destination {\n    return new MyDestination({\n      someConfig: flags.someConfig\n    });\n  }\n}\n```\n\n### Building the CLI for dev\n\nInstall deps and build the CLI:\n\n```\npnpm install\npnpm build\n```\n\n### Running the CLI for dev\n\nFrom within the `cli` directory:\n\n```\n./bin/dev.js start [OPTS]\n```\n\nFor example:\n\n```\n./bin/dev.js start tinybird --token e.XXX --endpoint https://api.tinybird.co --datasource bluebird_feed\n```\n\nNote: you need to do this inside the `cli` dir if you don't have `ts-node` installed globally. If you see a module not found error for `ts-node`, this is why.\n\n## Contributing\n\nJust submit a PR - all contributions are welcome!\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsdairs%2Fbluebird","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsdairs%2Fbluebird","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsdairs%2Fbluebird/lists"}