{"id":20141264,"url":"https://github.com/alekcz/firestream","last_synced_at":"2025-04-09T18:40:55.307Z","repository":{"id":62431255,"uuid":"213499370","full_name":"alekcz/firestream","owner":"alekcz","description":"Kafkaesque streams built on firebase","archived":false,"fork":false,"pushed_at":"2024-05-02T17:31:22.000Z","size":95,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":3,"default_branch":"master","last_synced_at":"2024-05-28T12:41:53.967Z","etag":null,"topics":["clojure","firebase","kafkaesque-streams","stream"],"latest_commit_sha":null,"homepage":null,"language":"Clojure","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"epl-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/alekcz.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-10-07T22:35:13.000Z","updated_at":"2024-05-29T23:55:01.259Z","dependencies_parsed_at":"2024-05-30T00:07:31.469Z","dependency_job_id":null,"html_url":"https://github.com/alekcz/firestream","commit_stats":{"total_commits":101,"total_committers":1,"mean_commits":101.0,"dds":0.0,"last_synced_commit":"3e7a596540b558c6bcfb0c9b8ca04c1698a1e2d3"},"previous_names":[],"tags_count":5,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alekcz%2Ffirestream","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alekcz%2Ffirestream/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alekcz%2Ffirestream/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alekcz%2Ffirestream/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/alekcz","download_url":"https://codeload.github.com/alekcz/firestream/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247428204,"owners_count":20937430,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["clojure","firebase","kafkaesque-streams","stream"],"created_at":"2024-11-13T21:56:52.777Z","updated_at":"2025-04-09T18:40:55.286Z","avatar_url":"https://github.com/alekcz.png","language":"Clojure","funding_links":[],"categories":[],"sub_categories":[],"readme":"# firestream\n\nKafkaesque streams built on firebase\n\n![Clojure CI](https://github.com/alekcz/firestream/workflows/Clojure%20CI/badge.svg) [![codecov](https://codecov.io/gh/alekcz/firestream/branch/master/graph/badge.svg)](https://codecov.io/gh/alekcz/firestream) \n\n[![Clojars Project](https://img.shields.io/clojars/v/alekcz/firestream.svg)](https://clojars.org/alekcz/firestream)\n\n## Design \n`firestream` is designed to provide kafakaesque streams with minimal hassle for micro-scale applications or MVPs. Once your application is no longer micro-scale using `firestream` is a great way to ensure bad things happen. `firestream` is aimed to give your application (or business) the runway grow until it can absorb the operational cost of `kafka`. \n\n### Schema \nAt both rest and in transit all messages are stored as stringified `json` objects. `firestream` itself works in `clojure` maps. \n\n### Topics and partitions\nPartitions are not provided for in `firestream`. If you really really really need partitions, it's probably time to switch to `kafka`.\n\n### Producers and consumers\nLike `kafka`, `firestream` allows for multiple consumer and producers. It however only allows for consumer groups with one consumer. If you really really really need more than one consume in a consumer group, it's probably time to switch to `kafka`.\n\n### Brokers and clusters\nThere is only one broker, that broker is your firebase instance. There is only one cluster\n\n### Interface\nThe design of `firestream`'s interface is inspired by [pyr's](https://github.com/pyr) somewhat opinionated client library for `kafka` [kinsky](https://github.com/pyr/kinsky).\n\n## Limits\nThe theoretical limits* of `firestream` (i.e. running it on the biggest machine you can find) are derived by using an 8th of the limits of firebase. For pico-scale applications or MVPs it's unlikely you'll hit the limits of firebase or `firestream`. Here they are anyway:\n\n- Individual producer throughput: ~2k per second (messages)\n- Maximum message size: 1MB (after `pr-str`)\n- Maximmum data transfer per read: 200MB\n- Maximum number of messages per topic: 9 million\n\n*It's quite likely that you can get more perf than the above, but better safe than sorry.\n\n[Firebase usage limits in case they change](https://firebase.google.com/docs/database/usage/limits)\n\n## Installation\n\nYou can grab `firestream` from clojars:    \n`[alekcz/firestream \"2.0.3\"]`\n\n\n### Setting up firebase\n\n1. You need to create a project on [firebase](https://firebase.google.com/) to get started. So do that first.\n2. Once you've created your project setup a Realtime Database.\n3. We don't want any frontends or non-admin apps to access our database, as this database will be at the core of our stream. So we need to deny all non-admin access using the [firebase securtiy rules](https://firebase.google.com/docs/database/security/quickstart). You can use the rules below.\n```javascript\n{\n  \"rules\": {\n    \".read\": false, //block all non-admin reads\n    \".write\": false //block all non-admin writes\n  }\n}\n```\n\n### Connecting your app to firebase\n\n1. Get the `json` file containing your creditials by following the instruction here [https://firebase.google.com/docs/admin/setup](https://firebase.google.com/docs/admin/setup)  \n\n2. Set the `GOOGLE_APPLICATION_CREDENTIALS` environment variable to the contents of your `json` key file. Using [google-credentials](https://github.com/alekcz/google-credentials) firestream will pull the credentials from the specified environment variable: `GOOGLE_APPLICATION_CREDENTIALS`. (Sometimes it may be necessary to remove all the line breaks and wrap the key contents in single quotes to escape all the special characters within it. e.g. GOOGLE_APPLICATION_CREDENTIALS='`contents-of-json`')\n\n3. You're now good to go.\n\n## Usage\n\n### Example\n```clojure\n(require '[firestream.core :as f])\n\n(let [c (f/producer {:env \"GOOGLE_APPLICATION_CREDENTIALS\"})]\n      (f/send! p \"emailqueue\" :k0 {:name \"Alexander the Great\" :email \"alex@macedon.gr\"})   \n\n(let [c (f/consumer {:env \"GOOGLE_APPLICATION_CREDENTIALS\" :group.id \"kafkaesque\"})]\n      (f/subscribe! c \"emailqueue\")\n      (f/poll! c 100)\n```\n\n### API\n\nThe `firestream` API has 5 functions. \n- `producer`: Create a producer\n- `send!`: Send new message to topic\n- `consumer`: Create a consumer\n- `subscribe!`: Subscribe to a topic\n- `poll!`: Read messages ready for consumption\n\n\n## Metrics\n\nThere are no metrics for the moment, but hopefully someday we'll get to the same level as [operatr.io](https://operatr.io/). \n\n## What next\n\nWhen you outgrow `firestream` and are ready for `kafka`, hit up the awesome folks at [troywest.com](https://troywest.com/) to get you started.  \n\n## License\n\nCopyright © 2019 Alexander Oloo\n\nThis program and the accompanying materials are made available under the\nterms of the Eclipse Public License 2.0 which is available at\nhttp://www.eclipse.org/legal/epl-2.0.\n\nThis Source Code may also be made available under the following Secondary\nLicenses when the conditions for such availability set forth in the Eclipse\nPublic License, v. 2.0 are satisfied: GNU General Public License as published by\nthe Free Software Foundation, either version 2 of the License, or (at your\noption) any later version, with the GNU Classpath Exception which is available\nat https://www.gnu.org/software/classpath/license.html.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Falekcz%2Ffirestream","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Falekcz%2Ffirestream","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Falekcz%2Ffirestream/lists"}