{"id":17967685,"url":"https://github.com/simonprickett/redis-compatible-bloom-filter","last_synced_at":"2026-04-11T08:04:11.325Z","repository":{"id":149219470,"uuid":"515964297","full_name":"simonprickett/redis-compatible-bloom-filter","owner":"simonprickett","description":"Attempt to build a Bloom Filter implementation using a Pimoroni Unicorn Hat on a Raspberry Pi that talks the RESP protocol.","archived":false,"fork":false,"pushed_at":"2022-12-21T17:10:32.000Z","size":342,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-04-03T21:33:47.197Z","etag":null,"topics":["bloom-filter","probabilistic-data-structures","python","redis"],"latest_commit_sha":null,"homepage":"https://simonprickett.dev/things-on-thursdays-livestreams/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/simonprickett.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-07-20T12:01:40.000Z","updated_at":"2024-07-05T11:57:10.000Z","dependencies_parsed_at":null,"dependency_job_id":"bf217f9a-9717-481c-ae00-2f59aac9e265","html_url":"https://github.com/simonprickett/redis-compatible-bloom-filter","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/simonprickett/redis-compatible-bloom-filter","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/simonprickett%2Fredis-compatible-bloom-filter","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/simonprickett%2Fredis-compatible-bloom-filter/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/simonprickett%2Fredis-compatible-bloom-filter/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/simonprickett%2Fredis-compatible-bloom-filter/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/simonprickett","download_url":"https://codeload.github.com/simonprickett/redis-compatible-bloom-filter/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/simonprickett%2Fredis-compatible-bloom-filter/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":259580861,"owners_count":22879740,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bloom-filter","probabilistic-data-structures","python","redis"],"created_at":"2024-10-29T14:09:16.699Z","updated_at":"2026-04-11T08:04:11.268Z","avatar_url":"https://github.com/simonprickett.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Redis Compatible Bloom Filter\n\nAttempt to build a Bloom Filter implementation using a [Pimoroni Unicorn HAT](https://shop.pimoroni.com/products/unicorn-hat?variant=932565325) on a Raspberry Pi that talks the RESP protocol.  This is the repo to support my livestream series \"Things on Thursdays\" ([see all episodes](https://simonprickett.dev/things-on-thursdays-livestreams/)).  This needs to be run on a Raspberry Pi with a Unicorn HAT attached.\n\n![Still image from the livestream showing the project working](redis_compatible_bloom.png)\n\nHere's the videos from this series:\n\n* [Episode One - RESP protocol and exploring Redis Sets](https://www.youtube.com/watch?v=uyjAFP73ttI)\n* [Episode Two - Working with the LEDs and Bloom Filter Commands](https://www.youtube.com/watch?v=Ym4g5iti3bo)\n\nThe current state of this is that it has basic support for `BF.ADD`, `BF.EXISTS` and `BF.MADD` commands with little error checking.\n\nIt uses [`nodemon`](https://www.npmjs.com/package/nodemon) to restart the process when you save changes, so you'll need both Python 3 and Node.js if you want to take advantage of that.  We'll cover software setup later.\n\n## Shopping List\n\nIf you want to build one of these, you'll need to have/buy the following items (links to specific suppliers, others are available - Google the items to find suppliers in your country to avoid foreign exchange and import costs):\n\n* A [Raspberry Pi](https://www.raspberrypi.com/products/) (Model 3A, 3B or any model 4) - get whichever you have or can find, as there's a shortage right now.  The 3B is a solid all rounder to use on other future projects too.\n* A decent power supply as this will have to power the Pi and the LEDs.  You'll want a [USB C](https://www.raspberrypi.com/products/type-c-power-supply/) one if using a Raspberry Pi 4, or a [micro USB version](https://www.raspberrypi.com/products/micro-usb-power-supply/) for all other Pi models.  Make sure to get one with the right wall plug for your country!\n* A [Micro SD card](https://www.mymemory.co.uk/memory/memory-cards/micro-sd.html) (8gb should do, larger won't hurt).  You may need a micro SD to regular SD card adapter to connect the card to your laptop/desktop machine to write the Pi OS image to it.  Many micro SD cards come with this adapter.\n* A [Pimoroni Unicorn HAT](https://shop.pimoroni.com/products/unicorn-hat?variant=932565325) (the 8 x 8 version)\n* Optional: A case for the Raspberry Pi with a lid that diffuses the LEDs a bit.  I use this one in smoke grey from Adafruit: ([case for Pi 3](https://www.adafruit.com/product/2256), [lid for Pi 3](https://www.adafruit.com/product/2244)).  They make these for the Pi Model A too, but it appears that it's not available for the Pi 4.\n\n## Rasperry Pi Setup\n\nRaspberry Pi have great documentation ([here](https://www.raspberrypi.com/documentation/computers/getting-started.html)) showing how to set up the operating system on a new Pi using their tool that will flash an operating system image to your micro SD card.  You'll want to pay attention to their instructions for how to configure your WIFI and give the Pi a hostname e.g. `bloom.local`.  It's also useful to turn on SSH (Secure Shell).\n\nWe'll assume from here on that your Pi is configured and connected to the network, has SSH enabled, and that you know its hostname or IP address and a username/password to connect to it.\n\n## Unicorn Hat Setup\n\nWith the Raspberry Pi turned off, orient the Unicorn HAT so that the LEDs cover the majority of the Raspberry Pi board, then press the Unicorn HAT onto the GPIO pins.  Ensure a firm connection.\n\n## Software Setup / Install Dependencies\n\n### Optional (for developing new functionality only)\n\nIf you want to use nodemon to automatically restart the Bloom Filter server every time you change the Python code, you'll need to install an up to date version of Node.js on the Pi.  If you just want to run the Bloom Filter code and use it, or don't feel you'll benefit from automatic restarts on change in your development process feel free to skip this step and go straight on to the \"Required\" section.\n\nGet a modern version of Node by [following the guide here](https://randomnerdtutorials.com/how-to-install-the-latest-version-of-node-js-in-raspberry-pi/).\n\n### Required\n\nConnect to your Raspberry Pi using SSH (`ssh \u003cusername\u003e@\u003chostname or ip address\u003e` e.g. `ssh pi@bloom.local`), then enter these commands - skip `npm install` if you opted not to use nodemon:\n\n```bash\ngit clone https://github.com/simonprickett/redis-compatible-bloom-filter.git\ncd redis-compatible-bloom-filter\npython3 -m venv venv\n. ./venv/bin/activate\n```\n\nIf you installed Node in the previous optional step, you'll also want to enter the following command to install nodemon:\n\n```\nnpm install\n```\n\n## Run the Server\n\nIf you installed `nodemon`, start the server on port 6379:\n\n```bash\nsudo bash\n./run.sh\n```\n\nOr without `nodemon`:\n\n```bash\nsudo bash\npython redis-server.py\n```\n\nNote that you need to use the `sudo` command to run the code as the `root` user because of the way that the Unicorn Hat LED SDK works.\n\n## Using the Bloom Filter with redis-cli:\n\nExample interactions using the real `redis-cli`, assuming the Raspberry Pi's IP address is `192.168.4.39` (yours will vary or you can alternatively use the hostname if you configured one e.g. `bloom.local`):\n\n```\nredis-cli -h 192.168.4.39 -p 6379\n192.168.4.39:6379\u003e bf.madd leds a b c\n1) (integer) 1\n2) (integer) 1\n3) (integer) 1\n(14.64s)\n192.168.4.39:6379\u003e bf.madd leds a b d e\n1) (integer) 0\n2) (integer) 0\n3) (integer) 1\n4) (integer) 1\n(29.20s)\n192.168.4.39:6379\u003e bf.exists leds r\n(integer) 0\n(2.45s)\n192.168.4.39:6379\u003e bf.exists leds d\n(integer) 1\n(7.31s)\n```\n\nNote that the key name isn't used in the code as there's only one Unicorn HAT attached to the Pi, which is used to represent a single Bloom Filter.  Key names are required but you can use any key name you like.\n\n## Using the Bloom Filter with a Redis Client:\n\nGiven some limitations (see later) you can write code using your choice of Redis client that uses the LED Bloom Filter.  It will run slowly as the filter is slow, but as far as the programmer's concerned it looks like normal code that uses Redis.  Here's an example with the popular [node-redis](https://github.com/redis/node-redis) client for Node.js:\n\n```javascript\nimport { createClient } from 'redis';\n\nconst KEY_NAME = 'led';\n\nconst client = createClient({\n  url: 'redis://192.168.4.39:6379/' // or redis://bloom.local:6379/ etc\n});\n\nawait client.connect();\n\nawait client.bf.add(KEY_NAME, 'steve');\nawait client.bf.add(KEY_NAME, 'josefin');\nawait client.bf.mAdd(KEY_NAME, [\n  'justin',\n  'nava',\n  'kyle'\n]);\n\nconsole.log(await client.bf.exists(KEY_NAME, 'kyle'));\nconsole.log(await client.bf.exists(KEY_NAME, 'elena'));\n\nawait client.disconnect();\n```\n\n## Limitations\n\nThis is a fun demo that's not intended for production use :)  It runs slowly so that you can watch the state changes on the LEDs.  It also doens't support [pipelining](https://redis.io/docs/manual/pipelining/) in the Redis protocol, so won't be able to talk to a client that optimizes network traffic to Redis with that.\n\nRemember also that this demo doesn't pay attention to the Redis key name that you pass it in the commands... any command with the correct syntax will be executed against the same Bloom Filter instance, as there's only one Unicorn HAT connected to the Pi storing and displaying the state of the filter.\n\nIf you want to run a fast and scalable Bloom Filter in Redis, check out [Redis Stack](https://redis.io/docs/stack/get-started/).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsimonprickett%2Fredis-compatible-bloom-filter","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsimonprickett%2Fredis-compatible-bloom-filter","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsimonprickett%2Fredis-compatible-bloom-filter/lists"}