{"id":20589966,"url":"https://github.com/doist/bitmapist","last_synced_at":"2026-01-07T22:14:16.417Z","repository":{"id":5214900,"uuid":"6390826","full_name":"Doist/bitmapist","owner":"Doist","description":"Powerful analytics and cohort library using Redis bitmaps","archived":false,"fork":false,"pushed_at":"2025-03-03T16:14:48.000Z","size":378,"stargazers_count":942,"open_issues_count":1,"forks_count":77,"subscribers_count":48,"default_branch":"main","last_synced_at":"2025-04-13T13:54:03.228Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":"heroku/heroku-buildpack-ruby","license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Doist.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":"AUTHORS","dei":null,"publiccode":null,"codemeta":null}},"created_at":"2012-10-25T16:50:33.000Z","updated_at":"2025-03-30T23:00:22.000Z","dependencies_parsed_at":"2024-06-18T20:13:40.577Z","dependency_job_id":"7dede69d-1625-43dc-8bc6-b65d69ab1ff9","html_url":"https://github.com/Doist/bitmapist","commit_stats":{"total_commits":153,"total_committers":19,"mean_commits":8.052631578947368,"dds":0.7450980392156863,"last_synced_commit":"c5c3ee659d1f3b75b3519fe22bb931fea6855884"},"previous_names":[],"tags_count":27,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Doist%2Fbitmapist","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Doist%2Fbitmapist/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Doist%2Fbitmapist/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Doist%2Fbitmapist/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Doist","download_url":"https://codeload.github.com/Doist/bitmapist/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254069596,"owners_count":22009558,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-16T07:33:11.158Z","updated_at":"2026-01-07T22:14:16.405Z","avatar_url":"https://github.com/Doist.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"![bitmapist](https://raw.githubusercontent.com/Doist/bitmapist/master/static/bitmapist.png \"bitmapist\")\n\n[![Build Status](https://travis-ci.org/Doist/bitmapist.svg?branch=master)](https://travis-ci.org/Doist/bitmapist)\n\n# Bitmapist: a powerful analytics library for Redis\n\nBitmapist makes it possible to implement real-time, highly scalable analytics. The library is very easy to use, enabling you to create reports easily.\n\nLeveraging Redis bitmaps, you can store events for millions of users using a very little amount of memory (megabytes).\n\n\u003e [!TIP]\n\u003e Instead of Redis as a backing store, consider using [bitmapist-server](https://github.com/Doist/bitmapist-server).\n\u003e\n\u003e It is our custom data store that exposes a (partial) Redis-compatible API, fully compatible with Bitmapist. It is 443x more memory efficient for this particular use case, improving scalability and cost-effectiveness.\n\n## Use cases\n\nBitmapist can answer questions like:\n\n- Has user 123 been online today? This week? This month?\n- Has user 123 performed action \"X\"?\n- How many users have been active this month? This hour?\n- How many unique users have performed action \"X\" this week?\n- How many % of users that were active last week are still active?\n- How many % of users that were active last month are still active this month?\n- What users performed action \"X\"?\n\nAdditionally, it can generate cohort graphs that can do following:\n\n- Cohort over user retention\n- How many % of users that were active last [days, weeks, months] are still active?\n- How many % of users that performed action X also performed action Y (and this over time)\n- And a lot of other things!\n\n### Caveat: Avoid large IDs\n\nYou should be careful about using large IDs as this will require larger amounts of memory. IDs should be in range `[0, 2^32)`.\n\n## Installation\n\nCan be installed very easily via:\n\n    $ pip install bitmapist\n\nOr, if you use `uv`:\n\n    $ uv add bitmapist\n\n## Usage and examples\n\nSetting things up:\n\n```python\nfrom datetime import datetime, timedelta, timezone\nfrom bitmapist import setup_redis, delete_all_events, mark_event,\\\n                      MonthEvents, WeekEvents, DayEvents, HourEvents,\\\n                      BitOpAnd, BitOpOr\n\nnow = datetime.now(tz=timezone.utc)\nlast_month = now - timedelta(days=30)\n```\n\nMark user 123 as active and has played a song:\n\n```python\nmark_event('active', 123)\nmark_event('song:played', 123)\n```\n\nAnswer if user 123 has been active this month:\n\n```python\nassert 123 in MonthEvents('active', now.year, now.month)\nassert 123 in MonthEvents('song:played', now.year, now.month)\nassert MonthEvents('active', now.year, now.month).has_events_marked() == True\n```\n\nHow many users have been active this week?:\n\n```python\niso_year, iso_week, _ = now.isocalendar()\nprint(len(WeekEvents('active', iso_year, iso_week)))\n```\n\nIterate over all users active this week:\n\n```python\nfor uid in WeekEvents('active'):\n    print(uid)\n```\n\nIf you're interested in \"current events\", you can omit extra `now.whatever`\narguments. Events will be populated with current time automatically.\n\nFor example, these two calls are equivalent:\n\n```python\n\nMonthEvents('active') == MonthEvents('active', now.year, now.month)\n\n```\n\nAdditionally, for the sake of uniformity, you can create an event from\nany datetime object with a `from_date` static method.\n\n```python\n\nMonthEvents('active').from_date(now) == MonthEvents('active', now.year, now.month)\n\n```\n\nGet the list of these users (user ids):\n\n```python\niso_year, iso_week, _ = now.isocalendar()\nprint(list(WeekEvents('active', iso_year, iso_week)))\n```\n\nThere are special methods `prev` and `next` returning \"sibling\" events and\nallowing you to walk through events in time without any sophisticated\niterators. A `delta` method allows you to \"jump\" forward or backward for\nmore than one step. Uniform API allows you to use all types of base events\n(from hour to year) with the same code.\n\n```python\n\ncurrent_month = MonthEvents()\nprev_month = current_month.prev()\nnext_month = current_month.next()\nyear_ago = current_month.delta(-12)\n\n```\n\nEvery event object has `period_start` and `period_end` methods to find a\ntime span of the event. This can be useful for caching values when the caching\nof \"events in future\" is not desirable:\n\n```python\n\nev = MonthEvent('active', dt)\nif ev.period_end() \u003c now:\n    cache.set('active_users_\u003c...\u003e', len(ev))\n\n```\n\nAs something new tracking hourly is disabled (to save memory!) To enable it as default do::\n\n```python\nimport bitmapist\nbitmapist.TRACK_HOURLY = True\n```\n\nAdditionally you can supply an extra argument to `mark_event` to bypass the default value::\n\n```python\nmark_event('active', 123, track_hourly=False)\n```\n\n### Unique events\n\nSometimes the date of the event makes little or no sense, for example,\nto filter out your premium accounts, or in A/B testing. There is a\n`UniqueEvents` model for this purpose. The model creates only one\nRedis key and doesn't depend on the date.\n\nYou can combine unique events with other types of events.\n\nA/B testing example:\n\n```python\n\nactive_today = DailyEvents('active')\na = UniqueEvents('signup_form:classic')\nb = UniqueEvents('signup_form:new')\n\nprint(\"Active users, signed up with classic form\", len(active \u0026 a))\nprint(\"Active users, signed up with new form\", len(active \u0026 b))\n```\n\nGeneric filter example\n\n```python\n\ndef premium_up(uid):\n    # called when user promoted to premium\n    ...\n    mark_unique('premium', uid)\n\n\ndef premium_down(uid):\n    # called when user loses the premium status\n    ...\n    unmark_unique('premium', uid)\n\nactive_today = DailyEvents('active')\npremium = UniqueEvents('premium')\n\n# Add extra Karma for all premium users active today,\n# just because today is a special day\nfor uid in premium \u0026 active_today:\n    add_extra_karma(uid)\n```\n\nTo get the best of two worlds you can mark unique event and regular\nbitmapist events at the same time.\n\n```python\ndef premium_up(uid):\n    # called when user promoted to premium\n    ...\n    mark_event('premium', uid, track_unique=True)\n\n```\n\n### Perform bit operations\n\nHow many users that have been active last month are still active this month?\n\n```python\nactive_2_months = BitOpAnd(\n    MonthEvents('active', last_month.year, last_month.month),\n    MonthEvents('active', now.year, now.month)\n)\nprint(len(active_2_months))\n\n# Is 123 active for 2 months?\nassert 123 in active_2_months\n```\n\nAlternatively, you can use standard Python syntax for bitwise operations.\n\n```python\nlast_month_event = MonthEvents('active', last_month.year, last_month.month)\nthis_month_event = MonthEvents('active', now.year, now.month)\nactive_two_months = last_month_event \u0026 this_month_event\n```\n\nOperators `\u0026`, `|`, `^` and `~` supported.\n\nWork with nested bit operations (imagine what you can do with this ;-))!\n\n```python\nactive_2_months = BitOpAnd(\n    BitOpAnd(\n        MonthEvents('active', last_month.year, last_month.month),\n        MonthEvents('active', now.year, now.month)\n    ),\n    MonthEvents('active', now.year, now.month)\n)\nprint(len(active_2_months))\nassert 123 in active_2_months\n\n# Delete the temporary AND operation\nactive_2_months.delete()\n```\n\n### Deleting\n\nIf you want to permanently remove marked events for any time period you can use the `delete()` method:\n\n```python\nlast_month_event = MonthEvents('active', last_month.year, last_month.month)\nlast_month_event.delete()\n```\n\nIf you want to remove all bitmapist events use:\n\n```python\nbitmapist.delete_all_events()\n```\n\nWhen using Bit Operations (ie `BitOpAnd`) you can (and probably should) delete the results unless you want them cached. There are different ways to go about this:\n\n```python\nactive_2_months = BitOpAnd(\n    MonthEvents('active', last_month.year, last_month.month),\n    MonthEvents('active', now.year, now.month)\n)\n# Delete the temporary AND operation\nactive_2_months.delete()\n\n# delete all bit operations created in runtime up to this point\nbitmapist.delete_runtime_bitop_keys()\n\n# delete all bit operations (slow if you have many millions of keys in Redis)\nbitmapist.delete_temporary_bitop_keys()\n```\n\n## Cohorts\n\nWith bitmapist cohort you can get a form and a table rendering of the data you keep in bitmapist. If this sounds confusing [please look at Mixpanel](https://mixpanel.com/retention/).\n\nHere's a simple example of how to generate a form and a rendering of the data you have inside bitmapist:\n\n```python\nfrom bitmapist import cohort\n\nhtml_form = cohort.render_html_form(\n    action_url='/_Cohort',\n    selections1=[ ('Are Active', 'user:active'), ],\n    selections2=[ ('Task completed', 'task:complete'), ]\n)\nprint(html_form)\n\ndates_data = cohort.get_dates_data(select1='user:active',\n                                   select2='task:complete',\n                                   time_group='days')\n\nhtml_data = cohort.render_html_data(dates_data,\n                                    time_group='days')\n\nprint(html_data)\n\n# All the arguments should come from the FORM element (html_form)\n# but to make things more clear I have filled them in directly\n```\n\nThis will render something similar to this:\n\n![bitmapist cohort screenshot](https://raw.githubusercontent.com/Doist/bitmapist/master/static/cohort_screenshot.png \"bitmapist cohort screenshot\")\n\n## References\n\nIf you want to read more about bitmaps please read following:\n\n- http://blog.getspool.com/2011/11/29/fast-easy-realtime-metrics-using-redis-bitmaps/\n- http://redis.io/commands/setbit\n- http://en.wikipedia.org/wiki/Bit_array\n- http://www.slideshare.net/crashlytics/crashlytics-on-redis-analytics\n\n## Contributing\n\nPlease see our guide [here](./CONTRIBUTING.md)\n\n## Local Development\n\nWe use `uv` for dependency management \u0026 packaging. Please see [here for setup instructions](https://docs.astral.sh/uv/getting-started/).\n\nOnce you have `uv` installed, you can run the following to install the dependencies in a virtual environment:\n\n```bash\nuv sync\n```\n\n## Testing\n\n### Quick Start with Docker (Recommended)\n\nThe easiest way to run tests locally is with Docker:\n\n```bash\n# Start both backend servers\ndocker compose up -d\n\n# Run tests\nuv run pytest\n\n# Stop servers when done\ndocker compose down\n```\n\nThis runs tests against both Redis and bitmapist-server backends automatically.\n\n### Alternative: Native Binaries\n\nTo run tests with native binaries, you'll need at least one backend server installed:\n\n**Redis:**\n- Install `redis-server` using your package manager\n- Ensure it's in your `PATH`, or set `BITMAPIST_REDIS_SERVER_PATH`\n\n**Bitmapist-server:**\n- Download from the [releases page](https://github.com/Doist/bitmapist-server/releases)\n- Ensure it's in your PATH, or set `BITMAPIST_SERVER_PATH`\n\nThen run:\n```bash\nuv run pytest\n```\n\nThe test suite auto-detects available backends and runs accordingly:\n- **Docker containers running?** Uses them\n- **Native binaries available?** Starts them automatically\n- **Nothing available?** Shows error\n\n### Configuration\n\n#### Environment Variables\n\nCustomize backend locations and ports if needed:\n\n```bash\n# Backend binary paths (optional - auto-detected from PATH by default)\nexport BITMAPIST_REDIS_SERVER_PATH=/custom/path/to/redis-server\nexport BITMAPIST_SERVER_PATH=/custom/path/to/bitmapist-server\n\n# Backend ports (optional - defaults shown)\nexport BITMAPIST_REDIS_PORT=6399\nexport BITMAPIST_SERVER_PORT=6400\n```\n\n#### Testing Specific Backends\n\n```bash\n# Test only Redis\nuv run pytest -k redis\n\n# Test only bitmapist-server\nuv run pytest -k bitmapist-server\n```\n\n## Releasing new versions\n\n1. Bump version in `pyproject.toml` (or use `uv version`)\n   ```sh\n   uv version --bump minor\n   ```\n1. Update the CHANGELOG\n1. Commit the changes with a commit message \"Version X.X.X\"\n   ```sh\n   git commit -m \"Version $(uv version --short)\"\n   ```\n1. Tag the current commit with `vX.X.X`\n   ```sh\n   git tag -a -m \"Release $(uv version --short)\" \"v$(uv version --short)\"\n   ```\n1. Create a new release on GitHub named `vX.X.X`\n1. GitHub Actions will publish the new version to PyPI for you\n\n## Legal\n\nCopyright: 2012 by Doist Ltd.\n\nLicense: BSD-3-Clause\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdoist%2Fbitmapist","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdoist%2Fbitmapist","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdoist%2Fbitmapist/lists"}