{"id":17190751,"url":"https://github.com/stestagg/bitfield-test","last_synced_at":"2026-03-18T23:57:03.017Z","repository":{"id":215423597,"uuid":"738872670","full_name":"stestagg/bitfield-test","owner":"stestagg","description":null,"archived":false,"fork":false,"pushed_at":"2024-01-04T13:27:24.000Z","size":384,"stargazers_count":0,"open_issues_count":2,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-01-30T05:27:32.863Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Cython","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/stestagg.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-01-04T08:39:39.000Z","updated_at":"2024-01-04T08:40:31.000Z","dependencies_parsed_at":null,"dependency_job_id":"e282ddb8-fdbd-4c69-928c-488cdbe4dc9e","html_url":"https://github.com/stestagg/bitfield-test","commit_stats":null,"previous_names":["stestagg/bitfield-test"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stestagg%2Fbitfield-test","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stestagg%2Fbitfield-test/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stestagg%2Fbitfield-test/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stestagg%2Fbitfield-test/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/stestagg","download_url":"https://codeload.github.com/stestagg/bitfield-test/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245407772,"owners_count":20610234,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-15T01:23:49.554Z","updated_at":"2026-01-05T17:06:47.031Z","avatar_url":"https://github.com/stestagg.png","language":"Cython","funding_links":[],"categories":[],"sub_categories":[],"readme":"bitfield [![Build Status](https://travis-ci.org/stestagg/bitfield.png)](https://travis-ci.org/stestagg/bitfield)\n========\n\nA Cython implemented fast positive integer set implementation, optimised for sets of sequential numbers.\n\n__WARNING__ : The serialisation mechanism is not currently portable,  This will be fixed soon\n\nInstallation\n---------\n\n```\n$ sudo easy_install bitfield\n```\n\nUsage\n-----\n\n```python\n\u003e\u003e\u003e import bitfield\n\u003e\u003e\u003e field = bitfield.Bitfield()\n\u003e\u003e\u003e field.add(100)\n\u003e\u003e\u003e print list(field)\n[100L]\n\u003e\u003e\u003e second = bitfield.Bitfield([2, 100])\n\u003e\u003e\u003e list(field | second)\n[2L, 100L]\n\n\u003e\u003e\u003e second.add(10000)\n\u003e\u003e\u003e second.pickle()\n'BZ:x\\x9cca@\\x00\\x01\\x86Q0\\nF\\xc1(\\x18N\\x80\\x11\\x00e\\xe0\\x00\\x16'\n\n\u003e\u003e\u003e large=bitfield.Bitfield(random.sample(xrange(1000000), 500000)) # 500,000 items, randomly distributed\n\u003e\u003e\u003e len(large)\n500000\n\u003e\u003e\u003e len(large.pickle())\n125049  # 122KB\n\n\u003e\u003e\u003e large=bitfield.Bitfield(xrange(1000000)) # 1 million items, all sequential\n\u003e\u003e\u003e len(large)\n1000000\n\u003e\u003e\u003e len(large.pickle())\n36 # \u003c40 bytes\n```\n\nBitfields support most of the same operations/usage as regular sets, see the tests for examples.\n\nDesign\n------\n\nBitfield was designed to efficiently handle tracking large sets of items.\n\nThe main design goals were:\n * Space-efficient serialisation format\n * Fast membership tests and set differences\n\nInternally, bitfield achieves this using a page-compressed 1-d bitmap.  \n\nWithin a page, a number is recorded as being present in the set by setting the n-th bit to 1.\nI.e. the set([1]) is recorded as ...00000010b, while set([1,4]) would be ...00010010b.\n\nThis works well for small sets, but the size of the bitfield tends towards (highest set member)/8 bytes as the largest number in the set increases. \n\nTo counter this, the bit field is split into chunks of 1 page (usually 4k).  If a particular page is empty(no set members in that range) or full, \nthen the bitfield is discarded, and represented by an EMPTY or FULL flag.\n\nTo improve lookup times and simplify set comparison, the bitfield always indexes items from 0.  \nTherefore, a set with a single item of 1,000,000,000 isn't going to be as fast as it could be.  This was an intentional design decision.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fstestagg%2Fbitfield-test","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fstestagg%2Fbitfield-test","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fstestagg%2Fbitfield-test/lists"}