{"id":50335826,"url":"https://github.com/travisbrown/hkvdb","last_synced_at":"2026-05-29T13:31:20.458Z","repository":{"id":43747645,"uuid":"454010015","full_name":"travisbrown/hkvdb","owner":"travisbrown","description":"hkvdb","archived":false,"fork":false,"pushed_at":"2023-02-06T19:07:57.000Z","size":31,"stargazers_count":4,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-10-11T07:57:58.544Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/travisbrown.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2022-01-31T13:07:13.000Z","updated_at":"2023-10-03T08:32:24.000Z","dependencies_parsed_at":"2022-08-21T21:00:38.054Z","dependency_job_id":null,"html_url":"https://github.com/travisbrown/hkvdb","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/travisbrown/hkvdb","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/travisbrown%2Fhkvdb","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/travisbrown%2Fhkvdb/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/travisbrown%2Fhkvdb/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/travisbrown%2Fhkvdb/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/travisbrown","download_url":"https://codeload.github.com/travisbrown/hkvdb/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/travisbrown%2Fhkvdb/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":33655440,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-05-29T02:00:06.066Z","response_time":107,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-05-29T13:31:19.055Z","updated_at":"2026-05-29T13:31:20.450Z","avatar_url":"https://github.com/travisbrown.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"# hkvdb\n\n[![Rust build status](https://img.shields.io/github/workflow/status/travisbrown/hkvdb/rust-ci.svg?label=rust)](https://github.com/travisbrown/hkvdb/actions)\n[![Coverage status](https://img.shields.io/codecov/c/github/travisbrown/hkvdb/main.svg)](https://codecov.io/github/travisbrown/hkvdb)\n\nPlease note that this software is **not** \"open source\",\nbut the source is available for use and modification by individuals, non-profit organizations, and worker-owned cooperatives\n(see the [license section](#license) below for details).\n\n## About\n\nThis is a tiny project that packages up some code I was using in several places.\nThe motivation is that in several projects recently I've needed a way to store\nhistorical observations associating some entity with a field value at a particular time.\nFor example, we might want to collect profile image URLs for a selection of Twitter\naccounts, while keeping track of the first and last dates that each profile image\nis known to have been used.\n\nWe could use a relational database for this kind of task, but this data is often just\nan intermediate step in some other process, and the structure is very minimal, so\nsetting up a database in something like Postgres (or even SQLite) feels like overkill.\nUsing a [RocksDB][rocksdb] store gives us a lightweight way to work with this data\nefficiently without much setup cost.\n\nFor example, it takes around 35 minutes to load profile image URLs from 180 million\nuser profile snapshots from a month of [Twitter Stream Grab][tsg] data from 2021,\nand the resulting store (containing 38,937,009 URLs for 31,393,631 accounts) is only 2.2 GB.\nLooking up data values is pretty fast, with very little startup time. For example,\nsearching for profile image URLs and dates of use for a thousand users only takes a few dozen\nmilliseconds on my machine:\n\n```\n$ time target/release/demo data/profile-image-demo-2021-08/ \u003c users-ids-1k.txt \u003e out.txt\n\nreal\t0m0.053s\nuser\t0m0.049s\nsys     0m0.034s\n```\n\nThe version of this implementation here currently only supports tracking observed date\nranges or instances (using epoch seconds). ~~It also only supports lookup by key (e.g.\nTwitter user ID in the example above), although I have code for indexing values that\nI'll integrate at some point~~ [I did a quick job of copying some of this over but it's\nstill untested].\n\n## Usage\n\nThere's not much to the API beyond `put` and `get`. For example, suppose we have some data\nlike this:\n\n```rust\n\nstruct UserSnapshot {\n    user_id: u64,\n    timestamp_s: u32,\n    screen_name: String,\n}\n\nlet snapshots = vec![\n    UserSnapshot {\n        user_id: 770781940341288960,\n        timestamp_s: 1577933499,\n        screen_name: \"RudyGiuliani\".to_string(),\n    },\n    UserSnapshot {\n        user_id: 770781940341288960,\n        timestamp_s: 1479920042,\n        screen_name: \"xxxxxxx37583982\".to_string(),\n    },\n    UserSnapshot {\n        user_id: 6510972,\n        timestamp_s: 1643648042,\n        screen_name: \"travisbrown\".to_string(),\n    },\n    // Millions of other user profile snapshots?\n];\n```\n\nWe can create a database and insert this user data with `put`:\n\n```rust\nuse hkvdb::Hkvdb;\n\nlet db: Hkvdb\u003cRange32\u003e = Hkvdb::new(\"profile-image-urls\")?;\n\nfor snapshot in snapshots {\n    db.put(\n        snapshot.user_id,\n        \u0026snapshot.screen_name,\n        snapshot.timestamp_s,\n    )?;\n}\n```\n\nAnd look up all data values associated with an ID with `get`:\n\n```rust\nlet values = db.get(770781940341288960)?;\n\nlet mut expected = HashMap::new();\nexpected.insert(\"xxxxxxx37583982\".to_string(), 1479920042.into());\nexpected.insert(\"RudyGiuliani\".to_string(), 1577933499.into());\n\nassert_eq!(values, expected);\n```\n\nWe can also optionally create an index and search the database by data value:\n\n```rust\ndb.make_index(CaseSensitivity::Insensitive)?;\n\nlet user_ids = db.search_ci(\"RuDYgiuLianI\")?;\n\nassert_eq!(user_ids, vec![770781940341288960]);\n```\n\nThat's about all you can do with it for now!\n\n## Details\n\nThe format is very simple, and should be easily readable from any RocksDB client or library.\nFor example, for the version with date ranges:\n\n```\n+----------------------------+---------+\n| key                        | value   |\n+-+--------+---------········+----+----+\n|0| id     | data            |1st |last|\n+-+--------+---------········+----+----+\n```\n\nThe version that stores all observed timestamps just has more stuff in the value part:\n\n```\n+----------------------------+--------------+\n| key                        | value        |\n+-+--------+---------········+----+----+····|\n|0| id     | data            |1st |2nd |etc.|\n+-+--------+---------········+----+----+····|\n```\n\nIf you generate the index, the additional rows look like this:\n\n```\n+-------------------+--------------------------+\n| key               | value                    |\n+-+---------········+--------+--------+········|\n|1| data            | id 1   | id 2   | etc.   |\n+-+---------········+--------+--------+········|\n```\n\nAll integers are stored using the big-endian byte ordering.\n\n## License\n\nThis software is published under the [Anti-Capitalist Software License][acsl] (v. 1.4).\n\n[acsl]: https://anticapitalist.software/\n[rocksdb]: https://rocksdb.org\n[tsg]: https://archive.org/details/twitterstream\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftravisbrown%2Fhkvdb","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftravisbrown%2Fhkvdb","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftravisbrown%2Fhkvdb/lists"}