{"id":20836326,"url":"https://github.com/igrigorik/mneme","last_synced_at":"2025-05-08T02:38:59.878Z","repository":{"id":66004973,"uuid":"1505384","full_name":"igrigorik/mneme","owner":"igrigorik","description":"Mneme is an HTTP web-service for recording and identifying previously seen records - aka, duplicate detection.","archived":false,"fork":false,"pushed_at":"2013-06-30T17:27:34.000Z","size":453,"stargazers_count":108,"open_issues_count":0,"forks_count":5,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-03-31T16:34:44.926Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"http://www.igvita.com/2011/03/24/mneme-scalable-duplicate-filtering-service","language":"Ruby","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/igrigorik.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2011-03-21T04:20:36.000Z","updated_at":"2024-01-04T15:52:32.000Z","dependencies_parsed_at":"2023-02-19T21:15:21.757Z","dependency_job_id":null,"html_url":"https://github.com/igrigorik/mneme","commit_stats":null,"previous_names":[],"tags_count":4,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/igrigorik%2Fmneme","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/igrigorik%2Fmneme/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/igrigorik%2Fmneme/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/igrigorik%2Fmneme/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/igrigorik","download_url":"https://codeload.github.com/igrigorik/mneme/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252988635,"owners_count":21836576,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-18T00:29:33.696Z","updated_at":"2025-05-08T02:38:59.859Z","avatar_url":"https://github.com/igrigorik.png","language":"Ruby","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Mneme\n\n    mneme (n.)  mne·me\n       1. Psychology: the retentive basis or basic principle in a mind or organism accounting for memory, persisting effect of memory of past events.\n       2. Mythology: the Muse of memory, one of the original three Muses. Cf.\"Aoede, Melete.\"\n\nMneme is an HTTP web-service for recording and identifying previously seen records - aka, duplicate detection. To achieve this goal in a scalable, and zero-maintenance manner, it is implemented via a collection of automatically rotated bloomfilters. By using a collection of bloomfilters, you can customize your false-positive error rate, as well as the amount of time you want your memory to perist (ex: remember all keys for the last 6 hours).\n\nTo minimize the require memory footprint, mneme does not store the actual key names, instead each specified key is hashed and mapped onto the bloomfilter. For data storage, we use Redis getbit/setbit to efficiently store and retrieve bit-level data for each key. Couple this with Goliath app-server, and you have an out-of-the-box, high-performance, customizable duplicate filter.\n\nFor more details: [Mneme: Scalable Duplicate Filtering Service](http://www.igvita.com/2011/03/24/mneme-scalable-duplicate-filtering-service )\n\n## Sample configuration\n\n    # example_config.rb\n\n    config['namespace'] = 'default' # namespace for your app (if you're sharing a redis instance)\n    config['periods'] = 3           # number of periods to store data for\n    config['length']  = 60          # length of a period in seconds (length = 60, periods = 3.. 180s worth of data)\n\n    config['size']    = 1000        # desired size of the bloomfilter\n    config['bits']    = 10          # number of bits allocated per key\n    config['hashes']  = 7           # number of times each key will be hashed\n    config['seed']    = 30          # seed value for the hash function\n\n    config['pool']    = 2           # number of concurrent Redis connections\n\nTo learn more about Bloom filter configuration: [Scalable Datasets: Bloom Filters in Ruby](http://www.igvita.com/2008/12/27/scalable-datasets-bloom-filters-in-ruby/)\n\n## Launching mneme\n\n    $\u003e redis-server\n    $\u003e gem install mneme\n    $\u003e mneme -p 9000 -sv -c config.rb   # run with -h to see all options\n\nThat's it! You now have a mneme web service running on port 9000. Let's try querying and inserting some data:\n\n    $\u003e curl \"http://127.0.0.1:9000?key=abcd\"\n    {\"found\":[],\"missing\":[\"abcd\"]}\n\n    # -d creates a POST request with key=abcd, aka insert into filter\n    $\u003e curl \"http://127.0.0.1:9000?key=abcd\" -d' '\n\n    $\u003e curl \"http://127.0.0.1:9000?key=abcd\"\n    {\"found\":[\"abcd\"],\"missing\":[]}\n\n## Performance \u0026 Memory requirements\n\n - [Redis](http://redis.io/) is used as an in-memory datastore of the bloomfilter\n - [Goliath](https://github.com/postrank-labs/goliath) provides the high-performance HTTP frontend\n - The speed of storing a new key is: *O(number of BF hashes) - aka, O(1)*\n - The speed of retrieving a key is: *O(number of filters * number of BF hashes) - aka, O(1)*\n\n - Sample ab benchmarks for single key lookup: [https://gist.github.com/895326](https://gist.github.com/895326)\n\nBloom filter is a space-efficient probabilistic data structure that is used to test whether an element is a member of a set. False positives are possible, but false negatives are not. Because we are using Redis as a backend, in-memory store for the filters, there is some extra overhead. Sample memory requirements:\n\n - 1.0% error rate for 1M items, 10 bits/item: 2.5 mb\n - 1.0% error rate for 150M items, 10 bits per item: 358.52 mb\n - 0.1% error rate for 150M items, 15 bits per item: 537.33 mb\n\nEx: If you wanted to store up to 24 hours (with 1 hour = 1 bloom filter) of keys, where each hour can have up to 1M keys, and you are willing to accept a 1.0% error rate, then your memory footprint is: 24 * 2.5mb = 60mb of memory. The footprint will not change after 24 hours, because Mneme will automatically rotate and delete old filters for you!\n\n### License\n\n(MIT License) - Copyright (c) 2011 Ilya Grigorik","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Figrigorik%2Fmneme","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Figrigorik%2Fmneme","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Figrigorik%2Fmneme/lists"}