{"id":21296952,"url":"https://github.com/reddec/file-stack","last_synced_at":"2025-03-15T17:27:45.332Z","repository":{"id":81702484,"uuid":"57661459","full_name":"reddec/file-stack","owner":"reddec","description":"File based stack","archived":false,"fork":false,"pushed_at":"2016-05-03T19:29:30.000Z","size":230,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-03-09T00:06:38.995Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/reddec.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2016-05-01T16:43:32.000Z","updated_at":"2019-02-28T10:19:04.000Z","dependencies_parsed_at":null,"dependency_job_id":"b1dfe95a-1555-43c2-9fee-4ad3e64cf453","html_url":"https://github.com/reddec/file-stack","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/reddec%2Ffile-stack","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/reddec%2Ffile-stack/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/reddec%2Ffile-stack/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/reddec%2Ffile-stack/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/reddec","download_url":"https://codeload.github.com/reddec/file-stack/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243765050,"owners_count":20344526,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-21T14:31:17.297Z","updated_at":"2025-03-15T17:27:45.296Z","avatar_url":"https://github.com/reddec.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# About\n\nThis is extended stack implementation on Go-lang with the file as storage\n\nFor API see http://godoc.org/github.com/reddec/file-stack\n\nUsed in [file-stack-db](http://github.com/reddec/file-stack-db)\n\n# Installation\n\nBy Go build\n\n    go get -u github.com/reddec/file-stack/cmd/...\n\n\nFor Debian/Centos by [packager.io](https://packager.io/gh/reddec/file-stack)\n\n# Introduction\n\n## Use-case\n\nWhen a lot of messages (events from IoT for example) with simple structure\nhave to be saved on disk for later usage with instant (or near that) access to\nlast message (I mean something like `get last state`). Processing of\nhistorical events may be non-realtime operation.\n\n## Requirements\n\nExpected load:\n\n* Rate: about 40'000+ messages/seconds\n* Message size:\n * 90% - less then 1KB (means status updates or notifications)\n * 10% - about 5-50MB (means camera images or batch updates)\n* Message contains:\n * Small (can easily fits into one memory block - about 4KB) header\n * Big body\n\nExpected environment:\n\n* Fast, big and reliable file storage - like SSD, enterprise HDD or NAS\n* Low-cost multi-cores computing unit - like virtual machine in Digital Ocean, Supermicro or Raspberry Pi 2\n\nExpected usage:\n\n* State query (aka `get last message`): about 5'000+ requests/seconds\n* Full historical dump: about 1 times/week\n* Full historical headers scan: about 0.5 time/second\n* Almost messages have not to be lost\n\nFormal minimal operations descriptions:\n\n* Put full message\n* Get last message\n* Iterate over headers\n\n## Possible decisions\n\nThere are several implementations: SQL databases, no-SQL, KV databases,\nin-memory storage and others.\n\nTested:\n\n### PostgreSQL\n\n* Tested version: *9.4*\n\n#### Pros\n\n* Stability\n* Clustering\n* Low resource usage\n\n#### Cons\n\n* Not enough speed (7000-8000 inserts/second on target hardware)\n\n### Cassandra DB\n\n#### Pros\n\n* Fastest solution in theory\n* Stability\n\n#### Cons\n\n* Awful memory usage - about 10 GB in idle state\n\n### LevelDB\n\n#### Pros\n\n* Stable\n* Fast\n\n#### Cons\n\n* Not enough fast (about 20000 inserts/second)\n* Too simple structure: can't iterate over keys without loading full item\n\n# Current implementation\n\n12-cores I7 5-gen, SSD 840 PRO\n\n* Push: 200'000 messages/second\n* Pop: 200'000 messages/second\n* Last message: 1'000'000 requests/second\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Freddec%2Ffile-stack","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Freddec%2Ffile-stack","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Freddec%2Ffile-stack/lists"}