{"id":15525283,"url":"https://github.com/donatj/unic","last_synced_at":"2025-07-09T20:36:48.443Z","repository":{"id":35000500,"uuid":"110744602","full_name":"donatj/unic","owner":"donatj","description":"Like UNIX `sort | uniq` except it's quicker and maintains order. Uses a Cuckoo Filter.","archived":false,"fork":false,"pushed_at":"2023-12-07T14:19:17.000Z","size":52,"stargazers_count":4,"open_issues_count":0,"forks_count":0,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-04-23T09:41:15.518Z","etag":null,"topics":["command-line-tool","unique"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/donatj.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2017-11-14T21:05:24.000Z","updated_at":"2023-08-24T15:44:50.000Z","dependencies_parsed_at":"2023-12-07T15:29:44.888Z","dependency_job_id":"537f5803-f421-480c-a1e3-6d3e4f4e2ad3","html_url":"https://github.com/donatj/unic","commit_stats":null,"previous_names":[],"tags_count":5,"template":false,"template_full_name":null,"purl":"pkg:github/donatj/unic","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/donatj%2Funic","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/donatj%2Funic/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/donatj%2Funic/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/donatj%2Funic/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/donatj","download_url":"https://codeload.github.com/donatj/unic/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/donatj%2Funic/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":264502992,"owners_count":23618674,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["command-line-tool","unique"],"created_at":"2024-10-02T10:56:17.444Z","updated_at":"2025-07-09T20:36:48.415Z","avatar_url":"https://github.com/donatj.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# unic\n\n[![Go Report Card](https://goreportcard.com/badge/github.com/donatj/unic)](https://goreportcard.com/report/github.com/donatj/unic)\n[![GoDoc](https://godoc.org/github.com/donatj/unic?status.svg)](https://godoc.org/github.com/donatj/unic)\n\nWorks like UNIX `sort | uniq` to provide global uniques except you don't have to sort first.\n\nWorks by using Cuckoo Filters - See: https://github.com/seiflotfy/cuckoofilter\n\n## Advantages over `sort | uniq`\n\n### Quicker output, lower memory footprint\n\n`sort` by definitions needs to buffer the entire input before it can begin outputing **anything**. This can use a lot of memory and prevents anything from getting output until the initial process completes.\n\n`unic` uses probabalistic filters (Cuckoo) to determine if the input has been seen before, and can begin output after the first line of input.\n\n### Original item order is kept\n\nGiven the list `3 1 2 1 2 3`, compare `sort | uniq` 's output\n\n```bash\n$ echo '3\\n1\\n2\\n1\\n2\\n3' | sort | uniq\n1\n2\n3\n```\n\nto `unic`\n\n```bash\necho '3\\n1\\n2\\n1\\n2\\n3' | unic\n3\n1\n2\n```\n\n## Disadvantages\n\n### Probabilistic Filtering\n\nAs `unic` works with Cuckoo Filters, there is a very small probability a line will be wrongly marked duplicate. Lines will **never** be incorrectly marked as unique due to the nature of the filter.\n\nIn cases where a false positive cannot ever be tolerated, `unic` **should not** be used.\n\n### Not compatible with all of `uniq`'s flags\n\n`unic` by nature does not buffer; thus some of `uniq`'s flags cannot be implemented.\n\nIn these cases, you should use `uniq`.\n\n## Installing\n\n### Binaries\n\nSee: [releases](https://github.com/donatj/unic/releases)\n\n### From Source\n\n```bash\n$ go install github.com/donatj/unic/cmd/unic@latest\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdonatj%2Funic","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdonatj%2Funic","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdonatj%2Funic/lists"}