{"id":26016596,"url":"https://github.com/brzzznko/uniqueiptracker","last_synced_at":"2025-08-28T05:14:02.373Z","repository":{"id":280498530,"uuid":"942200793","full_name":"brzzznko/UniqueIpTracker","owner":"brzzznko","description":"Tracks unique IPv4 addresses from large input files","archived":false,"fork":false,"pushed_at":"2025-03-06T18:34:37.000Z","size":74,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-06-04T22:16:00.245Z","etag":null,"topics":["bitwise","concurrency","java","junit","optimization","parallel-computing"],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/brzzznko.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-03-03T18:27:34.000Z","updated_at":"2025-03-06T18:34:41.000Z","dependencies_parsed_at":"2025-03-03T20:00:08.135Z","dependency_job_id":null,"html_url":"https://github.com/brzzznko/UniqueIpTracker","commit_stats":null,"previous_names":["brzzznko/uniqueiptracker"],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/brzzznko/UniqueIpTracker","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/brzzznko%2FUniqueIpTracker","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/brzzznko%2FUniqueIpTracker/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/brzzznko%2FUniqueIpTracker/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/brzzznko%2FUniqueIpTracker/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/brzzznko","download_url":"https://codeload.github.com/brzzznko/UniqueIpTracker/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/brzzznko%2FUniqueIpTracker/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":272444019,"owners_count":24936055,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-28T02:00:10.768Z","response_time":74,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bitwise","concurrency","java","junit","optimization","parallel-computing"],"created_at":"2025-03-06T04:22:12.828Z","updated_at":"2025-08-28T05:14:02.367Z","avatar_url":"https://github.com/brzzznko.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Unique IP Tracker\n\n![build status badge](https://github.com/brzzznko/UniqueIpTracker/actions/workflows/build.yml/badge.svg)\n[![Codacy Badge](https://app.codacy.com/project/badge/Grade/63887cb1260f40f3bce2b33ddc3736df)](https://app.codacy.com/gh/brzzznko/UniqueIpTracker/dashboard?utm_source=gh\u0026utm_medium=referral\u0026utm_content=\u0026utm_campaign=Badge_grade)\n[![Codacy Badge](https://app.codacy.com/project/badge/Coverage/63887cb1260f40f3bce2b33ddc3736df)](https://app.codacy.com/gh/brzzznko/UniqueIpTracker/dashboard?utm_source=gh\u0026utm_medium=referral\u0026utm_content=\u0026utm_campaign=Badge_coverage)\n\n## **Overview**\nThe Unique IP Tracker is a high-performance tool for processing large files and counting unique IP addresses. It supports multiple processing methods and is optimized for parallel execution.\n\n---\n\n## **Algorithm**\n\nThe Unique IP Tracker efficiently processes large files using bit operations and parallel processing techniques:\n\n1. Bit Array Representation: Uses a large bit array (`AtomicBitArrayIpTracker.class`) to store seen IP addresses efficiently in memory.\n2. IP to Long Conversion: Converts IPv4 addresses into a single long value to map them into the bit array.\n3. Parallel Processing Methods:\n   - Parallel Streams (default) - Uses Java's built-in parallel streams for high-speed processing. (most efficient)\n   - Chunks - Splits the file into chunks and processes them concurrently.\n   - Single-threaded - Processes line by line without parallel execution for minimal memory usage. \n4. Efficient I/O Handling: Streams the file without loading it fully into memory to support very large files.\n5. Bitwise Operations for Fast Lookups: Uses bitwise shifts and masks to set and check bits in the bit array, reducing memory overhead compared to HashSets.\n\n## **Features**\n- **Parallel Processing** - Uses Parallel Streams, multi-threading for fast execution.  \n- **Customizable Processing Modes** - Choose from `parallel-streams`, `chunks`, or `single-threaded`.  \n- **Memory Optimized** - Uses bit operations and bit arrays to efficiently track unique IPs while minimizing memory usage.  \n- **Command-Line Interface** - Easily select processing mode and input file.  \n- **Pre-built JAR Available** - Download from [GitHub Releases](https://github.com/brzzznko/unique-ip-tracker/releases).\n\n---\n\n## **Installation \u0026 Setup**\n\n### **Clone the Repository**\n```sh\ngit clone https://github.com/brzzznko/unique-ip-tracker.git\ncd unique-ip-tracker\n```\n\n### **Build the Project**\n```sh\n./gradlew clean shadowJar\n```\nThis generates a **fat JAR** in `build/libs/` containing all dependencies.\n\nAlternatively, download the latest **pre-built JAR** from [GitHub Releases](https://github.com/brzzznko/UniqueIpTracker/releases) and skip the build step.\n\n---\n\n## **Usage**\n\n### **Example Input File (`example.txt`)**\n```\n97.71.174.4\n97.71.173.241\n97.71.173.235\n161.71.174.27\n215.10.61.107\n```\n\n### **Run Locally**\n```sh\njava -jar build/libs/unique-ip-tracker-1.0.0.jar --processor=parallel-streams --filename=path/input.txt\n```\n**Available Processors:** `parallel-streams` (default), `chunks`, `single-threaded`\n\n### **Run in Docker**\n```sh\ndocker build -t unique-ip-tracker .  \ndocker run --rm  -v ${PWD}/ip_addresses:/app/example.txt unique-ip-tracker\n```\n\n---\nCheck how it handles this [file](https://ecwid-vgv-storage.s3.eu-central-1.amazonaws.com/ip_addresses.zip). Attention - the file weighs about 20Gb, and unzips to about 120Gb.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbrzzznko%2Funiqueiptracker","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbrzzznko%2Funiqueiptracker","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbrzzznko%2Funiqueiptracker/lists"}