https://github.com/zelr0x/icntr

Distinct occurrence counter for anything that can be turned into an unsigned integer
https://github.com/zelr0x/icntr

Last synced: 13 days ago
JSON representation

Distinct occurrence counter for anything that can be turned into an unsigned integer

• Host: GitHub
• URL: https://github.com/zelr0x/icntr
• Owner: zelr0x
• License: gpl-3.0
• Created: 2022-09-27T19:08:51.000Z (over 2 years ago)
• Default Branch: master
• Last Pushed: 2023-06-14T18:27:29.000Z (over 1 year ago)
• Last Synced: 2024-11-05T19:56:48.207Z (2 months ago)
• Language: Kotlin
• Size: 77.1 KB
• Stars: 0
• Watchers: 1
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

        
# icntr - Integer-Based Distinct Count Problem Solver

*icntr* is a distinct occurrence counter for anything that can be turned into an unsigned integer

## IPv4 Counter Example

This repository contains an example for finding unique IPv4 addresses in a large file.

It takes 25-35 minutes to find 1 billion unique addresses within a 120GB (8 billion lines, one address per line) file on a 4-core ryzen-3 3200G,

with enabled input validation and live progress tracking.

Memory consumption depends heavily on the number of workers and the chunk size.

Chunk size is a portion of the file (in lines) read into memory by each worker.

## IPv4 Counter CLI

The syntax is:

```shell

icntr FILE [number-of-workers:4] [chunk-size:100000]

```

Gradle example:

```shell

# *nix

gradlew run --args='./ip_addresses 4 100000'

# Windows

gradlew run --args="ip_addresses 4 100000"

```