{"id":37022245,"url":"https://github.com/sadikovi/spark-netflow","last_synced_at":"2026-01-14T02:39:02.256Z","repository":{"id":55517085,"uuid":"50008884","full_name":"sadikovi/spark-netflow","owner":"sadikovi","description":"NetFlow data source for Spark SQL and DataFrames","archived":false,"fork":false,"pushed_at":"2021-05-06T05:59:41.000Z","size":554,"stargazers_count":18,"open_issues_count":5,"forks_count":11,"subscribers_count":8,"default_branch":"master","last_synced_at":"2024-11-15T12:11:56.439Z","etag":null,"topics":["cisco","datasource","flow-tools","netflow","spark","sql"],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/sadikovi.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2016-01-20T06:12:10.000Z","updated_at":"2024-04-19T06:31:13.000Z","dependencies_parsed_at":"2022-08-15T02:10:53.770Z","dependency_job_id":null,"html_url":"https://github.com/sadikovi/spark-netflow","commit_stats":null,"previous_names":[],"tags_count":17,"template":false,"template_full_name":null,"purl":"pkg:github/sadikovi/spark-netflow","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sadikovi%2Fspark-netflow","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sadikovi%2Fspark-netflow/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sadikovi%2Fspark-netflow/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sadikovi%2Fspark-netflow/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/sadikovi","download_url":"https://codeload.github.com/sadikovi/spark-netflow/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sadikovi%2Fspark-netflow/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28408711,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-14T01:52:23.358Z","status":"online","status_checked_at":"2026-01-14T02:00:06.678Z","response_time":107,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cisco","datasource","flow-tools","netflow","spark","sql"],"created_at":"2026-01-14T02:39:01.547Z","updated_at":"2026-01-14T02:39:02.238Z","avatar_url":"https://github.com/sadikovi.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"# spark-netflow\nA library for reading NetFlow files from [Spark SQL](http://spark.apache.org/docs/latest/sql-programming-guide.html).\n\n[![Build Status](https://travis-ci.org/sadikovi/spark-netflow.svg?branch=master)](https://travis-ci.org/sadikovi/spark-netflow)\n[![codecov](https://codecov.io/gh/sadikovi/spark-netflow/branch/master/graph/badge.svg)](https://codecov.io/gh/sadikovi/spark-netflow)\n\n## Requirements\n| Spark version | spark-netflow latest version |\n|---------------|------------------------------|\n| 1.4.x | [1.3.1](http://spark-packages.org/package/sadikovi/spark-netflow) |\n| 1.5.x | [1.3.1](http://spark-packages.org/package/sadikovi/spark-netflow) |\n| 1.6.x | [1.3.1](http://spark-packages.org/package/sadikovi/spark-netflow) |\n| 2.0.x | [2.0.4](http://spark-packages.org/package/sadikovi/spark-netflow) |\n| 2.1.x | [2.0.4](http://spark-packages.org/package/sadikovi/spark-netflow) |\n| 3.0.x | [2.1.0](http://spark-packages.org/package/sadikovi/spark-netflow) |\n\n\u003e Documentation reflects changes in master branch, for documentation on a specific version, please\n\u003e select corresponding version tag or branch.\n\n## Linking\nThe spark-netflow library can be added to Spark by using the `--packages` command line option. For\nexample, run this to include it when starting the spark shell:\n```shell\n $SPARK_HOME/bin/spark-shell --packages com.github.sadikovi:spark-netflow_2.12:2.1.0\n```\nSee other available versions at http://spark-packages.org/package/sadikovi/spark-netflow.\n\n## Features\n- Column pruning\n- Predicate pushdown to the NetFlow file\n- Auto statistics based on file header information\n- Fields conversion (IP addresses, protocol, etc.)\n- NetFlow version 5 support ([list of columns](./docs/NETFLOW_V5.md))\n- NetFlow version 7 support ([list of columns](./docs/NETFLOW_V7.md))\n- Reading files from local file system and HDFS\n\n### Options\nCurrently supported options:\n\n| Name | Example | Description |\n|------|:-------:|-------------|\n| `version` | _5, 7_ | Version to use when parsing NetFlow files. This setting is optional, by default the package will resolve the version from provided files\n| `buffer` | _1024, 32Kb, 3Mb, etc_ | Buffer size for NetFlow compressed stream (default `1Mb`)\n| `stringify` | _true, false_ | Enables conversion of certain supported fields (e.g. IP, protocol) into human-readable format. If performance is essential, consider disabling the feature (default `true`)\n| `predicate-pushdown` | _true, false_ | Enables predicate pushdown at NetFlow library level (default `true`)\n\n### Dealing with corrupt files\nPackage supports Spark option `spark.sql.files.ignoreCorruptFiles`. When set to `true`, corrupt files\nare ignored (corrupt header, wrong format) or partially read (corrupt data block in a middle of a\nfile). By default, option is set to `false`, meaning exception will be raised when such file is\nencountered, this behaviour is similar to Spark.\n\n### Other NetFlow formats\nIf you would like to have the package support NetFlow files for other formats, e.g. NetFlow 9, feel free to open an issue or a pull request.\n\n## Example\n\n### Scala API\n```scala\n// You can provide only format, package will infer version from provided files,\n// or you can enforce version of the files with `version` option.\nval df = spark.read.format(\"com.github.sadikovi.spark.netflow\").load(\"...\")\n\n// You can read files from local file system or HDFS.\nval df = spark.read.format(\"com.github.sadikovi.spark.netflow\")\n  .option(\"version\", \"5\")\n  .load(\"file:/...\")\n  .select(\"srcip\", \"dstip\", \"packets\")\n\n// You can also specify buffer size when reading compressed NetFlow files.\nval df = spark.read.format(\"com.github.sadikovi.spark.netflow\")\n  .option(\"version\", \"5\")\n  .option(\"buffer\", \"2Mb\")\n  .load(\"hdfs://sandbox:8020/tmp/...\")\n```\n\nAlternatively you can use shortcuts for NetFlow files\n```scala\nimport com.github.sadikovi.spark.netflow._\n\n// This will read version 5 with default buffer size.\nval df = spark.read.netflow5(\"hdfs:/...\")\n\n// This will read version 7 without fields conversion.\nval df = spark.read.option(\"stringify\", \"false\").netflow7(\"file:/...\")\n```\n\n### Python API\n```python\ndf = spark.read.format(\"com.github.sadikovi.spark.netflow\") \\\n  .option(\"version\", \"5\") \\\n  .load(\"file:/...\") \\\n  .select(\"srcip\", \"srcport\")\n\nres = df.where(\"srcip \u003e 10\")\n```\n\n### SQL API\n```sql\nCREATE TEMPORARY TABLE ips\nUSING com.github.sadikovi.spark.netflow\nOPTIONS (path \"file:/...\", version \"5\");\n\nSELECT srcip, dstip, srcport, dstport FROM ips LIMIT 10;\n```\n\n## Building From Source\nThis library is built using `sbt`, to build a JAR file simply run `sbt package` from project root.\n\n## Testing\nRun `sbt test` from project root.\n\n## Running benchmark\nRun `sbt package` to package project, next run `spark-submit` with following options:\n```shell\n$ spark-submit --class com.github.sadikovi.spark.benchmark.NetFlowReadBenchmark \\\n  target/scala-2.12/spark-netflow_2.12-2.1.0.jar \\\n  --iterations 5 \\\n  --files 'file:/Users/sadikovi/developer/spark-netflow/temp/ftn/0[1,2,3]/ft*' \\\n  --version 5\n```\n\nLatest benchmarks:\n```\n- Iterations: 5\n- Files: file:/tmp/spark-netflow/files/0[1,2,3]/ft*\n- Version: 5\n\nJava HotSpot(TM) 64-Bit Server VM 1.7.0_80-b15 on Mac OS X 10.12.4\nIntel(R) Core(TM) i5-4258U CPU @ 2.40GHz\nNetFlow full scan:                       Best/Avg Time(ms)    Rate(M/s)   Per Row(ns)   Relative\n------------------------------------------------------------------------------------------------\nScan, stringify = F                            567 /  633          0.0       56726.7       1.0X\nScan, stringify = T                            968 / 1049          0.0       96824.6       0.6X\n\nJava HotSpot(TM) 64-Bit Server VM 1.7.0_80-b15 on Mac OS X 10.12.4\nIntel(R) Core(TM) i5-4258U CPU @ 2.40GHz\nNetFlow predicate scan:                  Best/Avg Time(ms)    Rate(M/s)   Per Row(ns)   Relative\n------------------------------------------------------------------------------------------------\nPredicate pushdown = F, high                  1148 / 1200          0.0      114845.4       1.0X\nPredicate pushdown = T, high                  1208 / 1257          0.0      120818.0       1.0X\nPredicate pushdown = F, low                    706 /  732          0.0       70559.3       1.6X\nPredicate pushdown = T, low                    226 /  243          0.0       22575.0       5.1X\n\nJava HotSpot(TM) 64-Bit Server VM 1.7.0_80-b15 on Mac OS X 10.12.4\nIntel(R) Core(TM) i5-4258U CPU @ 2.40GHz\nNetFlow aggregated report:               Best/Avg Time(ms)    Rate(M/s)   Per Row(ns)   Relative\n------------------------------------------------------------------------------------------------\nAggregated report                             2171 / 2270          0.0      217089.9       1.0X\n```\n\n## Using `netflowlib` library separately\nYou can use `netflowlib` without using `spark-netflow` package. Here some basic concepts and\nexamples:\n- `com.github.sadikovi.netflowlib.predicate.Columns.*` all available column types in the library,\ncheck out `com.github.sadikovi.netflowlib.version.*` classes to see what columns are already defined\nfor a specific NetFlow format.\n- `com.github.sadikovi.netflowlib.predicate.FilterApi` utility class to create predicates for\nNetFlow file\n- `com.github.sadikovi.netflowlib.statistics.StatisticsTypes` statistics that you can use to reduce\nboundaries of filter or allow filter to be evaluated before scanning the file. For example, library\ncreates statistics on time, so time filter can be resolved upfront\n- `com.github.sadikovi.netflowlib.NetFlowReader` main entry to work with NetFlow file, gives access\nto file header and iterator of rows, allows to pass additional predicate and statistics\n- `com.github.sadikovi.netflowlib.NetFlowHeader` header information can be accessed using this\nclass from `NetFlowReader.getHeader()`, see class for more information on flags available\n\nHere is the general usage pattern:\n```scala\nimport com.github.sadikovi.netflowlib.NetFlowReader\nimport com.github.sadikovi.netflowlib.version.NetFlowV5\n\n// Create input stream by opening NetFlow file, e.g. `fs.open(hadoopFile)`\nval stm: DataInputStream = ...\n// Prepare reader based on input stream and buffer size, you can use\n// overloaded alternative with default buffer size\nval reader = NetFlowReader.prepareReader(stm, 10000)\n// Check out header, optional\nval header = reader.getHeader()\n// Actual NetFlow version of the file\nval actualVersion = header.getFlowVersion()\n// Whether or not file is compressed\nval isCompressed = header.isCompressed()\n\n// This is list of fields that will be returned in iterator as values in\n// array (same order)\nval fields = Array(\n  NetFlowV5.FIELD_UNIX_SECS,\n  NetFlowV5.FIELD_SRCADDR,\n  NetFlowV5.FIELD_DSTADDR,\n  NetFlowV5.FIELD_SRCPORT,\n  NetFlowV5.FIELD_DSTPORT\n)\n\n// Build record buffer and iterator that you can use to get values.\n// Note that you can also use set of filters, if you want to get\n// particular records\nval recordBuffer = reader.prepareRecordBuffer(fields)\nval iter = recordBuffer.iterator()\n\nwhile (iter.hasNext) {\n  // print every row with values\n  println(iter.next)\n}\n```\n\nHere is an example of using predicate to keep certain records:\n```scala\nimport com.github.sadikovi.netflowlib.predicate.FilterApi\nval predicate = FilterApi.and(\n  FilterApi.eq(NetFlowV5.FIELD_SRCPORT, 123),\n  FilterApi.eq(NetFlowV5.FIELD_DSTPORT, 456)\n)\n\n...\nval recordBuffer = reader.prepareRecordBuffer(fields, predicate)\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsadikovi%2Fspark-netflow","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsadikovi%2Fspark-netflow","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsadikovi%2Fspark-netflow/lists"}