{"id":13775201,"url":"https://github.com/justinazoff/flow-indexer","last_synced_at":"2025-05-11T07:32:11.371Z","repository":{"id":57502243,"uuid":"51235313","full_name":"JustinAzoff/flow-indexer","owner":"JustinAzoff","description":"Flow-Indexer indexes flows found in chunked log files from bro,nfdump,syslog, or pcap files","archived":false,"fork":false,"pushed_at":"2024-05-09T17:29:11.000Z","size":220,"stargazers_count":44,"open_issues_count":4,"forks_count":6,"subscribers_count":5,"default_branch":"master","last_synced_at":"2024-08-03T17:11:18.031Z","etag":null,"topics":["bro","index","netflow","pcap","search-engine"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/JustinAzoff.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2016-02-07T04:27:20.000Z","updated_at":"2024-07-16T02:15:15.000Z","dependencies_parsed_at":"2024-06-20T02:28:22.794Z","dependency_job_id":null,"html_url":"https://github.com/JustinAzoff/flow-indexer","commit_stats":null,"previous_names":[],"tags_count":13,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JustinAzoff%2Fflow-indexer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JustinAzoff%2Fflow-indexer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JustinAzoff%2Fflow-indexer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JustinAzoff%2Fflow-indexer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/JustinAzoff","download_url":"https://codeload.github.com/JustinAzoff/flow-indexer/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":225027355,"owners_count":17409417,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bro","index","netflow","pcap","search-engine"],"created_at":"2024-08-03T17:01:35.217Z","updated_at":"2024-11-17T10:30:44.938Z","avatar_url":"https://github.com/JustinAzoff.png","language":"Go","funding_links":[],"categories":["\u003ca id=\"f13469c9891173804423be4403b2c4ff\"\u003e\u003c/a\u003epcap"],"sub_categories":["\u003ca id=\"eb49514924c3f4bf2acf6f3a4436af13\"\u003e\u003c/a\u003e未分类"],"readme":"Flow Indexer [![Build Status](https://travis-ci.org/JustinAzoff/flow-indexer.svg?branch=master)](https://travis-ci.org/JustinAzoff/flow-indexer)\n\n    flow-indexer indexes flows\n\n    Usage: \n      flow-indexer [command]\n\n    Available Commands: \n      compact     Compact the database\n      daemon      Start daemon\n      expandcidr  Expand a CIDR range from those seen in the database\n      index       Index flows\n      search      Search flows\n      help        Help about any command\n\n    Flags:\n          --dbpath=\"flows.db\": Database path\n      -h, --help[=false]: help for flow-indexer\n\n\n    Use \"flow-indexer [command] --help\" for more information about a command.\n\n# Quickstart\n\n## Install\n\n    $ export GOPATH=~/go\n    $ go get github.com/JustinAzoff/flow-indexer\n\n## Create configuration\n\n    $ cp ~/go/src/github.com/JustinAzoff/flow-indexer/example_config.json config.json\n    $ vi config.json # Adjust log paths and database paths.\n\nThe indexer configuration is as follows:\n\n* name - The name of the indexer. Keep this short and lowercase, as you will use it as an http query param.\n* backend - The backend log ip extractor to use. Choices: bro, bro\\_json, nfdump, syslog, pcap, and argus.\n* file\\_glob - The shell globbing pattern that should match all of your log files.\n* recent\\_file\\_glob - The strftime+shell globbing pattern that should match todays log files.\n* filename\\_to\\_database\\_regex - A regular expression applied to each filename used to extract information used to name the database.\n* database\\_root - Where databases will be written to.  Should be indexer specific.\n* datapath\\_path - The name of an individual database.  This can contain $variables set in filename\\_to\\_database\\_regex.\n\nThe deciding factor for how to partition the databases is how many unique ips\nyou see per day.  I suggest starting with monthly indexes.  If the indexing\nperformance takes a huge hit by the end of the month, switch to daily indexes.\n\n## Run initial index\n\nthe `indexall` command will expand file\\_glob and index any log file that matches.\n\n    $ ~/go/bin/flow-indexer indexall\n\n## Start Daemon\n\nOnce the initial index is complete, start the daemon.  Starting the daemon will\nexpand recent\\_file\\_glob and index any recently created log file that matches.\n\n    $ ~/go/bin/flow-indexer daemon\n\nIt will do this in a 60 second loop to keep itself up to date.\n\n## Query API\n\n    $ curl -s 'localhost:8080/search?i=conn\u0026q=1.2.3.0/24'\n    $ curl -s 'localhost:8080/dump?i=conn\u0026q=1.2.3.0/24'\n    $ curl -s 'localhost:8080/stats?i=conn\u0026q=1.2.3.0/24'\n\n## Service Configuration\nRunning flow-indexer as a service\n\n### systemd\n\nTo run flow-indexer as a service on a system using systemd, you can use the\n[provided flow-indexer.service file](./flow-indexer.service).\n\n### upstart\n\nIf you are planning to run flow-indexer as a service on a system that uses\nupstart, you may want to consider a conf file like the following in order\nto properly syslog stdout and stderr from flow-indexer, and to run as a\nnon-root user.\n```\n# flow-indexer - Flow Indexer\n#\n# flow-indexer is a service that indexes and allows retrieval of flows using bro logs\n\ndescription     \"Flow Indexer Daemon\"\n\nstart on runlevel [345]\nstop on runlevel [!345]\n\nsetuid flowindexer\nsetgid flowindexer\n\nexec /path/to/bin/flow-indexer daemon --config /path/to/flow-indexer/config.json 2\u003e\u00261 | logger -t flow-indexer\n```\n\n# Common Issues\nIn order to avoid `too many open files` errors, you may want to increase the\nnumber of open files you allow the user that flow-indexer runs as to have\naccess to.  This can be done by changing your `nofile` setting in\n`/etc/security/limits.conf` as shown below.\n```\nflowindexer soft nofile 65535\nflowindexer hard nofile 65535\n```\n\nLower level commands example\n============================\n\nNot really used anymore in practice, the daemon is the recommended way to use flow-indexer.\nBut these commands can be useful for testing and development.\n\nIndex flows\n-----------\n\n    ./flow-indexer --dbpath /tmp/f/flows.db index /tmp/f/conn*\n    2016/02/06 23:36:51 /tmp/f/conn.00:00:00-01:00:00.log.gz: Read 4260 lines in 24.392765ms\n    2016/02/06 23:36:51 /tmp/f/conn.00:00:00-01:00:00.log.gz: Wrote 281 unique ips in 2.215219ms\n    2016/02/06 23:36:51 /tmp/f/conn.01:00:00-02:00:00.log.gz: Read 4376 lines in 24.186168ms\n    2016/02/06 23:36:51 /tmp/f/conn.01:00:00-02:00:00.log.gz: Wrote 310 unique ips in 1.495277ms\n    [...]\n    2016/02/06 23:36:51 /tmp/f/conn.22:00:00-23:00:00.log.gz: Read 7799 lines in 18.350788ms\n    2016/02/06 23:36:51 /tmp/f/conn.22:00:00-23:00:00.log.gz: Wrote 775 unique ips in 5.155262ms\n    2016/02/06 23:36:51 /tmp/f/conn.23:00:00-00:00:00.log.gz: Read 5255 lines in 15.296847ms\n    2016/02/06 23:36:51 /tmp/f/conn.23:00:00-00:00:00.log.gz: Wrote 400 unique ips in 2.910344ms\n\nRe-Index flows\n--------------\n\n    ./flow-indexer --dbpath /tmp/f/flows.db index /tmp/f/conn*\n    2016/02/06 23:37:36 /tmp/f/conn.00:00:00-01:00:00.log.gz Already indexed\n    2016/02/06 23:37:36 /tmp/f/conn.01:00:00-02:00:00.log.gz Already indexed\n    2016/02/06 23:37:36 /tmp/f/conn.02:00:00-03:00:00.log.gz Already indexed\n    2016/02/06 23:37:36 /tmp/f/conn.03:00:00-04:00:00.log.gz Already indexed\n    [...]\n    2016/02/06 23:37:36 /tmp/f/conn.20:00:00-21:00:00.log.gz Already indexed\n    2016/02/06 23:37:36 /tmp/f/conn.21:00:00-22:00:00.log.gz Already indexed\n    2016/02/06 23:37:36 /tmp/f/conn.22:00:00-23:00:00.log.gz Already indexed\n    2016/02/06 23:37:36 /tmp/f/conn.23:00:00-00:00:00.log.gz Already indexed\n\nExpand CIDR Range\n-----------------\n\n    ./flow-indexer --dbpath /tmp/f/flows.db expandcidr 192.30.252.0/24\n    192.30.252.86\n    192.30.252.87\n    192.30.252.92\n    192.30.252.124\n    192.30.252.125\n    192.30.252.126\n    192.30.252.127\n    192.30.252.128\n    192.30.252.129\n    192.30.252.130\n    192.30.252.131\n    192.30.252.141\n\nSearch\n------\n\n    ./flow-indexer --dbpath /tmp/f/flows.db search 192.30.252.0/24\n    /tmp/f/conn.03:00:00-04:00:00.log.gz\n    /tmp/f/conn.04:00:00-05:00:00.log.gz\n    /tmp/f/conn.06:00:00-07:00:00.log.gz\n    /tmp/f/conn.14:00:00-15:00:00.log.gz\n    /tmp/f/conn.18:00:00-19:00:00.log.gz\n    /tmp/f/conn.20:00:00-21:00:00.log.gz\n    /tmp/f/conn.22:00:00-23:00:00.log.gz\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjustinazoff%2Fflow-indexer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjustinazoff%2Fflow-indexer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjustinazoff%2Fflow-indexer/lists"}