{"id":13844621,"url":"https://github.com/rapid7/dap","last_synced_at":"2025-10-25T23:40:43.024Z","repository":{"id":560221,"uuid":"14264205","full_name":"rapid7/dap","owner":"rapid7","description":"Data Analysis Pipeline","archived":false,"fork":false,"pushed_at":"2025-02-13T19:53:04.000Z","size":1602,"stargazers_count":159,"open_issues_count":13,"forks_count":38,"subscribers_count":33,"default_branch":"master","last_synced_at":"2025-05-10T09:38:18.469Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Ruby","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/rapid7.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2013-11-09T20:14:34.000Z","updated_at":"2025-02-13T19:53:07.000Z","dependencies_parsed_at":"2023-07-06T15:16:18.042Z","dependency_job_id":"85eb3c24-4fcc-4358-842f-230996e7d791","html_url":"https://github.com/rapid7/dap","commit_stats":{"total_commits":402,"total_committers":20,"mean_commits":20.1,"dds":0.5771144278606966,"last_synced_commit":"44050a6d2cf00677d56fa3a94df7943937ac9044"},"previous_names":[],"tags_count":57,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rapid7%2Fdap","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rapid7%2Fdap/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rapid7%2Fdap/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rapid7%2Fdap/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/rapid7","download_url":"https://codeload.github.com/rapid7/dap/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254319718,"owners_count":22051072,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-04T17:02:48.729Z","updated_at":"2025-10-07T17:57:14.526Z","avatar_url":"https://github.com/rapid7.png","language":"Ruby","funding_links":[],"categories":["Ruby (88)","Ruby"],"sub_categories":[],"readme":"# DAP: The Data Analysis Pipeline\n\n[![Gem Version](https://badge.fury.io/rb/dap.svg)](http://badge.fury.io/rb/dap)\n\nDAP was created to transform text-based data on the command-line, specializing in transforms that are annoying or difficult to do with existing tools.\n\nDAP reads data using an input plugin, transforms it through a series of filters, and prints it out again using an output plugin. Every record is treated as a document (aka: hash/dict) and filters are used to reduce, expand, and transform these documents as they pass through. Think of DAP as a mashup between sed, awk, grep, csvtool, and jq, with map/reduce capabilities.\n\nDAP was written to process terabyte-sized public scan datasets, such as those provided by https://scans.io/. Although DAP isn't particularly fast, it can be used across multiple cores (and machines) by splitting the input source and wrapping the execution with GNU Parallel.\n\n## Installation\n\n### Prerequisites\n\nDAP requires Ruby and is best suited for systems with a relatively current version with 2.6.x being the minimum requirement.\nIdeally, this will be managed with either\n[`rbenv`](https://github.com/rbenv/rbenv) or [`rvm`](https://rvm.io/) with the bundler gem also installed and up to date.\nUsing system managed/installed Rubies is possible but fraught with peril.\n\n#### Maxmind IP Location Databases\n\nIf you intend on using any of the `geo_ip*` or `geo_ip2*` filters, you must\ninstall the databases that provide the data for these filters.  If you do not\nintend on using these filters, you can skip this step.\n\n`dap` versions 1.4.x and later depend on [Maxmind's geoip2/geolite2\ndatabases](https://dev.maxmind.com/geoip/geoip2/geolite2/) to be able to append\ngeographic and related metadata to analyzed datasets.  In order to use this\nfunctionality you must put your copy of the relevant Maxmind databases in the\ncorrect location in `/var/lib/geoip2` or the `data` directory of your `dap`\ninstallation or override with an environment variable that specifies the full\npath to the database in question:\n\n* ASN: `GeoLite2-ASN.mmdb` (environment override: `GEOIP2_ASN_DATABASE_PATH`)\n* City: `GeoLite2-City.mmdb` (environment override: `GEOIP2_CITY_DATABASE_PATH`)\n* ISP: `GeoIP2-ISP.mmdb` (environment override: `GEOIP2_ISP_DATABASE_PATH`)\n\n*NOTE*: Prior to `dap` version 1.4.x there was a dependency on [Maxmind's geoip\ndatabase](http://dev.maxmind.com/geoip/legacy/downloadable/)\nto be able to append geographic metadata to analyzed datasets.  However, since\nthat time Maxmind has dropped support for these legacy databases.  If you\nintend to continue using this deprecated functionality, you must put your copy\nof the relevant Maxmind legacy databases in the correct location in\n`/var/lib/geoip` or the `data` directory of your `dap` installation or override\nwith an environment variable that specifies the full path to the database in question:\n\n* ASN: `GeoIPASNum.dat` (environment override in 1.4.x+: `GEOIP_ASN_DATABASE_PATH`)\n* City: `geoip_city.dat` (environment override in 1.4.x+: `GEOIP_CITY_DATABASE_PATH`)\n* Org: `geoip_org.dat` (environment override in 1.4.x+: `GEOIP_ORG_DATABASE_PATH`)\n\n### Ubuntu 16.04+\n\n```bash\nsudo apt-get install zlib1g-dev ruby ruby-dev gcc make ruby-bundler\ngem install dap\n```\n\n### OS X\n\n```bash\n# Install the GeoIP C library required by DAP\nbrew update\nbrew install geoip\n\ngem install dap\n```\n\n## Usage\n\nIn its simplest form, DAP takes input, applies zero or more filters which modify the input, and then outputs the result.  The input, filters and output are separated by plus signs (`+`).  As seen from `dap -h`:\n\n```shell\nUsage: dap  [input] + [filter] + [output]\n       --inputs\n       --outputs\n       --filters\n```\n\nTo see which input/output formats are supported and what filters are available, run `dap --inputs`,`dap --outputs` or `dap --filters`, respectively.\n\nThis example reads as input a single IP address from `STDIN` in line form, applies geo-ip transformations as a filter on that line, and then returns the output as JSON:\n\n```shell\n$   echo 8.8.8.8 | bin/dap + lines + geo_ip2_city line + json | jq .\n{\n  \"line\": \"8.8.8.8\",\n  \"line.geoip2.city.city.geoname_id\": \"0\",\n  \"line.geoip2.city.continent.code\": \"NA\",\n  \"line.geoip2.city.continent.geoname_id\": \"6255149\",\n  \"line.geoip2.city.country.geoname_id\": \"6252001\",\n  \"line.geoip2.city.country.iso_code\": \"US\",\n  \"line.geoip2.city.country.is_in_european_union\": \"false\",\n  \"line.geoip2.city.location.accuracy_radius\": \"1000\",\n  \"line.geoip2.city.location.latitude\": \"37.751\",\n  \"line.geoip2.city.location.longitude\": \"-97.822\",\n  \"line.geoip2.city.location.metro_code\": \"0\",\n  \"line.geoip2.city.location.time_zone\": \"America/Chicago\",\n  \"line.geoip2.city.postal.code\": \"\",\n  \"line.geoip2.city.registered_country.geoname_id\": \"6252001\",\n  \"line.geoip2.city.registered_country.iso_code\": \"US\",\n  \"line.geoip2.city.registered_country.is_in_european_union\": \"false\",\n  \"line.geoip2.city.represented_country.geoname_id\": \"0\",\n  \"line.geoip2.city.represented_country.iso_code\": \"\",\n  \"line.geoip2.city.represented_country.is_in_european_union\": \"false\",\n  \"line.geoip2.city.represented_country.type\": \"\",\n  \"line.geoip2.city.traits.is_anonymous_proxy\": \"false\",\n  \"line.geoip2.city.traits.is_satellite_provider\": \"false\",\n  \"line.geoip2.city.continent.name\": \"North America\",\n  \"line.geoip2.city.country.name\": \"United States\",\n  \"line.geoip2.city.registered_country.name\": \"United States\"\n}\n```\n\nThere are also several examples of how to use DAP along with sample datasets [here](samples).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frapid7%2Fdap","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Frapid7%2Fdap","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frapid7%2Fdap/lists"}