{"id":13540292,"url":"https://github.com/dirtbags/pcapdb","last_synced_at":"2026-01-27T18:13:49.552Z","repository":{"id":86015495,"uuid":"73320142","full_name":"dirtbags/pcapdb","owner":"dirtbags","description":"A Distributed, Search-Optimized Full Packet Capture System","archived":false,"fork":false,"pushed_at":"2023-04-17T19:33:39.000Z","size":11289,"stargazers_count":241,"open_issues_count":25,"forks_count":43,"subscribers_count":34,"default_branch":"master","last_synced_at":"2025-03-28T22:45:13.023Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/dirtbags.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2016-11-09T20:49:33.000Z","updated_at":"2025-02-03T11:13:46.000Z","dependencies_parsed_at":"2024-01-07T22:49:40.139Z","dependency_job_id":"2b23a305-8f0e-46b4-8771-7e43dd499089","html_url":"https://github.com/dirtbags/pcapdb","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dirtbags%2Fpcapdb","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dirtbags%2Fpcapdb/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dirtbags%2Fpcapdb/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dirtbags%2Fpcapdb/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/dirtbags","download_url":"https://codeload.github.com/dirtbags/pcapdb/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246774309,"owners_count":20831510,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-01T09:01:45.321Z","updated_at":"2026-01-27T18:13:49.474Z","avatar_url":"https://github.com/dirtbags.png","language":"JavaScript","funding_links":[],"categories":["\u003ca id=\"79499aeece9a2a9f64af6f61ee18cbea\"\u003e\u003c/a\u003e浏览嗅探\u0026\u0026流量拦截\u0026\u0026流量分析\u0026\u0026中间人","\u003ca id=\"7bf0f5839fb2827fdc1b93ae6ac7f53d\"\u003e\u003c/a\u003e工具"],"sub_categories":["\u003ca id=\"dde87061175108fc66b00ef665b1e7d0\"\u003e\u003c/a\u003epcap数据包","\u003ca id=\"b346105580b0240d693020ce8719ebca\"\u003e\u003c/a\u003e未分类"],"readme":"# Security Notice\n\nThis project has not been updated since 2019. There are multiple security vulnerabilities in dependency packages.\n\nWithout an investment in upgrading these dependencies,\nthis project should be considered\n**for archival use only**.\n\n\n\u003cimg src=\"https://cloud.githubusercontent.com/assets/22897558/22356169/8f6e0b16-e3ec-11e6-8695-2273424c2b06.png\" width=\"200\" /\u003e\n\n# Overview\n\nPcapDB is a distributed, search-optimized open source packet capture system. It was designed to\nreplace expensive, commercial appliances with off-the-shelf hardware and a free, easy to manage\nsoftware system. Captured packets are reorganized during capture by flow (an indefinite length\nsequence of packets with the same src/dst ips/ports and transport proto), indexed by flow, and\nsearched (again) by flow. The indexes for the captured packets are relatively tiny (typically less\nthan 1% the size of the captured data).\n\nFor hardware requirements, see [HARDWARE.md](HARDWARE.md).\n\n[Also see our fact sheet](FactSheet_PcapDBv2.pdf)\n\n## DESTDIR\nMany things in this file refer to DESTDIR as a pathname prefix. The default, and that used by future pcapdb packages, is `/var/pcapdb`.\n\n## Architectural Overview\nA PcapDB installation consists of a Search Head and one or more Capture Nodes. The Search Head can\nalso be a Capture Node, or it can be a VM somewhere else. Wherever it is, the Search Head must be\naccessible by the Capture Nodes, but there's no need for the Capture Nodes to be visible to the\nSearch Head.\n\n# 1. Requirements\nPcapDB is designed to work on Linux servers only. It was developed on both Redhat Enterprise and\nDebian systems, but its primary testbed has so far been Redhat based. While it has been verified to\nwork (with packages from non-default repositories) on RHEL 6, a more bleeding edge system (like\nRHEL/Centos 7, or the latest Debian/Ubuntu LTS) will greatly simplify the process of gathering dependencies.\n\n[sys_requirements.md](sys_requirements.md) contains a list of the packages required to run and build\npcapdb. They are easiest to install on modern Debian based machines.\n\nrequirements.txt contains python/pip requirements. They will automatically be installed via 'make install'.\n\n# 2. Installing\nTo build and install everything in /var/pcapdb/, run one of:\n```\nmake install-search-head\nmake install-capture-node\nmake install-monolithic\n```\n - Like with most Makefiles, you can set the DESTDIR environment variable to specify where to\n   install the system. `make install-search-head DESTDIR=/var/mypcaplocation`\n - This includes installing in place: `make install-capture-node DESTDIR=$(pwd)`. In this case, PcapDB\n   won't install system scripts for various needed components. You will have to run it manually, see\n   below.\n - If you're behind a proxy, you'll need to specify a proxy connection string using PROXY=host:port\n\tas part of the make command.\n - There's a bug in some 1.10.* versions of virtualenv that cause the install to fail. Specify the python3 virtualenv executable with using VIRTUALENV=\u003cvirtualenv path\u003e\n\nTo make your life easier, however, you should work make sure the indexing code builds cleanly by running 'make' in the 'indexer/' directory.\n\nPostgresql may install in a strange location, as noted in the 'indexer/README'. This can cause build\nfailures in certain pip installed packages. Add `PATH=$PATH:\u003cpgsql_bin_path\u003e` to the end of your\n'make install' command to fix this. For me, it is: `make install PATH=$PATH:/usr/pgsql-9.4/bin`.\n\n# 3. Setup\nAfter running 'make install', there are a few more steps to perform.\n\n## 3-0: Setup hugepages (optional)\nPcapdb uses 2MB hugepages to manage memory more efficiently. If a capture node has hugepages available,\nthey will be automatically consumed by the capture process. \n\nFirst, determine how much memory you want to devote to capture. I'd recommend about 70% of available\nsystem memory, which should be a minimum of 16G (64G or more are recommended). Then simply divide\nthat amount by 2M to get the number of hugepages. \n\nTo enable hugepages, add 'hugepages=\u003cnumber of pages\u003e hugepagesz=2M' to your /boot/grub/grub.conf\nand reboot. You may also want to add it to /etc/default/grub in the GRUB_CMDLINE_LINUX variable,\ndepending on your OS flavor (debian). \n\n## 3-1: Post-Install script\nThe core/bin/post-install.sh script will handle the vast majority of the system setup for you.\n - Only run once. It can cause issues if you run it multiple times. ~~It does so idempotently, so it can be run multiple times without breaking anything.~~ If you run it again, you may have to re-install completely. Moving the old PcapDB install files to another directory can cause problems as well.\n - Run without arguments to get the usage information.\n  - Basically, you want to give it arguments based on whether you're setting up a search head (-s),\n    a capture node (-c), or monolithic install (-c -s).\n  - You'll also have to give it the search head's IP.\n```\n/var/pcapdb/core/bin/post-install.sh [-c] [-s] \u003csearch_head_ip\u003e    \n```\n\nThis will set up the databases and rabbitmq.\n\n## 3-2 DESTDIR/etc/pcapdb.cfg\nThis is the main Pcapdb config file. You must set certain values before PcapDB will run at all.\nThere are a few things you need to set in here manually:\n - __(On capture nodes) The search head db password__\n - __(On capture nodes) The rabbitmq password__\n   - Both of the above should be in the search head's pcapdb.cfg file.\n - __(On search head) The local mailserver.__\n   - If you don't have one, I'd start with installing Postfix. It even has selectable install\n     settings that will configure it as a local mailserver for you.\n\n## 3-3 Add an admin user (Search Head Only)\nYou'll need to create an admin user.\n```\nsudo su - capture\n./bin/python core/manage.py add_user \u003cusername\u003e \u003cfirst_name\u003e \u003clast_name\u003e \u003cemail\u003e\n```\n - Usernames must be at least four characters long.\n - This will email you a link to use to set that user's password.\n  - (This is why email had to be set up).\n  - root@localhost is a reasonable email address, if you need it.\n  - *Note that manage.py also has a __createsuperuser__ command, which shouldn't be used.*\n\n## 3-4 Add the capture nodes to the postgres pg\\_hba.conf on the search head. \n This is needed if running a separate search head. See below.\n\n## 3-5 Set up a site. \n - You should be able to login with your admin account. \n - Click 'Admin', 'Capture Sites'. \n - Add a new capture site. The group name can be the same as the site name; the admin group should be different. \n - This adds a grouping of capture nodes to work with.\n\n## 3-6 Set up a capture node.\n - Click 'Admin', 'Capture Nodes'. \n - Select your site, and add a new capture by hostname. \n - The you have to do this even in monolithic mode. \n - If this fails, check logs/celery.log on the capture node. \n - The capture node must already be able to connect to rabbitmq/celery on the search head for this to work.\n\n## 3-7 Add capture node permissions\n - To be able to configure the capture node, you must set permissions for it. \n - Click 'Admin', 'Users'. \n - Select your user, and 'Add to Group'. Select the admin group for the site you added. \n - You can now edit the disks and configure capture. \n\n## 3-8 Configure disks.\nGo into the Disks view - Click 'Admin', 'Capture Nodes', and then the 'Disks' button on the node you\nwant to configure.\n\n### Index disks\n - You'll need one or two equally sized disks dedicated to indexing. Select the disks from the available 'Devices'\n   table, and click 'Create Index RAID'. It will take a few minutes.\n - If you choose two disks, it will create a RAID 1 of the two disks.\n\n### Capture disks\n You'll need to set up some groups of disks for capture. \n\n - Select some number of equally sized disks from the 'Devices' list, and click 'Create RAID5'. This\n   will create a new md device from those disks. (If your disk is an external RAID, you can skip\n   this step.)\n - Select your RAID, and click 'Initialize as Capture Disk'. This will format the RAID and add it to\n   the database. It should appear in the 'Capture Disks' table at the top of the page. \n - Repeat these steps to include as many capture disks as you like. PcapDB balances across them\n   according to their size, so they don't all have to end up as the same size. \n - You can also add dynamically re-assigned spares that will be used by any of your RAID's as needed, by\n   clicking a disk and selecting 'Add Spare'. \n - For each of the Capture Disks you added, select and enable them in the 'Capture Disks' table.\n\nDebugging note: Errors from this all go into the logs/celery.log on the capture node. \n\n## 3-9 Configure Capture\nGo into the Capture view - Click 'Admin', 'Capture Nodes', and then the 'Capture' button on the node you\nwant to configure.\n\n - Select the interface you'd like to enable for capture, and click the red circle. Capture will be\n   enabled on this device on the next capture restart. \n - On capture settings, you can enable PFRing ZC mode, if you have a license and a compatible\n   network interface. All used interfaces must be compatible. \n - The multi-queue mode has been tested, but not well. The queue slider is only used when in\n   PFring-ZC mode.\n - There's a bug in the capture settings; you must put a number (including zero) in the local memory\n   box. \n\nWhen you're ready, click 'Start'. \n\nDebugging note: Capture runner errors go into logs/django.log on the capture node. For some reason you may get error messages about the capture runner not trying to do anything. As root, on the capture node, run supervisorctl restart capture_runner. Logs for the actual capture process are supposed to be in /var/pcapdb/log/capture.log, but sometimes show up in /var/log/messages if syslog didn't get configured correctly.\n\n\n\n### Things that can, and have, gone wrong\n - If your host doesn't have a host name in DNS, you can set an IP in the 'search\\_head\\_host' variable\n   in the pcapdb.cfg file.\n\n## 3-5 pfring-zc drivers\nOne more thing. You should install the drivers specific to your capture card for pfring-zc. The\npackages from NTOP actually build the drivers for your kernel on the fly when installed, though\nyou may have to reinstall that package whenever you do a kernel update.  Building and installing\nfrom source is also fairly straightforward.\n\n# Using PcapDB\nNow that the system is installed and running, you have to set up a capture site, capture node, and some users.\n\n## Create a Capture Site\nEvery capture node belongs to a capture site.\n - Each capture site has its own group, only users in that group can search the site.\n - They also have an admin group. Users must be a member of this to admin (setup disk on, or start capture on) capture nodes in that site.\n - These can be LDAP groups.\n\n## Create a Capture Node\nAdd a capture node to your site.\n - If the search head is also a capture node, it will have to be added to.\n - If the buttons for disk and capture configuration aren't active, your user isn't an admin for the relevant site.\n\n## Setup disks\nPcapDB needs three types of disk:\n - An OS disk, which is barely used.\n - An index disk or disks. It's generally recommended to have two identical disks for this to set up a RAID 1.\n - A bunch of disk for capture storage.\n\n You've obviously already got an OS at this point. The other two can be any sort of hard disk, SSD, partition, or RAID. As long as the system thinks it's a block device, you should be fine.\n\n#### Note\n*The system is pretty dumb about what disks it considers to be block devices. Any /dev/sd or /dev/md device works, for now*\n\n### Add an Index Disk\nIn the disk management interface, select one or two identical disks to act as the index disk. If more than one disk is selected, they will automatically be grouped together in a RAID 1 configuration.\n - Click, 'create index disk' and the system will RAID, format, and name the disks appropriately.\n\n### Add a Capture Disk\nIn the disk management interface, you can build RAID 5 arrays, and then assign those (or individual disks) as capture disks.\n - While you can add all your disks individually as capture disks, it's recommended to RAID them for a bit of data safety.  RAID's of 9 disks are fairly reasonable.\n - Like with the index disk, select groups of disks, and click 'Create RAID 5'. Once that's done, you can add the\n   resulting disk as a capture disk.\n - Or you can add individual disks (or external RAIDs) as capture disks.\n - Once a capture disk is added, it must be activated above before it can be captured to.\n - You can also set disks as __spares__, which are shared across all RAIDS created by PcapDB.\n - If RAIDS are ever degraded, they should be put in REPAIR mode automatically. They'll still be available to search, but won't be written to until they're fixed.\n\n## Set a capture interface, and go.\nIn the capture interface, enable the interface or interfaces of your choice.\n - Each will get a separate thread (which will in turn be dedicated to its own processor. So you shouldn't try to capture on more interfaces than half your CPU's).\n - PFring mode is far less likely to drop packets than libpcap mode.\n - PFringZC mode is far less likely to drop packets than PFring mode, but requires a license from NTOP.\n\n# Details on the various subsystems\nPcapDB uses quite a few off-the-shelf open source systems, and it's useful to understand how those\npieces fit into the larger system. What follows is a detailed description of those systems, and how\nto set them up manually.\n\n## RabbitMQ\nRabbitMQ is a fast and efficient messaging system used to communicate simple messages between a\ndistributed network of hosts. As with Celery, RabbitMQ is really meant for distributing messages to\nthe 'first available' worker, but in PcapDB all of our messages are to a specific worker. As such,\nthe PcapDB rabbitMQ instance automatically creates a specific message queue for each Capture Node,\nas well as a queue for the Search Head. The command 'rabbitmqctl' gives visibility into the\ncurrently active queues. For further debugging/introspection, the rabbitmq admin plugin provides a\nweb interface that can be quite useful.\n\nRabbitMQ server need only be configured on the search head.\n```\n# Get rid of the guest user\nrabbitmqctl delete_user guest\n# Create the pcapdb user with a password\nrabbitmqctl add_user pcapdb \u003ca strong password\u003e\n# Allow the pcapdb user to set/view all queues\nrabbitmqctl set_permissions -p / pcapdb '.*' '.*' '.*'\n```\n\n## Database Setup\nThe setup varies significantly between the search head and capture nodes.\n\n### Add the 'capture' Role\nOn all pcapdb servers, add a 'capture' role with login privileges and a password. As the postgres user:\n```\nsudo su -postgres\ncreateuser capture -l -P\n```\n\nThe 'db\\_pass' variable in the pcapdb.cfg file should be set to the Search Head's db password on all\npcapdb hosts in the network.\n\n### On the Search Head\nCreate a database named \"pcapdb\":\n```\ncreatedb -O capture pcapdb\n```\n\nEdit the Search Head's \"pg\\_hba.conf\" (location varies) file to allow connections to Search Head DB from localhost. Also add a line allowing each capture node.\n```\nhost    pcapdb          capture         127.0.0.1/32            md5\nhost    pcapdb          capture         \u003ccapture node ip\u003e       md5\n```\n\nEdit the Search Head's postgresql.conf file so that it listens on its own IP:\n```\nlisten_addresses = 'localhost,\u003csearch head ip\u003e'\n```\n\n### On the Capture Nodes\nCreate a database named 'capture\\_node' on each capture node host:\n```\ncreatedb -O capture capture_node\n```\n\nSince the capture nodes connect via peer (unix socket) to their own database, no additional setup\nshould be needed.\n\n\n### Restart the postgresql server.\n```\n# On most systems...\nservice postgresql restart\n```\n\n### Install the Database Tables\nAfter restarting the postgres service, we'll need to install the database tables on each\nPcapDB host. From the PcapDB installation directory (typically /var/pcapdb):\n```\nsudo su - root\ncd /var/pcapdb\n./bin/python core/manage.py makemigrations stats_api login_api core task_api search_head_api\nlogin_gui search_head_gui capture_node_api\n\n# On the search head\n./bin/python core/manage.py migrate\n\n# On the capture nodes\n./bin/python core/manage.py migrate --database=capture_node\n```\n\n## Web server setup\nThe Makefile will generate a self-signed cert for your server for you if an installed one doesn't\nalready exist at '/etc/ssl/\u003cHOSTNAME\u003e.pem'. You should probably change that to something that isn't\nself-signed.\n\n## Firewall Notes\n - The Capture Nodes don't open any incoming ports for PcapDB, all communication is out to the Search Head.\n - Using IPtables or other system firewalls on the Capture Nodes is discouraged. Instead\n   put them on a normally inaccessible network. They shouldn't generally have any ports open other than ssh.\n - The Search Head needs to be accessible by the Capture Nodes on ports 443, 5432 (postgres), and\n   25672 (rabbitmq).\n   - It's ok to us IP tables on the Search Head (and will eventually be automatic)\n\n## Running the system\nIf you installed anywhere except 'in place', the system should attempt to run itself via\nsupervisord. __You'll have to restart some processes, as supervisord will have given up on them.__\n - The `supervisorctl` command can give you the status of the various components of the system. Capture has to be started manually from within the interface, so you shouldn't expect it to be running initially.\n - The `capture_runner` process, however, should be running. From within supervisorctl,\n\n - The `core/runserver` and `core/runcelery` scripts will be helpful when not running the system in\n   production.\n - Similarly, to run capture outside of production, use the capture_runner.py script:\n   `DESTDIR/bin/python DESTDIR/core/bin/capture_runner.py`\n   - DESTDIR/log/django.log will tell you the exact command used to start capture, if for some reason it's failing to start.\n   - DESTDIR/log/capture.log will usually give you some idea why capture is failing to run. If this file doesn't exist, either capture has never successfully ran at all, or rsyslog isn't forwarding the logs to the right place.\n\n\n### PostgreSQL\nWhile PcapDB is a database of packets, it uses postgres to take care of more mundane database tasks.\nThe Search Head has a unique database that houses information on users, capture nodes, celery\nresponse data, and aggregate statistics for the entire network of PcapDB capture nodes. All PcapDB\nhosts must be able to connect to the Search Head database.\n\nEach Capture Node also has a database that keeps track of the available disk chunks and indexes on\nthat node. This database is only accessible to the capture node itself.\n\nThis has to be set up manually. See above for more information.\n\n### Celery\nCelery is a system for distributing and scheduling tasks across a network of workers. PcapDB manages\nall of its communications with the Capture Nodes through Celery tasks, from initiating searches to\nmanaging disk arrays. The tasks are assigned and picked up by the appropriate host via RabbitMQ\nmessaging queues, and the responses are saved to the search head via the search head's database.\nCelery runs on both the Search Head and all Capture Nodes, though each host subscribes to different\ntask queues (see RabbitMQ below).\n\nCelery is configured automatically on system install, and the process is managed via supervisord.\n\n\n### uWSGI and Nginx\nThe web interface for PcapDB is built in the Python Django system, which is served via a unix\nsocket using uWSGI and persistant Python instances. Nginx handles all of the standard HTTP/HTTPS\nportions of the web service, and passes Django requests to uWSGI via its socket. (This is a pretty\nstandard way of doing things).\n\n - uWSGI and Nginx are automatically configured on install.\n - Nginx is managed as a standard system service.\n - uWSGI is managed via supervisord\n\n#### Certificates\nThe standard configuration for PcapDB and Nginx expects ssl certificates and a private key installed\nat:\n```\n/etc/ssl/\u003cHOSTNAME\u003e.pem\n/etc/ssl/\u003cHOSTNAME\u003e.prv\n```\n\nIf these don't already exist when running make install, self-signed certs will be created\nautomatically (with your input).\n\n\n# A Few Other Tasks\n\n## Install static files\nYou will need to install the static files for pcapdb.\n\n```\nsudo su - capture\n./bin/python core/manage.py collectstatic\n```\n# Making sure everything is working.\nAfter installing and setting up the database, there are a few things you can check to make sure\neverything is working.\n\n 1. Restart supervisord to reset the uwsgi and celery processes to pick up the new database configs.\n\n 2. Go to your webserver's root directory https://\u003cmyserver\u003e/, and you should get to the pcapdb\n login page.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdirtbags%2Fpcapdb","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdirtbags%2Fpcapdb","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdirtbags%2Fpcapdb/lists"}