{"id":20038558,"url":"https://github.com/yahoo/panoptes_docker","last_synced_at":"2025-10-24T21:06:07.357Z","repository":{"id":66000722,"uuid":"172540234","full_name":"yahoo/panoptes_docker","owner":"yahoo","description":"Containerized version of Panoptes for testing and experimentation.","archived":false,"fork":false,"pushed_at":"2023-05-01T16:45:11.000Z","size":531,"stargazers_count":10,"open_issues_count":1,"forks_count":10,"subscribers_count":9,"default_branch":"master","last_synced_at":"2025-04-08T18:51:38.778Z","etag":null,"topics":["big-data","container","docker","networking","telemetry"],"latest_commit_sha":null,"homepage":null,"language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/yahoo.png","metadata":{"files":{"readme":"Readme.md","changelog":null,"contributing":"Contributing.md","funding":null,"license":"LICENSE","code_of_conduct":"Code-of-Conduct.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-02-25T16:16:37.000Z","updated_at":"2023-05-12T06:05:36.000Z","dependencies_parsed_at":"2023-05-22T00:00:21.924Z","dependency_job_id":null,"html_url":"https://github.com/yahoo/panoptes_docker","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yahoo%2Fpanoptes_docker","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yahoo%2Fpanoptes_docker/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yahoo%2Fpanoptes_docker/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yahoo%2Fpanoptes_docker/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/yahoo","download_url":"https://codeload.github.com/yahoo/panoptes_docker/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252451828,"owners_count":21749992,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["big-data","container","docker","networking","telemetry"],"created_at":"2024-11-13T10:29:59.171Z","updated_at":"2025-10-24T21:06:02.335Z","avatar_url":"https://github.com/yahoo.png","language":"Shell","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Panoptes in Docker\n\n\u003e A Docker container for testing and observing Panoptes Functionality.\n\nPanoptes is a Python based network telemetry ecosystem that implements discovery, enrichment and polling. To learn more \nabout Panoptes, check out the project's [GitHub repository](https://github.com/yahoo/panoptes/). This repository \nprovides a workbench version of Panoptes inside a Docker container that is intended to be used for testing concepts and \nobserving code operations.\n\nThis does **not** follow best practices for deployment and shouldn't be used for production deployment. It's a very \nquick and dirty method to get a Panoptes instance up and running in a minimal amount of time with plenty of redundancy \nin place to allow enterprise scaling. While this isn't an efficient deployment, it does provide an accurate and \noperational build of the larger structure.\n\n## Quick Reference\nIf you're impatient, and simply want to get this up and running, check out the [quickstart guide](docker_quickstart.md).\n\n**Where to Get Help**\n\n[![Slack](https://img.shields.io/badge/slack-panoptescommunity-blue.svg?logo=slack)](https://panoptescommunity.slack.com/open)\n\n**Where to file Issues**\n\n[Issues related to Panoptes_docker](https://github.com/yahoo/panoptes_docker/issues)\n\n[Issues with Panoptes](https://github.com/yahoo/panoptes/issues)\n\n**Image Updates**\n\n[panoptes_docker PRs](https://github.com/yahoo/panoptes_docker/pulls)\n\n**Source of this Description**\n\n[The panoptes_docker Readme](https://github.com/yahoo/panoptes_docker/blob/master/Readme.md)\n\n**Maintained by**\n\nJames Diss: rexfury@verizonmedia.com\n\n**Supported Docker Versions**\n\nTested on [19.03.5](https://github.com/docker/docker-ce/releases/tag/v19.03.5)\n\n## Table of Contents\n\n- [Background](#background)\n- [Install](#install-and-setup)\n- [Usage](#usage)\n- [Configuration](#configuration)\n- [Contribute](#contribute)\n- [Additional Resources](#additional-resources)\n- [License](#license)\n\n\n## Background\n\nPanoptes is typically run in a distributed system, but it can be difficult to set up a working hive of the various \nmoving parts. Panoptes can be horizontally scaled for metric collection by adding more capacity to run collection, this \ndocker image simply cuts the total number of nodes down to *1*. This version also runs from a json file, but we use a \ncommand database internally and are working to extend this to CSV files.\n\nThe following image provides a visual representation of the services running inside this container.\n\n![Panoptes Block diagram](resources/panoptes_docker_block.svg?sanitize=true)\n\nThis container is based on Ubuntu:18.04, and includes Python3.6, open-jdk 8, redis, zookeeper, and influxdb. There are \nalso a few tools that are included for testing purposes, but are not necessary for operation of Panoptes:\n\n- Netcat - Used to check functionality of the services\n- tar - Used to view log archives.\n- snmpd - Provides an easy target to poll\n- nano - Makes it easier to open and modify configuration files and scripts.\n- daemontools - Handles output and schedules the various subcomponents of Panoptes. Panoptes typically relies on its \ngranular logging system to redirect to logging servers, but daemontools solves this requirement here.\n- Grafana - Included as a visualization tool. See [the section on Grafana](#visualize-with-grafana) for details.\n\nThe container also installs a specific version of Kafka, because of a hard dependency.  This is accomplished by \ndownloading and building the proper version from [the ASF archives](https://archive.apache.org/dist/kafka/). \n\nThe configuration files for the services included in this image are available to browse under the `resources/` directory \nstructure. Of particular interest will be the `resources/daemontools/panoptes_*.run` files, as these are the calls to \nthe various Panoptes components.\n\n## Install and Setup\nFirst, you will need to have docker [installed on your machine](https://docs.docker.com/install/).\n\nTo build the image, first clone the [source code](https://github.com/yahoo/panoptes_docker) and run `docker build` \nfrom inside the directory\n\n```\ngit clone https://github.com/yahoo/panoptes_docker.git \u0026\u0026 cd panoptes_docker\ndocker build . -t panoptes_docker\n```\n\n### Usage\n\nThis container requires quite a lot of processing power since it contains a distributed system that has been packed into \na small container. Because of this, you should consider limiting the resources available to the container.\n\nThe following command will run panoptes_docker inside a container:\n\n```\ndocker run -d \\\n    --sysctl net.core.somaxconn=511 \\\n    --name=\"panoptes_docker\" \\\n    --shm-size=2G \\\n    -p 127.0.0.1:8080:3000/tcp \\\n    panoptes_docker\n```\n\nThere is a five minute delay until the first metrics will show up.\n\n### Visualize with Grafana\n\nAfter launching the panoptes_docker container, you should be able to point a browser at http://127.0.0.1:8080 and see \nthe Grafana instance.\n\nUse the auth pair `admin/admin` to get in, and don't worry about a new password, it won't last longer than the \ncontainer.  Just skip it.\n\nThe datasource and dashboard should be available immediately, however, metrics gathering is delayed by 5 minutes. Your \ndashboard should look a little like this:\n\n![Localhost default dashboard](resources/grafana_default_board.png)\n\nNote that any changes you make to the dashboard won't survive the death of the container. If you want to learn more \nabout Grafana, [check out the docs on getting started](http://docs.grafana.org/guides/getting_started/).\n\n### Examine the Image\n\nUse `docker exec` to access the container for examination. \n\n```\ndocker exec -it panoptes_docker bash\n```\nIf you downloaded the prebuilt image, use `panoptes/panoptes_docker` in this example. This will drop you into the \ncontainer at `/home/panoptes`, and there will be some scripts available that expose the different services running in \nthe container and split the various log files into meaningful groups.\n\n![Scripts Ahoy](resources/home_panoptes_ls.png)\n\nAll of these scripts are copied into the docker container from `resources/misc`. If you're going to interact with the \npython scripts in the container, you need to operate in the proper a virtual environment. To do so, run the following \nafter you login to the container:\n\n```\nsource /home/panoptes_v/bin/activate\n```\n\n`conf` holds the configuration scripts for Panoptes. `logs` holds, well, logs and `run` contains lock files.\n\n#### Useful Scripts\n\n`show-services` will dump the PID and the uptime of each component.\n\n![show-services.sh](resources/panoptes_show_services.png)\n\n`show-influxdb.sh` first dumps the data that we're interested in, then tails the logs to show the following calls.  \nThis is generally the first step in troubleshooting because InfluxDB holds the measurement data.  Note: there is a 5 \nminute delay in the collection of stats, and the output can get a little large. To help with this, run:\n\n```\n/usr/bin/influx -database 'Panoptes' -format=csv -execute 'SHOW MEASUREMENTS' | more\n```\n\n![show-influxdb.sh](resources/panoptes_show_influxdb.png)\n\n`show-logs.sh` will tail all the relevant logs, but is of limited utility unless you really need a scrolling list of \nlog updates. Then again, perhaps you run a TV crime lab.\n\n`show-panoptes.sh`, `show-panoptes-agents.sh`, `show-panoptes-discovery.sh`, `show-panoptes-polling.sh` and \n`show-panoptes-schedulers.sh` show different aspects of the Panoptes system in operation by tailing the relevant logs.\n\nFinally, `show-snmpwalk.sh` effectively queries *localhost* for it's SNMP OIDs.  This is for checking, but also provides \na glimpse into the world of OIDs.\n\n\nOnce you're done, stop and remove the container before trying to run another.\n\n```\ndocker stop panoptes_docker \u0026\u0026 docker rm panoptes_docker\n```\n\n## Configuration\n\nPanoptes is run as a python module, so the relevant code is under site-packages in the \nvenv (`/home/panoptes_v/lib/site-packages`).  You can `activate` this environment from within the docker container.\n\n### Runtime\n\nYou can override the default [localhost.json](resources/panoptes/localhost.json) by supplying one externally and adding \na flag during the container runtime. The `-v` join effectively overlays the *default* localhost.json built into the \ncontainer.  This example uses a localhost.json at `/data/servers/panoptes/conf` on the host.  \nUse ```-v \u003csource_location\u003e:/home/panoptes/conf/localhost.json``` as a template.\n\nBoth `-e` variables are optional and default during build time to the values shown.\n\n`SNMP_SITE` defaults to `local`, and is one pair of the SNMP Secrets.\n\n`SNMP_COMM_STRING` defaults to `public`, but would be the 'community string' for the SNMP_SITE.\n\n```\ndocker run -d \\\n    --sysctl net.core.somaxconn=511 \\\n    --name=\"panoptes_docker\" \\\n    --shm-size=2G \\\n    -e SNMP_SITE=\"local\" \\\n    -e SNMP_COMM_STRING=\"public\" \\\n    -v /data/servers/panoptes/conf/localhost.json:/home/panoptes/conf/localhost.json \\\n    -p 127.0.0.1:8080:3000/tcp \\\n    panoptes_docker\n```\n\n### Plugins\n\nDuring the build of the container, the plugins from each subdirectory of `/resource/panoptes/plugins` are copied into \nthe container, so this is a good entry point into the structure.\n\nSee the [Panoptes Plugin](https://github.com/yahoo/panoptes/blob/master/docs/Concepts.md#plugins) page for details.\nYou'll also find instructions on building plugins at the [Plugin Guide](https://getpanoptes.io/docs/panoptes-reference/plugin-guide)\n\n## Additional Resources\n\n- [Panoptes Web Site](https://getpanoptes.io)\n- [Panoptes Source Code](https://github.com/yahoo/panoptes)\n- [Panoptes Documentation](https://github.com/yahoo/panoptes_documentation)\n- [Panoptes Concepts](https://github.com/yahoo/panoptes/blob/master/docs/Concepts.md)\n- [Panoptes Plugin Architecture](https://github.com/yahoo/panoptes/blob/master/docs/Concepts.md#plugins)\n- [Contributing to Panoptes](https://github.com/yahoo/panoptes/blob/master/docs/Contributing.md)\n- [Panoptes Configuration Samples](https://github.com/yahoo/panoptes/tree/master/examples/conf)\n- [Panoptes Plugin Examples](https://github.com/yahoo/panoptes/tree/master/examples/plugins)\n- [Open Sourcing Panoptes](https://yahoodevelopers.tumblr.com/post/178699833943/open-sourcing-panoptes-oaths-distributed-network)\n\n## Contribute\n\nPlease refer to [the contributing.md file](Contributing.md) for information about how to get involved. We welcome issues, \nquestions, and pull requests. Pull Requests are welcome.\n\n## License\n\nThis project is licensed under the terms of the Apache 2.0 open source license. Please refer to [LICENSE](LICENSE) for the full terms.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fyahoo%2Fpanoptes_docker","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fyahoo%2Fpanoptes_docker","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fyahoo%2Fpanoptes_docker/lists"}