{"id":20358870,"url":"https://github.com/fifemon/probes","last_synced_at":"2025-04-12T03:22:46.151Z","repository":{"id":145957992,"uuid":"54030051","full_name":"fifemon/probes","owner":"fifemon","description":"Scripts and configuration for loading information into Fifemon","archived":false,"fork":false,"pushed_at":"2019-01-25T20:38:44.000Z","size":26,"stargazers_count":6,"open_issues_count":2,"forks_count":7,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-03-25T22:51:29.572Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/fifemon.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2016-03-16T12:29:58.000Z","updated_at":"2019-05-23T15:31:15.000Z","dependencies_parsed_at":null,"dependency_job_id":"d3165503-2b19-4608-b0f2-fbe17d87c659","html_url":"https://github.com/fifemon/probes","commit_stats":null,"previous_names":[],"tags_count":4,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fifemon%2Fprobes","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fifemon%2Fprobes/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fifemon%2Fprobes/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fifemon%2Fprobes/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/fifemon","download_url":"https://codeload.github.com/fifemon/probes/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248510645,"owners_count":21116242,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-14T23:29:15.219Z","updated_at":"2025-04-12T03:22:46.137Z","avatar_url":"https://github.com/fifemon.png","language":"Python","readme":"**UPDATED CONDOR PROBE HAS BEEN MOVED TO [HEPiX-batchmonitoring/fifemon-condor-probe](https://github.com/HEPiX-batchmonitoring/fifemon-condor-probe)**\n\nFifemon\n=======\n\nCollect HTCondor statistics and report into time-series database. All modules \nsupport Graphite, and there is some support for InfluxDB.\n\nAdditionally report select job and slot Classads into Elasticsearch via Logstash.\n\nNote: this is a fork of the scripts used for monitoring the HTCondor pools\nat Fermilab, and while generally intended to be \"generic\" for any pool still \nmay require some tweaking to work well for your pool.\n\nCopyright Fermi National Accelerator Laboratory (FNAL/Fermilab). See LICENSE.txt.\n\nRequirements\n------------\n\n* Python 2.6 or greater recommended.\n* HTCondor libraries and Python bindings\n    * https://research.cs.wisc.edu/htcondor/downloads/\n* A running Graphite server (available in EPEL or via PIP) \n    * http://graphite.readthedocs.org/en/latest/\n\nFor current job and slot state:\n\n* A running Elasticsearch cluster\n    * https://www.elastic.co/products/elasticsearch\n* Logstash (tested with v2.0.0+)\n    * https://www.elastic.co/downloads/logstash\n\nInstallation\n------------\n\nAssuming HTCondor and Python virtualenv packages are already installed:\n\n    cd $INSTALLDIR\n    git clone https://github.com/fifemon/probes\n    cd probes\n    virtualenv --system-site-packages venv\n    source venv/bin/activate\n    pip install supervisor influxdb\n\nOptionally, for crash mails:\n\n    pip install superlance\n\nConfiguration\n-------------\n\n### Condor metrics probe\n\nExample probe config is in `etc/condor-probe.cfg`:\n\n    [probe]\n    interval = 240   # how often to send data in seconds\n    retries = 10     # how many times to retry condor queries\n    delay = 30       # seconds to wait beteeen retries\n    test = false     # if true, data is output to stdout and not sent downstream\n    once = false     # run one time and exit, i.e. for running wtih cron (not recommended)\n    \n    [graphite]\n    enable = true                           # enable output to graphite\n    host = localhost                        # graphite host\n    port = 2004                             # graphite pickle port\n    namespace = clusters.mypool             # base namespace for metrics\n    meta_namespace = probes.condor-mypool   # namespace for probe metrics\n    \n    [influxdb]\n    enable = false       # enable output to influxdb (not fully supported)\n    host = localhost     # influxdb host\n    port = 8086          # influxdb api port\n    db = test            # influxdb database\n    tags = foo:bar       # extra tags to include with all metrics (comma-separated key:value)\n    \n    [condor]\n    pool = localhost            # condor pool (collector) to query\n    post_pool_status = true     # collect basic daemon metrics\n    post_pool_slots = true      # collect slot metrics\n    post_pool_glideins = false  # collect glidein-specific metrics\n    post_pool_prio = false      # collect user priorities\n    post_pool_jobs = false      # collect job metrics\n    use_gsi_auth = false        # set true if collector requires authentication\n    X509_USER_CERT = \"\"         # location of X.509 certificate to authenticate to condor with\n    X509_USER_KEY = \"\"          # private key\n\n\n### Supervisor\n\nExample supervisor config is in `etc/supervisord.conf`, it can be used as-is for\nbasic usage. Reuqires some modification to enable crashmails or to report \njob and slot details to elsaticsearch (via logstash). \n\n### Job and slot state\n\nThe scripts that collect raw job and slot records into elasticsearch are much simpler than the metrics \nprobe - simply point at your pool with --pool and JSON records are output to stdout. We use logstash \nto pipe the output to Elasticsearch; see `etc/logstash-fifemon.conf`.\n\nRunning\n-------\n\nUsing uspervisor:\n\n    cd $INSTALLDIR/probes\n    source venv/bin/activate\n\nIf using influxdb:\n\n    export INFLUXDB_USERNAME=\u003cusername\u003e\n    export INFLUXDB_PASSWORD=\u003cpassword\u003e\n\nStart supervisor:\n\n    supervisord\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffifemon%2Fprobes","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ffifemon%2Fprobes","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffifemon%2Fprobes/lists"}