{"id":27004381,"url":"https://github.com/tacc/hpcperfstats","last_synced_at":"2025-09-18T08:56:12.857Z","repository":{"id":18131512,"uuid":"21212519","full_name":"TACC/HPCPerfStats","owner":"TACC","description":"HPCPerfStats (formerly TACC Stats) is an automated resource-usage monitoring and analysis package.","archived":false,"fork":false,"pushed_at":"2025-03-31T16:29:53.000Z","size":15241,"stargazers_count":46,"open_issues_count":40,"forks_count":15,"subscribers_count":13,"default_branch":"master","last_synced_at":"2025-03-31T17:37:38.257Z","etag":null,"topics":["hpc","metrics","performance-analysis","rabbitmq","xsede"],"latest_commit_sha":null,"homepage":"","language":"C","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"lgpl-2.1","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/TACC.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2014-06-25T17:42:03.000Z","updated_at":"2025-03-31T16:29:57.000Z","dependencies_parsed_at":"2023-01-11T20:28:08.751Z","dependency_job_id":"a8a958ec-d444-4dc4-be58-a027f88d5864","html_url":"https://github.com/TACC/HPCPerfStats","commit_stats":null,"previous_names":["tacc/hpcperfstats"],"tags_count":8,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TACC%2FHPCPerfStats","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TACC%2FHPCPerfStats/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TACC%2FHPCPerfStats/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TACC%2FHPCPerfStats/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/TACC","download_url":"https://codeload.github.com/TACC/HPCPerfStats/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247128739,"owners_count":20888235,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["hpc","metrics","performance-analysis","rabbitmq","xsede"],"created_at":"2025-04-04T06:16:31.766Z","updated_at":"2025-09-18T08:56:07.823Z","avatar_url":"https://github.com/TACC.png","language":"C","funding_links":[],"categories":[],"sub_categories":[],"readme":"HPCPerfStats\n-----------------\n(The package formerly known as TACC Stats)\n\nDescription\n-----------------\nThe hpcperfstats package provides the tools to monitor resource usage of HPC systems at multiple levels of resolution.\n\n[Collected Data Definitions](docs/attributes-definition.md)\n\nWe are currently in the proccess of updating our documentation. Please stand by.\n\n\u003c!---\nThe package is split into an `autotools`-based `monitor` subpackage and a Python `setuptools`-based `hpcperfstats` subpackage.  `monitor` performs the online data collection and transmission in a production environment while `hpcperfstats` performs the data curation and analysis in an offline environment.\n\nBuilding and installing the `hpcperfstats-2.3.5-1.el7.x86_64.rpm` package with the `hpcperfstats.spec` file will build and install a systemd service `hpcperfstats`.  This service launches a daemon with an overhead of 3% on a single core when configured to sample at a frequency of 1Hz.  It is typically configured to sample at 5 minute intervals, with samples taken at the start and end of every job as well. The TACC Stats daemon, `hpcperfstatsd`, is controlled by the `hpcperfstats` service and sends the data directly to a RabbitMQ server over the administrative ethernet network.  RabbitMQ must be installed and running on the server in order for the data to be received.\n\nInstalling the `hpcperfstats` module will setup a Django-based web application along with tools for extracting the data from the RabbitMQ server and feeding them into a PostgreSQL database.   \n\nCode Access\n-----------\nTo get access to the hpcperfstats source code clone this repository:\n\n    git clone https://github.com/TACC/hpcperfstats\n\n\n----------------------------------------------------------------------------\n\nInstallation\n--------\n#### `monitor` subpackage\n\nFirst ensure the RabbitMQ library and header file are installed on the build and compute nodes\n\n[librabbitmq-devel-0.5.2-1.el6.x86_64](ftp://fr2.rpmfind.net/linux/epel/6/x86_64/librabbitmq-devel-0.5.2-1.el6.x86_64.rpm)\n\n`./configure; make; make install` will then successfully build the `hpcperfstatsd` executable for many systems.  If Xeon Phi coprocessors are present on your system they can be monitored with the `--enable-mic` flag.  Additionally the configuration options, `--disable-infiniband`, `--disable-lustre`, `--disable-hardware` will disable infiniband, Lustre Filesystem, and Hardware Counter monitoring which are all enabled by default. Disabling RabbitMQ will result in a legacy build of `hpcperfstatsd` that relies on the shared filesystem to transmit data.  This mode is not recommended and currently used for testing purposes only.  If libraries or header files are not found than add their paths to the include and library paths with the `CPPFLAGS` and/or `LDFLAGS` vars as is standard in autoconf based installations.  \n\nThere will be a configuration file, `/etc/hpcperfstats/hpcperfstats.conf`, after installation.  This file contains the fields\n\n`server localhost`\n\n`queue default`\n\n`port 5672`\n\n`freq 600`\n\n\n`server` should be set to the hostname or IP hosting the RabbitMQ server, `queue` to the system/cluster name that is being monitored, `port` to the RabbitMQ port (5672 is default), and `freq` to the desired sampling frequency in seconds. The file and settings can be reloaded into a running `hpcperfstatsd` daemon with a SIGHUP signal.\n\nAn RPM can be built for deployment using  the `hpcperfstats.spec` file.  The most straightforward approach to build this is to setup your rpmbuild directory then run\n\n`rpmbuild -bb hpcperfstats.spec`\n\nThe `hpcperfstats.spec` file `sed`s the `hpcperfstats.conf` file to the correct server and queue. These can be changed by modifying these two lines \n\n`sed -i 's/localhost/stats.frontera.tacc.utexas.edu/' src/hpcperfstats.conf`\n\n`sed -i 's/default/frontera/' src/hpcperfstats.conf`\n\n`hpcperfstatsd` can be started, stopped, and restarted using `systemctl start hpcperfstats`, `systemctl stop hpcperfstats`, and `systemctl restart hpcperfstats`.\n\nIn order to notify `hpcperfstats` of a job beginning, echo the job id into `/var/run/TACC_jobid` on each node where the job is running.  It order to notify\nit of a job ending echo `-` into `/var/run/TACC_jobid` on each node where the job is running.  This can be accomplished in the job scheduler prolog and\nepilog for example.\n\n#### Job Scheduler Configuration\n-------\nIn order for hpcperfstats to correcly label records with JOBIDs it is required that\nthe job scheduler prolog and epilog contain the lines\n\n\n`echo $JOBID \u003e jobid_file`  \n\nand\n\n`echo - \u003e jobid_file`\n\nTo perform the pickling of this data it is also necessary to\ngenerate an accounting file that contains the following information\nin the following format\n\n`JobID|User|Account|Start|End|Submit|Partition|Timelimit|JobName|State|NNodes|ReqCPUS|NodeList`\n\nfor example,\n\n1837137|sharrell|project140208|2018-08-01T18:18:51|2018-08-02T11:44:51|2018-07-29T08:05:43|normal|1-00:00:00|jobname|COMPLETED|8|104|c420-[024,073],c421-[051-052,063-064,092-093]\n\nIf using SLURM the `sacct_gen.py` script that will be installed with the `hpcperfstats` subpackage may be used. \nThis script generates a file for each date with the name format `year-month-day.txt`, e.g. `2018-11-01.txt`.\n\n#### `hpcperfstats` subpackage\n To install TACC Stats on the machine where data will be processed, analyzed, and the webserver hosted follow these\n steps:\n \n1.  Download the package and setup the Python3 virtual environment. TACC Stats is Python3 dependent.\n```\n$ virtualenv machinename --system-site-packages\n$ cd machinename; source bin/activate\n$ git clone https://github.com/TACC/hpcperfstats\n```\n`hpcperfstats` is a pure Python package.  Dependencies should be automatically downloaded\nand installed when installed via `pip`.  The package must first be configured however \nin the `hpcperfstats.ini` file.  \n2.  The initialization file, `hpcperfstats.ini`, controls all the configuration options and has \nthe following content and descriptions\n```\n## Basic configuration options - modify these\n# machine       = unique name of machine/queue\n# server        = database and rmq server hostname\n# data_dir      = where data is stored\n[DEFAULT]\nmachine         = ls5\ndata_dir        = /hpc/hpcperfstats_site/%(machine)s\nserver          = tacc-stats02.tacc.utexas.edu\n\n## RabbitMQ Configuration\n# RMQ_SERVER    = RMQ server\n# RMQ_QUEUE     = RMQ server\n[RMQ]\nrmq_server      = %(server)s\nrmq_queue       = %(machine)s\n\n## Configuration for Web Portal Support\n[PORTAL]\nacct_path       = %(data_dir)s/accounting\narchive_dir     = %(data_dir)s/archive\nhost_name_ext   = %(machine)s.tacc.utexas.edu\ndbname          = %(machine)s_db\n```\nSet these paths as needed. The `accounting_path` will contain an accounting file for each date, e.g. `2018-11-01.txt`. The raw stats data will be stored in the `archive_dir` and processed stats data in the TimeScale database `dbname`.  `machine` should match the system name used in the RabbitMQ server `QUEUE` field and is the RabbitMQ `QUEUE` that the monitoring agent sends the data too.  This is the only field that needs to match settings in the `monitor` subpackage. `host_name_ext` is the extension required to each compute node hostname in order to build a FQDN. This will match to directory names created in the `archive_dir`. \n3.  Install `hpcperfstats`\n```\n$ pip install -e hpcperfstats/\n```\n4.  Start the RabbitMQ server reader in the background, e.g. \n```\n$ nohup listend.py \u003e /tmp/listend.log\n```\nRaw stats files will now be generated in the `archive_dir`.\n5.  A PostgreSQL database must be setup on the host.  To do this, after installation of PostgreSQL\nand the `hpcperfstats` Python module \n```\n$ sudo su - postgres\n$ psql\n# CREATE DATABASE machinename_db;\n# CREATE USER hpcperfstats WITH PASSWORD 'hpcperfstats';\n# ALTER ROLE hpcperfstats SET client_encoding TO 'utf8';\n# ALTER ROLE hpcperfstats SET default_transaction_isolation TO 'read committed';\n# ALTER ROLE hpcperfstats SET timezone TO 'UTC';\n# ALTER DATABASE machinename_db OWNER TO hpcperfstats;\n# GRANT ALL PRIVILEGES ON DATABASE machinename_db TO hpcperfstats;\n# \\q\n```\n\nthen\n\n```\n$ python manage.py migrate\n```\nThis will generate a table named `machinename_db` in your database.  \n\n6.  Setup cron jobs to process raw data and ingest into database.  Add the following to your \ncron file\n```\n*/15 * * * * source /home/sharrell/testing/bin/activate; job_pickles.py; update_db.py \u003e /tmp/ls5_update.log 2\u003e\u00261\n```\n7.  Next configure the Apache server (make sure it is installed and the `mod_wsgi` Apache module is installed)\nA sample configuration file, `/etc/httpd/conf.d/ls5.conf`, looks like\n```\nLoadModule wsgi_module /stats/stampede2/lib/python3.7/site-packages/mod_wsgi/server/mod_wsgi-py37.cpython-37m-x86_64-linux-gnu.so\nWSGISocketPrefix run/wsgi\n\n\u003cVirtualHost *:80\u003e\n\nServerAdmin sharrell@tacc.utexas.edu\nServerName stats.webserver.tacc.utexas.edu\nServerAlias stats.webserver.tacc.utexas.edu\n\nWSGIDaemonProcess s2-stats python-home=/stats/stampede2 python-path=/stats/stampede2/hpcperfstats:/stats/stampede2/lib/python3.7/site-packages user=sharrell\nWSGIProcessGroup s2-stats\nWSGIScriptAlias / /hpcperfstats/site/hpcperfstats_site/wsgi.py process-group=s2-stats\nWSGIApplicationGroup %{GLOBAL}\n\n\u003cDirectory /stats/stampede2/hpcperfstats/hpcperfstats/site/hpcperfstats_site\u003e\n\u003cFiles wsgi.py\u003e\nRequire all granted\n\u003c/Files\u003e\n\u003c/Directory\u003e\n\u003c/VirtualHost\u003e\n```\n8.  Start up Apache \n\n### Running `job_pickles.py`\n`job_pickles.py` can be run manually by:\n\n    $ ./job_pickles.py [start_date] [end_date] [-dir directory] [-jobids id0 id1 ... idn]\n\nwhere the 4 optional arguments have the following meaning\n\n  - `start_date`     : the start of the date range, e.g. `\"2013-09-25\"` (default is today)\n  - `end_date`       : the end of the date range, e.g. `\"2013-09-26\"` (default is `start_date`)\n  - `-dir`       : the directory to store pickled dictionaries (default is set in hpcperfstats.ini)\n  - `-jobids`     : individual jobids to pickle (default is all jobs)\n  \nNo arguments results in all jobs from the previous day getting pickled and stored in the `pickles_dir`\ndefined in `hpcperfstats.ini`. On Stampede argumentless `job_pickles.py` is run every 24 hours as a `cron` job\nset-up by the user.\n\n\n### Pickled data format: generated `job_pickles.py`\n\nPickled stats data will be placed in the directory specified by\n`pickles_dir`.  The pickled data is contained in a nested python\ndictionary with the following key layers:\n\n    job       : 1st key Job ID\n     host     : 2nd key Host node used by Job ID\n      type    : 3rd key TYPE specified in hpcperfstats\n       device : 4th key device belonging to type\n\nFor example, to access Job ID `101`'s stats data on host `c560-901` for\n`TYPE` `intel_snb` for device cpu number `0` from within a python script:\n\n    pickle_file = open('101','r')\n    jobid = pickle.load(pickle_file)\n    pickle_file.close()\n    jobid['c560-901']['intel_snb']['0']\n\nThe value accessed by this key is a 2D array, with rows corresponding to record times and\ncolumns to specific counters for the device.  To view the names for each counter add\n\n    jobid.get_schema('intel_snb')\n\nor for a short version\n\n    jobid.get_schema('intel_snb').desc\n---\u003e\n\nPublications\n-------\n[Comprehensive Resource Use Monitoring for HPC Systems with TACC Stats](http://doi.org/10.1109/HUST.2014.7)\n\n[Understanding application and system performance through system-wide monitoring](http://doi.org/10.1109/IPDPSW.2016.145)\n\n[![DOI](https://zenodo.org/badge/21212519.svg)](https://zenodo.org/badge/latestdoi/21212519)\n\n\nDevelopers and Maintainers\n-------\nAmit Ruhela  (\u003cmailto:aruhela@tacc.utexas.edu\u003e) \u003cbr /\u003e\nStephen Lien Harrell  (\u003cmailto:sharrell@tacc.utexas.edu\u003e) \u003cbr /\u003e\nSangamithra Goutham (\u003cmailto:sgoutham@tacc.utexas.edu\u003e) \u003cbr /\u003e\nChris Ramos (\u003cmailto:cramos@tacc.utexas.edu\u003e) \u003cbr /\u003e\n\nDeveloper Emeritus\n-------\nJohn Hammond \u003cbr /\u003e\nR. Todd Evans  \u003cbr /\u003e\nBill Barth \u003cbr /\u003e\nAlbert Lu \u003cbr /\u003e\nJunjie Li \u003cbr /\u003e\nJohn McCalpin \u003cbr /\u003e\n\n\n----------------------------------------------------------------------------\n\n## Copyright\n(C) 2011 University of Texas at Austin\n\n## License\n\nThis library is free software; you can redistribute it and/or\nmodify it under the terms of the GNU Lesser General Public\nLicense as published by the Free Software Foundation; either\nversion 2.1 of the License, or (at your option) any later version.\n\nThis library is distributed in the hope that it will be useful,\nbut WITHOUT ANY WARRANTY; without even the implied warranty of\nMERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU\nLesser General Public License for more details.\n\nYou should have received a copy of the GNU Lesser General Public\nLicense along with this library; if not, write to the Free Software\nFoundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA  02110-1301  USA\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftacc%2Fhpcperfstats","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftacc%2Fhpcperfstats","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftacc%2Fhpcperfstats/lists"}