{"id":27004451,"url":"https://github.com/tacc/demonspawn","last_synced_at":"2025-04-04T06:16:53.706Z","repository":{"id":50341621,"uuid":"289707418","full_name":"TACC/demonspawn","owner":"TACC","description":"A quick way of spawning many batch jobs","archived":false,"fork":false,"pushed_at":"2022-10-24T16:30:24.000Z","size":559,"stargazers_count":13,"open_issues_count":1,"forks_count":0,"subscribers_count":9,"default_branch":"master","last_synced_at":"2024-06-06T09:33:36.003Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/TACC.png","metadata":{"files":{"readme":"readme.md","changelog":"CHANGELOG","contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-08-23T14:55:33.000Z","updated_at":"2024-02-05T11:12:49.000Z","dependencies_parsed_at":"2023-01-20T10:46:08.181Z","dependency_job_id":null,"html_url":"https://github.com/TACC/demonspawn","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TACC%2Fdemonspawn","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TACC%2Fdemonspawn/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TACC%2Fdemonspawn/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TACC%2Fdemonspawn/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/TACC","download_url":"https://codeload.github.com/TACC/demonspawn/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247128739,"owners_count":20888235,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-04-04T06:16:53.008Z","updated_at":"2025-04-04T06:16:53.701Z","avatar_url":"https://github.com/TACC.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# DemonSpawn\nDemonSpawn is a tool for quickly firing off a large number of SLURM jobs. \n\nThis can be used\n\n* as a benchmarking tool for kernels / micro-apps\n* as a stress test doing parametrized runs of a single test\n* sanity test: after a system upgrade, see if everything still works.\n\nCopyright Victor Eijkhout 2020-2022\n\nFor questions and problems, please submit a github issue.\n\n## Quick example\n\nExample configuration file:\n\n    # slurm script parameters\n    system frontera\n    user eijkhout\n    account A-ccsc\n    modules intel/18.0.2 impi/18.0.2\n    \n    # point-to-point benchmark setup\n    queue development\n    nodes 2\n    ppn 1\n    # program to run\n    suite name:paw-ptp type:mpi dir:/home/me/paw p2p_*\n    \n    # collective benchmark setup\n    queue normal\n    nodes 4,8,12,16\n    ppn 1,5,10\n    # program to run\n    suite name:paw-col type:mpi dir:/home/me/paw coll_*\n    \n## Introduction\n\nDemonspawn is a script that schedules SLURM job. Invocation:\n\n    python3 spawn.py [ options ] configurationfile\n\nOption flags:\n\n* `-h --help` : print help and quit.\n* `-d --debug` : lots of debug output.\n* `-f --filesonly` : generate all script files, but do not submit.\n* `-o --outputdir` + `dir` : specify output directory; omitting this gives a standard output name that includes the current date.\n* `-r --regression` + `dir` : only run the regression tests on output generated in a previous run.\n* `-c --compare` + `dir` : compare regression results in current output directory, and one generated in a previous run.\n\nThe python script stays active until all submitted SLURM jobs have finished. This is strictly necessary only for handling regression tests after the jobs have finished, but the python script also handles proper closing of files. Thus it is a good idea to \n\n    nohup python3 spawn.py myconf.txt \u0026\n    \nThe configuration is specified split over the file on the commandline, and a `.spawnrc` file, which can be used for common options, such as your username, and the slurm account to bill your runs to. The current directory is search first for the `.spawnrc` file, and then the home directory. This makes it possible to have system dependent settings. Since configuration files and `rc` files have the exact same syntax, we will not distinguish between them, and mostly discuss the configuration file.\n\n### Macros\n\nThe configuration, specified in the file given on the commandline, is completely macro based. A macro is defined in a key/value line\n\n    key value\n    otherkey value1 value2 value3\n    \nMacros keys are evaluated with a `%[macro]` syntax, because why not have yet another sytax? Example:\n\n    key value\n    otherkey another-%[key]\n\nThis syntax can also be used to substitute environment variables, if the key is not explicitly defined.\n\nSome keys have special meanings; see below.\n\nThe keyword `suite` is special in that it only defines a benchmark suite, but also triggers its execution.\nThus, you can have multiple suites in one configuration file. Each is invoked with the current value of all macros, so you can redefine macros between suite invocations. See the example above, which uses different node and process counts for the point-to-point and collective tests.\n\n### File structure\n\nDemonspawn generates output:\n\n* A single log file for the full configuration will be created in the current directory. It is identifiable by having the current date in the name.\n* An output directory is generated based on the required `outputdir` key. This will contain subdirectories `scripts` and `output` with the SLURM scripts and their standard out/err respectively.\n* If you do regression, the output directory will also contain a single regression file for each `suite` line.\n\n## SLURM macros\n\nSome macros have special meaning for your SLURM script:\n\n* `system` is set to the current hostname. If you specify this macro, it is enforced that this run can only happen on that particular system.\n* `account` is used in your slurm script as the value of the `-A` or `--account` flag.\n* `modules` is the list of modules that is loaded at the beginning of your slurm script. The value of `%[modules]` has all spaces stripped. Special case: `modules restore foo` will cause the saveset `foo` to be restored.\n* `queue` is the value of the slurm `-p` flag: the partition, or queue, name where the jobs of the next suite will be submitted. The demonspawn manager will make sure that queue limits are not violated. The queue name has optional limits on the number of simultaneous jobs:\n\n    `queue somequeue limit:2`\n\n   Suggestion: specify queue limits in the `.spawnrc` file. The last specified queue will be used as the default, or you can explicitly choose a queue in the configuration file.\n    \n* `time` is a `hh:mm:ss` specification for the slurm `-t` flag.\n\nIt is possible to add custom `#SBATCH foo=bar` lines to a script. For this, put one or more lines\n\n    sbatch foo=bar\n\nin the configuration. This the only option that is cumulative: these options are gathered up and used for each script. Thus they can not be reset between `suite` lines.\n\n## Scaling setup\n\nFor an MPI run you want to specify:\n\n* `nodes` : node count for the testsuite, unless the suite itself overrides this. This is either a single number or a comma-separated list, for scalability studies.\n* `ppn` : number of processes-per-node. A single number or a comma-separated list.\n* `threads` : OpenMP thread count. Single number or comma-separated list. A negative value indicates a thread count such that the product of MPI processes and OpenMP threads equals `SLURM_CPUS_ON_NODE`. (A zero value means that no threading is used; this value is ignored.)\n\n## Suite setup\n\nSome macros related to running the benchmark programs.\n\n* `jobname` : this is by default \"`spawn`\". It is used for the name of the logfile. You can only once define this in your configuration. The logfile, by the way, has as time stamp, in case you re-use the output directory.\n* `env` : this is used to specify environment variables. At the moment this is strictly additive: each suite is started with the sum total of specified options at that point. Example:\n\n    `env PETSC_OPTIONS -ksp_max_it 100 -ksp_monitor`\n\nYou can have multiple test suites. A test suite is specified by the keyword:\n\n* `suite` : this is followed by a list of key:value pairs, followed by a list of programs, which can use wildcards\n\nExample:\n\n    suite name:paw-mpi-p2p type:mpi dir:%[pawdir] p2p_*\n\nThis line defines the suite with the current value of all macros. \n\nIf there is more than one suite in a configuration file, each suite is fully finished before the next one is started. This is convenient if the suite runs a shell script that does a custom recompilation. You can redefine macros for the next suite in the same configuration file.\n\nThe available keys are:\n \n* `name` : for identification purposes\n* `type` : choice of `seq` or `mpi`; MPI jobs are started with ibrun\n* `dir`  : location of the programs\n\nAfter these pairs, the programs are specified with wildcards but no path.\n\n## Directory macros\n\nIt is required that you define this macro:\n\n* `outputdir` is the directory in which subdirectories `scripts`, `output`, `regression` are created.\n\nIn order to create a unique output directory, the following macros are useful:\n\n* `pwd` is set to the current working directory where you are running python on the configuration file\n* `date` is set to current date-time.\n* `mpi` is set to `LMOD_FAMILY_COMPILER`.\n\nFor example:\n\n    outputdir %[pwd]/spawn-mycode-%[mpi]-%[date]\n\n## Regression\n\nIt is easy to run a regression on all output files of a suite by specifying the `regression` key. You can specify what to regress on:\n\n    regression grep:Result\n    \nThis will grep through each result file in the suite. You can also\n\n    regression line:last\n\nwith possible options `first`, `last`. Since a regression definition stays in effect for subsequent jobsuites, you can disable a previously specified regression with\n\n    regression none\n\nRegression results are written to a single file\n\n    %[outputdir]/regression-%[suitename].txt\n    \nAdditionally, each job regression goes into a separate file\n\n    %[outputdir]/regression/\u003cjobname\u003e.out\n    \nNote: the regression specification is part of the suite definition, so it needs to come *before* the `suite` line.\n\nFurther options:\n\n* `field:5` extract only the 5-th whitespace-separated field; this numbering is 1-based\n* `label:abcd` put a label in front of the regression line. This can be a literal string, or a macro. If multiple `label` options are given, they are all used, in the sequence specified, separated by a space character.\n\nIf you want to run a regression on already generated output, run the configuration again, but with the `-r` or `--regression` flag.\n\nYou can compare the regressions of two runs by using the `-c old_output_dir` option. This will compare the files in the `regression` subdirectory, leaving the results in a file `regression_compare`. \n\nNormally, regression comparison results in both values being written to the `regression_compare` file. However, numerical comparison is enabled by having an option `margin:10percent` in the `regression` line.\n\n## Limitations\n\n* Currently the software requires python version 3.8 or higher.\n* The `system` keyword only works at TACC\n* The `mpi` keyword depends on Lmod.\n\n## Changelog\n\n0.1 somewhere around `2021/12/01`: posted on reddit\n\n0.2 `2022/02/15`: adding module restore, `regressiononly` option\n\n0.3 logfile now goes into output dir, regression flag is now `regression`\n\n0.4 regressions go into separate files, new flag for regression comparing","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftacc%2Fdemonspawn","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftacc%2Fdemonspawn","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftacc%2Fdemonspawn/lists"}