{"id":39662186,"url":"https://github.com/whbrewer/spc","last_synced_at":"2026-01-18T09:24:55.645Z","repository":{"id":16009182,"uuid":"79125823","full_name":"whbrewer/spc","owner":"whbrewer","description":"The Scientific Platform for the Cloud (SPC) - migrate and run scientific applications in the cloud","archived":false,"fork":false,"pushed_at":"2026-01-13T20:35:15.000Z","size":14929,"stargazers_count":10,"open_issues_count":1,"forks_count":4,"subscribers_count":5,"default_branch":"main","last_synced_at":"2026-01-13T20:35:52.699Z","etag":null,"topics":["batch","cloud","docker","interface","paas","scheduling"],"latest_commit_sha":null,"homepage":"","language":"Jinja","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/whbrewer.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2017-01-16T14:21:52.000Z","updated_at":"2026-01-13T20:16:58.000Z","dependencies_parsed_at":"2022-09-07T20:20:24.044Z","dependency_job_id":"9aeb29fb-e332-493e-83a9-67432610ac99","html_url":"https://github.com/whbrewer/spc","commit_stats":null,"previous_names":[],"tags_count":12,"template":false,"template_full_name":null,"purl":"pkg:github/whbrewer/spc","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/whbrewer%2Fspc","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/whbrewer%2Fspc/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/whbrewer%2Fspc/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/whbrewer%2Fspc/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/whbrewer","download_url":"https://codeload.github.com/whbrewer/spc/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/whbrewer%2Fspc/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28534159,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-18T00:39:45.795Z","status":"online","status_checked_at":"2026-01-18T02:00:07.578Z","response_time":98,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["batch","cloud","docker","interface","paas","scheduling"],"created_at":"2026-01-18T09:24:55.537Z","updated_at":"2026-01-18T09:24:55.616Z","avatar_url":"https://github.com/whbrewer.png","language":"Jinja","funding_links":[],"categories":[],"sub_categories":[],"readme":"# SPC — Scientific Platform for the Cloud\n\n[![Python](https://img.shields.io/badge/python-3.8%2B-blue.svg)](https://www.python.org/)\n[![Docs](https://img.shields.io/badge/docs-Read%20the%20Docs-blue.svg)](http://spc.readthedocs.io)\n\n\u003e Note: See `NOTES.md` for important release notes.\n\nFull documentation is available online at: http://spc.readthedocs.io  \nMinimum Python requirement: **3.8** (tested on **3.12**).\n\n---\n\n## Overview\n\nThe **Scientific Platform for the Cloud (SPC)** is a Python-based platform/framework for rapidly migrating and running **scientific applications** in the cloud. SPC provides a web UI and workflow scaffolding for running parameterized simulations, tracking cases, managing files, executing jobs through a scheduler, and visualizing results (including plotting) in a standardized and repeatable way.\n\nThis repository contains the **Python 3 modernization** of SPC, built with:\n- **Flask + Jinja2** (web UI stack)\n- **pydal** (data layer)\n- **boto3** (AWS integration)\n\nThe original Python 2 / Bottle implementation is archived in the `legacy` branch:\nhttps://github.com/whbrewer/spc/tree/legacy\n\n---\n\n## Features (High-level)\n\n- Web-based UI for launching and monitoring scientific workflows\n- Standardized parameter entry and input deck generation\n- Case management and file management per run\n- Job submission and output redirection\n- Plot definitions and result visualization\n- Packaged “apps” that can be installed into SPC\n\n---\n\n## Best fit / Intended use\n\nSPC is ideally suited for scientific applications that:\n\n1. Use an **input deck** of common parameter types (ints, floats, strings, booleans) stored in standardized formats such as:\n   - INI, XML, JSON, YAML, TOML, or `Namelist.input`\n2. Require **non-trivial runtime** (though instantaneous jobs are supported)\n3. Produce **plots** and/or structured results\n4. Use **MPI** or MapReduce for parallelization (serial workflows are also supported)\n\nOther applications may be supported with additional pre-/post-processing.\n\n---\n\n## Screenshots\n\n![SPC UI](docs/_static/spc0.png)\n![SPC UI](docs/_static/spc1.png)\n![SPC UI](docs/_static/spc2.png)\n\n---\n\n## Requirements\n\nSPC assumes the following are available on your system:\n\n### System prerequisites\n- Python **3.8+**\n- `virtualenv` (or equivalent environment tooling)\n- A compiler toolchain (`gcc` / `clang`)\n- Python development headers (required to build packages such as `psutil`)\n  - Debian/Ubuntu: `python3-dev`\n  - RHEL/CentOS: `python3-devel`\n\n### Browser support\nSPC has been tested primarily with **Google Chrome** on Linux and macOS. Other environments may work but are not guaranteed.\n\n---\n\n## Quickstart\n\nInitialize SPC (installs dependencies and initializes the SPC database):\n\n```bash\n./spc init\n```\n\nStart the web server:\n\n```bash\n./spc run\n```\n\nOpen your browser at:\n\n- http://localhost:8580/\n\n\u003e Note: Port can be overridden in `config.py`.\n\n## Headless (CLI) mode\n\nYou can run SPC without the web UI using the CLI. A typical workflow:\n\n```bash\n# Submit a job\n./spc submit dna --params \"dna=ATCGATCGATCG\" --desc \"my experiment\"\n\n# Run the scheduler in a separate terminal\n./spc scheduler\n\n# Check status\n./spc status \u003ccase_id\u003e\n\n# View output\ncat user_data/cli/dna/\u003ccase_id\u003e/dna.out\n```\n\nFor an interactive REPL, use:\n\n```bash\n./spc shell\n```\n\n## How the scheduler works\n1. `spc submit` queues a job (`state='Q'`) in the database.\n2. `spc scheduler` polls for queued jobs, marks them running (`state='R'`), and runs the app command in the case directory.\n3. Output is redirected to `\u003capp\u003e.out`; completed jobs are marked `state='C'`.\n4. Max concurrent jobs honors `config.np`.\n\n---\n\n## Run the pre-installed example: DNA Analyzer\n\n1. **Activate App**  \n   Navigate to `Apps` → `Installed`.  \n   Click **Activate** for the DNA app, then return to `Activated`.\n\n2. **Enter Parameters**  \n   Open the `dna` app and enter a DNA string (or use the default).  \n   Click `confirm` to write the datafile to disk.\n\n3. **Start Job**  \n   Click `execute` to run the DNA analysis.  \n   SPC will submit the job to the scheduler and redirect to the output view.\n\n4. **Inspect Outputs**  \n   Use `files` to open the file manager.  \n   Use `output` to view redirected executable output.  \n   Use `download` to zip and download all case files.\n\n5. **View Plots**  \n   Click `plot` to define/view plots.  \n   Use `data` to view the plotted data files (also visible via `files`).\n\n---\n\n## Install packaged apps\n\nTo install an SPC packaged app (example: Mendel’s Accountant), run one of:\n\n### macOS (Apple Silicon)\n```bash\n./spc install https://github.com/whbrewer/spc-fmendel-plugin/releases/download/v2.0.1/fmendel-spc-darwin-arm64.zip\n```\n\n### Linux (x86_64)\n```bash\n./spc install https://github.com/whbrewer/spc-fmendel-plugin/releases/download/v2.0.1/fmendel-spc-linux-x86_64.zip\n```\n\n---\n\n## Documentation\n\n- Online documentation: http://spc.readthedocs.io\n- Local docs: see the `docs/` directory\n\n---\n\n## Developer setup (minimal)\n\nA lightweight development workflow:\n\n1. Clone the repo and enter it:\n```bash\ngit clone https://github.com/whbrewer/spc.git\ncd spc\n```\n\n2. Initialize SPC:\n```bash\n./spc init\n```\n\n3. Run the server:\n```bash\n./spc run\n```\n\n4. Open:\n- http://localhost:8580/\n\n### Running tests\n\nTests use pytest and run against a temporary database (no server required):\n\n```bash\n# Run all tests\n./venv/bin/pytest tests/ -v\n\n# Run a single test file\n./venv/bin/pytest tests/test_login.py -v\n\n# Run a single test\n./venv/bin/pytest tests/test_login.py::TestLogin::test_login_valid_credentials -v\n```\n\nTest coverage includes:\n- Login, registration, and authentication routes\n- App display, parameter submission, and job execution\n- Admin routes and app configuration\n- Job scheduler operations\n- Input file format parsers (INI, JSON, YAML, Namelist)\n\n---\n\n## Contributing\n\nContributions are welcome.\n\nSuggested workflow:\n1. Fork the repo\n2. Create a feature branch\n3. Make changes with clear commit messages\n4. Add/update documentation where needed\n5. Open a Pull Request\n\nIf you plan to contribute a new packaged app/plugin, please include:\n- A small example input deck\n- A minimal runnable executable or script stub\n- One representative plot/output artifact\n\n---\n\n## Citation\n\nIf you use SPC in academic work, please cite:\n\n\u003e W. Brewer, W. Scott, and J. Sanford, “An Integrated Cloud Platform for Rapid Interface Generation, Job Scheduling, Monitoring, Plotting, and Case Management of Scientific Applications”, Proc. of the International Conference on Cloud Computing Research and Innovation, Singapore, IEEE Press, October 2015, pp. 156–165. DOI: 10.1109/ICCCRI.2015.24\n\n---\n\n## Support / Questions\n\nFor questions, contact [Wes Brewer](https://www.ornl.gov/staff-profile/wes-h-brewer).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwhbrewer%2Fspc","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fwhbrewer%2Fspc","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwhbrewer%2Fspc/lists"}