Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/bsc-wdc/compss
COMP Superscalar (COMPSs) is a framework which aims to ease the development and execution of applications for distributed infrastructures, such as Clusters, Grids and Clouds.
https://github.com/bsc-wdc/compss
c distributed-computing docker hpc java pipeline-framework python singularity slurm workflow-management-system workflows
Last synced: 2 days ago
JSON representation
COMP Superscalar (COMPSs) is a framework which aims to ease the development and execution of applications for distributed infrastructures, such as Clusters, Grids and Clouds.
- Host: GitHub
- URL: https://github.com/bsc-wdc/compss
- Owner: bsc-wdc
- License: apache-2.0
- Created: 2018-03-05T16:44:51.000Z (almost 7 years ago)
- Default Branch: stable
- Last Pushed: 2025-01-01T23:45:32.000Z (14 days ago)
- Last Synced: 2025-01-06T17:15:01.765Z (9 days ago)
- Topics: c, distributed-computing, docker, hpc, java, pipeline-framework, python, singularity, slurm, workflow-management-system, workflows
- Language: Java
- Homepage: https://compss.bsc.es
- Size: 288 MB
- Stars: 48
- Watchers: 10
- Forks: 20
- Open Issues: 13
-
Metadata Files:
- Readme: README.md
- Changelog: changelog
- Contributing: CONTRIBUTING.md
- License: LICENSE
- Code of conduct: CODE_OF_CONDUCT.md
- Citation: CITATION.cff
Awesome Lists containing this project
README
COMPSs/PyCOMPSs Framework
Website •
Documentation •
Releases •
Slack •
✉
COMP Superscalar (COMPSs) is a programming model which aims to ease the development
of applications for distributed infrastructures, such as Clusters, Grids and Clouds.
COMP Superscalar also features a runtime system that exploits the inherent parallelism
of applications at execution time.
## Repository Structure
* **builders**: Packages, scripts for local installations, scripts for supercomputers
installation and package building scripts
* **compss** : Programming Model, Bindings and Runtime source code
* **dependencies** : Embeded dependencies
* **files** : Dependency files (i.e. paraver configurations)
* **tests** : Integration tests
* **utils** : Misc utils (i.e. OVA scripts, Docker generation, Storage implementations)
## Supported Systems
COMPSs/PyCOMPSs fully supports Linux systems for x86_64, amd64, ppc64, arm64 and riscv64 architectures. macOS systems are also supported with some limitations.
## Building From Sources
Follow the next steps to build COMPSs in your current machine.
### 1. Install dependencies
For an updated list of COMPSs dependencies and how to install them for different systems visit the [dependencies section](https://compss-doc.readthedocs.io/en/latest/Sections/01_Installation/01_Dependencies.html) of the COMPSs documentation website.
### 2. Get GIT submodules
Before installing COMPSs you need to download the git submodules that contain its dependencies. To do that execute the following two commands at the root of the repository.
```
./submodules_get.sh
```
### 3. Build COMPSs
**Note**: Remember to install the COMPSs dependencies and to get the GIT submodules before trying to build COMPSs from sources.
* Building COMPSs for all users (not supported in macOS)
```
cd builders/
INSTALL_DIR=/opt/COMPSs/
sudo -E ./buildlocal [options] ${INSTALL_DIR}
```
* Building COMPSs for current user
```
cd builders/
INSTALL_DIR=$HOME/opt/COMPSs/
./buildlocal [options] ${INSTALL_DIR}
```
For macOS:
```
cd builders/
alias libtoolize=/usr/local/bin/glibtoolize
alias readlink=/usr/local/bin/greadlink
export LIBTOOL=`which glibtool`
export LIBTOOLIZE=`which glibtoolize`
INSTALL_DIR=$HOME/opt/COMPSs/
./buildlocal -K -T -M ${INSTALL_DIR}
```
Many COMPSs modules can be activated/deactivated during the build using different options in the `buildlocal` command. You may check the available options by running the following command:
```
cd builders
./buildlocal -h
```
## Running docker tests
### 1. Install Docker and docker-py
Follow these instructions
- [Docker for Mac](https://store.docker.com/editions/community/docker-ce-desktop-mac). Or, if you prefer to use [Homebrew](https://brew.sh/).
- [Docker for Ubuntu](https://docs.docker.com/install/linux/docker-ce/ubuntu/#install-docker-ce-1).
- [Docker for Arch Linux](https://wiki.archlinux.org/index.php/Docker#Installation).
Add user to docker group to run docker as non-root user.
- [Instructions](https://docs.docker.com/install/linux/linux-postinstall/).
### 2. Build the docker image
Run the following command at the root of the project to build the image that will used for testing. The command create an image named **compss** and install the current branch into the image.
```
docker build --target=ci -t compss .
```
### 3. Run the tests
To run the tests inside the docker image use the script found in `./tests/scripts/docker_main`. This command is a wrapper for the `./main` test command
so it has de the syntax and options. For example, you can run the first test without retrials as follows:
```
./docker_main -R -t 1
```
The docker main command creates a new docker container each time you run it (replacing the last one used). It copies the current framework inside it
and runs its tests. **Note**: the testing scripts assumes you have named the testing image `compss`.
**Please be aware that:**
* Code changes affecting the tests sources, config files (e.g. `local.cfg`, and scripts (like `./local`) __will be__ visible inside the newly created container.
* Code changes affecting the installation __will not be__ visible in the installation because framework is not reinstalled. To do that rebuild the docker image as explained in step 3.
* If you run the command once, the container will be available for manual inspection (such as logs). You can log into in issuing `docker exec --user jenkins -it compss_test bash` and use the CLI as usual.
* To delete the created image issue `docker rmi compss`
* To delete the compss_test container use `docker rm -f compss_test`.
### 4. Run the tests locally on macOS
In order to run Jenkins tests locally on macOS, GNU sed (gsed) is needed. To install it, use:
```
brew install gsed
```
Some environment variables also need to be defined to ensure gsed is used instead of macOS sed.
```
export PATH="/usr/local/opt/gnu-sed/libexec/gnubin:$PATH
```
Finally, the `NIO_mac.cfg` file needs to be updated with any specific features of the local macOS environment, commonly variables such as the `java_home`, `compss_home` and `runcompss_opts` parameters should be updated.
## Support
For support please send and e-mail to [email protected]
(c) Workflows and Distributed Computing Group (WDC) - Department of Computer Science (CS) - Barcelona Supercomputing Center (BSC)