https://github.com/juliaaplavin/distributedadhoc.jl

Ad-hoc distributed computing utilities for quick parallel processing tasks.
https://github.com/juliaaplavin/distributedadhoc.jl

distributed-computing parallel-processing utilities

Last synced: 18 days ago
JSON representation

Ad-hoc distributed computing utilities for quick parallel processing tasks.

• Host: GitHub
• URL: https://github.com/juliaaplavin/distributedadhoc.jl
• Owner: JuliaAPlavin
• License: mit
• Created: 2024-02-18T17:56:42.000Z (about 2 years ago)
• Default Branch: master
• Last Pushed: 2025-03-16T12:17:04.000Z (12 months ago)
• Last Synced: 2025-10-20T07:54:56.786Z (5 months ago)
• Topics: distributed-computing, parallel-processing, utilities
• Language: Julia
• Size: 8.79 KB
• Stars: 1
• Watchers: 1
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

          
# DistributedAdhoc.jl

Easily use Julia `Distributed` in an ad-hoc way: on computers not dedicated to distributed processing.

Each machine can have different paths, different usernames and home directories, no shared filesystem.

The only requirement is that Julia is installed everywhere, and all machines can be added as `Distributed` workers.

`DistributedAdhoc` can send files, directories, and whole environments from the master process to all workers:

```julia

julia> using DistributedAdhoc  # reexports Distributed, adds no new exports

julia> addprocs(...)

# send a local file and a directory to worker #3

julia> remote_file = DistributedAdhoc.send_file("local_path/to/my_file.txt", 3)

"/tmp/jl_S1vUpx/my_file.txt"  # path of the file on worker #3

julia> remote_dir = DistributedAdhoc.send_dir("local_path/to/my_dir", 3; include=["file.txt", "data/*.csv"])

"/tmp/jl_fKgj4Y"  # path of the dir on worker #3

# send the current Julia environment to all workers, instantiate and activate it

# include source code in `src`, developed packages (in `./dev`), scripts (in `./scripts`), and CSV files (in `./data`)

julia> DistributedAdhoc.send_env_activate_everywhere(include=[

    "*.toml", "src/*.jl", "scripts/*.jl",

    "dev/*/*.toml", "dev/*/src/*.jl", "dev/*/src/*/*.jl",

    "data/*.csv",

])

```

More functions, options, and details: see `src/DistributedAdhoc.jl` or docstrings.