Projects in Awesome Lists tagged with distributed-computing

https://github.com/sw33tLie/fleex

Fleex makes it easy to create multiple VPS on cloud providers and use them to distribute workloads.

bug-bounty bugbounty digitalocean distributed-computing distributed-systems hacking hacking-tool hacktoberfest linode

Last synced: 05 Apr 2025

https://github.com/sciml/linearsolve.jl

LinearSolve.jl: High-Performance Unified Interface for Linear Solvers in Julia. Easily switch between factorization and Krylov methods, add preconditioners, and all in one interface.

amg differential-equations distributed-computing factorization gpu julia krylov-methods linear-algebra linear-solvers multigrid newton-krylov preconditioners scientific-machine-learning sciml

Last synced: 13 May 2025

https://github.com/maxpumperla/learning_ray

Notebooks for the O'Reilly book "Learning Ray"

data-science deep-learning distributed-computing machine-learning notebook python ray

Last synced: 07 Apr 2025

https://github.com/NexaAI/octopus-v4

AI for all: Build the large graph of the language models

distributed-computing graph large-language-models

Last synced: 23 Feb 2025

https://github.com/teivah/gosiris

An actor framework for Go

actor-model actors actorsystem amqp distributed-computing distributed-systems etcd kafka zipkin

Last synced: 29 Mar 2025

https://github.com/SciML/LinearSolve.jl

LinearSolve.jl: High-Performance Unified Interface for Linear Solvers in Julia. Easily switch between factorization and Krylov methods, add preconditioners, and all in one interface.

amg differential-equations distributed-computing factorization gpu julia krylov-methods linear-algebra linear-solvers multigrid newton-krylov preconditioners scientific-machine-learning sciml

Last synced: 04 May 2025

https://github.com/Bears-R-Us/arkouda

Arkouda (αρκούδα): Interactive Data Analytics at Supercomputing Scale :bear:

chapel data data-analysis data-science distributed-computing eda hpc python

Last synced: 20 Nov 2024

https://github.com/joergen7/cuneiform

Cuneiform distributed programming language

bioinformatics distributed-computing erlang functional-programming machine-learning otp parallelization workflow workflow-engine

Last synced: 16 May 2025

https://github.com/FleexSecurity/fleex

Fleex makes it easy to create multiple VPS on cloud providers and use them to distribute workloads.

bug-bounty bugbounty digitalocean distributed-computing distributed-systems hacking hacking-tool hacktoberfest linode

Last synced: 21 Nov 2024

https://github.com/scipr-lab/dizk

Java library for distributed zero knowledge proof systems

cryptography distributed-computing zero-knowledge-proofs zksnarks

Last synced: 01 May 2025

https://github.com/whitfin/local-cluster

Easy local cluster creation for Elixir to aid in unit testing

distributed-computing erlang-distribution local-clustering unit-testing

Last synced: 16 May 2025

https://github.com/r-lib/mirai

mirai - Minimalist Async Evaluation Framework for R

async asynchronous-tasks concurrency distributed-computing high-performance-computing parallel-computing r

Last synced: 05 Apr 2025

https://github.com/futureverse/future.apply

:rocket: R package: future.apply - Apply Function to Elements in Parallel using Futures

asynchronous distributed-computing future hpc hpc-clusters package parallel parallel-computing parallel-processing parallelization programming r

Last synced: 14 May 2025

https://github.com/ljk628/ML-Systems

papers on scalable and efficient machine learning systems

convolutional-neural-networks deep-learning distributed-computing machine-learning optimization

Last synced: 06 May 2025

https://github.com/gridhead/sysmon

An intuitive remotely-accessible system performance monitoring and task management tool for servers and headless Raspberry Pi setups.

distributed-computing performance-analysis raspberry-pi sysadmin-tool system-monitoring

Last synced: 04 Apr 2025

https://github.com/orleanscontrib/orleans.clustering.kubernetes

Orleans Membership provider for Kubernetes

actor-framework actor-model azure distributed-computing distributed-systems dotnet-core kubernetes kubernetes-api kubernetes-cluster orleans orleans-cluster orleans-framework virtual-actors

Last synced: 12 Apr 2025

https://github.com/syzer/js-spark

Realtime calculation distributed system. AKA distributed lodash

distributed distributed-computing multicore realtime spark

Last synced: 09 Apr 2025

https://github.com/catalyst-team/mlcomp

Distributed DAG (Directed acyclic graph) framework for machine learning with UI

artificial-intelligence automl computer-vision deep-learning distributed-computing infrastructure machine-learning python pytorch research

Last synced: 03 Mar 2025

https://github.com/lightforever/mlcomp

Distributed DAG (Directed acyclic graph) framework for machine learning with UI

artificial-intelligence automl computer-vision deep-learning distributed-computing infrastructure machine-learning python pytorch research

Last synced: 19 Dec 2024

https://github.com/pnxtech/hydra-express

A module which wraps Hydra and ExpressJS into a library for building distributed applications - such as microservices

distributed-computing express expressjs hydra hydra-express microservice redis

Last synced: 11 Apr 2025

https://github.com/OrleansContrib/Orleans.Clustering.Kubernetes

Orleans Membership provider for Kubernetes

actor-framework actor-model azure distributed-computing distributed-systems dotnet-core kubernetes kubernetes-api kubernetes-cluster orleans orleans-cluster orleans-framework virtual-actors

Last synced: 18 Apr 2025

https://github.com/daveschumaker/artbot-for-stable-diffusion

A front-end GUI for interacting with the AI Horde / Stable Diffusion distributed cluster

ai-art art-generator distributed-computing generative-art javascript nextjs nodejs reactjs stable-diffusion stable-diffusion-webui tailwindcss typescript

Last synced: 10 Apr 2025

https://github.com/papuSpartan/stable-diffusion-webui-distributed

Chains stable-diffusion-webui instances together to facilitate faster image generation.

automatic1111 distributed-computing multi-gpu stable-diffusion stable-diffusion-webui stable-diffusion-webui-plugin

Last synced: 28 Mar 2025

https://github.com/cda-group/arcon

State-first Streaming Applications in Rust

arrow data-analytics dataflow distributed-computing rust stream-processor

Last synced: 27 Apr 2025

https://github.com/mostafa-samir/klyng

A message-passing distributed computing framework for node.js

distributed-computing javascript message-passing nodejs parallel-computing

Last synced: 18 Apr 2025

https://github.com/varchar-io/nebula

A distributed block-based data storage and compute engine

access-control analytics big-data data-analysis data-visualization distributed-computing distributed-systems real-time

Last synced: 09 May 2025

https://github.com/xorbitsai/xoscar

Python actor framework for heterogeneous computing.

actor-model asynchronous distributed-computing

Last synced: 12 Jun 2025

https://github.com/sansa-stack/sansa-stack

Big Data RDF Processing and Analytics Stack built on Apache Spark and Apache Jena http://sansa-stack.github.io/SANSA-Stack/

apache-jena apache-spark distributed-computing flink rdf semantic-web spark

Last synced: 04 Apr 2025

https://github.com/SANSA-Stack/SANSA-Stack

Big Data RDF Processing and Analytics Stack built on Apache Spark and Apache Jena http://sansa-stack.github.io/SANSA-Stack/

apache-jena apache-spark distributed-computing flink rdf semantic-web spark

Last synced: 20 Nov 2024

https://github.com/dask/dask-cloudprovider

Cloud provider cluster managers for Dask. Supports AWS, Google Cloud Azure and more...

aws azure cloud dask deployment-tools digitalocean distributed-computing google-cloud hetzner ibm-cloud openstack

Last synced: 08 May 2025

https://github.com/openmole/openmole

Workflow engine for exploration of simulation models using high throughput computing

dirac distributed-computing egi grid hyperparameters modeling-tool parameter-estimation parameter-search parameter-tuning scala scientific-computing workflow workflow-engine

Last synced: 25 Jan 2025

https://github.com/shikokuchuo/mirai

mirai - Minimalist Async Evaluation Framework for R

asynchronous-tasks concurrency cran distributed-computing high-performance-computing parallel-programming promises r r-package rstats

Last synced: 29 Mar 2025

https://github.com/siboehm/shallowspeed

Small scale distributed training of sequential deep learning models, built on Numpy and MPI.

deep-learning distributed-computing pipelines

Last synced: 26 Jun 2025

https://github.com/lachlan2k/phatcrack

Modern web-based distributed hashcracking solution, built on hashcat

distributed-computing golang gpu-computing hacking hashcat hashcracking infosec pentesting security-tools vue

Last synced: 10 Apr 2025

https://github.com/czg1225/AsyncDiff

Official implementation of "AsyncDiff: Parallelizing Diffusion Models by Asynchronous Denoising"

diffusion-models distributed-computing efficient-inference inference-acceleration stable-diffusion text-to-image text-to-video training-free

Last synced: 13 Mar 2025

https://github.com/stateless-minds/cyber-stasis

Post-money economy simulator in the form of a free fictional game based on gift economy that tests the hypothesis of having a market system without any exchange be it barters or money.

decentralized-web distributed-computing economics-game economics-models economy-simulator gift-economy golang ipfs moneyless multiplayer p2p post-capitalism post-scarcity self-governance sharing-economy simulation simulator society-management web3 webassembly

Last synced: 05 Apr 2025

https://github.com/cubed-dev/cubed

Scalable array processing with bounded memory

distributed-computing ndimensional-arrays numpy python serverless xarray zarr

Last synced: 05 Apr 2025

https://github.com/spacewalkhq/raft-rs

An understandable, fast and scalable Raft Consensus implementation

byzantine-consensus consensus distributed-computing distributed-systems raft-consensus-algorithm rust

Last synced: 24 Nov 2024

https://github.com/haidra-org/horde-worker-regen

The default client software to create images for the AI-Horde

ai ai-horde distributed-computing generative-ai volunteer-computing

Last synced: 22 Apr 2025

https://github.com/asynkron/realtimemap-dotnet

A showcase for Proto.Actor - an ultra-fast distributed actors solution for Go, C#, and Java/Kotlin.

actors clustering distributed-computing distributed-systems proto-actor

Last synced: 12 May 2025

https://github.com/reiver/blockchain-reading-list

A reading list on blockchain and related technologies, targeted at technical people who want a deep understanding of those topics.

abstract-algebra bitcoin blockchain cryptography distributed-computing ethereum identity peer-to-peer reading-list whitepapers

Last synced: 11 Apr 2025

https://github.com/diracgrid/dirac

DIRAC Grid

distributed-computing grid interware workflow-management workload-management

Last synced: 14 May 2025

https://github.com/pennylaneai/pennylane-lightning

The Lightning plugin ecosystem provides fast quantum state-vector and tensor network simulators written in C++ for use with PennyLane.

cuda distributed-computing gpu hpc mpi openmp parallel quantum-computing quantum-machine-learning rocm

Last synced: 15 May 2025

https://github.com/erdewit/distex

Distributed process pool for Python

asyncio distributed-computing multiprocessing parallel-computing python task-queue

Last synced: 26 Mar 2025

https://github.com/PennyLaneAI/pennylane-lightning

The Lightning plugin ecosystem provides fast quantum state-vector and tensor network simulators written in C++ for use with PennyLane.

cuda distributed-computing gpu hpc mpi openmp parallel quantum-computing quantum-machine-learning rocm

Last synced: 11 May 2025

https://github.com/JasonThon/lightflus

A Lightweight, Cloud-Native Stateful Distributed Dataflow Engine

dataflow deno distributed-computing rust streaming typescript

Last synced: 27 Apr 2025

https://github.com/h2337/cppq

Simple, reliable & efficient distributed task queue for C++17

asynchronous-tasks background-jobs cpp cpp17 distributed-computing redis redis-queue task-queue tasks worker-pool

Last synced: 30 Apr 2025

https://github.com/ray-project/mobius

Mobius is an AI infrastructure platform for distributed online learning, including online sample processing, training and serving.

distributed-computing fusion machine-learning online-learning realtime training

Last synced: 14 Apr 2025

https://github.com/codepr/tasq

A simple task queue implementation to enqeue jobs on local or remote processes.

distributed-computing distributed-job distributed-systems job-scheduler queue redis remote-workers scheduled-jobs task-queue zmq

Last synced: 27 Nov 2024

https://github.com/aavache/llmwebcrawler

A Web Crawler based on LLMs implemented with Ray and Huggingface. The embeddings are saved into a vector database for fast clustering and retrieval. Use it for your RAG.

api distributed-computing fastapi huggingface large-language-models llm machine-learning milvus nlp pydantic python rag ray raylib transformer vector-database webcrawler webcrawling

Last synced: 14 Apr 2025

https://github.com/georgemac/adagio

Workflow Orchestrator

distributed-computing graph workflow-orchestrator

Last synced: 17 Dec 2024

https://github.com/alibaba/TePDist

TePDist (TEnsor Program DISTributed) is an HLO-level automatic distributed system for DL models.

auto-parallelization compiler deep-learning disthlo distributed-computing distributed-systems distributed-training high-performance-computing machine-learning rhino

Last synced: 04 Apr 2025

https://github.com/slai-labs/get-beam

Run GPU inference and training jobs on serverless infrastructure that scales with you.

artificial-intelligence cloud-computing cost-optimization data-science deep-learning distributed-computing gpu-acceleration gpu-computing hpc llm-serving llm-training machine-learning ml-infrastructure mlops python serverless serverless-architectures

Last synced: 18 Apr 2025

https://github.com/futureverse/doFuture

:rocket: R package: doFuture - Use Foreach to Parallelize via Future Framework

batchjobs batchtools biocparallel cran distributed-computing foreach hpc hpc-clusters package parallel plyr r

Last synced: 20 Dec 2024

https://github.com/HenrikBengtsson/doFuture

:rocket: R package: doFuture - Use Foreach to Parallelize via Future Framework

batchjobs batchtools biocparallel cran distributed-computing foreach hpc hpc-clusters package parallel plyr r

Last synced: 04 Dec 2024

https://github.com/HenrikBengtsson/future.batchtools

:rocket: R package future.batchtools: A Future API for Parallel and Distributed Processing using batchtools

distributed-computing hpc job-scheduler package parallel pbs r sge slurm torque

Last synced: 04 Dec 2024

https://github.com/mahmoudparsian/pyspark-algorithms

PySpark Algorithms Book: https://www.amazon.com/dp/B07X4B2218/ref=sr_1_2

algorithms big-data data data-abstractions data-science dataframe distributed-computing graphframes mapreduce monoid nosql partitioning pyspark pyspark-algorithms python rdd spark transformations

Last synced: 07 Apr 2025

https://github.com/henrikbengtsson/dofuture

:rocket: R package: doFuture - Use Foreach to Parallelize via Future Framework

batchjobs batchtools biocparallel cran distributed-computing foreach hpc hpc-clusters package parallel plyr r

Last synced: 08 Dec 2024

https://github.com/futureverse/future.batchtools

:rocket: R package future.batchtools: A Future API for Parallel and Distributed Processing using batchtools

distributed-computing hpc job-scheduler package parallel pbs r sge slurm torque

Last synced: 09 Apr 2025

https://github.com/henrikbengtsson/future.batchtools

:rocket: R package future.batchtools: A Future API for Parallel and Distributed Processing using batchtools

distributed-computing hpc job-scheduler package parallel pbs r sge slurm torque

Last synced: 08 Dec 2024

https://github.com/RBigData/remoter

Control a remote R session from your local R session.

distributed-computing pbdr r zeromq

Last synced: 04 Dec 2024

https://github.com/asynkron/protoactor-python

Proto Actor - Ultra fast distributed actors

actor-model actors akka clustering cross-platform distributed-computing protobuf

Last synced: 12 May 2025

https://github.com/futureverse/dofuture

:rocket: R package: doFuture - Use Foreach to Parallelize via Future Framework

batchjobs batchtools biocparallel cran distributed-computing foreach hpc hpc-clusters package parallel plyr r

Last synced: 07 Apr 2025

https://github.com/alibaba/easydist

Automated Parallelization System and Infrastructure for Multiple Ecosystems

automatic-parallelization deep-learning diffusion-models distributed-computing graph-neural-networks high-performance-computing large-language-models llama llm machine-learning

Last synced: 05 Apr 2025

https://github.com/turingcell/turingcell

TuringCell: Run Linux over Paxos/Raft

consensus distributed-computing distributed-systems paxos raft

Last synced: 20 Apr 2025

https://github.com/ehsanmok/spark-lp

Distributed Linear Programming Solver on top of Apache Spark

distributed-computing distributed-optimization high-performance linear-programming scala spark

Last synced: 10 Jan 2025

https://github.com/stellar-group/phylanx

An Asynchronous Distributed C++ Array Processing Toolkit

array-processing c-plus-plus cpp distributed distributed-computing hpx

Last synced: 03 Apr 2025

https://github.com/modular-ml/wrapyfi

Python Multi-Middleware Wrapper for Robotics MOM and RPC

0mq distributed-computing flask-socketio middleware mqtt p2p parallel-computing python robotics ros ros-galactic ros-humble ros-noetic ros2 socketio websocket wrapper yarp zenoh zeromq

Last synced: 09 Apr 2025

https://github.com/actonlang/acton

The Acton Programming Language

actor-model compiler distributed-computing hacktoberfest language programming-language

Last synced: 25 Nov 2024

https://github.com/attumm/redis-dict

Python dictionary with Redis as backend, built for large datasets. Simplifies Redis operations for large-scale and distributed systems. Supports various data types, namespacing, pipelining, and expiration.

dictionary distributed-computing distributed-systems python pythonic redis-dict

Last synced: 09 Apr 2025

https://github.com/thomasweise/distributedcomputingexamples

Example codes for my Distributed Computing course at Hefei University.

axis2 c communication distributed-computing glassfish hadoop html java java-rmi java-servlet javascript javaserver-pages json-rpc jsp mpi servlet-container socket web-services xml xml-document

Last synced: 07 May 2025

https://github.com/artyom-poptsov/guile-ssh

Guile-SSH is a library that provides access to the SSH protocol for GNU Guile programs.

c distributed-computing guile libssh lisp scheme ssh

Last synced: 15 Apr 2025

https://github.com/ow2-proactive/scheduling

Multi-platform Scheduling and Workflows Engine

activeeon distributed-computing java orchestration-framework proactive proactive-workflows resource-manager scheduler scheduling

Last synced: 06 Apr 2025

https://github.com/jgujerry/python-frameworks

Another curated list of Python frameworks

api artificial-intelligence cms data-workflow deep-learning devops distributed-computing enterprise-integrations frameworks machine-learning messaging parallel-computing pipeline python task-queue web-crawling webapp

Last synced: 26 Mar 2025

https://github.com/tf-encrypted/moose

Secure distributed dataflow framework for encrypted machine learning and data processing

cryptography data-science distributed-computing machine-learning privacy secure-computation

Last synced: 08 May 2025

https://github.com/capillariesio/capillaries

Distributed batch data processing framework

batch-processing cassandra dag distributed-computing distributed-systems go golang rabbitmq relational-algebra workflow-engine workflows

Last synced: 07 May 2025

https://github.com/in03/proxima

Transcode source media directly from DaVinci Resolve using multiple machines for encoding. Great for creating proxies quickly.

celery cli-app davinci-resolve distributed-computing parallel-computing python redis video-processing

Last synced: 09 May 2025

https://github.com/saturncloud/dask-pytorch-ddp

dask-pytorch-ddp is a Python package that makes it easy to train PyTorch models on dask clusters using distributed data parallel.

computer-vision dask deep-learning distributed-computing machine-learning nlp pytorch

Last synced: 08 May 2025

https://github.com/juliaparallel/slurmclustermanager.jl

Julia package for running code on Slurm clusters

distributed-computing julia slurm

Last synced: 05 May 2025

https://github.com/masa16/pwrake

Parallel Workflow extension for Rake, runs on multicores, clusters, clouds.

cluster distributed-computing gfarm parallel parallel-computing pwrake rake ruby scientific-computing workflow

Last synced: 13 May 2025

https://github.com/JuliaParallel/SlurmClusterManager.jl

Julia package for running code on Slurm clusters

distributed-computing julia slurm

Last synced: 31 Mar 2025

https://github.com/opencyphal/yakut

Simple CLI tool for diagnostics and debugging of Cyphal networks

aerospace cli command-line-tool cyphal data-distribution diagnostics distributed-computing drone dsdl embedded ethernet hacktoberfest networking opencyphal pubsub real-time robotics rpc uavcan vehicular-networks

Last synced: 07 May 2025

https://github.com/explosion/spacy-ray

☄️ Parallel and distributed training with spaCy and Ray

distributed-computing machine-learning natural-language-processing parallel-training ray spacy training

Last synced: 04 Feb 2025

https://github.com/cjus/hydra-cluster

Hydra compute cluster project

cluster distributed-computing hydra hydra-cluster microservice microservices raspberry raspberry-pi sbc

Last synced: 21 Nov 2024

https://github.com/chris-piekarski/python-json-socket

JSON messaging based socket interface with multi-threaded server and client

distributed-computing distributed-systems json networking python threading

Last synced: 29 Mar 2025

https://github.com/ledjon-behluli/orleanspaces

A virtual, fully-asynchronous, Tuple Space implementation backed by Orleans.

concurrent-programming distributed-computing dotnet orleans sba tuple-space

Last synced: 22 Nov 2024

https://github.com/ashwanthkumar/suuchi

सूचि - Toolkit to build Distributed Data Systems

distributed-computing rocksdb scala toolkit

Last synced: 11 Apr 2025

https://github.com/emadehsan/marsjs

MobileNet on TensorFlow.js - label images from Unsplash in browser

chrome-extension deep-learning distributed-computing image-labeling tensorflow-js volunteer-computing

Last synced: 01 May 2025

https://github.com/natefoo/slurm-drmaa

DRMAA for Slurm: Implementation of the DRMAA C bindings for Slurm

cluster clusters distributed-computing drmaa hpc resource-management slurm slurm-job-scheduler

Last synced: 05 Apr 2025

https://github.com/ml-tooling/lazycluster

🎛 Distributed machine learning made simple.

cluster dask distributed-computing hyperopt machine-learning python ssh

Last synced: 25 Nov 2024

https://github.com/bsc-wdc/compss

COMP Superscalar (COMPSs) is a framework which aims to ease the development and execution of applications for distributed infrastructures, such as Clusters, Grids and Clouds.

c distributed-computing docker hpc java pipeline-framework python singularity slurm workflow-management-system workflows

Last synced: 05 Apr 2025

https://github.com/jonathandinu/spark-ray-data-science

Supporting content (slides and exercises) for the Pearson video series covering best practices for developing scalable applications with Spark and Ray in the context of a data scientist's standard workflow.

artificial-intelligence data-science distributed-computing machine-learning python ray spark

Last synced: 08 May 2025

https://github.com/dcfjs/dcf

Yet another distributed compute framework

distributed-computing nodejs spark

Last synced: 09 Apr 2025

https://github.com/ambientsprotocol/whitepaper

📄 The Ambients protocol white paper

ambient-calculus decentralized distributed-computing functional-programming p2p process-calculus

Last synced: 06 Dec 2024

https://github.com/flatironinstitute/disbatch

Dynamic dispatch of a list of command-line tasks, locally or on a cluster. Supports retrying failed tasks, and adding/removing compute resources on-the-fly.

batch-processing distributed-computing

Last synced: 07 Apr 2025