Data

Tools

Indexes

Applications

Experiments

Awesome

An open API service indexing awesome lists of open source software.

Projects in Awesome Lists tagged with batch-processing

A curated list of projects in awesome lists tagged with batch-processing .

https://github.com/naibowang/easyspider

A visual no-code/code-free web crawler/spider易采集：一个可视化浏览器自动化测试/数据采集/爬虫软件，可以无代码图形化的设计和执行爬虫任务。别名：ServiceWrapper面向Web应用的智能化服务封装系统。

batch-processing batch-script code-free crawler data-collection frontend gui html input-parameters layman parameters robotics rpa scraper spider visual visualization visualprogramming web www

Last synced: 12 May 2025

https://github.com/NaiboWang/EasySpider

A visual no-code/code-free web crawler/spider易采集：一个可视化浏览器自动化测试/数据采集/爬虫软件，可以无代码图形化的设计和执行爬虫任务。别名：ServiceWrapper面向Web应用的智能化服务封装系统。

batch-processing batch-script code-free crawler data-collection frontend gui html input-parameters layman parameters robotics rpa scraper spider visual visualization visualprogramming web www

Last synced: 20 Mar 2025

https://github.com/firecrawl/firecrawl-mcp-server

🔥 Official Firecrawl MCP Server - Adds powerful web scraping and search to Cursor, Claude and any other LLM clients.

batch-processing claude content-extraction data-collection firecrawl firecrawl-ai javascript-rendering llm-tools mcp mcp-server model-context-protocol search-api web-crawler web-scraping

Last synced: 13 Nov 2025

https://github.com/spring-projects/spring-batch

Spring Batch is a framework for writing batch applications using Java and Spring

batch batch-processing java spring

Last synced: 12 May 2025

https://github.com/faastjs/faast.js

Serverless batch computing made simple.

batch batch-processing faas javascript library serverless typescript

Last synced: 14 May 2025

https://github.com/spring-attic/spring-cloud-dataflow

A microservices-based Streaming and Batch data processing in Cloud Foundry and Kubernetes

batch-processing cloud-native datapipelines microservices-architecture orchestration predictive-analytics stream-processing

Last synced: 14 Jun 2025

https://github.com/spring-cloud/spring-cloud-dataflow

A microservices-based Streaming and Batch data processing in Cloud Foundry and Kubernetes

batch-processing cloud-native datapipelines microservices-architecture orchestration predictive-analytics stream-processing

Last synced: 12 May 2025

https://github.com/iusztinpaul/energy-forecasting

🌀 𝗧𝗵𝗲 𝗙𝘂𝗹𝗹 𝗦𝘁𝗮𝗰𝗸 𝟳-𝗦𝘁𝗲𝗽𝘀 𝗠𝗟𝗢𝗽𝘀 𝗙𝗿𝗮𝗺𝗲𝘄𝗼𝗿𝗸 | 𝗟𝗲𝗮𝗿𝗻 𝗠𝗟𝗘 & 𝗠𝗟𝗢𝗽𝘀 for free by designing, building and deploying an end-to-end ML batch system ~ 𝘴𝘰𝘶𝘳𝘤𝘦 𝘤𝘰𝘥𝘦 + 2.5 𝘩𝘰𝘶𝘳𝘴 𝘰𝘧 𝘳𝘦𝘢𝘥𝘪𝘯𝘨 & 𝘷𝘪𝘥𝘦𝘰 𝘮𝘢𝘵𝘦𝘳𝘪𝘢𝘭𝘴

3-pipeline-design airflow batch-processing cicd data-versioning docker fastapi feature-store gcp github-actions great-expectations hopsworks ml-monitoring mlops model-registry poetry python sktime streamlit weights-and-biases

Last synced: 15 May 2025

https://github.com/alessandrofrancesconi/gimp-plugin-bimp

BIMP. Batch Image Manipulation Plugin for GIMP.

batch-processing c gimp image-manipulation plugin

Last synced: 20 Mar 2025

https://github.com/j-easy/easy-batch

The simple, stupid batch framework for Java

batch batch-processing java

Last synced: 16 Dec 2025

https://github.com/guillermo-navas-palencia/optbinning

Optimal binning: monotonic binning with constraints. Support batch & stream optimal binning. Scorecard modelling and counterfactual explanations.

batch-processing binning counterfactual-explanations credit-scoring mdlp optimization python scorecard stream streaming-data woe woebinning

Last synced: 15 Dec 2025

https://github.com/spring-cloud/spring-cloud-task

Short lived microservices with Spring Batch

batch-processing microservices short-lived-microservices

Last synced: 23 Jun 2025

https://github.com/gdquest/krita-batch-exporter

A Free Krita plugin to batch export layers and groups with maximum flexibility. Scale, trim, export multiple copies of your layers...

batch-processing exporter game-development graphic-design image-processing krita krita-plugin

Last synced: 06 Apr 2025

https://github.com/nilenso/goose

A powerful background job processing library for Clojure

asynchronous background-processing batch-processing clojure rabbitmq redis scheduler

Last synced: 08 May 2025

https://github.com/tintinweb/ida-batch_decompile

*Decompile All the Things* - IDA Batch Decompile plugin and script for Hex-Ray's IDA Pro that adds the ability to batch decompile multiple files and their imports with additional annotations (xref, stack var size) to the pseudocode .c file

batch-processing decompile ida ida-batch-decompile ida-plugin reverse-engineering

Last synced: 09 Apr 2025

https://github.com/btargac/excel-parser-processor

Automate downloads from Excel files in seconds. Simply does the tedious, repetitive operations for rows of Excel files and reports the results. It downloads files from URL(s) in column A, if a new filename is provided at column B it will rename before saving. It will even create sub folders if column C is filled with a valid folder name.

autodownloader batch batch-download batch-processing download downloader electron electronjs excel excel-based excel-download excel-import excel-parse exceldownload filedownloader imagedownloader nodejs rename subfolder

Last synced: 12 Apr 2025

https://github.com/apple/batch-processing-gateway

The gateway component to make Spark on K8s much easier for Spark users.

batch-processing k8s kubernetes spark

Last synced: 13 Apr 2025

https://github.com/yornaath/batshit

A batch manager that will deduplicate and batch requests for a certain data type made within a window. Useful to batch requests made from multiple react components that uses react-query

async batch-processing concurrency deduplication fetch react react-query tanstack typescript

Last synced: 04 Apr 2025

https://github.com/pszemraj/textsum

CLI & Python API to easily summarize text-based files with transformers

batch-processing inference inference-api pipeline summarization summary text text-to-text-transformer transformer transformers

Last synced: 04 Apr 2025

https://github.com/jberet/jsr352

Implementation of Jakarta Batch Specification and API

batch batch-processing jakarta java

Last synced: 14 May 2025

https://github.com/addyosmani/squish

Squish - Batch Browser-based Image Compression

batch batch-processing image-compression image-processing

Last synced: 27 Aug 2025

https://github.com/Refefer/Dampr

Python Data Processing library

batch-processing dataflow machine-learning mapreduce

Last synced: 19 Jul 2025

https://github.com/opencultureconsulting/openrefine-batch

Shell script to run OpenRefine in batch mode (import, transform, export). It orchestrates OpenRefine (server) and a python client that communicates with the OpenRefine API.

bash-script batch-processing code4lib docker etl openrefine

Last synced: 08 Apr 2025

https://github.com/seznam/euphoria

Euphoria is an open source Java API for creating unified big-data processing flows. It provides an engine independent programming model which can express both batch and stream transformations.

apache-flink apache-spark batch-processing big-data hadoop hdfs java-api kafka streaming-data unified-bigdata-processing

Last synced: 21 Aug 2025

https://github.com/scripbox/flume

A blazing fast job processing system backed by GenStage & Redis.

background-jobs batch-processing concurrent elixir-lang genstage rate-limiting redis scheduled-jobs

Last synced: 30 Mar 2025

https://github.com/furyolo/mistralocr

该仓库是一个基于Mistral API的文档识别工具，支持处理PDF和图片文件（如JPG、JPEG、PNG）。它提供图形用户界面和命令行界面，能够自动保存处理结果为Markdown格式，并支持配置文件管理和批量处理文件

batch-processing command-line document-recognition gui markdown mistral-api ocr python

Last synced: 15 Aug 2025

https://github.com/runlyio/core-dotnet

Multi-threaded batch processing and background jobs for .NET Core

background-jobs batch-processing job-queue job-scheduler multi-threading

Last synced: 14 Aug 2025

https://github.com/tacc/launcher

A simple utility for executing multiple sequential or multi-threaded applications in a single multi-node batch job

batch-processing heterogeneous hpc launcher parametric-submission shell tacc xsede

Last synced: 30 Jul 2025

https://github.com/flipkart-incubator/batchman

This library for Android will take any set of events and batch them up before sending it to the server. It also supports persisting the events on disk so that no event gets lost because of an app crash. Typically used for developing any in-house analytics sdk where you have to make a single api call to push events to the server but you want to optimize the calls so that the api call happens only once per x events, or say once per x minutes. It also supports exponential backoff in case of network failures

analytics android-library batch-manager batch-processing batching java job-scheduler networking persistence persistence-strategies retry serialization strategies

Last synced: 21 Apr 2025

https://github.com/taskforcesh/bullmq-proxy

HTTP based proxy for accessing BullMQ Queues

batch-processing jobscheduler queue-service queues

Last synced: 03 May 2025

https://github.com/kieranjol/IFIscripts

Detailed documentation is available here: http://ifiscripts.readthedocs.io/en/latest/index.html

batch-processing dcp digital-preservation ffv1 manifest metadata premis tiff-files transcode

Last synced: 18 Jul 2025

https://github.com/flatironinstitute/disbatch

Dynamic dispatch of a list of command-line tasks, locally or on a cluster. Supports retrying failed tasks, and adding/removing compute resources on-the-fly.

batch-processing distributed-computing

Last synced: 08 Sep 2025

https://github.com/pathway-labs/pathway-examples

Code examples for the Pathway programming framework

batch-processing examples kafka pathway python streaming

Last synced: 03 May 2025

https://github.com/aureliojargas/replace

Generic file search & replace tool (python)

batch-processing find-and-replace python3 regex

Last synced: 29 Mar 2025

https://github.com/raycad/stream-processing

Stream processing guidelines and examples using Apache Flink and Apache Spark

apache-flink apache-spark batch-processing data-analysis streaming

Last synced: 13 Jul 2025

https://github.com/josef-friedrich/mscxyz

Manipulate the XML based *.mscx and *.mscz files of the notation software MuseScore.

batch-processing musescore python python-3

Last synced: 19 Aug 2025

https://github.com/mersinvald/batch_resolve

Fast asynchronous DNS batch resolver

async batch-processing dns dns-client futures resolve rust tokio

Last synced: 28 Jul 2025

https://github.com/basvandijk/monad-batcher

An applicative monad that batches commands for later more efficient execution

batch-processing haskell-library monad

Last synced: 20 Mar 2025

https://github.com/tychenjiajun/art

AI-PP3 is a command-line tool that uses artificial intelligence to analyze RAW photos and generate optimized processing profiles (PP3 files) for RawTherapee.

ai arw batch-processing cli computer-vision cr2 dng image-processing multimodal nef photo-editing photography pp3 raw rawtherapee

Last synced: 11 Jun 2025

https://github.com/comcast/python-batch-runner

A tiny framework for building batch applications as a collection of tasks in a workflow.

batch batch-processing microframework python tiny-framework workflow-engine

Last synced: 03 Oct 2025

https://github.com/Comcast/python-batch-runner

A tiny framework for building batch applications as a collection of tasks in a workflow.

batch batch-processing microframework python tiny-framework workflow-engine

Last synced: 18 Jul 2025

https://github.com/dylanpieper/chatalot

Process a Lot of LLM Chats

batch batch-processing ellmer llm package r

Last synced: 27 Jul 2025

https://github.com/hussein-awala/async-batcher

A service to batch the http requests.

asyncio batch-processing fastapi grpc http-requests python

Last synced: 18 Sep 2025

https://github.com/vallahaye/go-batcher

Zero dependency generic Golang batcher

batch-processing generics go golang

Last synced: 13 Apr 2025

https://github.com/dogweather/raspberry-pi-python-github-runner

Sample configuration for using Raspberry Pi's as Python Github Runners

batch-processing ci distributed-systems github-actions

Last synced: 11 Nov 2025

https://github.com/0xibra/docker-jobs-bundle

Symfony bundle that offers a batch processing system with Docker containers.

batch-processing docker php-jobs symfony symfony-bundle

Last synced: 21 Apr 2025

https://github.com/0xIbra/docker-jobs-bundle

Symfony bundle that offers a batch processing system with Docker containers.

batch-processing docker php-jobs symfony symfony-bundle

Last synced: 14 Jun 2025

https://github.com/shixiangwang/ezcox

Easily Process a Batch of Cox Models

batch-processing cox-model r-package

Last synced: 12 Apr 2025

https://github.com/imazen/imageflow-go

Go bindings for Imageflow -> https://github.com/imazen/imageflow

batch-processing fast golang http image-processing rust

Last synced: 31 Aug 2025

https://kamilburda.github.io/batcher/

Batch Layer Processing and Export for GIMP 3.0.0-RC2 and Later

batch-processing gimp gimp-plugin gimp-plugins python

Last synced: 11 Apr 2025

https://github.com/kamilburda/batcher

Batch Layer Processing and Export for GIMP 2.99.18

batch-processing gimp gimp-2-99 gimp-plugin python

Last synced: 10 Jul 2025

https://github.com/mikestefanello/batcher

Type-safe, automatic, asynchronous batch processing.

batch batch-processing concurrency data goroutines

Last synced: 26 Sep 2025

https://github.com/romainmenke/simple-guetzli

Batch Guetzli compressions in a manageable fashion

batch-processing compressed guetzli jpeg-encoder

Last synced: 23 Mar 2025

https://github.com/photostructure/batch-cluster.js

Parallelized and efficient Node.js support for batch-mode child processes

batch-cluster batch-mode batch-processing nodejs parallelism spawn spawned-processes

Last synced: 10 May 2025

https://github.com/ksm26/efficiently-serving-llms

Learn the ins and outs of efficiently serving Large Language Models (LLMs). Dive into optimization techniques, including KV caching and Low Rank Adapters (LoRA), and gain hands-on experience with Predibase’s LoRAX framework inference server.

batch-processing deep-learning-techniques inference-optimization large-scale-deployment machine-learning-operations model-acceleration model-inference-service model-serving optimization-techniques performance-enhancement scalability-strategies server-optimization serving-infrastructure text-generation

Last synced: 02 Aug 2025

https://github.com/ryanjoachim/mcp-batchit

🚀 MCP aggregator for batching multiple tool calls into a single request. Reduces overhead, saves tokens, and simplifies complex operations in AI agent workflows.

ai-agents ai-tools automation batch-processing developer-tools llm-tools mcp model-context-protocol

Last synced: 17 Jun 2025

https://github.com/joshpaulie/nanoencoder

A simple FFmpeg-powered batch re-encoder for h.265 (HEVC)

batch-processing ffmpeg ffmpeg-wrapper hevc-encoder python

Last synced: 03 Jul 2025

https://github.com/jftuga/photo_id_resizer

Resize photo ID images using face recognition technology

batch-processing face-detection face-recognition golang image-processing photos profile-picture

Last synced: 26 Jul 2025

https://github.com/brutusin/wava

Linux batch scheduler supporting job memory usage limits

batch-processing cgroups job-queue job-scheduler linux scheduler

Last synced: 10 Apr 2025

https://github.com/elgopher/batch

Simple Go package for handling incoming requests in batches.

batch batch-processing go golang high-performance high-throughput

Last synced: 25 Dec 2025

https://github.com/shadawck/lrjson

Transform Lightroom standard file (like lrtemplate or xmp) into manipulable json file

adobe batch-processing export filter json lightroom lrtemplate photo preset

Last synced: 13 Apr 2025

https://github.com/aymane-maghouti/big-data-project

This project aims to predict smartphone prices using a combination of batch and stream processing techniques in a Big Data environment. The architecture follows the Lambda Architecture pattern, providing both real-time and batch processing capabilities to users.

apache-airflow apache-kafka apache-spark batch-processing big-data-projects hbase hdfs ingestion java lambda-architecture machine-learning postgresql-database powerbi pyspark python spring-boot streaming

Last synced: 29 Oct 2025

https://github.com/nathadriele/mlops-zoomcamp

The Zoomcamp MLOps Course covers tools like MLflow, Mage, Flask, Prometheus, Evidently, Grafana, Prefect, Terraform, and GitHub Actions. It emphasizes experiment tracking, model deployment, monitoring, CI/CD, and orchestration, culminating in an end-to-end project integrating best practices in MLOps.

aws batch-processing flask grafana mage-ai mlflow mlops model mongodb monitoring orchestration prefect streaming

Last synced: 10 Apr 2025

https://github.com/cmeeren/batchit

Call batched functions as normal functions

batch-processing batching fsharp

Last synced: 06 Apr 2025

https://github.com/thechymera/repositorg

Automatically reposit, organize, rename, and process large collections of files.

batch-processing file-management multimedia organizer

Last synced: 19 Apr 2025

https://github.com/brakmic/using-thrill

A complete Thrill environment with demo code

batch-processing big-data cpp cpp14 thrill

Last synced: 23 Apr 2025

https://github.com/ThaddeusJiang/nextjs-12-starter

You shouldn't start a wonderful app from scratch every time.

admin-system batch-processing e2e-testing job-scheduler nextjs storybook terraform

Last synced: 11 Mar 2025

https://github.com/mazzzystar/focus

Chrome Extension: One-click to batch open websites, double-click to close them.

batch-processing chrome-extension

Last synced: 28 Apr 2025

https://dylanpieper.github.io/hellmer/

Batch Processing for Chat Models

batch batch-processing ellmer llm package r

Last synced: 06 Apr 2025

https://github.com/prechy123/batch-tools

Batch tools provides various tools which are easy and free to use to work with files completely free and easy to use. Convert document format, background removal, qr code generator, video transcriber, pdf merger, html to pdf, pdf to jpg, json to csv and more!

batch-processing clean-code file-converter frontend image-converter image-processing nextjs14 pdf-tools tailwindcss vercel

Last synced: 15 Jul 2025