Data

Tools

Indexes

Applications

Experiments

Awesome

An open API service indexing awesome lists of open source software.

Projects in Awesome Lists tagged with batch-processing

A curated list of projects in awesome lists tagged with batch-processing .

https://github.com/phungvandat/calculate-ccu

Example calculate current ccu

batch-processing ccu golang hyperloglog redis

Last synced: 18 May 2026

https://github.com/ikauedev/spring-batch-examples

This repository contains a structured collection of modular examples developed with Spring Batch, with the aim of facilitating learning, prototyping and demonstrating real batch processing scenarios.

batch batch-processing

Last synced: 12 Sep 2025

https://github.com/dohabanoui/batch-processing-springbatch

L'objectif est de créer un job Spring Batch qui traite des commandes à partir d'un fichier CSV.

batch-job batch-processing spring-batch

Last synced: 21 Mar 2025

https://github.com/nicholaswmin/job-sequencer

Run strictly sequential jobs in Node.js whilst emitting W3C ProgressEvents

batch-processing node-js w3c-progress-events

Last synced: 19 May 2026

https://github.com/landerox/cloud-landerox-data

Reference architecture baseline for GCP data platforms (Apache Beam, BigQuery, Cloud Functions, Pub/Sub). Hybrid warehouse/lakehouse with batch + streaming, Medallion layering. Consumed by private runtime repos.

apache-beam batch-processing bigquery cloud-functions cloud-storage data-engineering data-platform dataform gcp google-cloud-dataflow iceberg lakehouse medallion-architecture opentelemetry pubsub python reference-architecture slsa streaming supply-chain-security

Last synced: 21 May 2026

https://github.com/nairgh/spring.batch

spring boot batch processing example

batch-processing spring spring-boot

Last synced: 02 Aug 2025

https://github.com/jesufemi-o/grokking-dlt-kafka

repo showing example of how to go from proof of concept to prod-ready setup with dlt for kafka sources

batch-processing dlthub etl-framework kafka

Last synced: 17 Aug 2025

https://github.com/sukitsubaki/nomino

Smart file renaming tool, that supports a variety of renaming strategies.

automation batch-processing bulk-rename-images cli file-renamer open-source python-script

Last synced: 20 Aug 2025

https://github.com/kostrykin/repype

Reproducible, efficient, flexible batch processing using pipelines for sustainable software experiments

batch-processing experiment-design pipeline python python-workflow scientific-computing sustainability sustainable-computing

Last synced: 24 Aug 2025

https://github.com/rkalnins/cdo-batch

Apply CDO operations to batches of files and manage output file locations

batch-processing cdo climate-data

Last synced: 07 Apr 2025

https://github.com/thevinh-ha-1710/airflow-weather-data-pipeline

An batch processing ETL data pipeline that collects real-time weather data for analysis and forecast

analysis batch-processing etl-pipeline weather-forecast

Last synced: 06 Mar 2025

https://github.com/ef2k/tempo

The time driven batch queue

batch-processing go golang queue

Last synced: 12 Jul 2025

https://github.com/victor-antoniassi/day-1_sales_data_generator

Generate daily sales data for the Chinook database. Perfect for building data engineering portfolios with realistic, continuously updating transactional data. Supports D-1 batch processing patterns used in production.

batch-processing chinook-database data-engineering data-generator neon neondb postgresql sample-database synthetic-data

Last synced: 16 May 2026

https://github.com/phac-nml/nf-sequenoscope

Streamlined Nextflow wrapper for the Sequenoscope toolkit. Simplifies complex metagenomic workflows with automated batch processing, allowing efficient comparative analysis from raw reads to visualization.

adaptive-sampling batch-processing data-visualization dsl2 metagenomics nextflow sequenoscope

Last synced: 16 Jan 2026

https://github.com/inc44/belilo

Belilo, which translates to 'whitewasher' in Russian, is a useful tool created with ❤️ using Rust. It quickly whitens images, providing a clean, uniform appearance. It's fast, efficient, and precise.

batch-processing cli image-optimization image-processing rust windows

Last synced: 12 May 2026

https://github.com/inc44/png_ect

PNG ECT, standing for PNG Efficient Compression Tool, is a useful tool created with ❤️ using Rust. It compresses PNG images quickly and efficiently, making them lighter without compromising quality. It's fast, efficient, and precise.

batch-processing cli image-optimization image-processing performance rust windows

Last synced: 19 May 2026

https://github.com/kamadulski/mp4_audio_extractor

A simple Python application to extract audio tracks (MP3 or AAC) from MP4 video files, supporting single file and batch processing with both GUI and Command-Line interfaces.

aac audio-extraction batch-processing cli ffmpeg gui mp3 mp4 python video-to-audio

Last synced: 02 Jul 2025

https://github.com/noarche/imagetoolaio

This script processes images in a specified directory. It can crop images, remove or retain metadata, save in a different format, compress, resize, and add metadata. It applies these operations to all images in the directory without needing confirmation for each file.

batch-processing exiftool metadata-editor photo-compressor photo-converter photo-crop photo-editing photo-resizer

Last synced: 25 Mar 2025

https://github.com/noarche/autocrop

Simple tool that batch crops 'to-fit' all transparent PNG or WEBP files. Windows Executable Available.

batch-processing crop-image image-crop image-cropping image-manipulation image-processing

Last synced: 25 Mar 2025

https://github.com/narius2030/lakehouse-solution-imcp

An end-to-end MLOps pipeline to develop, train, and deploy an Image Caption model that automatically generates captions for images based on diverse datasets

apache-airflow apache-kafka batch-processing lakehouse mlflow-tracking mlops polars spark-streaming stream-processing

Last synced: 28 Feb 2025

https://github.com/jwodder/forall

Operate on each project in a directory

batch-processing git repository-management rust

Last synced: 02 Apr 2025

https://github.com/brewkits/flutter_debounce_throttle

The Traffic Control System for your App Architecture. Unifies debounce, throttle, rate limiting & async concurrency control. Like ABS brakes for your app. Zero dependencies, 360+ tests, runs everywhere.

api-rate-limit async-debounce backpressure batch-processing concurrency-control dart dart-package debounce double-tap-prevention event-throttling flutter flutter-package pub-dev rate-limiter rate-limiting throttle token-bucket

Last synced: 21 Jan 2026

https://github.com/strmprivacy/blogpost-dss

An example notebook to work with STRM Privacy batch jobs and the Data Subjects API to easily retrieve data for Data Subject Requests (like a DSAR)

batch-processing gdpr gdpr-compliant privacy privacy-enhancing-technologies privacy-tools

Last synced: 23 Oct 2025

https://github.com/x1ao4/image-match-renamer

通过 python 脚本使用图片文件名与文本数据进行匹配，并按照匹配的数据重命名图片 / renaming images according to matched text data via python script

batch-processing batch-rename image-renamer image-renaming images matching rename rename-files rename-script renamer

Last synced: 30 Jan 2026

https://github.com/dadananjesha/azuredataengine

AzureDataEngine is a robust, scalable batch processing data architecture built on the Azure platform. It efficiently extracts, transforms, and loads massive datasets for machine learning applications, leveraging Azure Blob Storage, PostgreSQL, Databricks, and Key Vault to ensure reliability and maintainability.

azure batch-processing blob-storage databricks etl etl-framework key-vault postgresql-database spark vnet

Last synced: 15 Apr 2026

https://github.com/timgels/matroska-batch-flow

Matroska Batch Flow is a tool for batch processing Matroska files with ease. Quickly modify container properties across multiple files and streamline your workflow.

batch-processing cross-platform csharp dotnet matroska-files mkv uno-platform

Last synced: 01 Feb 2026

https://github.com/crper/py-image-compress-mcp

High-performance MCP image compression service for AI assistants with intelligent optimization and batch processing capabilities

batch-processing claude compression image-processing mcp mcp-server optimization pillow

Last synced: 15 Apr 2026

https://github.com/amirthfultehrani/browser-image-converter

Browser-based image converter, resizer, cropper, and rotator. No server uploads -- all processing happens locally! Built with HTML, Tailwind CSS, and Javascript. Supports PNG, JPEG, WEBP, GIF, and BMP.

batch-processing browser-based client-side html5 image-converter image-cropper image-manipulation image-processing image-resizer image-rotator javascript jpeg-converter mit-license no-server-upload open-source png-converter privacy-focused tailwindcss web-app web-development

Last synced: 16 Apr 2026

https://github.com/andrebfarias/conversor-video-para-ascii

Ferramenta de código aberto para conversão de arquivos de vídeo em arte ASCII usando OpenCV e processamento de chroma key. Suporta conversão em lote, reprodução em terminal com looping opcional e configuração personalizável via config.ini.

ascii-art batch-processing chroma-key configparser numpy open-sources-code-github opencv-python terminal-rendering video-conversion

Last synced: 19 Apr 2026

https://github.com/mrdavearms/bulk-pdf-extractor-and-generator

Batch-fill PDF forms from spreadsheet data — a Windows/Mac desktop app for educators and school leaders. No coding required.

australia batch-processing departmentofeducationvictoria desktop-app education pdf pdf-form-filler pdf-generator pyinstaller school-admin tkinter vcaa victoria

Last synced: 31 May 2026

https://github.com/rafalkaron/markup

Batch-convert Markdown and HTML files.

batch-processing cli-app dita html5 markdown md python3

Last synced: 18 Apr 2026

https://github.com/danielsan80/jobboy

JobBoy is little JobMan (JobManager), it manages batch processes

batch-processing jobboy packagist

Last synced: 21 Apr 2026

https://github.com/danielsan80/jobboy-example

A Symfony4 example project using JobBoy

batch-processing jobboy

Last synced: 21 Apr 2026

https://github.com/narius2030/mlops-image-captioning

An end-to-end MLOps pipeline to develop, train, and deploy an Image Caption model that automatically generates captions for images based on diverse datasets

apache-airflow apache-kafka batch-processing lakehouse mlflow-tracking mlops polars spark-streaming stream-processing

Last synced: 29 Apr 2026

https://github.com/aldythnahak/golang_console_delete_table_ms_sql_server

A console-based Go application to back up tables from a Microsoft SQL Server database. Supports selective optional deletion table.

batch-processing console-application golang goroutines mssqlserver sqlserver

Last synced: 29 Apr 2026

https://github.com/manhtdxxx/batch-and-stream-pipeline-via-lakehouse

This project demonstrates a modern Lakehouse architecture supporting both streaming and batch data pipelines, built on Apache Iceberg tables.

airflow batch-processing data-engineering data-visualization docker elt-pipeline hive-metastore iceberg kafka lakehouse medallion-architecture spark stream-processing superset trino

Last synced: 08 May 2026

https://github.com/koushik-elite/batch-processing

Batch Processing example for RNN, using variable batch size and sequence number

batch-processing python python3 pytorch recurrent-neural-networks

Last synced: 12 May 2026

https://github.com/akaliutau/hadoop-cluster

Batch data processing on the dockerized Hadoop cluster

batch-processing hadoop-cluster hdf5 hdfs java mapreduce

Last synced: 14 May 2026

https://github.com/nathannncurtis/ocr

Batch DICOM/TIFF/JPEG to searchable PDF with OCR. Docker, Tesseract, JBIG2, parallel processing.

batch-processing cpp docker ocr pdf tesseract

Last synced: 30 May 2026

https://github.com/faizpuad/dataengineeringproject-scalableolapsystemforcreditcradtransaction

Scalable OLAP system for credit card transaction analysis, leveraging AWS S3, Databricks, and dbt. Features end-to-end batch processing pipeline, medallion architecture, and interactive fraud detection dashboards. Demonstrates expertise in cloud-based data engineering and advanced analytical modeling for financial data.

batch-processing databricks dbt dbt-core pyspark python

Last synced: 13 Apr 2026

https://github.com/alansteinbarth/image-flow

Professional image converter supporting 15+ formats (HEIC→JPEG, PNG, WEBP, RAW). Batch processing, quality control, drag&drop UI, live preview. Cross-platform Python desktop app with modern interface. Portfolio project showcasing GUI development skills.

batch-processing cross-platform desktop-application drag-and-drop file-conversion gui-application heic-converter image-converter image-processing linux macos mit-license modern-ui open-source pillow portfolio-project python tkinter user-interface windows

Last synced: 13 Apr 2026

https://github.com/alchemist-aloha/explicit_util

A utility library for managing media files, especially focused on conversion, organization, and archival.

batch-processing media-management namer nfo-file rename-files transcribe whisper-cpp

Last synced: 30 Mar 2025

https://github.com/andresmorales08/recolor-app-icons

这是一个 Python 脚本，用于批量将应用程序图标重新着色为具有特定基础色调的极简、海报化风格。

batch-processing icons image-manipulation image-processing pillow png posterization python recolor script webp

Last synced: 15 Apr 2025

https://github.com/extractable-hoodedsheldrake431/deepseek_ocr_app

🖼️ Streamline your document processing with DeepSeek OCR, a modern app combining React and FastAPI for fast, accurate text extraction.

batch-processing computer-vision deepseek deepseek-ocr document-analysis fastapi image-recognition modern-ui nix nix-flake ocr ocr-recognition python pytorch responsive-design transformers vllm web-ui

Last synced: 13 Apr 2026

https://github.com/evitanrelta/github-markdown-batch-render

To batch-render Markdown strings via GitHub's API (using Octokit/REST). For mitigating GitHub's API's request rate-limit.

batch-processing batch-rendering gfm github-api github-flavored-markdown github-markdown markdown markdown-renderer octokit

Last synced: 10 Apr 2026

https://github.com/lectrician1/batchhighlightapp

Windows app to highlight specific words in a folder of Word and PDF documents

batch-processing highlighting microsoft-word microsoft-word-automation pdf pdf-highlighter windows-application

Last synced: 14 Mar 2025

https://github.com/alchemist-aloha/explicitutil

A utility library for managing media files, especially focused on conversion, organization, and archival.

batch-processing media-management namer nfo-file rename-files transcribe whisper-cpp

Last synced: 07 Apr 2025

https://github.com/raulil/juokse

Batch processor with Python like syntax

batch-processing shell-scripting

Last synced: 22 Jul 2025

https://github.com/snimmagadda1/spring-batch-rapid-starter

Quick start template for batch pipelines built with spring batch and orchestrated (optionally) with spring cloud dataflow

batch-processing java java11 jvm kubernetes spring-batch spring-boot spring-cloud spring-cloud-dataflow starter-template template

Last synced: 30 Mar 2025

https://github.com/howbizarre/add-stamp

A Nuxt 4 application that allows users to apply watermarks to bunch of images using WebAssembly (WASM) for high-performance image processing. And use Cloudlfare Worker as BaaS.

baas batch-processing browser cloudflare-workers image-processing images nuxt nuxt4 rust stamp tailwindcss typescript vue vue3 wasn watermark webassembly wrangler

Last synced: 11 Apr 2026

https://github.com/jayrbolton/hadrosaur

File management for batch compute work

batch-processing cluster-computing hpc hpc-applications

Last synced: 09 Nov 2025

https://github.com/npow/localbatch

Test AWS Batch workloads locally without an AWS account

aws-batch batch-processing docker fastapi integration-testing local-development metaflow python

Last synced: 23 May 2026

https://github.com/cainky/replacetext

Replaces text based on a dictionary, given user input to specify which direction (keys-to-values or values-to-keys)

batch-processing python text-processing text-replacement utility

Last synced: 27 Feb 2025

https://github.com/ahmed-naserelden/logistics-risk-intelligence

A full-stack data engineering project for real-time maritime and seismic monitoring. Integrates batch and streaming pipelines using Spark, Airflow, Kafka, and Snowflake. Enables analytics and dashboards for logistics, risk, and port operations.

batch-processing big-data hadoop hive lambda-architecture ods real-time-streaming snowflake spark

Last synced: 28 Jul 2025

https://github.com/meliheran/bufferq

A thread safe buffer queue mechanism that can be processed with multi thread logic. Usefull for periodicly loading batch records from database or etc. and to process these items on seperated multi threads.

batch batch-processing batch-sql-call buffer buffer-queues mq queue

Last synced: 08 Jan 2026

https://github.com/ryclarke/batch-tool

This tool provides convenient features for working on multiple git repositories simultaneously.

automation batch-processing tooling

Last synced: 12 Mar 2026

https://github.com/pandevim/my_editor

Your own editor in batch!

batch-processing editor shell-script

Last synced: 24 Apr 2026

https://github.com/giljr/how_to_storing_fiscal_data

You now have a production-ready Rails 8 app capable of parsing and storing fiscal data from RCAD-formatted files.

batch-processing fileutils fiscalization moving-processed-file rails8 rcad transactions upload-file

Last synced: 26 Jul 2025

https://github.com/garugaru/go-batcher

High performance, easy to use batcher written in Go with generics™

batch-processing batcher generics-in-golang golang golang-library

Last synced: 18 Oct 2025

https://github.com/euclidstellar/medplat-demo

The Unified Data Ingestion System is designed to handle both streaming and batch data ingestion with a modular architecture.

batch-processing data-ingestion kafka stream-processing-engine

Last synced: 24 Jul 2025

https://github.com/yasarsultan/taxi-trip-analysis

The NYC Taxi Trip Batch Data Pipeline automates processing of large-scale trip data using Apache Spark and Airflow, integrating AWS S3 and Google BigQuery for storage and analytics. It features scalable, containerized workflows with robust data validation.

airflow aws-s3 bash-script batch-processing bigquery data-lake data-warehouse docker python3 spark

Last synced: 10 Apr 2026

https://github.com/gabrielalmir/billion-csv-rs

billion-csv-rs is a Rust project designed to generate a CSV file with one billion records. Each record contains randomly generated data including name, age, birth date, height, weight, and gender.

batch-processing billion-csv csv etl rust

Last synced: 08 Aug 2025

https://github.com/palcarazm/batchjs

Dependencies free batch processing framework for NodeJS based on streams.

batch-processing framework nodejs streams-batch-processing-framework

Last synced: 17 Mar 2025

https://github.com/nshkrdotcom/flowstone_ai

FlowStone integration for altar_ai - AI-powered data pipeline assets with classify_each, enrich_each, embed_each helpers and unified telemetry

adapter ai asset batch-processing beam classification data-engineering data-pipeline dsl elixir embeddings etl flowstone hex-package llm machine-learning nshkr-ai-infra otp resource telemetry

Last synced: 13 Jan 2026

https://github.com/sheng1111/text2srt_tts

Convert text to speech and auto-generate SRT subtitles. A CLI tool for creating synced audio and captions from plain text using multilingual TTS.

audiobook batch-processing caption multilingual speech-synthesis srt subtitle-generator text-to-speech tts video-captioning

Last synced: 05 Oct 2025

https://github.com/meenbeese/md-convert

Basic wrapper around the MarkItDown library from Microsoft.

batch-processing desktop docx markdown markitdown pdf python

Last synced: 25 Apr 2026

https://github.com/mohamedawnallah/learning-apache-flink

Document my Apache Flink learning experience

apacheflink batch-processing java stream-processing

Last synced: 28 Jun 2025

https://github.com/play3rzer0/pspscripts

Scripts for batch image processing in Python based image editors.

batch-processing batch-script computer-vision graphics imaging python python-script

Last synced: 21 May 2026

https://github.com/brooks-code/jpeg-tidy

BRRER: a simple command-line tool to resize JPEGs, sequentially rename them, and strip all EXIF metadata — overwriting originals for fast, privacy-focused batch cleanup.

automation batch-processing exif image-tools metadata privacy privacy-protection privacy-tools shell shell-scripts

Last synced: 08 Oct 2025

https://github.com/rogers-cyber/csvtoexcel

Modern PySide6 desktop app to convert CSV files to Excel (.xlsx) with batch processing, preview, encoding support, and SQLite history tracking.

batch-processing csv csv-converter csv-to-excel data-tools desktop-app drag-and-drop encoding excel file-converter gui preview productivity pyside6 python sqlite xlsx xlsxwriter

Last synced: 25 Apr 2026

https://github.com/rogers-cyber/svgconverterpro

Professional SVG conversion and processing desktop app with batch export, ICO generation, Base64 encoding, real-time preview, and multi-threaded performance.

base64 batch-processing data-uri desktop-app image-processing svg svg-converter svg-to-ico svg-to-jpg svg-to-pdf svg-to-png svg-to-webp

Last synced: 25 Apr 2026

https://github.com/ydb-platform/ydb-parallel-processor

YDB Parallel Record Processor

batch-processing ydb

Last synced: 10 Oct 2025

https://github.com/comfortablesoftware/bip

Batch Image Processor (rescale, randomly distribute, redistribute, convert format, etc.) for some research needs

batch-processing convert images randomization rename rescale super-resolution

Last synced: 22 Jul 2025

https://github.com/plzzzzg/conditional-batch-executor

A batch executor that collects tasks and executes them when conditions are met for Go.

batch batch-processing go golang

Last synced: 14 Jan 2026

https://github.com/mekepi/pvgis-parallel-api-client

Parallel API client to fetch hourly solar radiation data from PVGIS for Brazilian cities

api-client batch-processing open-data parallel-computing photovoltaic pvgis python solar-energy solar-irradiance

Last synced: 11 Oct 2025

https://github.com/hfmrow/info-media-mkv-ed

Simple mkv info viewer with some limited editing features, titling, tag cleaner, default/forced track, head/tail video trimmer, aspect/ratio changer, more...

batch-processing cut editor extract golang gotk3 gui head-mkv-trimmer info-media-mkv-ed mediainfo mkv ratio-changer tail-mkv-trimmer

Last synced: 19 May 2026

https://github.com/marvin1099/duplicatebracketdeleter

Ever Saved a file as "FILENAME (1)" because "FILENAME" was used already. If yes this deletes and moves files to get rid of the brackets

batch-processing cleanup filename python rename

Last synced: 11 Oct 2025

https://github.com/bruce-mig/batch-processing

Scheduled concurrent batch processing using Spring Batch

batch-processing db-to-csv spring-batch-jobs

Last synced: 13 Sep 2025

https://github.com/syniverse/quickstart-batchautomation-java

Java example code supporting the Batch Automation Quick Start Guide for Syniverse Developer Community

batch-processing java quickstart

Last synced: 14 Oct 2025

https://github.com/mahmudnibir/deletetemp

🛠️This script deletes all temporary files from the system's %temp% directory with a single click.

bat batch-file batch-processing batch-script batchfile delete one-click system tempfiles

Last synced: 22 Jan 2026