Data

Tools

Indexes

Applications

Experiments

Awesome

An open API service indexing awesome lists of open source software.

Projects in Awesome Lists tagged with avro

A curated list of projects in awesome lists tagged with avro .

https://github.com/apache/avro

Apache Avro is a data serialization system.

avro bigdata c cplusplus csharp dotnet java perl php python ruby

Last synced: 09 Sep 2025

https://github.com/dflemstr/rq

Record Query - A tool for doing record analysis and transformation

avro command-line-tool javascript json lodash messagepack protobuf rust toml yaml

Last synced: 14 May 2025

https://github.com/pmacct/pmacct

pmacct is a small set of multi-purpose passive network monitoring tools [NetFlow IPFIX sFlow libpcap BGP BMP RPKI IGP Streaming Telemetry].

avro bgp bmp geoip2 ipfix json kafka libpcap mysql ndpi netflow nflog pmacct postgresql rabbitmq rpki sflow sql sqlite3

Last synced: 14 May 2025

https://github.com/linkedin/goavro

Goavro is a library that encodes and decodes Avro data.

avro golang

Last synced: 17 Aug 2025

https://github.com/bigdatagenomics/adam

ADAM is a genomics analysis platform with specialized file formats built using Apache Avro, Apache Spark, and Apache Parquet. Apache 2 licensed.

avro big-data bioinformatics genomics java parquet python r scala spark

Last synced: 19 Oct 2025

https://github.com/deviceinsight/kafkactl

Command Line Tool for managing Apache Kafka

apache-kafka avro cli fish golang kafka zsh

Last synced: 14 May 2025

https://github.com/cinchoo/choetl

ETL framework for .NET (Parser / Writer for CSV, Flat, Xml, JSON, Key-Value, Parquet, Yaml, Avro formatted files)

avro cinchoo-etl csharp csv dotnet etl etl-framework flat json keyvalue parquet parquet-files parser reader writer xml yaml

Last synced: 12 Apr 2025

https://github.com/harisekhon/devops-python-tools

80+ DevOps & Data CLI Tools - AWS, GCP, GCF Python Cloud Functions, Log Anonymizer, Spark, Hadoop, HBase, Hive, Impala, Linux, Docker, Spark Data Converters & Validators (Avro/Parquet/JSON/CSV/INI/XML/YAML), Travis CI, AWS CloudFormation, Elasticsearch, Solr etc.

avro aws cloudformation devops docker dockerhub elasticsearch gcf gcp hadoop hbase hdfs json linux parquet pyspark python solr spark travis-ci

Last synced: 13 Jun 2025

https://github.com/HariSekhon/DevOps-Python-tools

80+ DevOps & Data CLI Tools - AWS, GCP, GCF Python Cloud Functions, Log Anonymizer, Spark, Hadoop, HBase, Hive, Impala, Linux, Docker, Spark Data Converters & Validators (Avro/Parquet/JSON/CSV/INI/XML/YAML), Travis CI, AWS CloudFormation, Elasticsearch, Solr etc.

avro aws cloudformation devops docker dockerhub elasticsearch gcf gcp hadoop hbase hdfs json linux parquet pyspark python solr spark travis-ci

Last synced: 11 Apr 2025

https://github.com/Cinchoo/ChoETL

ETL framework for .NET (Parser / Writer for CSV, Flat, Xml, JSON, Key-Value, Parquet, Yaml, Avro formatted files)

avro cinchoo-etl csharp csv dotnet etl etl-framework flat json keyvalue parquet parquet-files parser reader writer xml yaml

Last synced: 14 Mar 2025

https://github.com/miguno/kafka-storm-starter

[PROJECT IS NO LONGER MAINTAINED] Code examples that show to integrate Apache Kafka 0.8+ with Apache Storm 0.9+ and Apache Spark Streaming 1.1+, while using Apache Avro as the data serialization format.

apache-avro apache-kafka apache-spark apache-storm avro integration kafka scala spark storm

Last synced: 17 Dec 2025

https://github.com/sksamuel/avro4s

Avro schema generation and serialization / deserialization for Scala

avro avro-schema coproduct scala scala-macros schema-generation serialization

Last synced: 13 May 2025

https://github.com/thekvs/cpp-serializers

Benchmark comparing various data serialization libraries (thrift, protobuf etc.) for C++

apache-avro avro boost c-plus-plus capn-proto cereal cpp flatbuffers msgpack performance-testing protobuf serialization thrift yas

Last synced: 11 Jan 2026

https://github.com/RandomFractals/vscode-data-preview

Data Preview 🈸 extension for importing 📤 viewing 🔎 slicing 🔪 dicing 🎲 charting 📊 & exporting 📥 large JSON array/config, YAML, Apache Arrow, Avro, Parquet & Excel data files

array arrow avro config csv data excel extension json parquet perspective viewer vscode yaml

Last synced: 08 Apr 2025

https://github.com/randomfractals/vscode-data-preview

Data Preview 🈸 extension for importing 📤 viewing 🔎 slicing 🔪 dicing 🎲 charting 📊 & exporting 📥 large JSON array/config, YAML, Apache Arrow, Avro, Parquet & Excel data files

array arrow avro config csv data excel extension json parquet perspective viewer vscode yaml

Last synced: 04 Apr 2025

https://github.com/zarusz/slimmessagebus

Lightweight message bus interface for .NET (pub/sub and request-response) with transport plugins for popular message brokers.

apache-kafka avro azure azure-event-hubs azure-service-bus bus c-sharp ddd dotnet kafka message-bus messaging mqtt pub-sub rabbitmq redis request-response

Last synced: 14 May 2025

https://github.com/netflix/iceberg

Iceberg is a table format for large, slow-moving tabular data

avro hadoop parquet spark

Last synced: 05 Aug 2025

https://github.com/lensesio/schema-registry-ui

Web tool for Avro Schema Registry |

avro kafka schema-registry

Last synced: 15 Dec 2025

https://github.com/hamba/avro

A fast Go Avro codec

avro avro-codec encoder-decoder golang

Last synced: 13 May 2025

https://github.com/spotify/ratatool

A tool for data sampling, data generation, and data diffing

avro bigquery parquet protobuf scala scalacheck

Last synced: 15 May 2025

https://github.com/uber/storagetapper

StorageTapper is a scalable realtime MySQL change data streaming, logical backup and logical replication service

avro cdc clickhouse etl hdfs json kafka msgpack mysql postgresql s3

Last synced: 11 Jun 2025

https://github.com/sksamuel/centurion

Kotlin Bigdata Toolkit

avro java kotlin orc parquet

Last synced: 19 Jul 2026

https://github.com/streamthoughts/kafka-connect-file-pulse

🔗 A multipurpose Kafka Connect connector that makes it easy to parse, transform and stream any file, in any format, into Apache Kafka

amazon-s3 avro azure-storage csv etl file-streaming google-cloud grok-filters kafka kafka-connect kafka-connector kafka-producer xml

Last synced: 12 Jan 2026

https://github.com/FasterXML/jackson-dataformats-binary

Uber-project for standard Jackson binary format backends: avro, cbor, ion, protobuf, smile

avro cbor hacktoberfest jackson-backends protobuf smile

Last synced: 03 May 2025

https://github.com/fasterxml/jackson-dataformats-binary

Uber-project for standard Jackson binary format backends: avro, cbor, ion, protobuf, smile

avro cbor hacktoberfest jackson-backends protobuf smile

Last synced: 13 May 2025

https://github.com/davidmc24/gradle-avro-plugin

A Gradle plugin to allow easily performing Java code generation for Apache Avro. It supports JSON schema declaration files, JSON protocol declaration files, and Avro IDL files.

avro gradle-plugin groovy java

Last synced: 04 Oct 2025

https://github.com/Eugene-Mark/bigdata-file-viewer

A cross-platform (Windows, MAC, Linux) desktop application to view common bigdata binary format like Parquet, ORC, AVRO, etc. Support local file system, HDFS, AWS S3, Azure Blob Storage ,etc.

avro bigdata hdfs orc parquet

Last synced: 20 Nov 2025

https://github.com/altinity/clickhouse-sink-connector

Replicate data from MySQL, Postgres and MongoDB to ClickHouse®

avro cdc clickhouse debezium kafka kafka-connect mongo mysql postgresql replication sqlserver

Last synced: 16 May 2025

https://github.com/Altinity/clickhouse-sink-connector

Replicate data from MySQL, Postgres and MongoDB to ClickHouse®

avro cdc clickhouse debezium kafka kafka-connect mongo mysql postgresql replication sqlserver

Last synced: 03 May 2025

https://github.com/riferrei/srclient

Golang Client for Schema Registry

apache avro cloud codec confluent go kafka registry schema

Last synced: 15 May 2025

https://github.com/absaoss/abris

Avro SerDe for Apache Spark structured APIs.

avro avro-schema kafka schema-registry spark

Last synced: 04 Apr 2025

https://github.com/marcosschroh/dataclasses-avroschema

Generate avro schemas from python dataclasses, Pydantic models and Faust Records. Code generation from avro schemas. Serialize/Deserialize python instances with avro schemas.

apache-avro avro avro-schemas code-generation faust-streaming json json-schema model pydantic python3 schema serialization

Last synced: 16 May 2025

https://github.com/rumbledb/rumble

⛈️ RumbleDB 1.23.0 "Mountain Ash" 🌳 for Apache Spark | Run queries on your large-scale, messy JSON-like data (JSON, text, CSV, Parquet, ROOT, AVRO, SVM...) | No install required (just a jar to download) | Declarative Machine Learning and more

avro azure csv data-science dataframes hdfs json jsoniq machine-learning nested parquet query query-engine s3 scale schemaless spark svm text yaml

Last synced: 03 Aug 2025

https://github.com/avro-kotlin/avro4k

Avro format support for Kotlin

avro kotlin schema serialization

Last synced: 23 Jan 2026

https://github.com/chabane/bigdata-playground

A complete example of a big data application using : Kubernetes (kops/aws), Apache Spark SQL/Streaming/MLib, Apache Flink, Scala, Python, Apache Kafka, Apache Hbase, Apache Parquet, Apache Avro, Apache Storm, Twitter Api, MongoDB, NodeJS, Angular, GraphQL

angular apache-flink apache-spark avro big-data docker graphql hadoop hbase kafka kops machine-learning mongodb nodejs parquet python scala spark-sql spark-streaming twitter-api

Last synced: 13 Apr 2025

https://github.com/mostafa/xk6-kafka

k6 extension to load test Apache Kafka with support for various serialization formats, SASL, TLS, compression, Schema Registry client and beyond

apache-kafka avro extension hacktoberfest json-schema jsonschema k6 kafka load-testing performance-testing sasl schema-registry xk6

Last synced: 01 Apr 2026

https://github.com/Chabane/bigdata-playground

A complete example of a big data application using : Kubernetes (kops/aws), Apache Spark SQL/Streaming/MLib, Apache Flink, Scala, Python, Apache Kafka, Apache Hbase, Apache Parquet, Apache Avro, Apache Storm, Twitter Api, MongoDB, NodeJS, Angular, GraphQL

angular apache-flink apache-spark avro big-data docker graphql hadoop hbase kafka kops machine-learning mongodb nodejs parquet python scala spark-sql spark-streaming twitter-api

Last synced: 28 Apr 2025

https://github.com/julianpeeters/avrohugger

Generate Scala case class definitions from Avro schemas

avro scala

Last synced: 10 Jul 2025

https://github.com/dasch/avro_turf

A library that makes it easier to use the Avro serialization format from Ruby.

avro avro-data ruby schema schema-registry

Last synced: 14 May 2025

https://github.com/MaterializeInc/datagen

Generate authentic looking mock data based on a SQL, JSON or Avro schema and produce to Kafka in JSON or Avro format.

avro kafka sql

Last synced: 13 Mar 2026

https://github.com/zio/zio-schema

Compositional, type-safe schema definitions, which enable auto-derivation of codecs and migrations.

avro bson json msg-pack protobuf scala thrift zio

Last synced: 08 Mar 2026

https://github.com/hpgrahsl/kafka-connect-mongodb

**Unofficial / Community** Kafka Connect MongoDB Sink Connector -> integrated 2019 into the official MongoDB Kafka Connector here: https://www.mongodb.com/kafka-connector

avro azure-cosmosdb bson cdc change-data-capture confluent-hub connector cosmosdb debezium json kafka kafka-connect mongodb sink sink-connector

Last synced: 28 Feb 2026

https://github.com/pennsignals/aptos

:sunny: A tool for validating data using JSON Schema and converting JSON Schema documents into different data-interchange formats

avro avro-schema cli command-line-tool data-interchange json-schema python3 schema-conversion validation

Last synced: 28 Mar 2025

https://github.com/materializeinc/datagen

Generate authentic looking mock data based on a SQL, JSON or Avro schema and produce to Kafka in JSON or Avro format.

avro kafka sql

Last synced: 04 Apr 2025

https://github.com/julianpeeters/sbt-avrohugger

sbt plugin for generating Scala sources for Apache Avro schemas and protocols.

avro sbt-plugin scala

Last synced: 17 Feb 2026

https://github.com/jrnd-io/jr

JR: streaming quality random data from the command line

avro datagen json-schema kafka kafka-producer mongodb protobuf random-generation redis schema-registry template

Last synced: 07 Oct 2025

https://github.com/ovotech/kafka-serialization

Lego bricks to build Apache Kafka serializers and deserializers

apache-kafka avro avro4s circe json json4s kaluza-to-migrate serialization spray-json

Last synced: 13 Apr 2025

https://github.com/ionutbalosin/ecommerce-app

eCommerce application is a platform where customers can find products, shop around using a cart, check out the products and initiate payments. It is used for the didactic purpose only, as a support project for my training.

api-driven-development asynchronous-logging avro caching debezium docker event-driven flywaydb kafka location-decoupling microservices openapi postgresql resilience resilience4j scalability schema-first-approach shared-nothing-database-approach spring-boot testcontainers

Last synced: 05 Apr 2025

https://github.com/ivangfr/springboot-kafka-connect-jdbc-streams

Project goal: Explore Kafka, Kafka Connect, and Kafka Streams. Components: store-api: Inserts/updates MySQL records. Source Connectors: Monitor MySQL changes, push messages to Kafka. Sink Connectors: Listen to Kafka, insert/update Elasticsearch docs. store-streams: Listens to Kafka, processes with Kafka Streams, pushes new messages to Kafka.

avro docker elasticsearch java jib json kafka kafka-connect kafka-streams mysql schema-registry spring-boot spring-cloud-stream spring-data-jpa spring-web-mvc springdoc-openapi

Last synced: 05 Apr 2025

https://github.com/indix/schemer

Schema registry for CSV, TSV, JSON, AVRO and Parquet schema. Supports schema inference and GraphQL API.

avro graphql-api json parquet schema-inference schema-registry spark tsv

Last synced: 27 Oct 2025

https://github.com/salsify/avro-builder

Ruby DSL to create Avro schemas

avro gem hacktoberfest

Last synced: 10 Jun 2025

https://github.com/fd4s/vulcan

Functional Avro for Scala

avro scala

Last synced: 04 Apr 2025

https://github.com/strech/avrora

A convenient Elixir library to work with Avro messages, schemas and Confluent® Schema Registry

avro avro-schema confluent elixir kafka schema-registry

Last synced: 15 May 2025

https://github.com/sevenmind/kaufmann_ex

Kafka backed service library.

avro concurrency elixir kafka microservice schema

Last synced: 07 May 2025

https://github.com/salsify/avro-schema-registry

Implementation of the Confluent Schema Registry API as a Rails application

avro hacktoberfest schema-registry

Last synced: 10 Jun 2025

https://github.com/salsify/avromatic

Generate Ruby models from Avro schemas

avro gem hacktoberfest

Last synced: 10 Jun 2025

https://github.com/Adelost/javascript-serialization-benchmark

Comparison and benchmark of JavaScript serialization libraries (Protocol Buffer, Avro, BSON, etc.)

avro benchmark binary-serialization comparison javascript performance protobuf protocol-buffers serialization

Last synced: 03 Apr 2025

https://github.com/ivangfr/springboot-kafka-connect-debezium-ksqldb

Experiment with Kafka, Debezium, and ksqlDB. research-service: Performs MySQL record manipulation. Source Connectors: Monitor MySQL changes, push messages to Kafka. Sink Connectors and kafka-research-consumer: Listen to Kafka, insert/update Elasticsearch. ksqlDB-Server: Listens to Kafka, performs joins, and pushes new messages to new Kafka topics.

avro confluent debezium elasticsearch java json kafka kafka-connect ksqldb mysql schema-registry spring-boot spring-data-jpa spring-kafka spring-web-mvc springdoc-openapi

Last synced: 08 Mar 2026

https://github.com/clemensv/avrotize

Avrotize is a command-line tool for converting data structure definitions between different schema formats, using Apache Avro Schema as the integration schema model.

asn1 avro avro-schema avro-schemas csharp java javascript json-schema protobuf protocol-buffers typescript xsd xsd-schema

Last synced: 07 Jun 2026

https://github.com/pozil/pub-sub-api-node-client

A node client for the Salesforce Pub/Sub API

avro grpc nodejs pubsub salesforce

Last synced: 16 May 2025

https://github.com/spotify/gcs-tools

GCS support for avro-tools, parquet-tools and protobuf

avro gcp gcs gcs-connector google-storage parquet protobuf

Last synced: 09 Apr 2025

https://github.com/beam-community/avro_ex

An Avro Library that emphasizes testability and ease of use.

avro avro-kafka avro-schema decoding encoding kafka schema

Last synced: 13 Mar 2026

https://github.com/nevillelyh/shapeless-datatype

Shapeless utilities for common data types

avro bigquery datastore google-cloud scala shapeless tensorflow

Last synced: 06 Apr 2025

https://github.com/flix-tech/avro-serde-php

Avro Serialisation/Deserialisation (SerDe) library for PHP 8.1+ with a Symfony Serializer integration

avro avro-format avro-schema confluent confluent-platform deserialization php serde serialization symfony symfony-serializer

Last synced: 08 Apr 2025

https://github.com/niqdev/kafka-scala-examples

Examples of Avro, Kafka, Schema Registry, Kafka Streams, Interactive Queries, KSQL, Kafka Connect in Scala

avro interactive-queries kafka kafka-connect kafka-scala kafka-streams ksql schema-registry

Last synced: 31 Oct 2025

https://github.com/mesosphere-backup/dcos-metrics

The metrics pipeline for DC/OS 1.9-1.11

avro dogstatsd metrics prometheus statsd

Last synced: 29 Apr 2025

https://github.com/onecricketeer/schema-registry-transfer-smt

Kafka Connect Transform to copy Avro schemas between Schema Registries

avro confluent kafka kafka-connect kafka-connect-smt schema-registry

Last synced: 02 Aug 2025

https://github.com/andrewinci/Insulator

A client UI to inspect Kafka topics, consume, produce and much more

apache-kafka avro consumer javafx kafka kafka-consumer kotlin schema topic tornadofx ui

Last synced: 03 Apr 2025

https://github.com/dangkaka/go-kafka-avro

A library provides consumer/producer to work with kafka, avro and schema registry

avro go kafka schema-registry

Last synced: 17 Jan 2026

https://github.com/goldmansachs/tablasco

Tablasco is a JUnit rule for comparing tables and Spark module for comparing large data sets

avro integration java junit regression spark tablasco testing

Last synced: 11 Apr 2025

https://github.com/farmdawgnation/registryless-avro-converter

An avro converter for Kafka Connect without a Schema Registry

avro avro-schema kafka kafka-connect

Last synced: 17 Oct 2025

https://github.com/supermodel/supermodel

Supermodel browser library and CLI tool for working with JSON Schema data models

avro graphql json json-ld json-schema openapi swagger

Last synced: 02 May 2026

https://github.com/apache/avro-rs

Rust SDK for Apache Avro - a data serialization system.

avro bigdata rust

Last synced: 09 Apr 2025

https://github.com/OneCricketeer/schema-registry-transfer-smt

Kafka Connect Transform to copy Avro schemas between Schema Registries

avro confluent kafka kafka-connect kafka-connect-smt schema-registry

Last synced: 11 May 2025

https://github.com/flix-tech/schema-registry-php-client

A PHP 8+ API client for the Confluent Schema Registry REST API based on Guzzle 7 - http://docs.confluent.io/current/schema-registry/docs/index.html

api-client avro avro-schema confluent event-driven-architecture guzzlehttp php schema-evolution schema-registry

Last synced: 04 Sep 2025

https://github.com/laserdisc-io/tamer

Standalone alternatives to Kafka Connect Connectors

avro doobie fp fs2 functional-programming jdbc kafka s3 scala schema-registry zio zio-streams

Last synced: 21 Sep 2025

https://github.com/khezen/avro

Apache AVRO for go

apache avro etl go golang redshift sql

Last synced: 21 Aug 2025

https://github.com/jpmorganchase/py-avro-schema

Generate Apache Avro schemas for Python types including standard library data-classes and Pydantic data models.

avro data dataclasses deserialization generate jpmorganchase kafka messaging pydantic python schema serialization types

Last synced: 28 Jun 2025

https://github.com/reproio/columnify

Make record oriented data to columnar format.

avro bigdata parquet

Last synced: 12 Jan 2026

https://github.com/joewood/avro-typescript

TypeScript Code Generator for Apache Avro Schema Types

avro avro-schema typescript typescript-library

Last synced: 24 Feb 2026

https://github.com/singlestore-labs/singlestore-logistics-sim

Scalable package delivery logistics simulator built using SingleStore and Vectorized Redpanda

avro demo kafka logistics redpanda simulation singlestore vectorized

Last synced: 06 Feb 2026