Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

https://github.com/HariSekhon/DevOps-Python-tools

80+ DevOps & Data CLI Tools - AWS, GCP, GCF Python Cloud Functions, Log Anonymizer, Spark, Hadoop, HBase, Hive, Impala, Linux, Docker, Spark Data Converters & Validators (Avro/Parquet/JSON/CSV/INI/XML/YAML), Travis CI, AWS CloudFormation, Elasticsearch, Solr etc.

avro aws cloudformation devops docker dockerhub elasticsearch gcf gcp hadoop hbase hdfs json linux parquet pyspark python solr spark travis-ci

Last synced: 27 Jun 2024

https://github.com/FasterXML/jackson-dataformats-binary

Uber-project for standard Jackson binary format backends: avro, cbor, ion, protobuf, smile

avro cbor hacktoberfest jackson-backends protobuf smile

Last synced: 20 Jun 2024

https://github.com/Chabane/bigdata-playground

A complete example of a big data application using : Kubernetes (kops/aws), Apache Spark SQL/Streaming/MLib, Apache Flink, Scala, Python, Apache Kafka, Apache Hbase, Apache Parquet, Apache Avro, Apache Storm, Twitter Api, MongoDB, NodeJS, Angular, GraphQL

angular apache-flink apache-spark avro big-data docker graphql hadoop hbase kafka kops machine-learning mongodb nodejs parquet python scala spark-sql spark-streaming twitter-api

Last synced: 17 Jun 2024

https://github.com/hpgrahsl/kafka-connect-mongodb

**Unofficial / Community** Kafka Connect MongoDB Sink Connector -> integrated 2019 into the official MongoDB Kafka Connector here: https://www.mongodb.com/kafka-connector

avro azure-cosmosdb bson cdc change-data-capture confluent-hub connector cosmosdb debezium json kafka kafka-connect mongodb sink sink-connector

Last synced: 07 Jun 2024

https://github.com/salsify/avromatic

Generate Ruby models from Avro schemas

avro gem hacktoberfest

Last synced: 02 Jun 2024

https://github.com/salsify/avro-schema-registry

Implementation of the Confluent Schema Registry API as a Rails application

avro hacktoberfest schema-registry

Last synced: 02 Jun 2024

https://github.com/salsify/avro-builder

Ruby DSL to create Avro schemas

avro gem hacktoberfest

Last synced: 02 Jun 2024

https://github.com/dasch/avro_turf

A library that makes it easier to use the Avro serialization format from Ruby.

avro avro-data ruby schema schema-registry

Last synced: 02 Jun 2024

https://github.com/supermodel/supermodel

Supermodel browser library and CLI tool for working with JSON Schema data models

avro graphql json json-ld json-schema openapi swagger

Last synced: 02 Jun 2024

https://github.com/niqdev/kafka-scala-examples

Examples of Avro, Kafka, Schema Registry, Kafka Streams, Interactive Queries, KSQL, Kafka Connect in Scala

avro interactive-queries kafka kafka-connect kafka-scala kafka-streams ksql schema-registry

Last synced: 01 Jun 2024

https://github.com/bolcom/avro-schema-viewer

Visualizer for Avro Schemas (.avsc) - Try it yourself at:

angular angular-material avro avro-schema

Last synced: 16 May 2024

https://github.com/bigdatagenomics/adam

ADAM is a genomics analysis platform with specialized file formats built using Apache Avro, Apache Spark, and Apache Parquet. Apache 2 licensed.

avro big-data bioinformatics genomics java parquet python r scala spark

Last synced: 13 May 2024

https://github.com/streamthoughts/kafka-connect-file-pulse

🔗 A multipurpose Kafka Connect connector that makes it easy to parse, transform and stream any file, in any format, into Apache Kafka

amazon-s3 avro azure-storage csv etl file-streaming google-cloud grok-filters kafka kafka-connect kafka-connector kafka-producer xml

Last synced: 07 May 2024

https://github.com/lensesio/schema-registry-ui

Web tool for Avro Schema Registry |

avro kafka schema-registry

Last synced: 07 May 2024

https://github.com/Cinchoo/ChoETL

ETL framework for .NET (Parser / Writer for CSV, Flat, Xml, JSON, Key-Value, Parquet, Yaml, Avro formatted files)

avro cinchoo-etl csharp csv dotnet etl etl-framework flat json keyvalue parquet parquet-files parser reader writer xml yaml

Last synced: 05 May 2024

https://github.com/thekvs/cpp-serializers

Benchmark comparing various data serialization libraries (thrift, protobuf etc.) for C++

apache-avro avro boost c-plus-plus capn-proto cereal cpp flatbuffers msgpack performance-testing protobuf serialization thrift yas

Last synced: 01 May 2024

https://github.com/indix/schemer

Schema registry for CSV, TSV, JSON, AVRO and Parquet schema. Supports schema inference and GraphQL API.

avro graphql-api json parquet schema-inference schema-registry spark tsv

Last synced: 30 Apr 2024

https://github.com/sksamuel/avro4s

Avro schema generation and serialization / deserialization for Scala

avro avro-schema coproduct scala scala-macros schema-generation serialization

Last synced: 30 Apr 2024

https://github.com/khezen/avro

Apache AVRO for go

apache avro etl go golang redshift sql

Last synced: 29 Apr 2024

https://github.com/apache/avro

Apache Avro is a data serialization system.

avro bigdata c cplusplus csharp dotnet java perl php python ruby rust

Last synced: 27 Apr 2024

https://github.com/zarusz/SlimMessageBus

Lightweight message bus interface for .NET (pub/sub and request-response) with transport plugins for popular message brokers.

apache-kafka avro azure azure-event-hubs azure-service-bus bus c-sharp ddd dotnet kafka message-bus messaging mqtt pub-sub rabbitmq redis request-response

Last synced: 25 Apr 2024

https://github.com/chezou/sparkavro

Load Avro data into Spark with sparklyr

avro r spark sparklyr

Last synced: 25 Apr 2024

https://github.com/malcolmgreaves/avro-codegen

Scala code generator for Avro schemas.

avro avro-schema codegen data scala serialization

Last synced: 22 Apr 2024

https://github.com/dflemstr/rq

Record Query - A tool for doing record analysis and transformation

avro command-line-tool javascript json lodash messagepack protobuf rust toml yaml

Last synced: 20 Apr 2024

https://github.com/pennsignals/aptos

:sunny: A tool for validating data using JSON Schema and converting JSON Schema documents into different data-interchange formats

avro avro-schema cli command-line-tool data-interchange json-schema python3 schema-conversion validation

Last synced: 15 Apr 2024

https://github.com/Altinity/clickhouse-sink-connector

Replicate data from MySQL, Postgres and MongoDB to ClickHouse

avro cdc clickhouse debezium kafka kafka-connect mongo mysql postgresql replication sqlserver

Last synced: 13 Apr 2024

https://github.com/Adelost/javascript-serialization-benchmark

Comparison and benchmark of JavaScript serialization libraries (Protocol Buffer, Avro, BSON, etc.)

avro benchmark binary-serialization comparison javascript performance protobuf protocol-buffers serialization

Last synced: 08 Apr 2024

https://github.com/piotr-yuxuan/dove

Infer specs from SpecificRecord Java classes of any Avro named type (record, fixed, or enum)

avro avro-format avro-schema clojure clojure-spec clojure-specs spec specific-schemas

Last synced: 08 Apr 2024

https://github.com/ovotech/kafka-avro-confluent

Kafka De/Serializer using avro and Confluent's Schema Registry

avro company-kaluza confluent kafka

Last synced: 08 Apr 2024

https://github.com/petersonjr/MetadataCrawler

A simple tool to extract metadata from relational databases

avro crawler database-schemas java jdbc metadata rdms relational-databases

Last synced: 01 Apr 2024

https://github.com/andygrove/bdt

Boring Data Tool

avro convert csv json parquet

Last synced: 01 Apr 2024

https://github.com/spotify/ratatool

A tool for data sampling, data generation, and data diffing

avro bigquery parquet protobuf scala scalacheck

Last synced: 31 Mar 2024

https://github.com/OneCricketeer/schema-registry-transfer-smt

Kafka Connect Transform to copy Avro schemas between Schema Registries

avro confluent kafka kafka-connect kafka-connect-smt schema-registry

Last synced: 26 Mar 2024

https://github.com/nodefluent/schema-registry

:orange_book: json & avro http schema registry backed by Kafka

avro json kafka node registry rest schema schema-registry

Last synced: 26 Mar 2024

https://github.com/deviceinsight/kafkactl

Command Line Tool for managing Apache Kafka

apache-kafka avro cli fish golang kafka zsh

Last synced: 22 Mar 2024

https://github.com/sevenmind/kaufmann_ex

Kafka backed service library.

avro concurrency elixir kafka microservice schema

Last synced: 22 Mar 2024

https://github.com/pmacct/pmacct

pmacct is a small set of multi-purpose passive network monitoring tools [NetFlow IPFIX sFlow libpcap BGP BMP RPKI IGP Streaming Telemetry].

avro bgp bmp geoip2 ipfix json kafka libpcap mysql ndpi netflow nflog pmacct postgresql rabbitmq rpki sflow sql sqlite3

Last synced: 22 Mar 2024

https://github.com/RandomFractals/vscode-data-preview

Data Preview 🈸 extension for importing 📤 viewing 🔎 slicing 🔪 dicing 🎲 charting 📊 & exporting 📥 large JSON array/config, YAML, Apache Arrow, Avro, Parquet & Excel data files

array arrow avro config csv data excel extension json parquet perspective viewer vscode yaml

Last synced: 17 Mar 2024

https://github.com/flix-tech/schema-registry-php-client

A PHP 8+ API client for the Confluent Schema Registry REST API based on Guzzle 7 - http://docs.confluent.io/current/schema-registry/docs/index.html

api-client avro avro-schema confluent event-driven-architecture guzzlehttp php schema-evolution schema-registry

Last synced: 17 Mar 2024

https://github.com/anair-it/schema-evolution

Demo - Schema evolution using Apache Avro and Google Protocol buffers

avro java protocol-buffers schema-evolution

Last synced: 17 Mar 2024

https://github.com/andrewinci/Insulator

A client UI to inspect Kafka topics, consume, produce and much more

apache-kafka avro consumer javafx kafka kafka-consumer kotlin schema topic tornadofx ui

Last synced: 16 Mar 2024