https://github.com/indix/schemer

Schema registry for CSV, TSV, JSON, AVRO and Parquet schema. Supports schema inference and GraphQL API.
https://github.com/indix/schemer

avro graphql-api json parquet schema-inference schema-registry spark tsv

Last synced: 29 days ago
JSON representation

Schema registry for CSV, TSV, JSON, AVRO and Parquet schema. Supports schema inference and GraphQL API.

• Host: GitHub
• URL: https://github.com/indix/schemer
• Owner: indix
• License: apache-2.0
• Created: 2017-11-24T10:35:28.000Z (almost 7 years ago)
• Default Branch: master
• Last Pushed: 2020-03-05T19:19:48.000Z (over 4 years ago)
• Last Synced: 2024-09-28T21:23:28.171Z (about 1 month ago)
• Topics: avro, graphql-api, json, parquet, schema-inference, schema-registry, spark, tsv
• Language: Scala
• Homepage:
• Size: 3.14 MB
• Stars: 111
• Watchers: 17
• Forks: 6
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

• awesome-ccamel - indix/schemer - Schema registry for CSV, TSV, JSON, AVRO and Parquet schema. Supports schema inference and GraphQL API. (Scala)
• awesome-scala - schemer - activity/y/indix/schemer) (Table of Contents / Big Data)

README

        
# schemer

[![Build Status](https://travis-ci.org/indix/schemer.svg?branch=master)](https://travis-ci.org/indix/schemer) [![Maven](https://maven-badges.herokuapp.com/maven-central/com.indix/schemer-core_2.11/badge.svg)](http://repo1.maven.org/maven2/com/indix/schemer-core_2.11/) [![Docker Pulls](https://img.shields.io/docker/pulls/indix/schemer-registry.svg)](https://hub.docker.com/r/indix/schemer-registry/)



    



Schema registry with support for CSV, TSV, AVRO, JSON and Parquet. Has ability to infer schema from a given data source.

## Schemer UI [WIP]



    



Schemer UI is the wizard based frontend for Schemer. It provides a wizard based schema creation and versioning workflow apart from browsing and search capabilities. It is a work in progress. [More screens](schemer-ui.md)

## Schemer Core

`schemer-core` is the core library that implements most of the logic needed to understand the supported schema types along with the schema inference. To use `schemer-core` directly, just add it to your dependencies:

```

libraryDependencies += "com.indix" %% "schemer" % "v0.2.3"

```

## Schemer Registry

`schemer-registry` is a schema registry for storing the metadata about schema and schema versions. It provides a GraphQL API for adding, viewing and inferring schemas.

Schemer Registry is available as a [docker image at DockeHub](https://hub.docker.com/r/indix/schemer-registry/)

### Running Locally

Local docker based PostgreSQL can be run as follows:

```

docker run -e POSTGRES_USER=schemer -e POSTGRES_PASSWORD=schemer -e PGDATA=/var/lib/postgresql/data/pgdata -e POSTGRES_DB=schemer -v $(pwd)/schemer_db:/var/lib/postgresql/data/pgdata -p 5432:5432 postgres:9.5.0

```

Remove `schmer_db` folder to clear all data and start from scratch.

The registry service can be run using `sbt`:

```bash

sbt "project registry" ~reStart

```