{"id":21185590,"url":"https://github.com/raystack/depot","last_synced_at":"2025-07-10T01:30:42.585Z","repository":{"id":37858276,"uuid":"483131207","full_name":"raystack/depot","owner":"raystack","description":"Depot contains various common sink implementations and publishes them as a library. This library will be used in firehose, daggers or any other application which wants to send data to destinations.","archived":false,"fork":false,"pushed_at":"2024-09-18T06:00:57.000Z","size":1029,"stargazers_count":9,"open_issues_count":6,"forks_count":4,"subscribers_count":8,"default_branch":"main","last_synced_at":"2024-09-18T08:26:02.500Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/raystack.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":"docs/roadmap.md","authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-04-19T06:55:28.000Z","updated_at":"2024-03-31T14:25:45.000Z","dependencies_parsed_at":"2024-09-18T08:16:41.758Z","dependency_job_id":null,"html_url":"https://github.com/raystack/depot","commit_stats":null,"previous_names":["raystack/depot","odpf/depot"],"tags_count":27,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/raystack%2Fdepot","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/raystack%2Fdepot/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/raystack%2Fdepot/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/raystack%2Fdepot/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/raystack","download_url":"https://codeload.github.com/raystack/depot/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":225606462,"owners_count":17495551,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-20T18:17:13.190Z","updated_at":"2024-11-20T18:17:13.812Z","avatar_url":"https://github.com/raystack.png","language":"Java","readme":"# Depot\n\nDepot contains various common sink implementations and publishes them as a library. This library will be used in\nfirehose, daggers or any other application which wants to send data to destinations such as service endpoints (HTTP or\nGRPC)\n\u0026 managed databases (Postgres, BigQuery, InfluxDB, Redis, Elasticsearch, Prometheus, MongoDB etc.)\n\n## Key Features\n\n- Instrumentation support with statsd\n- Log Sink\n- Bigquery Sink\n\nDepot is a sink connector, which acts as a bridge between data processing systems and real sink. The APIs in this\nlibrary can be used to push data to various sinks. Common sinks implementations will be added in this repo.\n\n## Requirements\n\n- java8 or higher\n- gradle\n\n## How to use\n\nExplore the following resources to get started\n\n- [Reference](docs/reference) contains details about configurations of metrics and various sinks\n- [Contribute](docs/contribute/contribution.md) contains resources for anyone who wants to contribute.\n\n### Build and run tests\n\n```sh\n# Building the jar\n$ ./gradlew clean build\n\n# Running unit tests\n$ ./gradlew test\n\n# Run code quality checks\n$ ./gradlew checkstyleMain checkstyleTest\n\n#Cleaning the build\n$ ./gradlew clean\n```\n\n### Maven and gradle dependency\n\n```xml\n\n\u003cdependency\u003e\n    \u003cgroupId\u003eorg.raystack\u003c/groupId\u003e\n    \u003cartifactId\u003edepot\u003c/artifactId\u003e\n    \u003cversion\u003eversion\u003c/version\u003e\n\u003c/dependency\u003e\n```\n\n```sh\nimplementation group: 'org.raystack', name: 'depot', version: 'version'\n```\n\n### Usage example:\n\n```java\npublic interface Sink extends Closeable {\n    SinkResponse pushToSink(List\u003cMessage\u003e messages) throws SinkException;\n}\n```\n\nSink implementations will normally have a factory class. The application using this library can create and use sinks by\nusing pseudocode snippet below. For example BigquerySink.\n\n```java\nclass MyClass {\n    void createSink() {\n        factory = new BigQuerySinkFactory();\n        factory.init();\n        sink = factory.create();\n    }\n\n    void sendMessages() {\n        response = sink.pushToSink(listOfMessage);\n        if (response.hasErrors()) {\n            // handle errors.\n        }\n    }\n}\n```\n\n### Data types\n\nCurrently, sink connector library is supporting protobuf and Json format. We can set the datatype of `Message` by\nsetting `SINK_CONNECTOR_SCHEMA_DATA_TYPE`. Each datatype has parsers which takes care of deserialization.\n\n### Adding a new Sink\n\nEach sink will have to implement `Sink` interface. The pushToSink take a batch of messages and return a response\nwith error list.\n\n### Configurations\n\nPlease check the docs folder for details.\n\n## License\n\nFirehose is [Apache 2.0](LICENSE) licensed.\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fraystack%2Fdepot","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fraystack%2Fdepot","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fraystack%2Fdepot/lists"}