{"id":23940292,"url":"https://github.com/sneaksanddata/arcane-framework-scala","last_synced_at":"2026-06-08T17:31:02.121Z","repository":{"id":270696090,"uuid":"911178542","full_name":"SneaksAndData/arcane-framework-scala","owner":"SneaksAndData","description":"Arcane — A simple ZIO-based data streaming framework implemented in Scala","archived":false,"fork":false,"pushed_at":"2025-02-17T10:53:29.000Z","size":96,"stargazers_count":1,"open_issues_count":3,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-02-17T11:22:16.019Z","etag":null,"topics":["actor","arcane","data-ingestion","reactive","scala","streaming","zio"],"latest_commit_sha":null,"homepage":"","language":"Scala","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/SneaksAndData.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2025-01-02T12:19:59.000Z","updated_at":"2025-02-17T10:53:09.000Z","dependencies_parsed_at":"2025-02-07T13:35:13.527Z","dependency_job_id":null,"html_url":"https://github.com/SneaksAndData/arcane-framework-scala","commit_stats":null,"previous_names":["sneaksanddata/arcane-framework-scala"],"tags_count":14,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SneaksAndData%2Farcane-framework-scala","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SneaksAndData%2Farcane-framework-scala/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SneaksAndData%2Farcane-framework-scala/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SneaksAndData%2Farcane-framework-scala/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/SneaksAndData","download_url":"https://codeload.github.com/SneaksAndData/arcane-framework-scala/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":240415381,"owners_count":19797620,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["actor","arcane","data-ingestion","reactive","scala","streaming","zio"],"created_at":"2025-01-06T03:14:38.912Z","updated_at":"2026-06-08T17:31:02.114Z","avatar_url":"https://github.com/SneaksAndData.png","language":"Scala","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Arcane Framework (Scala - ZIO)\nArcane — A simple ZIO-based data streaming framework that seamlessly runs locally and on Kubernetes with the support of [Arcane Operator](https://github.com/sneaksanddata/arcane-operator)\n\n## Creating plugins\nPlugins utilize framework to create specific source-to-Iceberg streaming applications.\nDevelopers should create a Scala3 project for each plugin, containing stream context definition, source settings definition and `main.scala`.\n\n### Stream source definition\nFirst, you should create a serializable proxy for plugin source configuration:\n```scala 3\ncase class MySourceSettings(\n    override val buffering: DefaultSourceBufferingSettings,\n    override val fieldSelectionRule: DefaultFieldSelectionRuleSettings,\n    override val configuration: DefaultMySourceSettings\n) extends StreamSourceSettings derives ReadWriter:\n  override type SourceSettingsType = DefaultMySourceSettings\n```\n\nwhere `DefaultMySourceSettings` should be defined in the framework, but you can also create one in the plugin repository.\n\n### Stream context definition\nMost plugins can rely on framework defaults and the stream source created on the previous step:\n```scala 3\nimport com.sneaksanddata.arcane.framework.models.app.{DefaultPluginStreamContext, PluginStreamContext}\nimport com.sneaksanddata.arcane.framework.models.settings.observability.DefaultObservabilitySettings\nimport com.sneaksanddata.arcane.framework.models.settings.sink.DefaultSinkSettings\nimport com.sneaksanddata.arcane.framework.models.settings.staging.DefaultStagingSettings\nimport com.sneaksanddata.arcane.framework.models.settings.streaming.{\n  DefaultStreamModeSettings,\n  DefaultThroughputSettings\n}\n\ncase class MyPluginStreamContext(\n    @key(\"observability\") private val observabilityIn: DefaultObservabilitySettings,\n    @key(\"staging\") private val stagingIn: DefaultStagingSettings,\n    @key(\"streamMode\") private val streamModeIn: DefaultStreamModeSettings,\n    @key(\"sink\") private val sinkIn: DefaultSinkSettings,\n    @key(\"throughput\") private val throughputIn: DefaultThroughputSettings,\n    override val source: MyPluginSourceSettings\n) extends DefaultPluginStreamContext(observabilityIn, stagingIn, streamModeIn, sinkIn, throughputIn) derives ReadWriter:\n  // TODO: should be implemented when Operator supports overrides\n  override def merge(other: Option[PluginStreamContext]): PluginStreamContext = this\n\nobject MyPluginStreamContext:\n  def apply(value: String): MyPluginStreamContext =\n    PluginStreamContext[MyPluginStreamContext](value)\n\n  // ZLayer for injecting the stream context singleton\n  lazy val layer\n      : ZLayer[Any, Throwable, PluginStreamContext \u0026 DatagramSocketConfig \u0026 MetricsConfig \u0026 DatadogPublisherConfig] =\n    PluginStreamContext.getLayer[MyPluginStreamContext]\n```\n\n### Entrypoint\n\nNow you can add `main.scala` and work is done:\n\n```scala 3\npackage com.sneaksanddata.arcane.sql_server_change_tracking\n\nimport models.app.MicrosoftSqlServerPluginStreamContext\n\nimport com.sneaksanddata.arcane.framework.logging.ZIOLogAnnotations.zlog\nimport com.sneaksanddata.arcane.framework.models.schemas.ArcaneSchema\nimport com.sneaksanddata.arcane.framework.services.app.base.{StreamLifetimeService, StreamRunnerService}\nimport com.sneaksanddata.arcane.framework.services.app.{GenericStreamRunnerService, PosixStreamLifetimeService}\nimport com.sneaksanddata.arcane.framework.services.base.SchemaProvider\nimport com.sneaksanddata.arcane.framework.services.bootstrap.DefaultStreamBootstrapper\nimport com.sneaksanddata.arcane.framework.services.filters.{ColumnSummaryFieldsFilteringService, FieldsFilteringService}\nimport com.sneaksanddata.arcane.framework.services.iceberg.{\n  IcebergEntityManager,\n  IcebergS3CatalogWriter,\n  IcebergTablePropertyManager\n}\nimport com.sneaksanddata.arcane.framework.services.merging.JdbcMergeServiceClient\nimport com.sneaksanddata.arcane.framework.services.metrics.{GlobalMetricTagProvider, DeclaredMetrics}\nimport com.sneaksanddata.arcane.framework.services.streaming.data_providers.backfill.{\n  GenericBackfillStreamingMergeDataProvider,\n  GenericBackfillStreamingOverwriteDataProvider\n}\nimport com.sneaksanddata.arcane.framework.services.streaming.graph_builders.{\n  StreamGraphResolver,\n  DefaultStreamingGraphBuilder\n}\nimport com.sneaksanddata.arcane.framework.services.streaming.processors.batch_processors.backfill.{\n  BackfillOverwriteBatchProcessor,\n  BackfillOverwriteWatermarkProcessor\n}\nimport com.sneaksanddata.arcane.framework.services.streaming.processors.batch_processors.streaming.{\n  DisposeBatchProcessor,\n  MergeBatchProcessor,\n  WatermarkProcessor\n}\nimport com.sneaksanddata.arcane.framework.services.streaming.processors.transformers.{\n  FieldFilteringTransformer,\n  StagingProcessor\n}\nimport com.sneaksanddata.arcane.framework.services.streaming.throughput.base.ThroughputShaperBuilder\nimport zio.logging.backend.SLF4J\nimport zio.metrics.connectors.MetricsConfig\nimport zio.metrics.connectors.datadog.DatadogPublisherConfig\nimport zio.metrics.connectors.statsd.DatagramSocketConfig\nimport zio.metrics.jvm.DefaultJvmMetrics\nimport zio.{Runtime, ZIO, ZIOAppDefault, ZLayer}\n\nobject main extends ZIOAppDefault {\n\n  override val bootstrap: ZLayer[Any, Nothing, Unit] = Runtime.removeDefaultLoggers \u003e\u003e\u003e SLF4J.slf4j\n\n  val appLayer: ZIO[StreamRunnerService, Throwable, Unit] = for\n    _ \u003c- zlog(\"Application starting\")\n    streamRunner \u003c- ZIO.service[StreamRunnerService]\n    _ \u003c- streamRunner.run\n  yield ()\n\n  private lazy val streamRunner = appLayer.provide(\n    GenericStreamRunnerService.layer,\n    StreamGraphResolver.composedLayer,\n    DisposeBatchProcessor.layer,\n    FieldFilteringTransformer.layer,\n    MergeBatchProcessor.layer,\n    StagingProcessor.layer,\n    FieldsFilteringService.layer,\n    MyPluginStreamContext.layer,\n    PosixStreamLifetimeService.layer,\n    MyPluginDataProvider.layer, // Plugin implementation of a SourceDataProvider\n    IcebergS3CatalogWriter.layer,\n    IcebergEntityManager.sinkLayer,\n    IcebergEntityManager.stagingLayer,\n    IcebergTablePropertyManager.stagingLayer,\n    IcebergTablePropertyManager.sinkLayer,\n    JdbcMergeServiceClient.layer,\n    MyPluginStreamingDataProvider.layer, // Plugin implementation of a StreamingDataProvider \n    MyPluginHookManager.layer, // Plugin implementation of a HookManager\n    BackfillOverwriteBatchProcessor.layer,\n    GenericBackfillStreamingOverwriteDataProvider.layer,\n    GenericBackfillStreamingMergeDataProvider.layer,\n    DefaultStreamingGraphBuilder.backfillSubStreamLayer,\n    MyPluginBackfillOverwriteBatchFactory.layer, // Plugin implementation of a BackfillOverwriteBatchFactory\n    DeclaredMetrics.layer,\n    WatermarkProcessor.layer,\n    BackfillOverwriteWatermarkProcessor.layer,\n    DefaultStreamBootstrapper.layer,\n    ThroughputShaperBuilder.layer,\n    GlobalMetricTagProvider.layer,\n    DefaultJvmMetrics.liveV2.unit\n  )\n\n  @main\n  def run: ZIO[Any, Throwable, Unit] =\n    val app = streamRunner\n\n    app.catchAllCause { cause =\u003e\n      for {\n        _ \u003c- zlog(s\"Application failed: ${cause.squashTrace.getMessage}\", cause)\n        _ \u003c- exit(zio.ExitCode(1))\n      } yield ()\n    }\n}\n\n```","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsneaksanddata%2Farcane-framework-scala","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsneaksanddata%2Farcane-framework-scala","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsneaksanddata%2Farcane-framework-scala/lists"}